Python for Data Engineering
Intermediate 15 min read Python for Data Engineering

File I/O and Formats

Get bytes in and out without losing anything

Reading and writing CSV, JSON, Parquet and Avro correctly — encodings, chunking, compression and atomic writes — is where pipelines most often lose data or speed.

CSVJSONParquetAvrochunkingcompressionatomic writes
Prerequisites: Exception Handling
Before we start

What you will learn

  • Read/write CSV, JSON and Parquet with the right encodings and types
  • Use chunked reads for large files
  • Write atomically (temp file + rename) and handle compression
Introduction

What is this?

Every pipeline begins and ends with files. The formats — CSV, JSON, Parquet, Avro — differ in compression, schema fidelity and cost. Correct I/O means: right encoding, right types, atomic writes, and streaming reads when the file outgrows memory.

Motivation

Why do you need it?

  • A CSV with a trailing comma or BOM silently mis-parses until data is wrong.
  • Parquet stores typed, compressed, columnar data — the de-facto lake format — while JSON re-parses text every read.
  • Atomic writes (temp + rename) mean readers never see half-written files.
Think about it

Mental model

Imagine

Shipping containers vs loose mail

Think of

Parquet/columnar vs CSV/text

Loose envelopes (CSV) arrive in any order, easily crumpled, re-read by a human every time. Containers (Parquet) are packed, labeled with a manifest (schema), and unloaded efficiently by machines.

Mechanics

How it works

  • CSV: Row-oriented text; read with `csv` module / pandas, minding delimiter, quotes, encoding and header row.
  • JSON: nested but text; `json.loads` per line for NDJSON; schema not enforced.
  • Parquet: columnar binary with embedded schema + compression (snappy/zstd); fast column subset reads.
  • Avro: row-oriented binary with an explicit schema — great for Kafka's Schema Registry model.
  • Chunking (`read_csv(chunksize=...)`) streams rows so RAM bounds, not file size, defines feasibility.

Choosing a format by use case.

Human edits / interop
Analytics at scale
Streaming with schema
Huge single files
Write atomically
Hands-on

Examples

Atomic, compressed CSV write

Atomic, compressed CSV write
import os, tempfile
def atomic_write(df, path):
    with tempfile.NamedTemporaryFile("w", suffix=".csv", delete=False) as tmp:
        df.to_csv(tmp.name, index=False)
    os.replace(tmp.name, path)  # readers see only the complete file

Writes land in a temp path and are renamed into place, so a crash mid-write never leaves a partial file at the target.

Real world

In production

  • A nightly export writes `orders_YYYYMMDD.parquet` with snappy to S3, then the catalog registers the new partition.
  • A webhook consumer logs each payload as a line of NDJSON for later replay and debug.
  • An extractor streams a 30 GB CSV in chunks into a DuckDB/warehouse instead of reading it whole.
Guidelines

Best practices

  • Prefer Parquet over CSV/JSON for any analytics data — schema, compression and columnar reads.
  • Always specify encoding='utf-8' for both read and write; be explicit, never default.
  • Write atomically for anything a reader could observe mid-write.
  • Chunk or stream reads for files approaching memory; cap buffers explicitly.
Watch out

Common mistakes

  • Opening files in binary/text interchangeably and breaking parsing or encodings.
  • Writing directly to the final path, risking torn files readers interpret as valid-but-empty.
  • Storing analytics data as JSON because it is 'flexible', then paying schema-less pain at query time.
  • Forgetting that gzip on CSV defeats columnar skipping — Parquet beats compress('csv.gz') for analytics.
Root cause analysis

Common issues

Error
UnicodeDecodeError on a 'text' file
Root cause

The file is actually latin-1/UTF-16 or contains binary sequences

Fix

Detect encoding (charset-normalizer) or open with encoding='utf-8', errors='replace' and log warns.

Error
Readers see partially-written files
Root cause

Writing in place without atomic replacement

Fix

Write to a temp file then os.replace; or write to `.part` and rename on success.

When things break

Debugging tips

  • When a load 'shifts' columns, re-read the raw file and inspect the first 3 lines with a hexdump or repr.
  • Check byte size after writing — a 0-byte 'success' almost always means opened the wrong path.
  • Diff row counts between source and target per partition to spot dropped/bad rows.
Interviews

Interview checkpoint

Practice

Hands-on lab

Round-Trip a Dataset Across Formats Safely
30 min · intermediate
In progress
Objectives
  • Convert a small CSV into typed, compressed Parquet
  • Prove atomic write behavior with a partial-write simulation
  • Re-read Parquet and verify types and row counts survived

Round-Trip a Dataset Across Formats Safely

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A 40 GB CSV must feed analytics. Which path is best?

2

A success marker rename guarantees readers never observe a half-written dataset.

Go deeper

Official documentation

pyarrow — Parquet C++
docs
Avro specification
spec