Python for Data Engineering
Intermediate 14 min read Python for Data Engineering

numpy Essentials

The C-speed foundation under every dataframe

numpy provides the typed arrays under pandas and almost every data library. Knowing it explains why vectorization is fast and where memory actually goes.

numpyarraysvectorizationbroadcastingmemorydtype
Prerequisites: Python Data Structures
Before we start

What you will learn

  • Build and manipulate typed numpy arrays
  • Explain broadcasting and use it correctly
  • Explain the memory cost of object dtypes and copies
Introduction

What is this?

numpy is a C library exposed to Python that stores homogenous typed arrays and operates on them as whole vectors. pandas DataFrames are built on numpy buffers; every vectorized operation you love is numpy under the hood.

Motivation

Why do you need it?

  • Understanding numpy explains how pandas gets its speed and where it loses it.
  • Broadcasting is the missing intuition that unlocks concise numeric transforms.
  • Dtype choice controls memory: one 8-byte float vs one Python-object pointer with full object overhead.
Think about it

Mental model

Imagine

A conveyor belt of identical boxes

Think of

A numpy array

A homogeneous array lets a machine handle every box identically at once — but the boxes must be the same shape and type. Break that rule and you fall back to shuffling each box by hand.

Mechanics

How it works

  • `np.array([...])` allocates one contiguous buffer with a fixed dtype and shape.
  • Vectorized ops (`arr * 2`) loop in C over the buffer, not in Python.
  • Broadcasting stretches smaller arrays across bigger ones when shapes are compatible.
  • Views share a buffer (`arr[::2]`, `arr.reshape`) while copies allocate new memory.
  • Mixed types force object dtype — Python objects, per-element overhead, 10-100x slower.

Why vectorization is fast: one C pass vs N Python calls.

Python loop
numpy op
Broadcast
Result
Hands-on

Examples

Broadcasting a scalar across an array

Broadcasting a scalar across an array
import numpy as np
a = np.array([10.0, 20.0, 30.0])
b = np.array([[1, 2], [3, 4]])
print(a * 0.08)          # scalar broadcast
print(b.sum(axis=1))     # keepdims meaning: row sums

Scalar multiplication and axis-sum are single C operations, not loops.

Real world

In production

  • A performance-sampling layer computes percentiles with np.percentile over millions of floats in milliseconds.
  • A masking step uses boolean arrays (`arr[mask]`) to select rows cheaply before a pandas transform.
  • A time-bucket encoder converts epoch arrays to bucket indices with integer arithmetic — no Python loop.
Guidelines

Best practices

  • Keep arrays homogenous — matching dtypes are the whole performance story.
  • Know whether you are viewing or copying (`arr.view()`, `.reshape()` return views) to bound memory.
  • Use `np.where`, `np.select` and boolean masks instead of Python conditionals.
  • Freeze numpy in requirements; its C ABI changes can break prebuilt wheels.
Watch out

Common mistakes

  • Building an array with `np.array([...])` from Python objects in a loop, then wondering why it's slow.
  • Copying huge arrays implicitly (e.g. `arr[:] = ...` vs `arr = arr + 1` allocating new memory).
  • Forgetting that mixed types become object arrays — silently 50x slower.
  • Using `np` indexing in a Python loop instead of vectorized selection.
Root cause analysis

Common issues

Error
MemoryError with a modest-sized array
Root cause

Object dtype (Python floats) or accidental copies using 100 bytes/element

Fix

Coerce to np.float32/float64 explicitly and audit in-place ops.

Error
Broadcasting error: operands could not be broadcast together
Root cause

Array shapes incompatible for an operation

Fix

Check .shape; align with .reshape/.expand_dims so trailing dimensions match.

When things break

Debugging tips

  • Check `arr.dtype` and `arr.itemsize` when something is mysteriously slow or large.
  • If an op degenerates to Python objects, inspect the dtype of the input arrays first.
  • Use `np.broadcast_to` errors to understand shape mismatches — they name the misaligned axis.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What determines the memory footprint of a numpy array?

2

A slice like arr[::2] always copies the underlying data into new memory.

Go deeper

Official documentation

NumPy user guide
docs
NumPy broadcasting docs
docs