Cluster Schedulers and Resource Management
Who hogs which machine, and how they share it
Cluster schedulers (YARN, Kubernetes, Mesos) allocate CPUs and memory to jobs from a shared pool. Understand the abstraction — drivers asking for resources, cluster managers granting containers — and you can run Spark anywhere.
What you will learn
- Explain the resource-negotiation model shared by all schedulers
- Compare YARN, Kubernetes and standalone scheduling trade-offs
- Tune driver/executor resources relative to your workload
What is this?
A cluster scheduler is the middleman between jobs and machines. When a Spark driver starts, it requests containers (CPU + memory) from YARN or Kubernetes; the scheduler places them on nodes, monitors them and reclaims them when the job ends. Different schedulers trade off latency, isolation, and ecosystem fit — but the resource model is identical.
Why do you need it?
- Shared clusters are cheaper — several teams use one hardware pool instead of each keeping its own.
- Fair allocation prevents one runaway job from starving reporting.
- Knowing the scheduler explains real clusters on-prem, on EMR and everywhere else.
- Autoscaling and cost controls live at the scheduler layer.
Mental model
A hotel assigning rooms
Resource scheduling (YARN/K8s)
The reservation desk (driver) requests rooms (containers); housekeeping places you, tracks you, and reclaims the room at checkout. The hotel can book the same rooms to a different guest tomorrow; the scheduler does the same for executors.
How it works
- A driver submits a resource request (cores, memory) to the cluster manager.
- The manager allocates containers on nodes matching the request.
- Executors register with the driver and start processing tasks.
- The scheduler monitors health; on container loss it re-allots and the job recovers via lineage.
- Queues and quotas (YARN) or namespaces/limits (K8s) provide multi-tenant fairness.
Scheduling Spark on a cluster.
In production
- A company runs Spark on Kubernetes with namespaces per team, autoscaling executors during the ETL peak.
- On-prem YARN queues guarantee the finance batch its nightly capacity despite a streaming team's jobs.
- An autoscaling cluster manager spins brokers to zero at night, cutting the bill by half.
Best practices
- Size executors modestly (e.g. 2-8 cores) to balance parallelism and memory isolation.
- Set memory overhead high enough for JVM/pyspark/python processes or you'll get OOM kills.
- Use queues/namespaces to protect SLAs in shared clusters.
- Turn on dynamic allocation when workloads vary; skip it for predictable production jobs.
Common mistakes
- Asking for one giant executor — limited parallelism and wasted idle cores.
- Zero memory-overhead, then 'killed by YARN/K8s' errors for Python overhead.
- Confusing the driver's memory with the executors', starving the driver)
Common issues
Executors keep getting killed by memory limitsJVM/pyspark processes exceed container memory; overhead set too low
Raise spark.executor.memoryOverhead and container memory, or shrink per-executor vcores.
Job waits forever in the queueScheduler queue exhausted, or capacity policy limits your team
Check queue capacity/quota and the fair-share policy; start smaller or wait off-peak.
Adding nodes doesn't helpDynamic allocation off, so new nodes aren't given containers
Enable dynamic allocation or set minimum executors to match the workload.
Debugging tips
- Container kill = always check the memory overhead AND the executor's actual footprint.
- Fair-share throttling shows as slow queue time; look at the scheduler UI before blaming the job.
- K8s: check pod evictions/listRequests before profiling the job code.
- Compare requested vs used resources — a mismatch tells you to resize containers, not tune code.
Interview checkpoint
Quick quiz
Which is NOT a cluster scheduler Spark commonly runs on?
When an executor container dies, the scheduler re-launches it and the driver can recover via lineage.