Capacity and Cost Engineering

• Size and auto-tune compute capacity against utilization-SLA targets
• Attribute cost to workloads and set budgets with alert tiers
• Plan growth: project next-quarter capacity from trend, not panic

• Idle 64-node warehouses and 28k tiny files both cost money while delivering nothing.
• Auto-scaling without a budget lets one backfill bill the whole month.
• Capacity under-sizing breaks SLOs (query waits, stream lag) that consumers feel as outages.
• Predictable cost keeps the platform funded: finance trusts it, teams optimize it.

The airline buys capacity for peak travel days, leases for bursts, and retires planes that fly empty — every seat is a cost. Data capacity is the same: shape compute to the load curve, charge the intense days, cut the idle engines.

• Utilization-to-SLO mapping: define what utilization% keeps query-wait SLIs green.
• Right-size: start with workload history; resize warehouses near the kneecap of utilization vs cost.
• Auto-scaling policy: scale on backlog/target utilization, cap at a budgeted max, suspend at idle.
• Reserved vs on-demand: continuous base load on reserved capacity; bursts on-demand.
• Attribution: tag warehouses/jobs/tables by workload and team; produce daily cost-by-tag.
• Budget stop: hard caps + burn alerts; oversized runs are blocked or warned before they bill.

• A retail company autoscales its ETL warehouse on queue depth: 200 nodes at monthly-refresh peaks, 4 at 3am — the utilization SLO stays green and the bill reflects the curve.
• A fintech reserves steady-node capacity for its 24/7 streaming consumers and uses on-demand for the ad-hoc BI surge, cutting blended cost 30%.
• A startup tags every job by team; the ML team's backfill shows up as 60% of spend and gets budgeted, alerted and optimized.

• Size against the load curve and SLIs, not against the worst month of the year by feel.
• Cap autoscaling and suspend idle — two cheap policies that control the bill's tail.
• Publish per-warehouse/per-tag cost so optimization targets the biggest lines first.
• Set an annual growth model: trend capacity ~ volume; budget next quarter from that, reviewed monthly.

• Auto-scaling unbounded 'to be safe' — the bill becomes a surprise tax.
• Sizing for peak and paying for idle instead of matching the curve.
• No attribution, so nobody can say why spend doubled — and nobody fixes it.
• Forecasting capacity by gut during growth spurts, then hitting an SLO wall mid-quarter.

• SLO breach + high utilization → under-provisioned; raise the cap or reserve more — but check it's load, not a bad query.
• Bill spike → cost-by-tag report across hours; overnight backfills and ad-hoc giants are the usual lines.
• Idle spend → list warehouses by utilization; anything <40% idle-cut or auto-suspend.
• Growth surprise → compare capacity forecast vs actual volume monthly; divergence means the model is wrong, fix it early.