Kubernetes
Advanced 15 min read Kubernetes

Managed Kubernetes in Production

The platform is managed; the risk is yours

EKS/AKS/GKE run clusters reliably; you still own upgrades, node pools, backups, security policies, quotas, and observability. This is the 'managed ≠ hands-off' mental shift plus platform guardrails.

EKSAKSGKEcluster upgradesaddonsquotascostrunbook
Prerequisites: All prior Kubernetes concepts
Before we start

What you will learn

  • Scope the split: provider-managed vs your-owned responsibilities
  • Plan version/upgrade cadence and add-on hygiene
  • Build production guardrails: quotas, backups, policies
In 60 seconds

Provider manages the control plane, etcd HA, certificate rotation, and patching. You own workload readiness, version skew, RBAC, PodSecurity standards, network policy, backups, CostAllocation, HA testing, and runbooks.

Introduction

What is this?

Provider-owned: apiserver/etcd, control-plane HA + upgrades, node autoscaling, managed add-ons, CSI provisioning. Your-owned: workload deployment hygiene, cluster version policy (max N-2), controlled upgrades (BLUE-green node pools, kaniko/rollout phase), PDBs, quotas (ResourceQuota+LimitRange per namespace), PodSecurity (PodSecurity admission standard), network policies, ClusterPolicy (Kyverno/OPA), backups (Velero + cluster snapshot), region DR, RTO tests, budgets (per-cluster cost tags, rightsizing). Node pool operations: cordon/drain, replacement pool + upgrade, spot-vs-ondemand. Ops maturity: ephemeral dev clusters, runbooks (p95 recovery steps), dashboards (SLI green/yellow/red), on-call podcast with postmortem culture.

Motivation

Why do you need it?

  • A cluster is a product: version drift, weak policies, and no restore = production fragility.
  • Upgrades must happen; delaying > N-2 compounds with every release.
  • Managed control plane means health is now a policy question (PDBs, quotas, policies), not an infra tray.
  • Cost goes exponential without per-namespace quotas and spot-aware sizing.
Think about it

Mental model

Imagine

Renting a fully-serviced office building but still owning your fire drills, security keys and disaster plan

Think of

Managed Kubernetes ownership

The building manager (cloud) fixes AC and elevators; your company must still file safety plans (RBAC/policies), practice exits (DR drills), and know who holds which keys.

Mechanics

How it works

  • Constants check: version supported, add-ons healthy, quotas scoped.
  • Upgrade: new control plane version swapped by provider; you sequence node pool replacement + workloads.
  • Drift guards: PodSecurity admission, NetworkPolicies, Kyverno default-deny for privileged.
  • Backup: Velero snapshots namespace + PVCs on schedule to object storage with restore tested.
  • On-call: alerts map to curated runbooks; every incident's learnings update policies/code.

Production operating loop.

Guardrails
Upgrades
Backups
Observability
On-call
Hands-on

Examples

Per-namespace quota that keeps prod honest

Per-namespace quota that keeps prod honest
apiVersion: v1
kind: ResourceQuota
metadata: { name: prod-quota, namespace: prod }
spec:
  hard:
    requests.cpu: "4"
    limits.cpu: "8"
    requests.memory: 8Gi
    count/pods: "40"

Quantified entitlement: a namespace that can't silently balloon beyond its budget — a CostControl + blast-radius tool simultaneously.

Real world

In production

  • Controlled prod upgrade: new node pool on the new version, drain one-by-one respecting PDBs.
  • Velero backup nightly + quarterly restore drill to a scratch cluster.
  • Kyverno default-deny privileged; enforce runAsNonRoot for Receivables anyway.
  • Per-environment clusters with identical IaC modulo region/spot flags.
Guidelines

Best practices

  • Keep prod ≤ N-1 (or N-2) and follow upgrade notes; never skiplag 2+.
  • Test upgrades on staging plus a synthetic canary namespace first.
  • Set quotas + limit ranges before onboarding teams (hard to retrofit).
  • Automate restores; a backup nobody can restore is a screenshot.
  • Review cost per namespace weekly; node pool rightsizing is a monthly ritual.
Watch out

Common mistakes

  • 'It's managed' → skipping runbooks/backups/policies until an incident proves otherwise.
  • Running in-place kubectl edits on prod that drift from IaC.
  • No PDBs: worker drain during an upgrade kills in-flight requests.
  • Ignore upgrade Deprecations warnings until a 404 festival on the CLI.
When things break

Debugging tips

  • Cluster unhealthy: check control-plane page/provider status first, then kube-system pods.
  • Upgrade stuck drain: cordoned node with a Pod without PDB/grace → kubectl drain --force carefully.
  • Cost spike: correlate per-namespace nodePool cost tags and the HPA maxReplicas.
  • Restore confusion: keep a DR runbook that says exactly which PVC/CSI snapshot to restore first.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Provider-managed on managed clusters includes:

2

The single most protective habit in managed clusters is:

Go deeper

Official documentation

EKS best practices
docs
AKS production baseline
docs
GKE operations
docs
Velero backups
docs