Kubernetes — Production Guide
Complete Production Implementation Guide
Version: 1.30+ | Category: Orchestration | Level: All
Kubernetes is the industry-standard container orchestration platform. This guide covers everything from cluster setup to day-2 production operations — the way senior engineers actually run it.
Table of Contents
- Architecture Deep Dive
- Production Cluster Setup
- Workloads — Production Patterns
- Networking
- Storage
- RBAC & Security
- Autoscaling
- Observability
- Upgrade Strategy
- Production Gotchas
- Essential Commands Cheatsheet
1. Architecture Deep Dive {#architecture}
Control Plane Components
| Component | Role | Production Notes | |---|---|---| | kube-apiserver | All REST operations, auth/authz gateway | Run 3+ replicas behind a load balancer | | etcd | Distributed key-value store for all cluster state | Needs NVMe SSD, regular backups, odd number of nodes | | kube-scheduler | Assigns pods to nodes (filter + score) | Extendable with custom schedulers | | kube-controller-manager | Runs all built-in controllers (Deployment, Node, etc.) | Single leader-elected process | | cloud-controller-manager | Talks to cloud provider APIs (AWS, GCP, Azure) | Required for LoadBalancer services, node lifecycle |
Node Components
| Component | Role | |---|---| | kubelet | Node agent — talks to API server, manages containers via CRI | | kube-proxy | Maintains network rules (iptables/ipvs) for Service IPs | | Container Runtime | containerd (default), CRI-O — runs actual containers | | CNI Plugin | Handles pod networking — Cilium (recommended), Calico, Flannel |
What happens when you run kubectl apply
kubectl → kube-apiserver (authn/authz)
→ Admission Controllers (MutatingWebhook → ValidatingWebhook)
→ etcd (persisted)
→ Controller Manager (creates ReplicaSet → creates Pods)
→ Scheduler (assigns Pod.spec.nodeName)
→ kubelet on target node (pulls image, starts container via CRI)
→ CNI (sets up pod networking)
→ kube-proxy (updates iptables for Services)
2. Production Cluster Setup {#setup}
Minimum Production Requirements
| Component | Minimum | Recommended | |---|---|---| | Control plane nodes | 3 (HA) | 3 (odd number always) | | Worker nodes | 3 | Start with 3, use autoscaler | | etcd disk | 50GB SSD | NVMe, <10ms p99 latency | | Control plane CPU | 2 vCPU | 4 vCPU | | Control plane RAM | 4GB | 8GB |
kubeadm HA Cluster Init
bash1# On first control plane node2kubeadm init \3 --control-plane-endpoint "k8s-lb.internal:6443" \4 --upload-certs \5 --pod-network-cidr=10.244.0.0/16 \6 --service-cidr=10.96.0.0/12 \7 --kubernetes-version=v1.30.0 \8 --apiserver-cert-extra-sans="k8s-lb.internal,10.0.1.10"910# Save the output! You need:11# 1. kubeadm join command for control plane nodes12# 2. kubeadm join command for worker nodes13# 3. --certificate-key value1415# Setup kubeconfig16mkdir -p $HOME/.kube17cp -i /etc/kubernetes/admin.conf $HOME/.kube/config18chown $(id -u):$(id -g) $HOME/.kube/config1920# Install Cilium CNI (best choice for production)21helm repo add cilium https://helm.cilium.io/22helm install cilium cilium/cilium \23 --version 1.15.0 \24 --namespace kube-system \25 --set kubeProxyReplacement=true \26 --set k8sServiceHost="k8s-lb.internal" \27 --set k8sServicePort=6443 \28 --set hubble.enabled=true \29 --set hubble.relay.enabled=true \30 --set hubble.ui.enabled=true3132# Join additional control plane nodes33kubeadm join k8s-lb.internal:6443 \34 --token <token> \35 --discovery-token-ca-cert-hash sha256:<hash> \36 --control-plane \37 --certificate-key <cert-key>3839# Join worker nodes40kubeadm join k8s-lb.internal:6443 \41 --token <token> \42 --discovery-token-ca-cert-hash sha256:<hash>
etcd Backup (run daily via CronJob)
bash1ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \2 --endpoints=https://127.0.0.1:2379 \3 --cacert=/etc/kubernetes/pki/etcd/ca.crt \4 --cert=/etc/kubernetes/pki/etcd/server.crt \5 --key=/etc/kubernetes/pki/etcd/server.key67# Verify backup8ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-latest.db --write-out=table
3. Workloads — Production Patterns {#workloads}
Production-Grade Deployment
yaml1apiVersion: apps/v12kind: Deployment3metadata:4 name: api-server5 namespace: production6 labels:7 app: api-server8 version: "1.2.0"9 team: platform10spec:11 replicas: 312 revisionHistoryLimit: 513 selector:14 matchLabels:15 app: api-server16 strategy:17 type: RollingUpdate18 rollingUpdate:19 maxUnavailable: 0 # Never kill pod before replacement is ready20 maxSurge: 1 # One extra pod during update21 template:22 metadata:23 labels:24 app: api-server25 version: "1.2.0"26 annotations:27 prometheus.io/scrape: "true"28 prometheus.io/port: "8080"29 prometheus.io/path: "/metrics"30 spec:31 # Spread pods across nodes and AZs32 topologySpreadConstraints:33 - maxSkew: 134 topologyKey: kubernetes.io/hostname35 whenUnsatisfiable: DoNotSchedule36 labelSelector:37 matchLabels:38 app: api-server39 - maxSkew: 140 topologyKey: topology.kubernetes.io/zone41 whenUnsatisfiable: ScheduleAnyway42 labelSelector:43 matchLabels:44 app: api-server4546 # Graceful shutdown47 terminationGracePeriodSeconds: 604849 # Don't run as root50 securityContext:51 runAsNonRoot: true52 runAsUser: 100053 fsGroup: 200054 seccompProfile:55 type: RuntimeDefault5657 containers:58 - name: api59 image: myregistry/api:1.2.060 imagePullPolicy: IfNotPresent61 ports:62 - name: http63 containerPort: 808064 protocol: TCP6566 # ALWAYS set both requests and limits for memory67 # For CPU: set requests, be careful with limits (throttling)68 resources:69 requests:70 cpu: "100m"71 memory: "256Mi"72 limits:73 memory: "512Mi" # No CPU limit — causes throttling7475 # Readiness: when to send traffic76 readinessProbe:77 httpGet:78 path: /ready79 port: 808080 initialDelaySeconds: 581 periodSeconds: 582 failureThreshold: 383 successThreshold: 18485 # Liveness: when to restart86 livenessProbe:87 httpGet:88 path: /healthz89 port: 808090 initialDelaySeconds: 30 # After app fully starts91 periodSeconds: 1592 failureThreshold: 393 timeoutSeconds: 59495 # Startup: for slow-starting apps (prevents liveness killing it)96 startupProbe:97 httpGet:98 path: /healthz99 port: 8080100 failureThreshold: 30 # 30 * 10s = 5 min max startup101 periodSeconds: 10102103 env:104 - name: PORT105 value: "8080"106 - name: DB_PASSWORD107 valueFrom:108 secretKeyRef:109 name: api-secrets110 key: db-password111 - name: POD_NAME112 valueFrom:113 fieldRef:114 fieldPath: metadata.name115 - name: POD_NAMESPACE116 valueFrom:117 fieldRef:118 fieldPath: metadata.namespace119120 # Graceful shutdown handler121 lifecycle:122 preStop:123 exec:124 command: ["/bin/sh", "-c", "sleep 5"] # Let LB drain connections125126 securityContext:127 allowPrivilegeEscalation: false128 readOnlyRootFilesystem: true129 capabilities:130 drop: ["ALL"]131132 volumeMounts:133 - name: tmp134 mountPath: /tmp135 - name: config136 mountPath: /app/config137 readOnly: true138139 volumes:140 - name: tmp141 emptyDir: {}142 - name: config143 configMap:144 name: api-config145146 # Pull from private registry147 imagePullSecrets:148 - name: registry-credentials
PodDisruptionBudget — Protect Against Node Drains
yaml1apiVersion: policy/v12kind: PodDisruptionBudget3metadata:4 name: api-server-pdb5 namespace: production6spec:7 minAvailable: 2 # Always keep 2 pods running8 # OR: maxUnavailable: 19 selector:10 matchLabels:11 app: api-server
HorizontalPodAutoscaler with Custom Metrics
yaml1apiVersion: autoscaling/v22kind: HorizontalPodAutoscaler3metadata:4 name: api-server-hpa5 namespace: production6spec:7 scaleTargetRef:8 apiVersion: apps/v19 kind: Deployment10 name: api-server11 minReplicas: 312 maxReplicas: 5013 metrics:14 - type: Resource15 resource:16 name: cpu17 target:18 type: Utilization19 averageUtilization: 7020 - type: Resource21 resource:22 name: memory23 target:24 type: Utilization25 averageUtilization: 8026 - type: Pods27 pods:28 metric:29 name: http_requests_per_second30 target:31 type: AverageValue32 averageValue: "1000"33 behavior:34 scaleDown:35 stabilizationWindowSeconds: 300 # Wait 5 min before scaling down36 policies:37 - type: Percent38 value: 2539 periodSeconds: 6040 scaleUp:41 stabilizationWindowSeconds: 0 # Scale up immediately42 policies:43 - type: Percent44 value: 10045 periodSeconds: 15
CronJob — Production Pattern
yaml1apiVersion: batch/v12kind: CronJob3metadata:4 name: db-backup5 namespace: production6spec:7 schedule: "0 2 * * *" # 2 AM daily8 concurrencyPolicy: Forbid # Don't run if previous still running9 successfulJobsHistoryLimit: 310 failedJobsHistoryLimit: 511 startingDeadlineSeconds: 3600 # Don't start if >1hr late12 jobTemplate:13 spec:14 backoffLimit: 215 activeDeadlineSeconds: 3600 # Kill if runs >1hr16 template:17 spec:18 restartPolicy: OnFailure19 containers:20 - name: backup21 image: myregistry/db-backup:latest22 resources:23 requests:24 cpu: "100m"25 memory: "128Mi"26 limits:27 memory: "256Mi"
4. Networking {#networking}
Service Types — When to Use Each
yaml1# ClusterIP — internal only (default, most common)2apiVersion: v13kind: Service4metadata:5 name: api-service6spec:7 type: ClusterIP8 selector:9 app: api-server10 ports:11 - port: 8012 targetPort: 80801314---15# NodePort — avoid in production, use LoadBalancer or Ingress instead16# Range: 30000-327671718---19# LoadBalancer — creates cloud LB (AWS NLB, GCP LB, Azure LB)20apiVersion: v121kind: Service22metadata:23 name: api-lb24 annotations:25 # AWS NLB instead of CLB26 service.beta.kubernetes.io/aws-load-balancer-type: "nlb"27 service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"28spec:29 type: LoadBalancer30 selector:31 app: api-server32 ports:33 - port: 44334 targetPort: 8080
Production Ingress with TLS
yaml1apiVersion: networking.k8s.io/v12kind: Ingress3metadata:4 name: api-ingress5 namespace: production6 annotations:7 nginx.ingress.kubernetes.io/ssl-redirect: "true"8 nginx.ingress.kubernetes.io/proxy-body-size: "10m"9 nginx.ingress.kubernetes.io/proxy-read-timeout: "60"10 nginx.ingress.kubernetes.io/rate-limit: "100" # req/min per IP11 cert-manager.io/cluster-issuer: "letsencrypt-prod"12spec:13 ingressClassName: nginx14 tls:15 - hosts:16 - api.example.com17 secretName: api-tls-cert18 rules:19 - host: api.example.com20 http:21 paths:22 - path: /23 pathType: Prefix24 backend:25 service:26 name: api-service27 port:28 number: 80
Network Policy — Zero Trust
yaml1# Default deny all ingress and egress2apiVersion: networking.k8s.io/v13kind: NetworkPolicy4metadata:5 name: default-deny-all6 namespace: production7spec:8 podSelector: {}9 policyTypes:10 - Ingress11 - Egress1213---14# Allow specific traffic15apiVersion: networking.k8s.io/v116kind: NetworkPolicy17metadata:18 name: api-server-netpol19 namespace: production20spec:21 podSelector:22 matchLabels:23 app: api-server24 policyTypes:25 - Ingress26 - Egress27 ingress:28 - from:29 - namespaceSelector:30 matchLabels:31 name: ingress-nginx32 ports:33 - protocol: TCP34 port: 808035 egress:36 - to:37 - podSelector:38 matchLabels:39 app: postgres40 ports:41 - protocol: TCP42 port: 543243 - to: [] # Allow DNS44 ports:45 - protocol: UDP46 port: 53
5. Storage {#storage}
StorageClass for AWS EBS gp3
yaml1apiVersion: storage.k8s.io/v12kind: StorageClass3metadata:4 name: gp35 annotations:6 storageclass.kubernetes.io/is-default-class: "true"7provisioner: ebs.csi.aws.com8volumeBindingMode: WaitForFirstConsumer # Don't provision until pod scheduled9reclaimPolicy: Retain # Don't delete PV when PVC deleted10allowVolumeExpansion: true11parameters:12 type: gp313 iops: "3000"14 throughput: "125"15 encrypted: "true"
StatefulSet with Persistent Storage
yaml1apiVersion: apps/v12kind: StatefulSet3metadata:4 name: postgres5 namespace: production6spec:7 serviceName: postgres-headless8 replicas: 39 selector:10 matchLabels:11 app: postgres12 template:13 metadata:14 labels:15 app: postgres16 spec:17 containers:18 - name: postgres19 image: postgres:16-alpine20 ports:21 - containerPort: 543222 resources:23 requests:24 cpu: "500m"25 memory: "1Gi"26 limits:27 memory: "2Gi"28 env:29 - name: POSTGRES_PASSWORD30 valueFrom:31 secretKeyRef:32 name: postgres-secret33 key: password34 - name: PGDATA35 value: /var/lib/postgresql/data/pgdata36 volumeMounts:37 - name: postgres-data38 mountPath: /var/lib/postgresql/data39 volumeClaimTemplates:40 - metadata:41 name: postgres-data42 spec:43 accessModes: ["ReadWriteOnce"]44 storageClassName: gp345 resources:46 requests:47 storage: 100Gi
6. RBAC & Security {#security}
Least-Privilege ServiceAccount
yaml1# ServiceAccount2apiVersion: v13kind: ServiceAccount4metadata:5 name: api-server-sa6 namespace: production7 annotations:8 # AWS IRSA — let pod assume IAM role without storing credentials9 eks.amazonaws.com/role-arn: "arn:aws:iam::123456789:role/api-server-role"10automountServiceAccountToken: false # Don't mount unless needed1112---13# Role — namespace-scoped14apiVersion: rbac.authorization.k8s.io/v115kind: Role16metadata:17 name: api-server-role18 namespace: production19rules:20- apiGroups: [""]21 resources: ["configmaps"]22 verbs: ["get", "list", "watch"]23- apiGroups: [""]24 resources: ["secrets"]25 resourceNames: ["api-secrets"] # Only this specific secret26 verbs: ["get"]2728---29# RoleBinding30apiVersion: rbac.authorization.k8s.io/v131kind: RoleBinding32metadata:33 name: api-server-rolebinding34 namespace: production35subjects:36- kind: ServiceAccount37 name: api-server-sa38 namespace: production39roleRef:40 kind: Role41 apiGroup: rbac.authorization.k8s.io42 name: api-server-role
OPA Gatekeeper — Enforce Policies
yaml1# Require all pods to have resource limits2apiVersion: constraints.gatekeeper.sh/v1beta13kind: K8sRequiredResources4metadata:5 name: require-resource-limits6spec:7 match:8 kinds:9 - apiGroups: [""]10 kinds: ["Pod"]11 excludedNamespaces: ["kube-system", "cert-manager", "monitoring"]12 parameters:13 limits: ["cpu", "memory"]14 requests: ["cpu", "memory"]
7. Autoscaling {#autoscaling}
Karpenter — Better Than Cluster Autoscaler
yaml1# NodePool — define what nodes Karpenter can provision2apiVersion: karpenter.sh/v1beta13kind: NodePool4metadata:5 name: default6spec:7 template:8 spec:9 nodeClassRef:10 name: default11 requirements:12 - key: karpenter.sh/capacity-type13 operator: In14 values: ["spot", "on-demand"]15 - key: node.kubernetes.io/instance-type16 operator: In17 values: ["m5.large", "m5.xlarge", "m5.2xlarge", "m6i.large", "m6i.xlarge"]18 - key: topology.kubernetes.io/zone19 operator: In20 values: ["us-east-1a", "us-east-1b", "us-east-1c"]21 limits:22 cpu: "1000"23 memory: "4000Gi"24 disruption:25 consolidationPolicy: WhenUnderutilized26 consolidateAfter: 30s27 expireAfter: 720h # Rotate nodes every 30 days
Vertical Pod Autoscaler (Recommendation Mode)
yaml1apiVersion: autoscaling.k8s.io/v12kind: VerticalPodAutoscaler3metadata:4 name: api-server-vpa5 namespace: production6spec:7 targetRef:8 apiVersion: apps/v19 kind: Deployment10 name: api-server11 updatePolicy:12 updateMode: "Off" # Recommendation only, don't auto-apply13 resourcePolicy:14 containerPolicies:15 - containerName: api16 minAllowed:17 cpu: "50m"18 memory: "64Mi"19 maxAllowed:20 cpu: "4"21 memory: "8Gi"
8. Observability {#observability}
Prometheus ServiceMonitor
yaml1apiVersion: monitoring.coreos.com/v12kind: ServiceMonitor3metadata:4 name: api-server-monitor5 namespace: monitoring6 labels:7 release: kube-prometheus-stack8spec:9 selector:10 matchLabels:11 app: api-server12 namespaceSelector:13 matchNames:14 - production15 endpoints:16 - port: http17 path: /metrics18 interval: 15s19 scrapeTimeout: 10s
PrometheusRule — Alerting
yaml1apiVersion: monitoring.coreos.com/v12kind: PrometheusRule3metadata:4 name: api-server-alerts5 namespace: monitoring6spec:7 groups:8 - name: api-server9 interval: 30s10 rules:11 - alert: HighErrorRate12 expr: |13 sum(rate(http_requests_total{job="api-server",status=~"5.."}[5m]))14 /15 sum(rate(http_requests_total{job="api-server"}[5m])) > 0.0116 for: 5m17 labels:18 severity: critical19 team: platform20 annotations:21 summary: "API error rate > 1%"22 description: "Error rate is {{ $value | humanizePercentage }}"23 runbook: "https://runbooks.internal/api-high-error-rate"2425 - alert: PodCrashLooping26 expr: |27 rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 528 for: 0m29 labels:30 severity: critical31 annotations:32 summary: "Pod {{ $labels.pod }} is crash looping"
9. Upgrade Strategy {#upgrades}
bash1# Step 1: Check upgrade path (never skip minor versions)2# 1.28 → 1.29 → 1.30 (not 1.28 → 1.30 directly)34# Step 2: Update kubeadm5apt-mark unhold kubeadm && apt-get install kubeadm=1.30.0-00 && apt-mark hold kubeadm67# Step 3: Verify upgrade plan8kubeadm upgrade plan910# Step 4: Apply upgrade to control plane11kubeadm upgrade apply v1.30.01213# Step 5: Upgrade kubelet on each node (one at a time)14kubectl drain <node> --ignore-daemonsets --delete-emptydir-data15apt-mark unhold kubelet kubectl && \16 apt-get install kubelet=1.30.0-00 kubectl=1.30.0-00 && \17 apt-mark hold kubelet kubectl18systemctl daemon-reload && systemctl restart kubelet19kubectl uncordon <node>2021# Step 6: Verify22kubectl get nodes23kubectl version
10. Production Gotchas {#gotchas}
| Gotcha | Impact | Fix |
|---|---|---|
| CPU limits cause throttling | App becomes slow under load even with spare CPU | Remove CPU limits, keep only requests |
| etcd on slow disk | Slow API server, timeouts, leader elections fail | Use NVMe SSD, keep p99 latency <10ms |
| Default ndots:5 | Every DNS lookup does 5 queries (slow) | Set dnsConfig.options: ndots: 2 |
| No PodDisruptionBudget | Rolling restarts kill all pods of a deployment | Always create PDB for production workloads |
| ImagePullPolicy Always | Pulls image on every pod start, slow and fragile | Use IfNotPresent with immutable image tags |
| Running as root | Security risk, many exploits target this | Set runAsNonRoot: true, runAsUser: 1000 |
| No terminationGracePeriod tuning | In-flight requests killed during shutdown | Set period > max request timeout + 5s buffer |
| Default resource requests only | Node oversubscription, OOMKill storms | Set both requests AND limits for memory |
| NodePort for production traffic | Bypasses ingress, no TLS, port exposure | Always use Ingress + LoadBalancer |
| Secrets in environment variables | Appear in kubectl describe, logs | Use secret volumes or external secrets operator |
11. Essential Commands Cheatsheet {#cheatsheet}
bash1# ── Cluster Info ──────────────────────────────2kubectl cluster-info3kubectl get nodes -o wide4kubectl top nodes5kubectl get events --sort-by='.lastTimestamp' -A67# ── Pods ──────────────────────────────────────8kubectl get pods -A -o wide9kubectl describe pod <pod> -n <ns>10kubectl logs <pod> -n <ns> --previous # Previous container logs11kubectl logs <pod> -n <ns> -c <container> -f # Follow multi-container12kubectl exec -it <pod> -- /bin/sh # Shell into pod13kubectl debug <pod> -it --image=busybox # Debug without modifying pod14kubectl top pod <pod> --containers # Per-container resource usage1516# ── Deployments ───────────────────────────────17kubectl rollout status deployment/<name>18kubectl rollout history deployment/<name>19kubectl rollout undo deployment/<name> # Rollback20kubectl rollout undo deployment/<name> --to-revision=221kubectl set image deployment/<name> <container>=<image>:<tag>22kubectl scale deployment/<name> --replicas=52324# ── Troubleshooting ───────────────────────────25kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'26kubectl get events --field-selector involvedObject.name=<pod>27kubectl auth can-i create pods --as=system:serviceaccount:production:api-sa28kubectl get ep <service> # Check service endpoints2930# ── Resources ─────────────────────────────────31kubectl api-resources --verbs=list -o name32kubectl explain deployment.spec.template.spec.containers33kubectl diff -f deployment.yaml # Diff before apply3435# ── Context / Namespace ───────────────────────36kubectl config get-contexts37kubectl config use-context <context>38kubectl config set-context --current --namespace=production39kubens production # If kubectx installed4041# ── Quick resource creation ───────────────────42kubectl create deployment nginx --image=nginx --replicas=3 --dry-run=client -o yaml43kubectl create service clusterip nginx --tcp=80:80 --dry-run=client -o yaml44kubectl create secret generic mysecret --from-literal=password=secret12345kubectl create configmap myconfig --from-file=config.yaml4647# ── Port forwarding ───────────────────────────48kubectl port-forward svc/<service> 8080:8049kubectl port-forward pod/<pod> 8080:80805051# ── Useful aliases ────────────────────────────52alias k=kubectl53alias kgp='kubectl get pods'54alias kgpa='kubectl get pods -A'55alias kdp='kubectl describe pod'56alias klogs='kubectl logs'