Cloud Security
Intermediate 13 min read Cloud Security

Secrets Management

Live vaults, not hardcoded hope

Secrets — API keys, DB passwords, tokens — need a live store with rotation and audit, not environment files. Secrets Manager, Key Vault, and Secret Manager share the pattern: versioned values, rotation schedules, granular access, and client caching with graceful re-read.

secretSecrets ManagerKey VaultSecret ManagerrotationVaultlease
Prerequisites: Encryption at Rest & In Transit
Before we start

What you will learn

  • Move secrets out of code/config into a managed store
  • Configure rotation + audit + minimum access
  • Integrate at runtime (env, mount, SDK) without downtime
In 60 seconds

A secrets manager stores name→versioned value, rotates on schedule, grants access via identity, and records every read. Runtime clients fetch or cache; rotation causes a version bump that downstream systems re-read safely.

Introduction

What is this?

Products: AWS Secrets Manager, Azure Key Vault (secrets), GCP Secret Manager, HashiCorp Vault (flexible + dynamic leases), Kubernetes + External Secrets/SealedSecrets. Features: versioning, rotation (lambda-trigger auto-rotation; Key Vault near-expiry notifications), access control (IAM/KV RBAC/SM policies), secret rotation handlers (DB creds + app creds), replication (cross-region), audit (get/set/describe), recovery (soft delete/deletion protection), caching (SDKs cache with TTL; files mounted at the kubelet). Anti-secrets: env vars in actions/config files, base64 'duh' secrets, image-layer history, notebook leakage. Dynamic secrets: Vault short-lived leases (TTL-scoped DB creds).

Motivation

Why do you need it?

  • Hardcoded secrets rotate never; a store makes rotation a function call.
  • Audit trails point to exactly who/what read a secret — irreplaceable in incidents.
  • Dynamic leases shrink the lifetime window of any leaked credential.
  • K8s-native integration keeps secrets out of images and git history.
Think about it

Mental model

Imagine

A bank's vault with scheduled re-keying and a guard logbook

Think of

Secrets manager

The vault rotates its combination on a date (rotation), everyone with a key signs the logbook (audit), and nobody walks around with the master combo pinned to their shirt (no hardcoded secrets).

Mechanics

How it works

  • You create a secret with versions (current + previous).
  • Identity-checked callers read the secret (SDK/CLI/mount).
  • Rotation handler updates a new version atomically; applications observe the version change.
  • Pipeline/CI fetches secrets at build via integration (actions/vault role), not repo files.
  • K8s: External Secrets syncs store values into a Secret object on schedule; pods mount files/env.

Secret lifecycle.

Create
Access
Rotate
Re-read
Audit
Hands-on

Examples

Runtime read with version tolerance (SDK-ish)

Runtime read with version tolerance (SDK-ish)
# app on boot: fetch once, cache for TTL, re-fetch on version change
const { secretValue } = await secrets.getSecretValue({ SecretId: 'prod/db' })
let cached = secretValue
// rotation-aware:
if (versionChanged(cached)) cached = await secrets.getSecretValue(...)
# or kube: mount -> file watcher re-reads on rotation

Caching with a version check turns rotation into a seamless background event for your app.

Real world

In production

  • DB credentials rotated weekly with a lambda; app pools reconnect on version bump.
  • CI fetch from store with ephemeral audience instead of yaml 'db_password:'.
  • K8s workloads get Secrets via External Secrets sync, rotated without pod restarts.
Guidelines

Best practices

  • Treat any secret found in a repo/log as compromised; rotate it now.
  • Scope secret permissions to the exact caller/service; never 'all secrets' access.
  • Rotation must be tested: rotate in staging, watch re-reads + reconnects.
  • Enable deletion protection/soft delete and cross-region replication where compliance needs it.
  • Mount secrets as files or envs via the runtime, never as image layers.
Watch out

Common mistakes

  • S3 bucket with a JSON of secrets because 'it works'.
  • Rotation handler that breaks app pools (no re-read strategy) → first rotation = outage.
  • One omnipotent secret owner token used everywhere (no per-secret audit).
  • Secrets Manager without deletion protection → a wrong `delete-secret` is an incident.
When things break

Debugging tips

  • 'No permission' reading secret → check the identity's IAM/KeyVault policy for that secret's name, not the account.
  • Rotation changed but app still uses old value → caching TTL; implement version re-read.
  • Secret shows previous version → rotation handler timing/failure; inspect function logs.
  • Cross-region replication lag → replication status field per secret.
Interviews

Interview checkpoint

Practice

Hands-on lab

Rotate a Secret Without the App Blinking
40 min · intermediate
In progress
Objectives
  • Create + version a secret locally, with rotation simulation
  • Read current value with a version-aware caching app
  • Simulate rotation and prove a clean re-read

Rotate a Secret Without the App Blinking

40 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The first action after finding a secret in git is:

2

Deletion protection exists to:

Go deeper

Official documentation

AWS Secrets Manager
docs
Azure Key Vault
docs
GCP Secret Manager
docs
HashiCorp Vault secrets
docs