Secrets Management
Secrets are the crown jewels of your infrastructure. Learn why secrets leak into git and images, and how to store, inject, rotate and audit them with a dedicated secrets manager.
What is this?
A secret is any credential that grants access: API keys, tokens, passwords, certificates, database credentials. Secrets management is the discipline of storing them encrypted in a dedicated store, injecting them at runtime, rotating them regularly, and auditing every read. The enemy is secrets committed to git, baked into images, or hardcoded in config.
Why do you need it?
- A committed or baked-in secret is a permanent compromise: attackers mine git history and image layers continuously.
- Hardcoded secrets cannot be rotated quickly or revoked per-team, and nobody knows who has them.
- Central stores give you one audit trail, one encryption boundary and one rotation workflow.
- Short-lived, automatically rotated credentials turn a leaked secret from a disaster into a minor incident.
- Compliance frameworks (SOC 2, PCI-DSS) explicitly require protection and rotation of secrets.
Mental model
A house key under the doormat
A secret committed to a repo
Everyone knows to look under the mat. Git history and image layers are the doormat of the modern era — attackers check them first.
How it works
- A secrets manager (Vault, cloud KMS + Secret Manager) stores secrets encrypted at rest with centrally managed keys.
- Applications authenticate to the store using a short-lived token or workload identity, not a long-lived key.
- Secrets are fetched at startup or mounted as files, never baked into images or config.
- Dynamic secrets are minted on demand with leases — each app instance gets unique, expiring credentials.
- Rotation replaces values on a schedule and propagates them to consumers, with read/access logging throughout.
Runtime secret delivery, from store to process.
Key commands
grep -rniE "(password|secret|api[_-]?key|token)" . --exclude-dir=.gitScan a repo for plaintext credential patterns.
vault kv get secret/ci/pipelineFetch a secret value from a Vault KV store (audited).
vault kv put secret/ci/pipeline db_password=...Write a secret into the encrypted store.
git log --all --oneline -- .envCheck whether a sensitive file ever entered git history.
env | grep -iE "key|token|secret" | sortReview which secrets are currently exported in the environment.
Examples
Find secrets before they ship
grep -rn -iE "(password|api[_-]?key|secret|token)=" . --exclude-dir=node_modules | head -10git log --oneline --all | head -5git show $(git log --all --oneline | tail -1 | cut -d' ' -f1) -- .env 2>/dev/nullThe grep scans the tree for credential-looking lines; the git commands check whether a secret may already be in history. If grep finds anything, treat the secret as compromised.
In production
- Kubernetes mounts secrets as files via an external-secrets operator instead of hardcoding them in manifests.
- CI pipelines pull credentials from Vault or the provider's secret store at job start, never from repo variables.
- Database and cloud credentials are dynamic and short-lived; rotation is automatic.
- Secret scanning runs on every push and blocks the merge when a known pattern is detected.
- Read access to secrets is logged and periodically reviewed, so leaks are discoverable.
Best practices
- Never commit secrets; use a scanner in CI to enforce it.
- Store secrets in a dedicated, encrypted secrets manager, not in the repo or image.
- Inject at runtime (env, mounted files, vault agent) rather than baking into artifacts.
- Rotate secrets on a schedule and immediately after any suspected exposure.
- Use short-lived, scoped credentials so one leaked secret has a tiny blast radius.
Common mistakes
- Committing .env files just for local dev.
- Using ENV instructions in Dockerfiles, which bake the value into the image.
- Storing secrets in repo variables and calling it secret management.
- Rotating a secret without updating all consumers, causing outages.
- Reusing the same key across environments and services.
Common issues
Scanner blocks a merge on a false positiveA test string or example value matches a credential pattern
Use allowed-by-path rules for fixtures, or generate obviously-fake test values that scanners ignore.
App crashes after rotation because it still uses the old valueThe app caches the secret or the consumer was not updated
Make consumers fetch at startup or support watch/reload, and run rotation playbooks before cutting over.
Vault returns 403 permission deniedThe workload identity lacks a policy for that path
Grant the exact read policy to the service's role and verify the token is valid and unexpired.
Debugging tips
- Run a secret scanner locally on the exact branch that failed CI to reproduce the finding.
- Check image layers with docker history when a secret is suspected inside a container.
- Verify token revocation actually worked by attempting one request with the old credential.
- Watch the secrets manager audit log to see which service read a secret and when.
Interview checkpoint
Hands-on lab
- Commit a secret by accident
- Remove it from the working tree
- Prove it survives in git history
- Understand rotation as the real fix
Secrets in Git History
25 min · intermediate
Quick quiz
Where is the safest place to store a production API key?
Deleting a file from a git repo removes it from git history.
What is the correct first response to a suspected secret leak?
Cheat sheet
grep -rniE '(key|secret|token)='Scan for plaintext patternsgit log --all -- .envCheck history for sensitive filesdocker history IMGInspect image layers for baked secretssecret scanner in CIBlock merges on credential patternsvault kv get secret/appFetch a value with audit trailvault kv put secret/app k=vStore a new valueexternal-secretsSync Vault/cloud secrets into Kubernetesdynamic + leasedMint expiring credentials per workload