Terraform — Production Guide
Complete Production Implementation Guide
Version: 1.8.x | Category: Infrastructure as Code | Level: Intermediate+
Terraform is the industry standard for infrastructure as code. This guide covers production patterns — state management, modules, workspaces, CI/CD integration, and advanced techniques used at scale.
Table of Contents
- Core Concepts
- Remote State — The Right Way
- Module Architecture
- Workspaces vs Directory Structure
- Production AWS VPC Example
- CI/CD Integration
- Testing Terraform
- Import & Refactoring
- Production Gotchas
- Essential Commands
1. Core Concepts {#concepts}
The Terraform Workflow
Write → Init → Plan → Apply → Destroy
- Write: HCL configuration files (
.tf) - Init: Download providers, configure backend
- Plan: Dry run — shows exactly what will change
- Apply: Execute the plan (creates/updates/destroys resources)
- Destroy: Remove all managed resources (careful!)
File Structure Convention
infrastructure/
├── modules/ # Reusable modules
│ ├── vpc/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ ├── outputs.tf
│ │ └── README.md
│ ├── eks/
│ └── rds/
├── environments/
│ ├── production/
│ │ ├── main.tf # Calls modules
│ │ ├── variables.tf
│ │ ├── terraform.tfvars # Env-specific values (gitignored for secrets)
│ │ ├── backend.tf # Remote state config
│ │ └── providers.tf
│ ├── staging/
│ └── development/
└── global/ # Cross-env resources (Route53, IAM roles)
├── main.tf
└── backend.tf
2. Remote State — The Right Way {#state}
Bootstrap State Backend (S3 + DynamoDB)
hcl1# Create this ONCE manually or with a bootstrap script2# terraform/bootstrap/main.tf34resource "aws_s3_bucket" "tfstate" {5 bucket = "mycompany-terraform-state-prod"67 # Prevent accidental deletion8 lifecycle {9 prevent_destroy = true10 }11}1213resource "aws_s3_bucket_versioning" "tfstate" {14 bucket = aws_s3_bucket.tfstate.id15 versioning_configuration {16 status = "Enabled" # Critical: rollback if state corrupted17 }18}1920resource "aws_s3_bucket_server_side_encryption_configuration" "tfstate" {21 bucket = aws_s3_bucket.tfstate.id22 rule {23 apply_server_side_encryption_by_default {24 sse_algorithm = "AES256"25 }26 }27}2829resource "aws_s3_bucket_public_access_block" "tfstate" {30 bucket = aws_s3_bucket.tfstate.id31 block_public_acls = true32 block_public_policy = true33 ignore_public_acls = true34 restrict_public_buckets = true35}3637resource "aws_dynamodb_table" "tf_locks" {38 name = "terraform-state-locks"39 billing_mode = "PAY_PER_REQUEST"40 hash_key = "LockID"4142 attribute {43 name = "LockID"44 type = "S"45 }4647 server_side_encryption {48 enabled = true49 }50}
Backend Configuration
hcl1# environments/production/backend.tf2terraform {3 required_version = ">= 1.6.0, < 2.0.0"45 required_providers {6 aws = {7 source = "hashicorp/aws"8 version = "~> 5.0" # Tilde: allow 5.x but not 6.x9 }10 kubernetes = {11 source = "hashicorp/kubernetes"12 version = "~> 2.0"13 }14 }1516 backend "s3" {17 bucket = "mycompany-terraform-state-prod"18 key = "production/main/terraform.tfstate"19 region = "us-east-1"20 encrypt = true21 dynamodb_table = "terraform-state-locks"2223 # Optional: role assumption for cross-account24 role_arn = "arn:aws:iam::PROD_ACCOUNT_ID:role/TerraformRole"25 }26}2728provider "aws" {29 region = var.aws_region3031 default_tags {32 tags = {33 Environment = var.environment34 ManagedBy = "Terraform"35 Repository = "mycompany/infrastructure"36 }37 }38}
3. Module Architecture {#modules}
Writing a Reusable Module
hcl1# modules/rds/variables.tf2variable "identifier" {3 description = "RDS instance identifier"4 type = string5}67variable "engine_version" {8 description = "PostgreSQL version"9 type = string10 default = "16.2"11}1213variable "instance_class" {14 description = "RDS instance type"15 type = string16 default = "db.t3.medium"17}1819variable "allocated_storage" {20 description = "Storage in GB"21 type = number22 default = 1002324 validation {25 condition = var.allocated_storage >= 2026 error_message = "Minimum storage is 20GB"27 }28}2930variable "multi_az" {31 description = "Enable Multi-AZ for HA"32 type = bool33 default = true34}3536variable "vpc_id" {37 type = string38}3940variable "subnet_ids" {41 type = list(string)42}4344variable "allowed_security_group_ids" {45 type = list(string)46 default = []47}4849variable "tags" {50 type = map(string)51 default = {}52}
hcl1# modules/rds/main.tf2resource "aws_db_subnet_group" "this" {3 name = "${var.identifier}-subnet-group"4 subnet_ids = var.subnet_ids5 tags = var.tags6}78resource "aws_security_group" "rds" {9 name = "${var.identifier}-rds-sg"10 description = "Security group for ${var.identifier} RDS"11 vpc_id = var.vpc_id1213 ingress {14 from_port = 543215 to_port = 543216 protocol = "tcp"17 security_groups = var.allowed_security_group_ids18 description = "PostgreSQL from application"19 }2021 egress {22 from_port = 023 to_port = 024 protocol = "-1"25 cidr_blocks = ["0.0.0.0/0"]26 }2728 tags = var.tags29}3031resource "random_password" "master" {32 length = 3233 special = true34 override_special = "!#$%&*()-_=+[]{}<>:?"35}3637resource "aws_secretsmanager_secret" "rds_password" {38 name = "${var.identifier}/master-password"39 recovery_window_in_days = 740 tags = var.tags41}4243resource "aws_secretsmanager_secret_version" "rds_password" {44 secret_id = aws_secretsmanager_secret.rds_password.id45 secret_string = jsonencode({46 username = "postgres"47 password = random_password.master.result48 host = aws_db_instance.this.address49 port = 543250 dbname = var.identifier51 })52}5354resource "aws_db_instance" "this" {55 identifier = var.identifier56 engine = "postgres"57 engine_version = var.engine_version58 instance_class = var.instance_class59 allocated_storage = var.allocated_storage60 storage_type = "gp3"61 storage_encrypted = true6263 db_name = replace(var.identifier, "-", "_")64 username = "postgres"65 password = random_password.master.result6667 db_subnet_group_name = aws_db_subnet_group.this.name68 vpc_security_group_ids = [aws_security_group.rds.id]6970 multi_az = var.multi_az71 publicly_accessible = false72 deletion_protection = true # Prevent accidental deletion!7374 backup_retention_period = 3075 backup_window = "03:00-04:00"76 maintenance_window = "Mon:04:00-Mon:05:00"7778 performance_insights_enabled = true79 performance_insights_retention_period = 780 monitoring_interval = 6081 monitoring_role_arn = aws_iam_role.rds_monitoring.arn8283 auto_minor_version_upgrade = true84 copy_tags_to_snapshot = true8586 tags = var.tags8788 lifecycle {89 prevent_destroy = true90 ignore_changes = [password] # Managed by rotation91 }92}
hcl1# modules/rds/outputs.tf2output "endpoint" {3 description = "RDS endpoint"4 value = aws_db_instance.this.address5 sensitive = false6}78output "port" {9 value = aws_db_instance.this.port10}1112output "secret_arn" {13 description = "Secrets Manager ARN containing connection details"14 value = aws_secretsmanager_secret.rds_password.arn15}1617output "security_group_id" {18 value = aws_security_group.rds.id19}
Calling the Module
hcl1# environments/production/main.tf2module "rds" {3 source = "../../modules/rds"45 identifier = "myapp-production"6 instance_class = "db.r6g.xlarge"7 multi_az = true8 allocated_storage = 500910 vpc_id = module.vpc.vpc_id11 subnet_ids = module.vpc.database_subnet_ids1213 allowed_security_group_ids = [14 module.eks.worker_security_group_id,15 ]1617 tags = local.common_tags18}
4. Workspaces vs Directory Structure {#workspaces}
Recommendation: Use Directory Structure (Not Workspaces)
Workspaces are confusing and risky for production. One accidental terraform workspace select production in the wrong directory is catastrophic. Instead:
# ✅ Good: Separate directories per environment
environments/
production/ → separate state file, separate apply, no risk of cross-contamination
staging/ → separate state file
development/ → separate state file
# ❌ Risky: Workspaces
# One directory, switching between workspaces
# Risk: apply prod state while thinking you're in staging
When Workspaces Are OK
Use workspaces only for truly identical environments (e.g., ephemeral preview environments per PR) where state contamination doesn't matter:
bash1terraform workspace new pr-4562terraform apply -var="environment=pr-456"3# On PR close:4terraform destroy5terraform workspace delete pr-456
5. Production AWS VPC Example {#vpc}
hcl1# environments/production/vpc.tf2module "vpc" {3 source = "terraform-aws-modules/vpc/aws"4 version = "~> 5.5"56 name = "${local.name_prefix}-vpc"7 cidr = "10.0.0.0/16"89 azs = ["us-east-1a", "us-east-1b", "us-east-1c"]1011 # 3-tier subnet design12 private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]13 public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]14 database_subnets = ["10.0.201.0/24", "10.0.202.0/24", "10.0.203.0/24"]1516 # One NAT GW per AZ for true HA (3x cost vs single NAT)17 enable_nat_gateway = true18 single_nat_gateway = false # HA: one per AZ19 one_nat_gateway_per_az = true2021 enable_dns_hostnames = true22 enable_dns_support = true2324 # VPC Flow Logs — required for security compliance25 enable_flow_log = true26 create_flow_log_cloudwatch_iam_role = true27 create_flow_log_cloudwatch_log_group = true28 flow_log_max_aggregation_interval = 602930 # Tags required for EKS31 public_subnet_tags = {32 "kubernetes.io/role/elb" = 133 }34 private_subnet_tags = {35 "kubernetes.io/role/internal-elb" = 136 "karpenter.sh/discovery" = "${local.name_prefix}-eks"37 }3839 tags = local.common_tags40}
6. CI/CD Integration {#cicd}
GitHub Actions Pipeline
yaml1# .github/workflows/terraform.yml2name: Terraform34on:5 push:6 branches: [main]7 paths: ['infrastructure/**']8 pull_request:9 branches: [main]10 paths: ['infrastructure/**']1112env:13 TF_VERSION: "1.8.0"14 WORKING_DIR: "infrastructure/environments/production"1516jobs:17 plan:18 name: Terraform Plan19 runs-on: ubuntu-latest20 permissions:21 contents: read22 pull-requests: write23 id-token: write # OIDC for AWS2425 steps:26 - uses: actions/checkout@v42728 - name: Configure AWS credentials (OIDC — no long-lived keys!)29 uses: aws-actions/configure-aws-credentials@v430 with:31 role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/GitHubActionsRole32 aws-region: us-east-13334 - uses: hashicorp/setup-terraform@v335 with:36 terraform_version: ${{ env.TF_VERSION }}3738 - name: Terraform Format Check39 run: terraform fmt -check -recursive40 working-directory: ${{ env.WORKING_DIR }}4142 - name: Terraform Init43 run: terraform init -input=false44 working-directory: ${{ env.WORKING_DIR }}4546 - name: Terraform Validate47 run: terraform validate48 working-directory: ${{ env.WORKING_DIR }}4950 - name: Terraform Plan51 id: plan52 run: |53 terraform plan -input=false -no-color -out=tfplan \54 -var-file="terraform.tfvars" 2>&1 | tee plan_output.txt55 echo "exitcode=${PIPESTATUS[0]}" >> $GITHUB_OUTPUT56 working-directory: ${{ env.WORKING_DIR }}57 continue-on-error: true5859 - name: Post Plan to PR60 uses: actions/github-script@v761 if: github.event_name == 'pull_request'62 with:63 script: |64 const fs = require('fs');65 const plan = fs.readFileSync('${{ env.WORKING_DIR }}/plan_output.txt', 'utf8');66 const truncated = plan.length > 60000 ? plan.substring(0, 60000) + '\n... truncated' : plan;67 github.rest.issues.createComment({68 issue_number: context.issue.number,69 owner: context.repo.owner,70 repo: context.repo.repo,71 body: `## Terraform Plan\n\`\`\`hcl\n${truncated}\n\`\`\``72 });7374 - name: Fail if plan failed75 if: steps.plan.outputs.exitcode == '1'76 run: exit 17778 apply:79 name: Terraform Apply80 runs-on: ubuntu-latest81 needs: plan82 if: github.ref == 'refs/heads/main' && github.event_name == 'push'83 environment: production # Requires manual approval in GitHub8485 steps:86 - uses: actions/checkout@v487 - uses: aws-actions/configure-aws-credentials@v488 with:89 role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/GitHubActionsRole90 aws-region: us-east-191 - uses: hashicorp/setup-terraform@v392 with:93 terraform_version: ${{ env.TF_VERSION }}94 - run: terraform init -input=false95 working-directory: ${{ env.WORKING_DIR }}96 - run: terraform apply -auto-approve -input=false -var-file="terraform.tfvars"97 working-directory: ${{ env.WORKING_DIR }}
7. Testing Terraform {#testing}
bash1# Static analysis with tflint2tflint --init3tflint --recursive45# Security scanning with tfsec6tfsec .78# Policy as code with Checkov9checkov -d . --framework terraform1011# Built-in test framework (Terraform 1.6+)12# tests/vpc_test.tftest.hcl13run "creates_vpc_with_correct_cidr" {14 command = plan1516 assert {17 condition = module.vpc.vpc_cidr_block == "10.0.0.0/16"18 error_message = "VPC CIDR must be 10.0.0.0/16"19 }20}
8. Import & Refactoring {#import}
bash1# Import existing resource into state2terraform import aws_s3_bucket.my_bucket my-existing-bucket-name34# Generate config for existing resource (Terraform 1.5+)5terraform plan -generate-config-out=generated.tf67# Move resource in state without destroying (refactoring)8# terraform.tf9moved {10 from = aws_s3_bucket.old_name11 to = module.storage.aws_s3_bucket.this12}1314# Remove resource from state without destroying15terraform state rm aws_s3_bucket.my_bucket1617# List all resources in state18terraform state list1920# Show specific resource state21terraform state show aws_s3_bucket.my_bucket
9. Production Gotchas {#gotchas}
| Gotcha | Impact | Fix |
|---|---|---|
| terraform destroy without -target | Destroys ALL managed resources | Always use -target for partial destroys |
| Provider version not pinned | Minor version update breaks resources | Always pin: version = "~> 5.0" |
| Storing secrets in .tfvars | Secrets committed to git | Use Vault, AWS Secrets Manager, or env vars |
| No prevent_destroy lifecycle | Accidental deletion of DBs, S3 buckets | Add lifecycle { prevent_destroy = true } to critical resources |
| Force-unlock without verification | Corrupts state if another plan is running | Verify no running plan before terraform force-unlock |
| Not using -out flag in CI | Apply may differ from plan if resources changed between plan and apply | Always: plan -out=tfplan, apply tfplan |
| count vs for_each for resources | Removing item from middle of count list destroys and recreates all after it | Use for_each with maps instead of count with lists |
| Hardcoded region | Breaks when deploying to another region | Use variables for all region-specific values |
| Large state files | Slow plans, hard to debug | Split into multiple state files per concern |
| terraform apply -auto-approve in production | No human review of changes | Always require manual approval for production |
10. Essential Commands {#commands}
bash1# ── Core workflow ─────────────────────────────2terraform init # Initialize (always run after any change to providers/backend)3terraform init -upgrade # Upgrade provider versions4terraform validate # Check syntax5terraform fmt -recursive # Format all .tf files6terraform plan -var-file=prod.tfvars # Dry run7terraform plan -out=tfplan # Save plan for later apply8terraform apply tfplan # Apply saved plan9terraform apply -target=module.vpc # Apply specific resource10terraform destroy -target=aws_instance.test # Destroy specific resource1112# ── State management ──────────────────────────13terraform state list # List all resources14terraform state show <resource> # Show resource details15terraform state mv <old> <new> # Move/rename in state16terraform state rm <resource> # Remove from state (keeps real resource)17terraform state pull > backup.tfstate # Backup state18terraform force-unlock <LOCK_ID> # Force release stuck lock1920# ── Inspection ────────────────────────────────21terraform output # Show all outputs22terraform output -json # JSON format23terraform show # Human-readable state24terraform graph | dot -Tsvg > graph.svg # Visualize dependencies2526# ── Debug ─────────────────────────────────────27TF_LOG=DEBUG terraform plan # Verbose logging28TF_LOG_PATH=./terraform.log terraform plan # Log to file29terraform console # Interactive expression testing