Terraform — Production Guide

Complete Production Implementation Guide

Version: 1.8.x | Category: Infrastructure as Code | Level: Intermediate+

Terraform is the industry standard for infrastructure as code. This guide covers production patterns — state management, modules, workspaces, CI/CD integration, and advanced techniques used at scale.


Table of Contents

  1. Core Concepts
  2. Remote State — The Right Way
  3. Module Architecture
  4. Workspaces vs Directory Structure
  5. Production AWS VPC Example
  6. CI/CD Integration
  7. Testing Terraform
  8. Import & Refactoring
  9. Production Gotchas
  10. Essential Commands

1. Core Concepts {#concepts}

The Terraform Workflow

Write → Init → Plan → Apply → Destroy
  • Write: HCL configuration files (.tf)
  • Init: Download providers, configure backend
  • Plan: Dry run — shows exactly what will change
  • Apply: Execute the plan (creates/updates/destroys resources)
  • Destroy: Remove all managed resources (careful!)

File Structure Convention

infrastructure/
├── modules/                  # Reusable modules
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── eks/
│   └── rds/
├── environments/
│   ├── production/
│   │   ├── main.tf           # Calls modules
│   │   ├── variables.tf
│   │   ├── terraform.tfvars  # Env-specific values (gitignored for secrets)
│   │   ├── backend.tf        # Remote state config
│   │   └── providers.tf
│   ├── staging/
│   └── development/
└── global/                   # Cross-env resources (Route53, IAM roles)
    ├── main.tf
    └── backend.tf

2. Remote State — The Right Way {#state}

Bootstrap State Backend (S3 + DynamoDB)

hcl
1# Create this ONCE manually or with a bootstrap script
2# terraform/bootstrap/main.tf
3
4resource "aws_s3_bucket" "tfstate" {
5 bucket = "mycompany-terraform-state-prod"
6
7 # Prevent accidental deletion
8 lifecycle {
9 prevent_destroy = true
10 }
11}
12
13resource "aws_s3_bucket_versioning" "tfstate" {
14 bucket = aws_s3_bucket.tfstate.id
15 versioning_configuration {
16 status = "Enabled" # Critical: rollback if state corrupted
17 }
18}
19
20resource "aws_s3_bucket_server_side_encryption_configuration" "tfstate" {
21 bucket = aws_s3_bucket.tfstate.id
22 rule {
23 apply_server_side_encryption_by_default {
24 sse_algorithm = "AES256"
25 }
26 }
27}
28
29resource "aws_s3_bucket_public_access_block" "tfstate" {
30 bucket = aws_s3_bucket.tfstate.id
31 block_public_acls = true
32 block_public_policy = true
33 ignore_public_acls = true
34 restrict_public_buckets = true
35}
36
37resource "aws_dynamodb_table" "tf_locks" {
38 name = "terraform-state-locks"
39 billing_mode = "PAY_PER_REQUEST"
40 hash_key = "LockID"
41
42 attribute {
43 name = "LockID"
44 type = "S"
45 }
46
47 server_side_encryption {
48 enabled = true
49 }
50}

Backend Configuration

hcl
1# environments/production/backend.tf
2terraform {
3 required_version = ">= 1.6.0, < 2.0.0"
4
5 required_providers {
6 aws = {
7 source = "hashicorp/aws"
8 version = "~> 5.0" # Tilde: allow 5.x but not 6.x
9 }
10 kubernetes = {
11 source = "hashicorp/kubernetes"
12 version = "~> 2.0"
13 }
14 }
15
16 backend "s3" {
17 bucket = "mycompany-terraform-state-prod"
18 key = "production/main/terraform.tfstate"
19 region = "us-east-1"
20 encrypt = true
21 dynamodb_table = "terraform-state-locks"
22
23 # Optional: role assumption for cross-account
24 role_arn = "arn:aws:iam::PROD_ACCOUNT_ID:role/TerraformRole"
25 }
26}
27
28provider "aws" {
29 region = var.aws_region
30
31 default_tags {
32 tags = {
33 Environment = var.environment
34 ManagedBy = "Terraform"
35 Repository = "mycompany/infrastructure"
36 }
37 }
38}

3. Module Architecture {#modules}

Writing a Reusable Module

hcl
1# modules/rds/variables.tf
2variable "identifier" {
3 description = "RDS instance identifier"
4 type = string
5}
6
7variable "engine_version" {
8 description = "PostgreSQL version"
9 type = string
10 default = "16.2"
11}
12
13variable "instance_class" {
14 description = "RDS instance type"
15 type = string
16 default = "db.t3.medium"
17}
18
19variable "allocated_storage" {
20 description = "Storage in GB"
21 type = number
22 default = 100
23
24 validation {
25 condition = var.allocated_storage >= 20
26 error_message = "Minimum storage is 20GB"
27 }
28}
29
30variable "multi_az" {
31 description = "Enable Multi-AZ for HA"
32 type = bool
33 default = true
34}
35
36variable "vpc_id" {
37 type = string
38}
39
40variable "subnet_ids" {
41 type = list(string)
42}
43
44variable "allowed_security_group_ids" {
45 type = list(string)
46 default = []
47}
48
49variable "tags" {
50 type = map(string)
51 default = {}
52}
hcl
1# modules/rds/main.tf
2resource "aws_db_subnet_group" "this" {
3 name = "${var.identifier}-subnet-group"
4 subnet_ids = var.subnet_ids
5 tags = var.tags
6}
7
8resource "aws_security_group" "rds" {
9 name = "${var.identifier}-rds-sg"
10 description = "Security group for ${var.identifier} RDS"
11 vpc_id = var.vpc_id
12
13 ingress {
14 from_port = 5432
15 to_port = 5432
16 protocol = "tcp"
17 security_groups = var.allowed_security_group_ids
18 description = "PostgreSQL from application"
19 }
20
21 egress {
22 from_port = 0
23 to_port = 0
24 protocol = "-1"
25 cidr_blocks = ["0.0.0.0/0"]
26 }
27
28 tags = var.tags
29}
30
31resource "random_password" "master" {
32 length = 32
33 special = true
34 override_special = "!#$%&*()-_=+[]{}<>:?"
35}
36
37resource "aws_secretsmanager_secret" "rds_password" {
38 name = "${var.identifier}/master-password"
39 recovery_window_in_days = 7
40 tags = var.tags
41}
42
43resource "aws_secretsmanager_secret_version" "rds_password" {
44 secret_id = aws_secretsmanager_secret.rds_password.id
45 secret_string = jsonencode({
46 username = "postgres"
47 password = random_password.master.result
48 host = aws_db_instance.this.address
49 port = 5432
50 dbname = var.identifier
51 })
52}
53
54resource "aws_db_instance" "this" {
55 identifier = var.identifier
56 engine = "postgres"
57 engine_version = var.engine_version
58 instance_class = var.instance_class
59 allocated_storage = var.allocated_storage
60 storage_type = "gp3"
61 storage_encrypted = true
62
63 db_name = replace(var.identifier, "-", "_")
64 username = "postgres"
65 password = random_password.master.result
66
67 db_subnet_group_name = aws_db_subnet_group.this.name
68 vpc_security_group_ids = [aws_security_group.rds.id]
69
70 multi_az = var.multi_az
71 publicly_accessible = false
72 deletion_protection = true # Prevent accidental deletion!
73
74 backup_retention_period = 30
75 backup_window = "03:00-04:00"
76 maintenance_window = "Mon:04:00-Mon:05:00"
77
78 performance_insights_enabled = true
79 performance_insights_retention_period = 7
80 monitoring_interval = 60
81 monitoring_role_arn = aws_iam_role.rds_monitoring.arn
82
83 auto_minor_version_upgrade = true
84 copy_tags_to_snapshot = true
85
86 tags = var.tags
87
88 lifecycle {
89 prevent_destroy = true
90 ignore_changes = [password] # Managed by rotation
91 }
92}
hcl
1# modules/rds/outputs.tf
2output "endpoint" {
3 description = "RDS endpoint"
4 value = aws_db_instance.this.address
5 sensitive = false
6}
7
8output "port" {
9 value = aws_db_instance.this.port
10}
11
12output "secret_arn" {
13 description = "Secrets Manager ARN containing connection details"
14 value = aws_secretsmanager_secret.rds_password.arn
15}
16
17output "security_group_id" {
18 value = aws_security_group.rds.id
19}

Calling the Module

hcl
1# environments/production/main.tf
2module "rds" {
3 source = "../../modules/rds"
4
5 identifier = "myapp-production"
6 instance_class = "db.r6g.xlarge"
7 multi_az = true
8 allocated_storage = 500
9
10 vpc_id = module.vpc.vpc_id
11 subnet_ids = module.vpc.database_subnet_ids
12
13 allowed_security_group_ids = [
14 module.eks.worker_security_group_id,
15 ]
16
17 tags = local.common_tags
18}

4. Workspaces vs Directory Structure {#workspaces}

Recommendation: Use Directory Structure (Not Workspaces)

Workspaces are confusing and risky for production. One accidental terraform workspace select production in the wrong directory is catastrophic. Instead:

# ✅ Good: Separate directories per environment
environments/
  production/   → separate state file, separate apply, no risk of cross-contamination
  staging/      → separate state file
  development/  → separate state file

# ❌ Risky: Workspaces
# One directory, switching between workspaces
# Risk: apply prod state while thinking you're in staging

When Workspaces Are OK

Use workspaces only for truly identical environments (e.g., ephemeral preview environments per PR) where state contamination doesn't matter:

bash
1terraform workspace new pr-456
2terraform apply -var="environment=pr-456"
3# On PR close:
4terraform destroy
5terraform workspace delete pr-456

5. Production AWS VPC Example {#vpc}

hcl
1# environments/production/vpc.tf
2module "vpc" {
3 source = "terraform-aws-modules/vpc/aws"
4 version = "~> 5.5"
5
6 name = "${local.name_prefix}-vpc"
7 cidr = "10.0.0.0/16"
8
9 azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
10
11 # 3-tier subnet design
12 private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
13 public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
14 database_subnets = ["10.0.201.0/24", "10.0.202.0/24", "10.0.203.0/24"]
15
16 # One NAT GW per AZ for true HA (3x cost vs single NAT)
17 enable_nat_gateway = true
18 single_nat_gateway = false # HA: one per AZ
19 one_nat_gateway_per_az = true
20
21 enable_dns_hostnames = true
22 enable_dns_support = true
23
24 # VPC Flow Logs — required for security compliance
25 enable_flow_log = true
26 create_flow_log_cloudwatch_iam_role = true
27 create_flow_log_cloudwatch_log_group = true
28 flow_log_max_aggregation_interval = 60
29
30 # Tags required for EKS
31 public_subnet_tags = {
32 "kubernetes.io/role/elb" = 1
33 }
34 private_subnet_tags = {
35 "kubernetes.io/role/internal-elb" = 1
36 "karpenter.sh/discovery" = "${local.name_prefix}-eks"
37 }
38
39 tags = local.common_tags
40}

6. CI/CD Integration {#cicd}

GitHub Actions Pipeline

yaml
1# .github/workflows/terraform.yml
2name: Terraform
3
4on:
5 push:
6 branches: [main]
7 paths: ['infrastructure/**']
8 pull_request:
9 branches: [main]
10 paths: ['infrastructure/**']
11
12env:
13 TF_VERSION: "1.8.0"
14 WORKING_DIR: "infrastructure/environments/production"
15
16jobs:
17 plan:
18 name: Terraform Plan
19 runs-on: ubuntu-latest
20 permissions:
21 contents: read
22 pull-requests: write
23 id-token: write # OIDC for AWS
24
25 steps:
26 - uses: actions/checkout@v4
27
28 - name: Configure AWS credentials (OIDC — no long-lived keys!)
29 uses: aws-actions/configure-aws-credentials@v4
30 with:
31 role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/GitHubActionsRole
32 aws-region: us-east-1
33
34 - uses: hashicorp/setup-terraform@v3
35 with:
36 terraform_version: ${{ env.TF_VERSION }}
37
38 - name: Terraform Format Check
39 run: terraform fmt -check -recursive
40 working-directory: ${{ env.WORKING_DIR }}
41
42 - name: Terraform Init
43 run: terraform init -input=false
44 working-directory: ${{ env.WORKING_DIR }}
45
46 - name: Terraform Validate
47 run: terraform validate
48 working-directory: ${{ env.WORKING_DIR }}
49
50 - name: Terraform Plan
51 id: plan
52 run: |
53 terraform plan -input=false -no-color -out=tfplan \
54 -var-file="terraform.tfvars" 2>&1 | tee plan_output.txt
55 echo "exitcode=${PIPESTATUS[0]}" >> $GITHUB_OUTPUT
56 working-directory: ${{ env.WORKING_DIR }}
57 continue-on-error: true
58
59 - name: Post Plan to PR
60 uses: actions/github-script@v7
61 if: github.event_name == 'pull_request'
62 with:
63 script: |
64 const fs = require('fs');
65 const plan = fs.readFileSync('${{ env.WORKING_DIR }}/plan_output.txt', 'utf8');
66 const truncated = plan.length > 60000 ? plan.substring(0, 60000) + '\n... truncated' : plan;
67 github.rest.issues.createComment({
68 issue_number: context.issue.number,
69 owner: context.repo.owner,
70 repo: context.repo.repo,
71 body: `## Terraform Plan\n\`\`\`hcl\n${truncated}\n\`\`\``
72 });
73
74 - name: Fail if plan failed
75 if: steps.plan.outputs.exitcode == '1'
76 run: exit 1
77
78 apply:
79 name: Terraform Apply
80 runs-on: ubuntu-latest
81 needs: plan
82 if: github.ref == 'refs/heads/main' && github.event_name == 'push'
83 environment: production # Requires manual approval in GitHub
84
85 steps:
86 - uses: actions/checkout@v4
87 - uses: aws-actions/configure-aws-credentials@v4
88 with:
89 role-to-assume: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/GitHubActionsRole
90 aws-region: us-east-1
91 - uses: hashicorp/setup-terraform@v3
92 with:
93 terraform_version: ${{ env.TF_VERSION }}
94 - run: terraform init -input=false
95 working-directory: ${{ env.WORKING_DIR }}
96 - run: terraform apply -auto-approve -input=false -var-file="terraform.tfvars"
97 working-directory: ${{ env.WORKING_DIR }}

7. Testing Terraform {#testing}

bash
1# Static analysis with tflint
2tflint --init
3tflint --recursive
4
5# Security scanning with tfsec
6tfsec .
7
8# Policy as code with Checkov
9checkov -d . --framework terraform
10
11# Built-in test framework (Terraform 1.6+)
12# tests/vpc_test.tftest.hcl
13run "creates_vpc_with_correct_cidr" {
14 command = plan
15
16 assert {
17 condition = module.vpc.vpc_cidr_block == "10.0.0.0/16"
18 error_message = "VPC CIDR must be 10.0.0.0/16"
19 }
20}

8. Import & Refactoring {#import}

bash
1# Import existing resource into state
2terraform import aws_s3_bucket.my_bucket my-existing-bucket-name
3
4# Generate config for existing resource (Terraform 1.5+)
5terraform plan -generate-config-out=generated.tf
6
7# Move resource in state without destroying (refactoring)
8# terraform.tf
9moved {
10 from = aws_s3_bucket.old_name
11 to = module.storage.aws_s3_bucket.this
12}
13
14# Remove resource from state without destroying
15terraform state rm aws_s3_bucket.my_bucket
16
17# List all resources in state
18terraform state list
19
20# Show specific resource state
21terraform state show aws_s3_bucket.my_bucket

9. Production Gotchas {#gotchas}

| Gotcha | Impact | Fix | |---|---|---| | terraform destroy without -target | Destroys ALL managed resources | Always use -target for partial destroys | | Provider version not pinned | Minor version update breaks resources | Always pin: version = "~> 5.0" | | Storing secrets in .tfvars | Secrets committed to git | Use Vault, AWS Secrets Manager, or env vars | | No prevent_destroy lifecycle | Accidental deletion of DBs, S3 buckets | Add lifecycle { prevent_destroy = true } to critical resources | | Force-unlock without verification | Corrupts state if another plan is running | Verify no running plan before terraform force-unlock | | Not using -out flag in CI | Apply may differ from plan if resources changed between plan and apply | Always: plan -out=tfplan, apply tfplan | | count vs for_each for resources | Removing item from middle of count list destroys and recreates all after it | Use for_each with maps instead of count with lists | | Hardcoded region | Breaks when deploying to another region | Use variables for all region-specific values | | Large state files | Slow plans, hard to debug | Split into multiple state files per concern | | terraform apply -auto-approve in production | No human review of changes | Always require manual approval for production |


10. Essential Commands {#commands}

bash
1# ── Core workflow ─────────────────────────────
2terraform init # Initialize (always run after any change to providers/backend)
3terraform init -upgrade # Upgrade provider versions
4terraform validate # Check syntax
5terraform fmt -recursive # Format all .tf files
6terraform plan -var-file=prod.tfvars # Dry run
7terraform plan -out=tfplan # Save plan for later apply
8terraform apply tfplan # Apply saved plan
9terraform apply -target=module.vpc # Apply specific resource
10terraform destroy -target=aws_instance.test # Destroy specific resource
11
12# ── State management ──────────────────────────
13terraform state list # List all resources
14terraform state show <resource> # Show resource details
15terraform state mv <old> <new> # Move/rename in state
16terraform state rm <resource> # Remove from state (keeps real resource)
17terraform state pull > backup.tfstate # Backup state
18terraform force-unlock <LOCK_ID> # Force release stuck lock
19
20# ── Inspection ────────────────────────────────
21terraform output # Show all outputs
22terraform output -json # JSON format
23terraform show # Human-readable state
24terraform graph | dot -Tsvg > graph.svg # Visualize dependencies
25
26# ── Debug ─────────────────────────────────────
27TF_LOG=DEBUG terraform plan # Verbose logging
28TF_LOG_PATH=./terraform.log terraform plan # Log to file
29terraform console # Interactive expression testing