highProductionRED#lambda#dlq#retries#dead-letter

Lambda error retry storm — DLQ filling

Each poisoned event processed twice (synchronous + async retry)

Likely source:scalingavailability

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Each poisoned event processed twice (synchronous + async retry)
  • Dead-letter queue ApproximateNumberOfMessages climbing
  • Error rate steady on AWS/Lambda Errors metric

EventSourceMapping retries are asynchronous and friends of an outage: two retry chains can run the same event twice. If the handler isn't idempotent, add dedup, and always route non-retryable errors to a DLQ, not back into retries.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Error count

    aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Errors --dimensions Name=FunctionName,Value=<FUNCTION> --start-time <ISO> --end-time <ISO> --period 300 --statistics Sum

  • DLQ depth

    aws sqs get-queue-attributes --queue-url <DLQ-URL> --attribute-names ApproximateNumberOfMessages

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Same exception text on every retry
  • No DLQ configured so events die silently after N attempts

Confirm before fixing

aws lambda get-function-configuration --function-name <FUNCTION> --query 'DeadLetterConfig'
READ

Is a DLQ configured at all?

Fix

aws lambda update-function-configuration --function-name <FUNCTION> --dead-letter-config TargetArn=arn:aws:sqs:<REGION>:<ACCOUNT>:<DLQ>
CHANGE

Send the poison pill to a DLQ after retries

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws logs filter-log-events --log-group-name /aws/lambda/<FUNCTION> --filter-pattern "ERROR" --query 'events[0].message'
READ

Capture the exception

2
aws lambda get-function-configuration --function-name <FUNCTION> --query '{Dlq:DeadLetterConfig,Concurrency:ReservedConcurrentExecutions}'
READ

Current protection settings

3
aws lambda put-function-concurrency --function-name <FUNCTION> --reserved-concurrent-executions <LIMIT>
CHANGE

Throttle the storm immediately

4
aws lambda update-function-configuration --function-name <FUNCTION> --dead-letter-config TargetArn=arn:aws:sqs:<REGION>:<ACCOUNT>:<DLQ>
CHANGE

Redirect poison events to the DLQ for triage

5
aws lambda invoke --function-name <FUNCTION> --payload file://repro.json /dev/stdout
READ

Reproduce with the poison payload after fixing the code

Prevention

  • Treat parse/validation errors as terminal — raise immediately, never 'return err' for retry.
  • Configure a DLQ per function and alarm on its depth.
  • Set reserved concurrency to a sane cap so a single bug can't blow concurrent-executions limits.

Related runbooks

Same stack, same failure family — worth a skim.

Lambda function timing out — exceeded configured timeouthigh

CloudWatch log line: 'Task timed out after X seconds'

AWSIntermediateRED3 causes
Open runbook
Messages stuck in SQS dead-letter queuehigh

DLQ ApproximateNumberOfMessages keeps rising or flat at a ceiling

AWSAdvanced2 causes
Open runbook
Lambda execution role not authorized — invoke failshigh

Invoke returns 403 'The provided execution role does not have permissions'

AWSFoundation3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.