highAdvancedSymptom-first#sqs#dlq#messages#consumers

Messages stuck in SQS dead-letter queue

DLQ ApproximateNumberOfMessages keeps rising or flat at a ceiling

Likely source:availabilityconfiguration

Focused incident mode — strips away everything but this runbook.

Symptoms

  • DLQ ApproximateNumberOfMessages keeps rising or flat at a ceiling
  • Backlog in the main queue drains but messages keep redriving
  • Consumer logs show a repeated parse/processing error per message

A DLQ that never drains is a design flag: the handler is not idempotent or not tolerant of poison payloads. Fix the consumer, then redrive — never attach end-user traffic to a queue whose DLQ fills.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • DLQ depth

    aws sqs get-queue-attributes --queue-url <DLQ-URL> --attribute-names ApproximateNumberOfMessages

  • Redrive policy

    aws sqs get-queue-attributes --queue-url <QUEUE-URL> --attribute-names RedrivePolicy

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • Same message body in DLQ repeatedly
  • Consumer logs show a parse/validation failure

Confirm before fixing

aws sqs receive-message --queue-url <DLQ-URL> --max-number-of-messages 1 --query 'Messages[0].Body'
READ

The exact offending payload

Fix

aws sqs start-message-move-task --source-arn arn:aws:sqs:<REGION>:<ACCOUNT>:<DLQ>
CHANGE

Move the batch to a quarantine queue (not the live queue) until fixed

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
aws sqs get-queue-attributes --queue-url <DLQ-URL> --attribute-names ApproximateNumberOfMessages
READ

Baseline depth

2
aws sqs receive-message --queue-url <DLQ-URL> --max-number-of-messages 1 --attribute-names All
READ

Pull one message and read its body + receive count

3
aws sqs start-message-move-task --source-arn arn:aws:sqs:<REGION>:<ACCOUNT>:<DLQ>
CHANGE

Freeze the backlog in a quarantine queue

4
aws sqs set-queue-attributes --queue-url <QUEUE-URL> --attributes VisibilityTimeout=<LARGER-SECONDS>
CHANGE

Fix retry/visibility config if that's the cause

Prevention

  • Make consumers idempotent and validate payloads before processing.
  • Set maxReceiveCount to 3-5 with a visibility timeout above worst-case execution.
  • Alarm on DLQ ApproximateNumberOfMessages > 0 immediately.

Related runbooks

Same stack, same failure family — worth a skim.

Lambda error retry storm — DLQ fillinghigh

Each poisoned event processed twice (synchronous + async retry)

AWSProductionRED2 causes
Open runbook
SNS delivery failing to Lambda or SQS subscribershigh

Topic publish succeeds but subscribers never receive

AWSIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.