mediumIntermediateSymptom-first#deleted#binary#proc#exe#upgrade

Binary deleted but process still running — live process from deleted file

Upgraded binary but the running process still shows old version

Likely source:configurationdeployment

Focused incident mode — strips away everything but this runbook.

Symptoms

  • Upgraded binary but the running process still shows old version
  • ls -l /proc/<PID>/exe shows (deleted) after the path
  • lsof +L1 shows the deleted binary as an open file

A deleted binary is held in memory until the process exits. Disk space is freed immediately, but the old code keeps running. This is a classic gotcha during upgrades.

Evidence to grab

Collect these artifacts before you change anything — they prove (or rule out) each cause.

  • Process exe symlink

    ls -l /proc/<PID>/exe

  • Deleted open files

    lsof +L1 2>/dev/null | grep deleted | head -10

Diagnostic flow

Run the steps in order, marking each claim confirmed or refuted. The evidence you gather feeds the ranked causes below.

Guided decision tree is ready.

Steps with commands, expected output, and pass/fail branches.

Ranked root causes

Evidence checklist

  • ls -l /proc/<PID>/exe shows (deleted)
  • A new version of the binary exists on disk

Confirm before fixing

ls -la /proc/<PID>/exe && ls -la <new-binary-path>
READ

Compare the running binary against the new one

Fix

systemctl restart <service>
CHANGE

Restart the service to load the new binary

Resolution runbook

Apply from top to bottom after the cause is confirmed. Review every command before running it — risk tags are shown on each.

1
ls -l /proc/<PID>/exe
READ

Confirm the binary is deleted

2
md5sum /proc/<PID>/exe 2>/dev/null
READ

Hash the running binary to verify it matches expectations

3
systemctl restart <service>
CHANGE

Restart the service to load the current binary from disk

Prevention

  • Use systemctl restart (not rm + cp) for binary upgrades.
  • Implement rolling restarts so the new binary is loaded before the old process is stopped.
  • Alert on processes running from deleted binaries via a cron check.
  • Use package managers or symlinks for atomic binary swaps.

Related runbooks

Same stack, same failure family — worth a skim.

Deleted open file holding disk space — phantom disk usagehigh

df shows full disk but du shows less total usage

LinuxIntermediateUSE2 causes
Open runbook
Process ignores SIGTERM — cannot be gracefully stoppedmedium

systemctl stop <service> hangs for the full timeout

LinuxIntermediate3 causes
Open runbook

Commands marked READ CHANGE DESTRUCTIVE — never run DESTRUCTIVE commands against production data you cannot restore. Copy → verify → execute.