As of: 2026-07-05 — migrated from `postmortems/2026-04-15-immich-crash.md`

Postmortem: Immich Crash & Service Degradation

Date: 2026-04-15 Duration: Unknown start → resolved ~22:30 CST Severity: High — Immich completely unavailable, photos inaccessible Status: Resolved

Summary

Immich became unavailable due to a Docker bridge networking bug that left the server container unable to reach Redis. When the server was restarted to fix this, a second problem emerged: 11,744 assets in the database had missing source files on disk, causing the thumbnail generation job queue to storm-crash the process with exit code 139 (SIGSEGV) on every boot. The server then entered a persistent crash-restart loop. Root causes were identified, the bad assets were cleaned up in the database, and the server was restored to full health.

Timeline

Time (CST)	Event
Unknown	Docker bridge veth bug silently breaks immich_server → immich_redis routing
Unknown	Server begins crash-restart loop (Redis unreachable, EHOSTUNREACH)
~21:00	User reports Immich down; authenticated successfully but immediately logged out
~21:05	Investigation: `docker ps` shows immich_server "Up 46 seconds" — crash-looping
~21:10	Root cause 1 identified: `EHOSTUNREACH 172.18.0.3:6379` — server can't reach Redis despite same bridge network
~21:10	Root cause 2 identified: `immich_postgres` health check failing — stale checksum failure counter from April 4
~21:10	Fix 1: `pg_stat_reset()` clears stale checksum counter → postgres becomes healthy
~21:15	Fix 2: `docker compose down && docker compose up -d` on immich stack — recreates bridge network, resolves veth bug
~21:20	Server comes up healthy. User attempts thumbnail regeneration from admin UI
~21:22	Server crashes immediately with exit code 139 (SIGSEGV)
~21:25	Root cause 3 identified: thumbnail job queue flooded with 11,744+ jobs for missing files; sharp native library segfaults under parallel failure storm
~21:30	Redis thumbnail queue cleared; server stabilises
~21:35	Analysis: 34,397 library assets in DB; only 22,634 files on disk — 11,744 missing
~21:40	sharp tested against all 22,634 on-disk image files — only 4 failures (XMP sidecar files, not real images)
~21:45	Fix 3: 11,744 missing assets marked `status=trashed` in DB; 4 XMP entries deleted
~22:00	Server confirmed stable, all assets healthy, thumbnail generation safe to re-run

Root Causes

1. Docker Bridge veth Bug

What happened: After a prior crash-restart cycle, the immich_server container's virtual Ethernet interface lost routing to other containers on the same Docker bridge network. From the server's perspective, Redis at 172.18.0.3 was unreachable (EHOSTUNREACH), despite Redis being healthy and all containers nominally on the same immich_default bridge.

Why it happened: This is a known Docker bug where the kernel's bridge forwarding state for a container's veth pair becomes stale after a crash-restart cycle. docker ps shows the container as "Up" but inter-container routing is broken. The bug does not self-heal.

Detection: docker logs immich_server showed continuous EHOSTUNREACH 172.18.0.3:6379. Host could ping 172.18.0.3 fine; the server container could not.

Fix: docker compose down && docker compose up -d — tearing down and recreating the network is the only fix. docker compose restart does not work as it doesn't recreate the bridge.

2. Stale Postgres Checksum Failure Counter

What happened: immich_postgres was marked (unhealthy) by Docker, which prevented immich_server from starting (it has depends_on: database: condition: service_healthy).

Why it happened: A single page checksum failure occurred on April 4 (11 days prior). The failure count accumulates in pg_stat_database and is never reset automatically. The health check queries SUM(checksum_failures) and exits non-zero if > 0, so the container stayed unhealthy indefinitely even though the database was functioning normally and no data corruption was present.

Detection: docker inspect immich_postgres --format 'Template:Json .State.Health' showed checksum failure count is 1 with last failure 2026-04-04.

Fix: docker exec immich_postgres psql -U postgres -d immich -c "SELECT pg_stat_reset();" resets all per-database statistics including the checksum counter. The health check immediately passed on the next interval.

Note: A real checksum failure is worth investigating — it can indicate a dying disk. In this case the single failure was old and isolated; the database was consistent and Immich was functioning (HTTP 200) right up until the networking broke. Monitor for recurrence.

3. Missing Asset Crash Loop (Primary Crash Cause)

What happened: When thumbnail generation was triggered, the BullMQ job queue was flooded with ~11,744 jobs for assets whose source files no longer exist on disk. The Node.js worker processes fired these jobs in high concurrency. The sharp native image library (C++ via N-API) segfaulted under the load of simultaneous "Input file is missing" failure paths, killing the entire Node.js process with exit code 139 (SIGSEGV). Docker's restart: always policy restarted the server, which immediately picked up the queued jobs and crashed again — a persistent crash loop.

Why the files were missing: 34,397 assets were stored in the "uploaded library" path (/usr/src/app/upload/library/). Only 22,634 files exist on disk. The gap of 11,744 likely accumulated over time as photos were moved, deleted from the host filesystem, or sourced from a device (e.g., iPhone backup migration) that was later reorganised without removing the Immich DB records.

Detection: <syntaxhighlight lang="text"> docker logs immich_server → exit code 139 (SIGSEGV) find /raid1/.../library -type f | wc -l → 22,634 SELECT COUNT(*) FROM asset WHERE originalPath LIKE '/usr/src/app/upload/library/%' → 34,397 </syntaxhighlight>

Fix (immediate): Clear the Redis thumbnail queue to stop the crash loop: <syntaxhighlight lang="bash"> docker exec immich_redis redis-cli KEYS "immich_bull:thumbnailGeneration:*" | \

 xargs docker exec immich_redis redis-cli DEL

</syntaxhighlight>

Fix (permanent): Mark missing assets as trashed so Immich never queues thumbnail generation for them again: <syntaxhighlight lang="bash">

Pipe missing asset IDs into postgres to bulk-trash

cat /tmp/immich-missing-ids.txt | docker exec -i immich_postgres psql -U postgres -d immich -c " CREATE TEMP TABLE missing_ids (id uuid); COPY missing_ids FROM STDIN; UPDATE asset SET \"deletedAt\" = NOW(), status = 'trashed'

 WHERE id IN (SELECT id FROM missing_ids);"

</syntaxhighlight>

Additionally, 4 XMP sidecar files (.jpg.xmp) had been incorrectly indexed as image assets and deleted from the DB: <syntaxhighlight lang="bash"> docker exec immich_postgres psql -U postgres -d immich -c \

 "DELETE FROM asset WHERE \"originalPath\" LIKE '%.xmp';"

</syntaxhighlight>

What Was NOT Broken

HEIC files from Mac auto-upload — All passed sharp's metadata() test. Pillow (used for the corruption scan) doesn't support HEIC natively; those "failures" were false positives.
External library photos (70,840 assets in /app/local-images/) — Library scan confirmed 0 offlined, all intact.
Thumbnails on disk — 190,882 thumbnail files were present and correctly structured. No regeneration was needed for most assets.

Impact

Area	Impact
Photo browsing	Complete outage — all images showed "Error loading image"
New uploads	Unavailable during crash loop
External library	Intact, no data loss
Uploaded library	11,744 assets with no source files trashed (files were already gone before today)
Other services	None — isolated to Immich stack

Diagnostic Commands Reference

Check all Immich container statuses

docker ps --filter "name=immich" --format "table Template:.Names\tTemplate:.Status"

Check crash exit code

docker inspect immich_server --format 'Template:.State.ExitCode'

139 = SIGSEGV (native crash), 1 = normal error, 0 = clean exit

Check postgres health detail

docker inspect immich_postgres --format 'Template:Json .State.Health' | python3 -m json.tool | tail -20

Clear thumbnail queue (safe to run anytime — jobs will re-queue from admin UI on demand)

docker exec immich_redis redis-cli KEYS "immich_bull:thumbnailGeneration:*" | \

 xargs docker exec immich_redis redis-cli DEL

Count assets by storage location

docker exec immich_postgres psql -U postgres -d immich -t -c " SELECT

 SUM(CASE WHEN \"originalPath\" LIKE '/usr/src/app/upload/library/%' THEN 1 ELSE 0 END) as uploaded,
 SUM(CASE WHEN \"originalPath\" LIKE '/app/local-images/%' THEN 1 ELSE 0 END) as external,
 COUNT(*) FILTER (WHERE status = 'trashed') as trashed

FROM asset;"

Find assets with missing files (run on host, translate paths)
See: /home/mnw/Developer/SystemResiliency/scripts/immich-find-broken.py

Reset stale postgres checksum counter

docker exec immich_postgres psql -U postgres -d immich -c "SELECT pg_stat_reset();" </syntaxhighlight>

Follow-up Actions

Action	Priority	Status
Run thumbnail generation from Immich admin → Jobs	High	Pending
Empty Immich trash (11,744 trashed assets)	Medium	Pending
Investigate why 11,744 uploaded assets lost their source files	Low	Pending
Monitor postgres checksum failures — recurrence may indicate disk issue	Medium	Ongoing
Monthly rolling restarts workflow (n8n `OlzWTPDsCoO9oEo7`) now active	Done	Complete

Prevention

The monthly rolling restart workflow deployed today (n8n ID: OlzWTPDsCoO9oEo7) performs a full compose down && up -d on all service stacks on the 1st of each month at 2 AM CST. This proactively recreates Docker bridge networks before the veth bug can silently accumulate, and sends per-service status via Matrix (Zordon). The postgres checksum issue and missing asset problem are not recurring risks given the fixes applied.