Wikibot: Migrated from postmortems/2026-04-15-immich-crash.md

2026-07-05T18:17:36Z

Migrated from postmortems/2026-04-15-immich-crash.md

New page

''As of: 2026-07-05 — migrated from `postmortems/2026-04-15-immich-crash.md`''

= Postmortem: Immich Crash & Service Degradation =
'''Date:''' 2026-04-15
'''Duration:''' Unknown start → resolved ~22:30 CST
'''Severity:''' High — Immich completely unavailable, photos inaccessible
'''Status:''' Resolved

----

== Summary ==

Immich became unavailable due to a Docker bridge networking bug that left the server container unable to reach Redis. When the server was restarted to fix this, a second problem emerged: 11,744 assets in the database had missing source files on disk, causing the thumbnail generation job queue to storm-crash the process with exit code 139 (SIGSEGV) on every boot. The server then entered a persistent crash-restart loop. Root causes were identified, the bad assets were cleaned up in the database, and the server was restored to full health.

----

== Timeline ==

{| class="wikitable"
|-
! Time (CST)
! Event
|-
| Unknown
| Docker bridge veth bug silently breaks immich_server → immich_redis routing
|-
| Unknown
| Server begins crash-restart loop (Redis unreachable, EHOSTUNREACH)
|-
| ~21:00
| User reports Immich down; authenticated successfully but immediately logged out
|-
| ~21:05
| Investigation: <code>docker ps</code> shows immich_server "Up 46 seconds" — crash-looping
|-
| ~21:10
| Root cause 1 identified: <code>EHOSTUNREACH 172.18.0.3:6379</code> — server can't reach Redis despite same bridge network
|-
| ~21:10
| Root cause 2 identified: <code>immich_postgres</code> health check failing — stale checksum failure counter from April 4
|-
| ~21:10
| Fix 1: <code>pg_stat_reset()</code> clears stale checksum counter → postgres becomes healthy
|-
| ~21:15
| Fix 2: <code>docker compose down && docker compose up -d</code> on immich stack — recreates bridge network, resolves veth bug
|-
| ~21:20
| Server comes up healthy. User attempts thumbnail regeneration from admin UI
|-
| ~21:22
| Server crashes immediately with exit code 139 (SIGSEGV)
|-
| ~21:25
| Root cause 3 identified: thumbnail job queue flooded with 11,744+ jobs for missing files; sharp native library segfaults under parallel failure storm
|-
| ~21:30
| Redis thumbnail queue cleared; server stabilises
|-
| ~21:35
| Analysis: 34,397 library assets in DB; only 22,634 files on disk — 11,744 missing
|-
| ~21:40
| sharp tested against all 22,634 on-disk image files — only 4 failures (XMP sidecar files, not real images)
|-
| ~21:45
| Fix 3: 11,744 missing assets marked <code>status=trashed</code> in DB; 4 XMP entries deleted
|-
| ~22:00
| Server confirmed stable, all assets healthy, thumbnail generation safe to re-run
|}

----

== Root Causes ==

=== 1. Docker Bridge veth Bug ===
'''What happened:''' After a prior crash-restart cycle, the immich_server container's virtual Ethernet interface lost routing to other containers on the same Docker bridge network. From the server's perspective, Redis at <code>172.18.0.3</code> was unreachable (<code>EHOSTUNREACH</code>), despite Redis being healthy and all containers nominally on the same <code>immich_default</code> bridge.

'''Why it happened:''' This is a known Docker bug where the kernel's bridge forwarding state for a container's veth pair becomes stale after a crash-restart cycle. <code>docker ps</code> shows the container as "Up" but inter-container routing is broken. The bug does not self-heal.

'''Detection:''' <code>docker logs immich_server</code> showed continuous <code>EHOSTUNREACH 172.18.0.3:6379</code>. Host could ping 172.18.0.3 fine; the server container could not.

'''Fix:''' <code>docker compose down && docker compose up -d</code> — tearing down and recreating the network is the only fix. <code>docker compose restart</code> does '''not''' work as it doesn't recreate the bridge.

----

=== 2. Stale Postgres Checksum Failure Counter ===
'''What happened:''' <code>immich_postgres</code> was marked <code>(unhealthy)</code> by Docker, which prevented <code>immich_server</code> from starting (it has <code>depends_on: database: condition: service_healthy</code>).

'''Why it happened:''' A single page checksum failure occurred on April 4 (11 days prior). The failure count accumulates in <code>pg_stat_database</code> and is never reset automatically. The health check queries <code>SUM(checksum_failures)</code> and exits non-zero if > 0, so the container stayed unhealthy indefinitely even though the database was functioning normally and no data corruption was present.

'''Detection:''' <code>docker inspect immich_postgres --format '{{json .State.Health}}'</code> showed <code>checksum failure count is 1</code> with last failure <code>2026-04-04</code>.

'''Fix:''' <code>docker exec immich_postgres psql -U postgres -d immich -c "SELECT pg_stat_reset();"</code> resets all per-database statistics including the checksum counter. The health check immediately passed on the next interval.

'''Note:''' A real checksum failure is worth investigating — it can indicate a dying disk. In this case the single failure was old and isolated; the database was consistent and Immich was functioning (HTTP 200) right up until the networking broke. Monitor for recurrence.

----

=== 3. Missing Asset Crash Loop (Primary Crash Cause) ===
'''What happened:''' When thumbnail generation was triggered, the BullMQ job queue was flooded with ~11,744 jobs for assets whose source files no longer exist on disk. The Node.js worker processes fired these jobs in high concurrency. The <code>sharp</code> native image library (C++ via N-API) segfaulted under the load of simultaneous "Input file is missing" failure paths, killing the entire Node.js process with exit code 139 (SIGSEGV). Docker's <code>restart: always</code> policy restarted the server, which immediately picked up the queued jobs and crashed again — a persistent crash loop.

'''Why the files were missing:''' 34,397 assets were stored in the "uploaded library" path (<code>/usr/src/app/upload/library/</code>). Only 22,634 files exist on disk. The gap of 11,744 likely accumulated over time as photos were moved, deleted from the host filesystem, or sourced from a device (e.g., iPhone backup migration) that was later reorganised without removing the Immich DB records.

'''Detection:'''
<syntaxhighlight lang="text">
docker logs immich_server → exit code 139 (SIGSEGV)
find /raid1/.../library -type f | wc -l → 22,634
SELECT COUNT(*) FROM asset WHERE originalPath LIKE '/usr/src/app/upload/library/%' → 34,397
</syntaxhighlight>

'''Fix (immediate):''' Clear the Redis thumbnail queue to stop the crash loop:
<syntaxhighlight lang="bash">
docker exec immich_redis redis-cli KEYS "immich_bull:thumbnailGeneration:*" | \
xargs docker exec immich_redis redis-cli DEL
</syntaxhighlight>

'''Fix (permanent):''' Mark missing assets as trashed so Immich never queues thumbnail generation for them again:
<syntaxhighlight lang="bash">
# Pipe missing asset IDs into postgres to bulk-trash
cat /tmp/immich-missing-ids.txt | docker exec -i immich_postgres psql -U postgres -d immich -c "
CREATE TEMP TABLE missing_ids (id uuid);
COPY missing_ids FROM STDIN;
UPDATE asset SET \"deletedAt\" = NOW(), status = 'trashed'
WHERE id IN (SELECT id FROM missing_ids);"
</syntaxhighlight>

Additionally, 4 XMP sidecar files (<code>.jpg.xmp</code>) had been incorrectly indexed as image assets and deleted from the DB:
<syntaxhighlight lang="bash">
docker exec immich_postgres psql -U postgres -d immich -c \
"DELETE FROM asset WHERE \"originalPath\" LIKE '%.xmp';"
</syntaxhighlight>

----

== What Was NOT Broken ==

* '''HEIC files from Mac auto-upload''' — All passed sharp's <code>metadata()</code> test. Pillow (used for the corruption scan) doesn't support HEIC natively; those "failures" were false positives.
* '''External library photos''' (70,840 assets in <code>/app/local-images/</code>) — Library scan confirmed 0 offlined, all intact.
* '''Thumbnails on disk''' — 190,882 thumbnail files were present and correctly structured. No regeneration was needed for most assets.

----

== Impact ==

{| class="wikitable"
|-
! Area
! Impact
|-
| Photo browsing
| Complete outage — all images showed "Error loading image"
|-
| New uploads
| Unavailable during crash loop
|-
| External library
| Intact, no data loss
|-
| Uploaded library
| 11,744 assets with no source files trashed (files were already gone before today)
|-
| Other services
| None — isolated to Immich stack
|}

----

== Diagnostic Commands Reference ==

<syntaxhighlight lang="bash">
# Check all Immich container statuses
docker ps --filter "name=immich" --format "table {{.Names}}\t{{.Status}}"

# Check crash exit code
docker inspect immich_server --format '{{.State.ExitCode}}'
# 139 = SIGSEGV (native crash), 1 = normal error, 0 = clean exit

# Check postgres health detail
docker inspect immich_postgres --format '{{json .State.Health}}' | python3 -m json.tool | tail -20

# Clear thumbnail queue (safe to run anytime — jobs will re-queue from admin UI on demand)
docker exec immich_redis redis-cli KEYS "immich_bull:thumbnailGeneration:*" | \
xargs docker exec immich_redis redis-cli DEL

# Count assets by storage location
docker exec immich_postgres psql -U postgres -d immich -t -c "
SELECT
SUM(CASE WHEN \"originalPath\" LIKE '/usr/src/app/upload/library/%' THEN 1 ELSE 0 END) as uploaded,
SUM(CASE WHEN \"originalPath\" LIKE '/app/local-images/%' THEN 1 ELSE 0 END) as external,
COUNT(*) FILTER (WHERE status = 'trashed') as trashed
FROM asset;"

# Find assets with missing files (run on host, translate paths)
# See: /home/mnw/Developer/SystemResiliency/scripts/immich-find-broken.py

# Reset stale postgres checksum counter
docker exec immich_postgres psql -U postgres -d immich -c "SELECT pg_stat_reset();"
</syntaxhighlight>

----

== Follow-up Actions ==

{| class="wikitable"
|-
! Action
! Priority
! Status
|-
| Run thumbnail generation from Immich admin → Jobs
| High
| Pending
|-
| Empty Immich trash (11,744 trashed assets)
| Medium
| Pending
|-
| Investigate why 11,744 uploaded assets lost their source files
| Low
| Pending
|-
| Monitor postgres checksum failures — recurrence may indicate disk issue
| Medium
| Ongoing
|-
| Monthly rolling restarts workflow (n8n <code>OlzWTPDsCoO9oEo7</code>) now active
| Done
| Complete
|}

----

== Prevention ==

The '''monthly rolling restart workflow''' deployed today (n8n ID: <code>OlzWTPDsCoO9oEo7</code>) performs a full <code>compose down && up -d</code> on all service stacks on the 1st of each month at 2 AM CST. This proactively recreates Docker bridge networks before the veth bug can silently accumulate, and sends per-service status via Matrix (Zordon). The postgres checksum issue and missing asset problem are not recurring risks given the fixes applied.

Postmortems/2026-04-15 Immich Crash - Revision history

Wikibot: Migrated from postmortems/2026-04-15-immich-crash.md