As of: 2026-07-05 — migrated from `backup-stack/HOW_IT_WORKS.md` and `DISASTER_RECOVERY.md`. Canonical DR home is `~/Developer/DisasterRecovery/` — this page is the operational summary, not the full cold-start bootstrap guide.

How the Backup System Works

A beginner-friendly guide to understanding your new backup system.

The Big Picture

<syntaxhighlight lang="text"> Your Server (stronghold)

        │
        │ 1. Dump databases to files
        ▼
  ┌─────────────┐
  │ PostgreSQL  │──► .sql.gz files
  │ Containers  │    (in /library/backup-stack/dumps/)
  └─────────────┘
        │
        │ 2. Restic reads all your data
        ▼
  ┌─────────────┐
  │   Restic    │──► Deduplicates, compresses, encrypts
  │  Container  │
  └─────────────┘
        │
        │ 3. Uploads to cloud storage
        ▼
  ┌─────────────┐
  │   Hetzner   │  Your encrypted backups live here
  │ Storage Box │  (in Germany, €3.81/month)
  └─────────────┘

</syntaxhighlight>

What Gets Backed Up

What	Where It Lives	Size	Why It Matters
Authentik DB	PostgreSQL dump	14MB	Your SSO - lose this, lose all logins
Vaultwarden	SQLite file	356KB	Your passwords!
Immich photos	/raid1/.../Immich-Photos	149GB	Family photos
Immich DB	PostgreSQL dump	437MB	Photo metadata, faces, albums
Jellyfin	/var/lib/jellyfin	92GB	Watch history, metadata
Nextcloud	Config + PostgreSQL	~1GB	Files, calendar, contacts
Open-WebUI	SQLite + uploads	1.7GB	AI chat history
Memos, Linkwarden, Mailcow	Various	Small	Notes, bookmarks, email
MediaWiki	Config bind mount + MariaDB dump + images volume	~300KB	The wilsoz wiki — infra docs, runbooks, ADRs

The Components

1. Restic Container (`restic-backup`)

A small container that knows how to:

Read your data from mounted directories
Encrypt everything with your password
Upload to Hetzner via SFTP
Deduplicate (only upload changed data)

2. Database Dump Script (`host-dump-databases.sh`)

Runs on the host (not in container) because it needs Docker access. <syntaxhighlight lang="text"> authentik-postgresql → authentik-20260201.sql.gz nextcloud-db → nextcloud-20260201.sql.gz immich_postgres → immich-20260201.sql.gz linkwarden_postgres → linkwarden-20260201.sql.gz </syntaxhighlight>

3. Backup Script (`backup-to-hetzner.sh`)

Runs inside the container:

Connects to Hetzner via SSH (port 23, key auth)
Reads all your data directories
Uploads new/changed files
Prunes old snapshots (keeps 7 daily, 4 weekly, 6 monthly)

4. Systemd Timers

Schedules everything to run automatically:

3:00 AM - Full backup
6:00 AM - Integrity check
Sunday 7:00 AM - Restore test

How Restic Works (The Magic)

Deduplication

Restic breaks files into chunks. If you have 1000 photos and add 1 new one:

First backup: uploads all 1000 photos (~149GB)
Second backup: uploads only the 1 new photo (~5MB)

Encryption

Everything is encrypted with your password before leaving your server.

Hetzner can't read your data
Without the password, backups are useless
Password: E+mVDSjj8VsB5Feklelsq/BAJ3FmMX3E (stored in .env)

Snapshots

Each backup creates a "snapshot" - a point-in-time view of your data. <syntaxhighlight lang="text"> $ restic snapshots ID Time Host Tags ───────────────────────────────────────────────── a1b2c3d4 2026-02-01 23:50 stronghold full e5f6g7h8 2026-02-02 03:00 stronghold full ... </syntaxhighlight>

You can restore any snapshot, not just the latest.

How to Check on Backups

Is the backup running?

<syntaxhighlight lang="bash"> docker logs restic-backup </syntaxhighlight>

How big is my backup repo?

<syntaxhighlight lang="bash"> docker exec restic-backup restic stats </syntaxhighlight>

List all snapshots

<syntaxhighlight lang="bash"> docker exec restic-backup restic snapshots </syntaxhighlight>

Check repo integrity

<syntaxhighlight lang="bash"> docker exec restic-backup restic check </syntaxhighlight>

How to Restore

Restore everything (disaster recovery)

<syntaxhighlight lang="bash"> docker exec restic-backup restic restore latest --target /tmp/full-restore </syntaxhighlight>

Restore one file

<syntaxhighlight lang="bash"> docker exec restic-backup restic restore latest \

 --target /tmp/restore \
 --include "/data/vaultwarden/data/db.sqlite3"

</syntaxhighlight>

Restore from specific date

List snapshots to find the ID

docker exec restic-backup restic snapshots

Restore that snapshot

docker exec restic-backup restic restore a1b2c3d4 --target /tmp/restore </syntaxhighlight>

Important Files

<syntaxhighlight lang="text"> /library/backup-stack/ ├── docker-compose.yml # Container definition ├── .env # Passwords and settings (SECRET!) ├── ssh-config # SSH settings for Hetzner ├── dumps/ # Database dumps live here │ ├── authentik-20260201.sql.gz │ ├── immich-20260201.sql.gz │ └── ... ├── scripts/ │ ├── host-dump-databases.sh # Dumps DBs (runs on host) │ ├── backup-to-hetzner.sh # Main backup (runs in container) │ └── validate-backups.sh # Tests backups └── systemd/ # Timer definitions </syntaxhighlight>

What Happens If...

Power goes out during backup?

Restic is crash-safe. Next backup continues from where it left off.

Hetzner goes down?

Your local data is fine. Backups resume when Hetzner is back.

You delete a file by accident?

Restore it from a previous snapshot.

Your server dies completely?

Get new server
Install Docker
Connect to Hetzner
Run restic restore latest
Rebuild containers with restored data

You forget the password?

Game over. The backups are encrypted. No password = no data. Save it somewhere safe (Vaultwarden, password manager, printed paper in safe).

Costs

Item	Cost
Hetzner Storage Box 1TB	€3.81/month
Everything else	Free (self-hosted)

Summary

Every night at 3 AM:

- Databases get dumped to .sql.gz files
- Restic encrypts and uploads everything to Germany
- Old backups get cleaned up automatically

Every morning at 6 AM:

- System checks backup integrity
- Alerts you via ntfy if something's wrong

Every Sunday:

- Does a test restore to verify backups actually work

Your data is now protected. Sleep well.

Disaster Recovery Runbook

Last Updated: 2026-06-21 Print this and store in a safe place.

Keep DR current: after adding/changing/removing any service, run

scripts/dr-drift-check.sh and act on whatever it flags. See

DR_MAINTENANCE.md for the full workflow. Don't skip this — gradual drift

is how DR plans silently rot until the day you actually need them.

Quick Reference

Stronghold (primary)

Item	Value
Hetzner Storage Box	`u540853.your-storagebox.de` port 23
Repo path	`sftp:[email protected]:./backups` (port 23 comes from SSH config, not the URL — see note)
SSH Key	`~/.ssh/id_rsa` (on stronghold)
Restic Password	In `/library/backup-stack/.env` (`HETZNER_PASSWORD`)
Backup Schedule	Daily 03:00 CST
Retention	7 daily, 4 weekly, 6 monthly

VPS (wilsoz.com)

Item	Value
Hetzner Storage Box	same box, separate repo
Repo path	`sftp:[email protected]:./backups-vps` (port 23 comes from SSH config, not the URL — see note)
SSH Key	`~/.ssh/id_ed25519` (on VPS)
Restic Password	same `HETZNER_PASSWORD` — stored in `/etc/restic-vps.env` on VPS
Backup Schedule	Daily 04:30 UTC (offset from stronghold)
Retention	7 daily, 4 weekly, 6 monthly
Covers	Forgejo data + repos, Atuin (pg_dump), ntfy, nginx, Let's Encrypt certs, `/opt/` custom scripts

⚠️ SFTP repo URL gotcha (port vs. path): In restic's single-colon SFTP form

(sftp:user@host:path) everything after the host colon is the repository path —

there is no port in the URL. The storage box's port 23 must be set in SSH config

(~/.ssh/config → Port 23 for host u540853.your-storagebox.de), which both

stronghold and the restic-backup container already have. Writing the port into the

URL as :23/backups makes restic look for a repo at the literal path 23/backups and

fail with "repository does not exist". Always use the relative :./backups /

:./backups-vps form. (This exact mistake caused false "VPS: no snapshots found"

alerts from the n8n Backup Status Alert (SSH) workflow until 2026-06-29.)

What's Backed Up

Service Data (files)

Path in Backup	Source on Host	Contents
`/data/authentik`	`/library/authentik-app/`	SSO/identity provider config
`/data/vaultwarden`	`/library/vaultwarden-app/`	Password vault (SQLite DB + attachments)
`/data/nextcloud`	`/library/nextcloud-app/`	File sync config + data
`/data/immich-config`	`/library/immich-app/`	Photo manager config
`/data/immich-photos`	`/raid1/Consolidated/Picture/Immich-Photos/`	All photos (149G)
`/data/jellyfin`	`/var/lib/jellyfin/`	Media server metadata (92G)
`/data/jellyfin-etc`	`/etc/jellyfin/`	Jellyfin system config
`/data/memos`	`/library/memo-app/`	Notes app
`/data/linkwarden`	`/library/linkwarden-app/`	Bookmark manager
`/data/open-webui`	`/library/open-webui-app/`	AI chat interface
`/data/matrix-chat`	`/library/matrix-chat/`	Synapse + bridge configs + signing key
`/data/stalwart`	`/library/stalwart-app/data/`	Mail server data
`/data/snappymail`	`/library/snappymail-app/data/`	Webmail data
`/data/mailcow`	`/opt/mailcow-dockerized/`	Legacy mail (to be archived)
`/data/home`	`/home/mnw/`	Home dir: dotfiles, scripts, Documents, source, SSH/GPG keys (selective excludes)
`/data/developer`	`/home/mnw/Developer/`	All dev projects (7.9G) — also covered by `/data/home`
`/data/beets`	`/raid1/my-tunes/.beets/`	Music library DB
`/data/openbao`	`/library/openbao-app/`	OpenBao secret store — incl. `.init-output.json` (root token + unseal key) and `.scripts-token` / `.claude-token`. Bootstrap-critical: must be restored and unsealed before services that depend on secrets.
`/data/ezbookkeeping`	`/library/ezbookkeeping-app/data/`	Personal finance tracker (SQLite + uploads)
`/data/omada`	`/library/omada-app/`	Omada SDN controller — switch/AP config & topology

Database Dumps (in `/dumps/`)

File Pattern	Source Container	Database
`authentik-YYYYMMDD.sql.gz`	`authentik-app_postgresql_1`	Authentik identity DB
`nextcloud-YYYYMMDD.sql.gz`	`nextcloud-db`	Nextcloud DB
`immich-YYYYMMDD.sql.gz`	`immich_postgres`	Immich photo DB (~460MB compressed)
`linkwarden-YYYYMMDD.sql.gz`	`linkwarden_postgres`	Linkwarden bookmark DB
`matrix-YYYYMMDD.sql.gz`	`matrix-postgres`	Synapse + bridge DBs (~1MB compressed)

NOT Backed Up (re-acquirable)

Jellyfin media files (movies, TV)
Ollama models (~20G, re-downloadable)
Steam games
VirtualBox VMs
node_modules, .venv, .git/objects
Wyoming voice models (piper-data-app, whisper-data-app) — redownloadable
Prometheus metric history, Grafana dashboards data, uptime-kuma — re-buildable from compose

⚠️ Known Gaps — Data Not Currently Backed Up

Severity	What	Where	Notes
HIGH	AudioBookshelf config + metadata	`/usr/share/audiobookshelf/config/` (38M), `/usr/share/audiobookshelf/metadata/` (936M)	systemd service, not Docker — not yet bind-mounted into restic container. Lose library DB, progress, and cover art on rebuild.

Previously flagged gaps (all closed as of 2026-06-21): Nextcloud user files, Matrix media, Mealie, n8n named volumes, uptime-kuma, Grafana, AdGuard — all now covered by bind mounts in the restic container compose.

Recovery Scenarios

Scenario 1: Single Service Failure

Example: Immich database corrupted, need to restore just Immich.

1. Stop the service

cd /library/immich-app && docker compose down

2. List available snapshots

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

3. Restore service data from latest snapshot

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
  restic restore latest --target /tmp/restore --include /data/immich-config'

4. Copy restored data to host

docker cp restic-backup:/tmp/restore/data/immich-config /tmp/immich-restore

5. Replace the broken data

mv /library/immich-app /library/immich-app.broken mv /tmp/immich-restore /library/immich-app

6. Restore the database dump

docker cp restic-backup:/tmp/restore/dumps/immich-YYYYMMDD.sql.gz /tmp/ gunzip /tmp/immich-YYYYMMDD.sql.gz

Start just the database container

docker compose up -d immich_postgres

Wait for it to be ready, then restore

docker exec -i immich_postgres psql -U postgres immich < /tmp/immich-YYYYMMDD.sql

7. Start the full service

docker compose up -d

8. Clean up

docker exec restic-backup rm -rf /tmp/restore </syntaxhighlight>

Scenario 2: Full Server Rebuild

Starting from: Fresh Linux install on new hardware.

Bootstrap Order (DO NOT SHUFFLE)

Many services depend on OpenBao (secret store) and Authentik (SSO). Bring them up in this order so each layer has its dependencies in place:

<syntaxhighlight lang="text"> 1. Host: Docker, restic, Tailscale, SSH key 2. Restore everything from Hetzner via restic (raw files) 3. OpenBao ← unseal it first; it holds tokens other services need 4. Authentik (DB dump → start) ← SSO for everything 5. Vaultwarden ← human-facing creds; needed for me-the-operator 6. Restore Claude Code access (bw CLI, MCP wrappers) ← so future-you can drive recovery 7. Postgres-backed services (Immich, Nextcloud, Matrix, Linkwarden): DB dump → start 8. Stalwart + SnappyMail (mail) 9. Everything else (n8n, Grafana, Mealie, etc.) </syntaxhighlight>

Step 1: Install Prerequisites

Install Docker

curl -fsSL https://get.docker.com | sh sudo usermod -aG docker mnw

Install restic

sudo apt install restic

Set Docker MTU for Tailscale compatibility
Add to /etc/docker/daemon.json:
{"mtu": 1280}

sudo systemctl restart docker </syntaxhighlight>

Step 2: Restore SSH Keys

You need the SSH key to access Hetzner. If you have a copy:

<syntaxhighlight lang="bash"> mkdir -p ~/.ssh

Copy id_rsa from your backup/safe location

chmod 600 ~/.ssh/id_rsa </syntaxhighlight>

If you don't have the key, generate a new one and add it to Hetzner: <syntaxhighlight lang="bash"> ssh-keygen -t rsa -b 4096

Add public key via Hetzner Robot panel → Storage Box → SSH Keys

</syntaxhighlight>

Step 3: Bootstrap Restic

Create the backup stack directory

sudo mkdir -p /library/backup-stack cd /library/backup-stack

Create .env with Hetzner credentials:

cat > .env << 'EOF' HETZNER_REPO=sftp:[email protected]:23/backups HETZNER_PASSWORD=<your-restic-password> NTFY_TOKEN=tk_0fhigfiglbjw9r8sk9jv4hpr7q67f EOF

Test connection

export $(grep -v '^#' .env | xargs) restic -r "$HETZNER_REPO" snapshots </syntaxhighlight>

Step 4: Restore Everything

Create target directories

sudo mkdir -p /library /raid1/Consolidated/Picture/Immich-Photos

Restore all service data

restic -r "$HETZNER_REPO" restore latest --target /

This puts files at their original paths:
/data/authentik → needs moving to /library/authentik-app/
/data/developer → needs moving to /home/mnw/Developer/
etc.

Move restored data to correct locations

sudo mv /data/authentik /library/authentik-app sudo mv /data/vaultwarden /library/vaultwarden-app sudo mv /data/nextcloud /library/nextcloud-app sudo mv /data/immich-config /library/immich-app sudo mv /data/immich-photos/* /raid1/Consolidated/Picture/Immich-Photos/ sudo mv /data/jellyfin /var/lib/jellyfin sudo mv /data/jellyfin-etc /etc/jellyfin sudo mv /data/memos /library/memo-app sudo mv /data/linkwarden /library/linkwarden-app sudo mv /data/open-webui /library/open-webui-app sudo mv /data/matrix-chat /library/matrix-chat sudo mv /data/stalwart /library/stalwart-app/data sudo mv /data/snappymail /library/snappymail-app/data sudo mv /data/developer /home/mnw/Developer sudo mv /data/beets /raid1/my-tunes/.beets sudo chown -R mnw:mnw /home/mnw/Developer </syntaxhighlight>

Step 4b: Bootstrap OpenBao (Secret Store) — REQUIRED before DBs

OpenBao holds tokens and secrets that downstream services + scripts read at startup.

1. Move restored data into place

sudo mv /data/openbao /library/openbao-app sudo chown -R 1000:1000 /library/openbao-app # OpenBao runs as uid 1000

2. Start the container

cd /library/openbao-app && docker compose up -d

3. Unseal — uses the unseal key from .init-output.json (restored from backup)

sudo /usr/local/bin/openbao-unseal.sh

If that script isn't restored yet, manually:
ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
UNSEAL_KEY=$(jq -r .unseal_keys_b64[0] /library/openbao-app/.init-output.json)
docker exec openbao bao operator unseal "$UNSEAL_KEY"

4. Install the auto-unseal systemd unit so it survives reboots

sudo install -m 0755 scripts/openbao/openbao-unseal.sh /usr/local/bin/ sudo install -m 0644 scripts/openbao/openbao-unseal.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable --now openbao-unseal.service

5. Verify by reading a known secret

TOKEN=$(cat /library/openbao-app/.scripts-token) curl -sf http://127.0.0.1:8200/v1/secret/data/ntfy -H "X-Vault-Token: $TOKEN" | jq .data.data </syntaxhighlight>

If .init-output.json is lost: OpenBao is unrecoverable without it. Print it now (or keep it in a sealed envelope) — it contains the root token AND the unseal key. Without these, you cannot decrypt anything in the Raft store.

Step 5: Restore Databases

For each service, start just the DB container, restore the dump, then start the rest.
Example for Authentik:

cd /library/authentik-app docker compose up -d postgresql sleep 10 gunzip -c /dumps/authentik-YYYYMMDD.sql.gz | docker exec -i authentik-app_postgresql_1 psql -U authentik authentik docker compose up -d

Repeat for: nextcloud, immich, linkwarden, matrix

</syntaxhighlight>

Step 6: Restore System Services

Jellyfin (systemd, not Docker)

sudo apt install jellyfin

Data already restored to /var/lib/jellyfin and /etc/jellyfin

AudioBookshelf
Install from package, data at /usr/share/audiobookshelf

</syntaxhighlight>

Step 6b: Restore Claude Code Access (Vaultwarden MCP)

Once Vaultwarden is up (Step 5 + nginx routed), restore the bw CLI and MCP wrappers so a Claude session can help you finish recovery from this point on.

1. Install bw CLI (snap is the official Bitwarden source)

sudo snap install bw

2. Install Bitwarden MCP server (npm, globally)

npm install -g @bitwarden/mcp-server

3. Restore wrappers — these are in /data/home/.config/claude/ from backup

mkdir -p ~/.config/claude sudo cp /data/home/.config/claude/{vaultwarden.env,bw-mcp-start.sh,bw-get.py} ~/.config/claude/ chmod 600 ~/.config/claude/vaultwarden.env chmod 700 ~/.config/claude/bw-mcp-start.sh ~/.config/claude/bw-get.py

4. Verify bw can talk to Vaultwarden (not the public Bitwarden cloud!)

bw config server # MUST print https://vault.wilsoz.com — if not, run: bw config server https://vault.wilsoz.com bw login --apikey # uses BW_CLIENTID + BW_CLIENTSECRET from env

5. Wire MCPs into ~/.claude.json (vault + bitwarden entries)
See services/VAULTWARDEN.md for the exact JSON structure if it needs to be rebuilt.

</syntaxhighlight>

See services/VAULTWARDEN.md for full setup including the mcp-server-bitwarden symlink bug workaround.

Step 7: Re-establish Networking

Install Tailscale

curl -fsSL https://tailscale.com/install.sh | sh sudo tailscale up

VPS nginx configs point to Tailscale IPs — update if IP changes
VPS SSH: ssh -p 1981 [email protected]

</syntaxhighlight>

Scenario 3: Restore Specific Files

Restore a single file from a specific snapshot

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
  restic restore SNAPSHOT_ID --target /tmp/restore --include /data/developer/SystemResiliency/STATUS.md'

docker cp restic-backup:/tmp/restore/data/developer/SystemResiliency/STATUS.md /tmp/ </syntaxhighlight>

Scenario 4: Matrix Chat Recovery

Matrix requires special care — the signing key is critical.

1. Restore Matrix data

restic restore latest --target /tmp/restore --include /data/matrix-chat

2. Critical files:
- synapse/signing.key — IRREPLACEABLE, needed for federation
- synapse/homeserver.yaml — server config
- bridges/*/config.yaml — bridge configs
- bridges/*/registration.yaml — bridge registrations

3. Restore database

gunzip -c /dumps/matrix-YYYYMMDD.sql.gz | docker exec -i matrix-postgres psql -U synapse

4. Restart stack

cd /library/matrix-chat && docker compose up -d </syntaxhighlight>

Verification Commands

List all snapshots

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

Check repository integrity

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic check'

Check repo size on Hetzner

ssh -p 23 [email protected] "du -sh backups/"

List files in a snapshot

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
  restic ls latest /data/matrix-chat'

Verify a DB dump

gunzip -t /library/backup-stack/dumps/immich-YYYYMMDD.sql.gz && echo "OK" || echo "CORRUPT"

Manual backup trigger

sudo /library/backup-stack/scripts/host-dump-databases.sh docker exec restic-backup /scripts/backup-to-hetzner.sh </syntaxhighlight>

Automated Monitoring

Check	Frequency	Alert Channel
Backup age (n8n)	Every 6h	ntfy (nc700x-alerts) + Matrix
Integrity check (restic)	Daily 06:00	ntfy if FAIL
Restore test (restic)	Weekly Sunday 07:00	ntfy if FAIL
Disk space (n8n)	Every 4h	ntfy (nc700x-alerts) + Matrix

Key Credentials

Keep a copy of these offline:

Credential	Location
Hetzner SSH key	`~/.ssh/id_rsa` on stronghold
Restic password	`/library/backup-stack/.env`
Hetzner account	Robot panel, u540853
ntfy token	`tk_0fhigfiglbjw9r8sk9jv4hpr7q67f`
Authentik admin	`https://key.wilsoz.com/if/admin/`
VPS SSH	`ssh -p 1981 [email protected]`
Synapse shared secret	In `homeserver.yaml` line 50
Matrix signing key	`/library/matrix-chat/synapse/signing.key` — IRREPLACEABLE

Restore Test Log

2026-02-23 — False Failure Alerts + Upload Anomaly Detection

Problem 1 — False failure alert after meta-array removal: validate-backups.sh still contained a restic check block against $METAARRAY_REPO / $METAARRAY_PASSWORD. Both vars were removed from .env when meta-array was dropped on 2026-02-22. Empty repo var → restic failed → ERRORS++ → ntfy nc700x-alerts fired every day.

Fix: Removed the dead meta-array block from validate-backups.sh (lines 44–52).

Problem 2 — Large upload on first run after container recreation: Restic container was recreated during meta-array removal, clearing the local cache. Without cache, restic reported "no parent snapshot found" and re-scanned all 441,884 files (all showed as "new"). Chunk dedup still worked correctly — only 11.5 GiB of 314 GiB was actually uploaded. Confirmed by upload anomaly report: Stalwart ~8k new email files (ongoing Fastmail getmail import) + /data/home (all new to the repo without a parent reference). Normal next run.

Enhancement — Upload anomaly detection: Added detect_upload_anomaly() to run-backup.sh:

Tracks upload size per run in /dumps/upload-history.txt (rolling 30 entries)
If upload >2.5× rolling average AND >500 MiB, runs restic diff <prev> <cur> and summarises changes by service path
POSTs JSON payload to n8n webhook (whO5LBZHQHCXXG9I) which formats and sends to Matrix
ntfy not used — this is informational, not an emergency

2026-02-22 — Recurring SSH Timeout Incident

Problem: Backups failing with subprocess ssh: Timeout / Fatal: unable to save snapshot: context canceled. Hetzner SFTP server drops idle connections (~5 min) during restic's local scan phase (22+ min). Compounded by 96 accumulated index files (documented threshold: 49) because prune was also failing.

Secondary cause removed: meta-array (ma.sdf.org) secondary backup target was unreliable. Since run-backup.sh used set -e, any metaarray failure marked the entire job as failed even when Hetzner backup succeeded. Removed entirely.

Fixes applied:

restic repair index — consolidated 96 → 3 index files
restic forget --prune — cleaned up retention
Added -o sftp.args="-oServerAliveInterval=10 -oServerAliveCountMax=36" to backup-to-hetzner.sh — 10s keepalives, 6-min tolerance
Added --retry-lock 10m to backup-to-hetzner.sh
Removed backup-to-metaarray.sh and all meta-array config
Added /home/mnw → /data/home to backup scope (dotfiles, Documents, source, SSH keys)

If SSH timeouts recur: It means index files are accumulating again (prune is failing silently). Run: <syntaxhighlight lang="bash"> docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic repair index' docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune' </syntaxhighlight>

2026-02-16 — First Manual Test

Test: Restore files from latest snapshot (ae89d11d), compare against live originals.

File	Restored	Compared	Result
`matrix-chat/synapse/homeserver.yaml`	3,022 bytes	vs live	IDENTICAL
`matrix-chat/synapse/signing.key`	59 bytes	vs live	IDENTICAL
`matrix-chat/docker-compose.yml`	3,232 bytes	vs live	IDENTICAL
`matrix-chat/bridges/whatsapp/config.yaml`	33,990 bytes	vs live	IDENTICAL
`developer/SystemResiliency/STATUS.md`	Restored	pre-edit version	Expected diff (edited today)
DB dump: `immich-20260213.sql.gz`	463 MB	gunzip -t	VALID
DB dump: `matrix-20260215.sql.gz`	1 MB	gunzip -t	VALID
DB dump: `authentik-20260213.sql.gz`	16 MB	gunzip -t	VALID

Issues Found:

DB dumps from Feb 14-15 were 20 bytes (empty) — container's run-backup.sh was calling dump-databases.sh inside the container where docker CLI doesn't exist, overwriting the good host-side dumps.
Fixed: Deployed updated run-backup.sh that verifies dumps instead of re-dumping. Fresh dumps created manually and confirmed valid.

Verdict: PASS — restore process works. Files are intact and match live originals.

How the Backup System Works

The Big Picture

What Gets Backed Up

The Components

1. Restic Container (restic-backup)

2. Database Dump Script (host-dump-databases.sh)

3. Backup Script (backup-to-hetzner.sh)

4. Systemd Timers

How Restic Works (The Magic)

Deduplication

Encryption

Snapshots

How to Check on Backups

Is the backup running?

How big is my backup repo?

List all snapshots

Check repo integrity

How to Restore

Restore everything (disaster recovery)

Restore one file

Restore from specific date

Important Files

What Happens If...

Power goes out during backup?

Hetzner goes down?

You delete a file by accident?

Your server dies completely?

You forget the password?

Costs

Summary

Disaster Recovery Runbook

Disaster Recovery Runbook

Quick Reference

Stronghold (primary)

VPS (wilsoz.com)

What's Backed Up

Service Data (files)

Database Dumps (in /dumps/)

NOT Backed Up (re-acquirable)

⚠️ Known Gaps — Data Not Currently Backed Up

Recovery Scenarios

Scenario 1: Single Service Failure

Scenario 2: Full Server Rebuild

Bootstrap Order (DO NOT SHUFFLE)

Step 1: Install Prerequisites

Step 2: Restore SSH Keys

Step 3: Bootstrap Restic

Step 4: Restore Everything

Step 4b: Bootstrap OpenBao (Secret Store) — REQUIRED before DBs

Step 5: Restore Databases

Step 6: Restore System Services

Step 6b: Restore Claude Code Access (Vaultwarden MCP)

Step 7: Re-establish Networking

Scenario 3: Restore Specific Files

Scenario 4: Matrix Chat Recovery

Verification Commands

Automated Monitoring

Key Credentials

Restore Test Log

2026-02-23 — False Failure Alerts + Upload Anomaly Detection

2026-02-22 — Recurring SSH Timeout Incident

2026-02-16 — First Manual Test

1. Restic Container (`restic-backup`)

2. Database Dump Script (`host-dump-databases.sh`)

3. Backup Script (`backup-to-hetzner.sh`)

Database Dumps (in `/dumps/`)