Identity/AuthentikMatrix

As of: 2026-07-05 — migrated from INFRASTRUCTURE_INVENTORY.md's Authentication Matrix section

How the Backup System Works

A beginner-friendly guide to understanding your new backup system.

The Big Picture

<syntaxhighlight lang="text"> Your Server (stronghold)

        │
        │ 1. Dump databases to files
        ▼
  ┌─────────────┐
  │ PostgreSQL  │──► .sql.gz files
  │ Containers  │    (in /library/backup-stack/dumps/)
  └─────────────┘
        │
        │ 2. Restic reads all your data
        ▼
  ┌─────────────┐
  │   Restic    │──► Deduplicates, compresses, encrypts
  │  Container  │
  └─────────────┘
        │
        │ 3. Uploads to cloud storage
        ▼
  ┌─────────────┐
  │   Hetzner   │  Your encrypted backups live here
  │ Storage Box │  (in Germany, €3.81/month)
  └─────────────┘

</syntaxhighlight>

What Gets Backed Up

What	Where It Lives	Size	Why It Matters
Authentik DB	PostgreSQL dump	14MB	Your SSO - lose this, lose all logins
Vaultwarden	SQLite file	356KB	Your passwords!
Immich photos	/raid1/.../Immich-Photos	149GB	Family photos
Immich DB	PostgreSQL dump	437MB	Photo metadata, faces, albums
Jellyfin	/var/lib/jellyfin	92GB	Watch history, metadata
Nextcloud	Config + PostgreSQL	~1GB	Files, calendar, contacts
Open-WebUI	SQLite + uploads	1.7GB	AI chat history
Memos, Linkwarden, Mailcow	Various	Small	Notes, bookmarks, email
MediaWiki	Config bind mount + MariaDB dump + images volume	~300KB	The wilsoz wiki — infra docs, runbooks, ADRs

The Components

1. Restic Container (restic-backup)

A small container that knows how to:

• Read your data from mounted directories
• Encrypt everything with your password
• Upload to Hetzner via SFTP
• Deduplicate (only upload changed data)

2. Database Dump Script (host-dump-databases.sh)

Runs on the host (not in container) because it needs Docker access. <syntaxhighlight lang="text"> authentik-postgresql → authentik-20260201.sql.gz nextcloud-db → nextcloud-20260201.sql.gz immich_postgres → immich-20260201.sql.gz linkwarden_postgres → linkwarden-20260201.sql.gz </syntaxhighlight>

3. Backup Script (backup-to-hetzner.sh)

Runs inside the container:

1. Connects to Hetzner via SSH (port 23, key auth)
2. Reads all your data directories
3. Uploads new/changed files
4. Prunes old snapshots (keeps 7 daily, 4 weekly, 6 monthly)

4. Systemd Timers

Schedules everything to run automatically:

• 3:00 AM - Full backup
• 6:00 AM - Integrity check
• Sunday 7:00 AM - Restore test

How Restic Works (The Magic)

Deduplication

Restic breaks files into chunks. If you have 1000 photos and add 1 new one:

• First backup: uploads all 1000 photos (~149GB)
• Second backup: uploads only the 1 new photo (~5MB)

Encryption

Everything is encrypted with your password before leaving your server.

• Hetzner can't read your data
• Without the password, backups are useless
• Password: E+mVDSjj8VsB5Feklelsq/BAJ3FmMX3E (stored in .env)

Snapshots

Each backup creates a "snapshot" - a point-in-time view of your data. <syntaxhighlight lang="text"> $ restic snapshots ID Time Host Tags ───────────────────────────────────────────────── a1b2c3d4 2026-02-01 23:50 stronghold full e5f6g7h8 2026-02-02 03:00 stronghold full ... </syntaxhighlight>

You can restore any snapshot, not just the latest.

How to Check on Backups

Is the backup running?

<syntaxhighlight lang="bash"> docker logs restic-backup </syntaxhighlight>

How big is my backup repo?

<syntaxhighlight lang="bash"> docker exec restic-backup restic stats </syntaxhighlight>

List all snapshots

<syntaxhighlight lang="bash"> docker exec restic-backup restic snapshots </syntaxhighlight>

Check repo integrity

<syntaxhighlight lang="bash"> docker exec restic-backup restic check </syntaxhighlight>

How to Restore

Restore everything (disaster recovery)

<syntaxhighlight lang="bash"> docker exec restic-backup restic restore latest --target /tmp/full-restore </syntaxhighlight>

Restore one file

<syntaxhighlight lang="bash"> docker exec restic-backup restic restore latest \

 --target /tmp/restore \
 --include "/data/vaultwarden/data/db.sqlite3"

</syntaxhighlight>

Restore from specific date

<syntaxhighlight lang="bash">

1. List snapshots to find the ID

docker exec restic-backup restic snapshots

1. Restore that snapshot

docker exec restic-backup restic restore a1b2c3d4 --target /tmp/restore </syntaxhighlight>

Important Files

<syntaxhighlight lang="text"> /library/backup-stack/ ├── docker-compose.yml # Container definition ├── .env # Passwords and settings (SECRET!) ├── ssh-config # SSH settings for Hetzner ├── dumps/ # Database dumps live here │ ├── authentik-20260201.sql.gz │ ├── immich-20260201.sql.gz │ └── ... ├── scripts/ │ ├── host-dump-databases.sh # Dumps DBs (runs on host) │ ├── backup-to-hetzner.sh # Main backup (runs in container) │ └── validate-backups.sh # Tests backups └── systemd/ # Timer definitions </syntaxhighlight>

What Happens If...

Power goes out during backup?

Restic is crash-safe. Next backup continues from where it left off.

Hetzner goes down?

Your local data is fine. Backups resume when Hetzner is back.

You delete a file by accident?

Restore it from a previous snapshot.

Your server dies completely?

1. Get new server
2. Install Docker
3. Connect to Hetzner
4. Run restic restore latest
5. Rebuild containers with restored data

You forget the password?

Game over. The backups are encrypted. No password = no data. Save it somewhere safe (Vaultwarden, password manager, printed paper in safe).

Costs

Item	Cost
Hetzner Storage Box 1TB	€3.81/month
Everything else	Free (self-hosted)

Summary

1. Every night at 3 AM:

• • Databases get dumped to .sql.gz files
  • Restic encrypts and uploads everything to Germany
  • Old backups get cleaned up automatically

1. Every morning at 6 AM:

• • System checks backup integrity
  • Alerts you via ntfy if something's wrong

1. Every Sunday:

• • Does a test restore to verify backups actually work

Your data is now protected. Sleep well.

---

Disaster Recovery Runbook

Last Updated: 2026-06-21 Print this and store in a safe place.

Keep DR current: after adding/changing/removing any service, run

scripts/dr-drift-check.sh and act on whatever it flags. See

DR_MAINTENANCE.md for the full workflow. Don't skip this — gradual drift

is how DR plans silently rot until the day you actually need them.

---

Quick Reference

Stronghold (primary)

Item	Value
Hetzner Storage Box	u540853.your-storagebox.de port 23
Repo path	sftp:[email protected]:./backups (port 23 comes from SSH config, not the URL — see note)
SSH Key	~/.ssh/id_rsa (on stronghold)
Restic Password	In /library/backup-stack/.env (HETZNER_PASSWORD)
Backup Schedule	Daily 03:00 CST
Retention	7 daily, 4 weekly, 6 monthly

VPS (wilsoz.com)

Item	Value
Hetzner Storage Box	same box, separate repo
Repo path	sftp:[email protected]:./backups-vps (port 23 comes from SSH config, not the URL — see note)
SSH Key	~/.ssh/id_ed25519 (on VPS)
Restic Password	same HETZNER_PASSWORD — stored in /etc/restic-vps.env on VPS
Backup Schedule	Daily 04:30 UTC (offset from stronghold)
Retention	7 daily, 4 weekly, 6 monthly
Covers	Forgejo data + repos, Atuin (pg_dump), ntfy, nginx, Let's Encrypt certs, /opt/ custom scripts

⚠️ SFTP repo URL gotcha (port vs. path): In restic's single-colon SFTP form

(sftp:user@host:path) everything after the host colon is the repository path —

there is no port in the URL. The storage box's port 23 must be set in SSH config

(~/.ssh/config → Port 23 for host u540853.your-storagebox.de), which both

stronghold and the restic-backup container already have. Writing the port into the

URL as :23/backups makes restic look for a repo at the literal path 23/backups and

fail with "repository does not exist". Always use the relative :./backups /

:./backups-vps form. (This exact mistake caused false "VPS: no snapshots found"

alerts from the n8n Backup Status Alert (SSH) workflow until 2026-06-29.)

---

What's Backed Up

Service Data (files)

Path in Backup	Source on Host	Contents
/data/authentik	/library/authentik-app/	SSO/identity provider config
/data/vaultwarden	/library/vaultwarden-app/	Password vault (SQLite DB + attachments)
/data/nextcloud	/library/nextcloud-app/	File sync config + data
/data/immich-config	/library/immich-app/	Photo manager config
/data/immich-photos	/raid1/Consolidated/Picture/Immich-Photos/	All photos (149G)
/data/jellyfin	/var/lib/jellyfin/	Media server metadata (92G)
/data/jellyfin-etc	/etc/jellyfin/	Jellyfin system config
/data/memos	/library/memo-app/	Notes app
/data/linkwarden	/library/linkwarden-app/	Bookmark manager
/data/open-webui	/library/open-webui-app/	AI chat interface
/data/matrix-chat	/library/matrix-chat/	Synapse + bridge configs + signing key
/data/stalwart	/library/stalwart-app/data/	Mail server data
/data/snappymail	/library/snappymail-app/data/	Webmail data
/data/mailcow	/opt/mailcow-dockerized/	Legacy mail (to be archived)
/data/home	/home/mnw/	Home dir: dotfiles, scripts, Documents, source, SSH/GPG keys (selective excludes)
/data/developer	/home/mnw/Developer/	All dev projects (7.9G) — also covered by /data/home
/data/beets	/raid1/my-tunes/.beets/	Music library DB
/data/openbao	/library/openbao-app/	OpenBao secret store — incl. .init-output.json (root token + unseal key) and .scripts-token / .claude-token. Bootstrap-critical: must be restored and unsealed before services that depend on secrets.
/data/ezbookkeeping	/library/ezbookkeeping-app/data/	Personal finance tracker (SQLite + uploads)
/data/omada	/library/omada-app/	Omada SDN controller — switch/AP config & topology

Database Dumps (in /dumps/)

File Pattern	Source Container	Database
authentik-YYYYMMDD.sql.gz	authentik-app_postgresql_1	Authentik identity DB
nextcloud-YYYYMMDD.sql.gz	nextcloud-db	Nextcloud DB
immich-YYYYMMDD.sql.gz	immich_postgres	Immich photo DB (~460MB compressed)
linkwarden-YYYYMMDD.sql.gz	linkwarden_postgres	Linkwarden bookmark DB
matrix-YYYYMMDD.sql.gz	matrix-postgres	Synapse + bridge DBs (~1MB compressed)

NOT Backed Up (re-acquirable)

• Jellyfin media files (movies, TV)
• Ollama models (~20G, re-downloadable)
• Steam games
• VirtualBox VMs
• node_modules, .venv, .git/objects
• Wyoming voice models (piper-data-app, whisper-data-app) — redownloadable
• Prometheus metric history, Grafana dashboards data, uptime-kuma — re-buildable from compose

⚠️ Known Gaps — Data Not Currently Backed Up

Severity	What	Where	Notes
HIGH	AudioBookshelf config + metadata	/usr/share/audiobookshelf/config/ (38M), /usr/share/audiobookshelf/metadata/ (936M)	systemd service, not Docker — not yet bind-mounted into restic container. Lose library DB, progress, and cover art on rebuild.

Previously flagged gaps (all closed as of 2026-06-21): Nextcloud user files, Matrix media, Mealie, n8n named volumes, uptime-kuma, Grafana, AdGuard — all now covered by bind mounts in the restic container compose.

---

Recovery Scenarios

Scenario 1: Single Service Failure

Example: Immich database corrupted, need to restore just Immich.

<syntaxhighlight lang="bash">

1. 1. Stop the service

cd /library/immich-app && docker compose down

1. 2. List available snapshots

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

1. 3. Restore service data from latest snapshot

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
  restic restore latest --target /tmp/restore --include /data/immich-config'

1. 4. Copy restored data to host

docker cp restic-backup:/tmp/restore/data/immich-config /tmp/immich-restore

1. 5. Replace the broken data

mv /library/immich-app /library/immich-app.broken mv /tmp/immich-restore /library/immich-app

1. 6. Restore the database dump

docker cp restic-backup:/tmp/restore/dumps/immich-YYYYMMDD.sql.gz /tmp/ gunzip /tmp/immich-YYYYMMDD.sql.gz

1. Start just the database container

docker compose up -d immich_postgres

1. Wait for it to be ready, then restore

docker exec -i immich_postgres psql -U postgres immich < /tmp/immich-YYYYMMDD.sql

1. 7. Start the full service

docker compose up -d

1. 8. Clean up

docker exec restic-backup rm -rf /tmp/restore </syntaxhighlight>

Scenario 2: Full Server Rebuild

Starting from: Fresh Linux install on new hardware.

Bootstrap Order (DO NOT SHUFFLE)

Many services depend on OpenBao (secret store) and Authentik (SSO). Bring them up in this order so each layer has its dependencies in place:

<syntaxhighlight lang="text"> 1. Host: Docker, restic, Tailscale, SSH key 2. Restore everything from Hetzner via restic (raw files) 3. OpenBao ← unseal it first; it holds tokens other services need 4. Authentik (DB dump → start) ← SSO for everything 5. Vaultwarden ← human-facing creds; needed for me-the-operator 6. Restore Claude Code access (bw CLI, MCP wrappers) ← so future-you can drive recovery 7. Postgres-backed services (Immich, Nextcloud, Matrix, Linkwarden): DB dump → start 8. Stalwart + SnappyMail (mail) 9. Everything else (n8n, Grafana, Mealie, etc.) </syntaxhighlight>

Step 1: Install Prerequisites

<syntaxhighlight lang="bash">

1. Install Docker

curl -fsSL https://get.docker.com | sh sudo usermod -aG docker mnw

1. Install restic

sudo apt install restic

1. Set Docker MTU for Tailscale compatibility
2. Add to /etc/docker/daemon.json:
3. {"mtu": 1280}

sudo systemctl restart docker </syntaxhighlight>

Step 2: Restore SSH Keys

You need the SSH key to access Hetzner. If you have a copy:

<syntaxhighlight lang="bash"> mkdir -p ~/.ssh

1. Copy id_rsa from your backup/safe location

chmod 600 ~/.ssh/id_rsa </syntaxhighlight>

If you don't have the key, generate a new one and add it to Hetzner: <syntaxhighlight lang="bash"> ssh-keygen -t rsa -b 4096

1. Add public key via Hetzner Robot panel → Storage Box → SSH Keys

</syntaxhighlight>

Step 3: Bootstrap Restic

<syntaxhighlight lang="bash">

1. Create the backup stack directory

sudo mkdir -p /library/backup-stack cd /library/backup-stack

1. Create .env with Hetzner credentials:

cat > .env << 'EOF' HETZNER_REPO=sftp:[email protected]:23/backups HETZNER_PASSWORD=<your-restic-password> NTFY_TOKEN=tk_0fhigfiglbjw9r8sk9jv4hpr7q67f EOF

1. Test connection

export $(grep -v '^#' .env | xargs) restic -r "$HETZNER_REPO" snapshots </syntaxhighlight>

Step 4: Restore Everything

<syntaxhighlight lang="bash">

1. Create target directories

sudo mkdir -p /library /raid1/Consolidated/Picture/Immich-Photos

1. Restore all service data

restic -r "$HETZNER_REPO" restore latest --target /

1. This puts files at their original paths:
2. /data/authentik → needs moving to /library/authentik-app/
3. /data/developer → needs moving to /home/mnw/Developer/
4. etc.

1. Move restored data to correct locations

sudo mv /data/authentik /library/authentik-app sudo mv /data/vaultwarden /library/vaultwarden-app sudo mv /data/nextcloud /library/nextcloud-app sudo mv /data/immich-config /library/immich-app sudo mv /data/immich-photos/* /raid1/Consolidated/Picture/Immich-Photos/ sudo mv /data/jellyfin /var/lib/jellyfin sudo mv /data/jellyfin-etc /etc/jellyfin sudo mv /data/memos /library/memo-app sudo mv /data/linkwarden /library/linkwarden-app sudo mv /data/open-webui /library/open-webui-app sudo mv /data/matrix-chat /library/matrix-chat sudo mv /data/stalwart /library/stalwart-app/data sudo mv /data/snappymail /library/snappymail-app/data sudo mv /data/developer /home/mnw/Developer sudo mv /data/beets /raid1/my-tunes/.beets sudo chown -R mnw:mnw /home/mnw/Developer </syntaxhighlight>

Step 4b: Bootstrap OpenBao (Secret Store) — REQUIRED before DBs

OpenBao holds tokens and secrets that downstream services + scripts read at startup.

<syntaxhighlight lang="bash">

1. 1. Move restored data into place

sudo mv /data/openbao /library/openbao-app sudo chown -R 1000:1000 /library/openbao-app # OpenBao runs as uid 1000

1. 2. Start the container

cd /library/openbao-app && docker compose up -d

1. 3. Unseal — uses the unseal key from .init-output.json (restored from backup)

sudo /usr/local/bin/openbao-unseal.sh

1. If that script isn't restored yet, manually:
2. ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
3. UNSEAL_KEY=$(jq -r .unseal_keys_b64[0] /library/openbao-app/.init-output.json)
4. docker exec openbao bao operator unseal "$UNSEAL_KEY"

1. 4. Install the auto-unseal systemd unit so it survives reboots

sudo install -m 0755 scripts/openbao/openbao-unseal.sh /usr/local/bin/ sudo install -m 0644 scripts/openbao/openbao-unseal.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable --now openbao-unseal.service

1. 5. Verify by reading a known secret

TOKEN=$(cat /library/openbao-app/.scripts-token) curl -sf http://127.0.0.1:8200/v1/secret/data/ntfy -H "X-Vault-Token: $TOKEN" | jq .data.data </syntaxhighlight>

If .init-output.json is lost: OpenBao is unrecoverable without it. Print it now (or keep it in a sealed envelope) — it contains the root token AND the unseal key. Without these, you cannot decrypt anything in the Raft store.

Step 5: Restore Databases

<syntaxhighlight lang="bash">

1. For each service, start just the DB container, restore the dump, then start the rest.
2. Example for Authentik:

cd /library/authentik-app docker compose up -d postgresql sleep 10 gunzip -c /dumps/authentik-YYYYMMDD.sql.gz | docker exec -i authentik-app_postgresql_1 psql -U authentik authentik docker compose up -d

1. Repeat for: nextcloud, immich, linkwarden, matrix

</syntaxhighlight>

Step 6: Restore System Services

<syntaxhighlight lang="bash">

1. Jellyfin (systemd, not Docker)

sudo apt install jellyfin

1. Data already restored to /var/lib/jellyfin and /etc/jellyfin

1. AudioBookshelf
2. Install from package, data at /usr/share/audiobookshelf

</syntaxhighlight>

Step 6b: Restore Claude Code Access (Vaultwarden MCP)

Once Vaultwarden is up (Step 5 + nginx routed), restore the bw CLI and MCP wrappers so a Claude session can help you finish recovery from this point on.

<syntaxhighlight lang="bash">

1. 1. Install bw CLI (snap is the official Bitwarden source)

sudo snap install bw

1. 2. Install Bitwarden MCP server (npm, globally)

npm install -g @bitwarden/mcp-server

1. 3. Restore wrappers — these are in /data/home/.config/claude/ from backup

mkdir -p ~/.config/claude sudo cp /data/home/.config/claude/{vaultwarden.env,bw-mcp-start.sh,bw-get.py} ~/.config/claude/ chmod 600 ~/.config/claude/vaultwarden.env chmod 700 ~/.config/claude/bw-mcp-start.sh ~/.config/claude/bw-get.py

1. 4. Verify bw can talk to Vaultwarden (not the public Bitwarden cloud!)

bw config server # MUST print https://vault.wilsoz.com — if not, run: bw config server https://vault.wilsoz.com bw login --apikey # uses BW_CLIENTID + BW_CLIENTSECRET from env

1. 5. Wire MCPs into ~/.claude.json (vault + bitwarden entries)
2. See services/VAULTWARDEN.md for the exact JSON structure if it needs to be rebuilt.

</syntaxhighlight>

See services/VAULTWARDEN.md for full setup including the mcp-server-bitwarden symlink bug workaround.

Step 7: Re-establish Networking

<syntaxhighlight lang="bash">

1. Install Tailscale

curl -fsSL https://tailscale.com/install.sh | sh sudo tailscale up

1. VPS nginx configs point to Tailscale IPs — update if IP changes
2. VPS SSH: ssh -p 1981 [email protected]

</syntaxhighlight>

Scenario 3: Restore Specific Files

<syntaxhighlight lang="bash">

1. Restore a single file from a specific snapshot

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
  restic restore SNAPSHOT_ID --target /tmp/restore --include /data/developer/SystemResiliency/STATUS.md'

docker cp restic-backup:/tmp/restore/data/developer/SystemResiliency/STATUS.md /tmp/ </syntaxhighlight>

Scenario 4: Matrix Chat Recovery

Matrix requires special care — the signing key is critical.

<syntaxhighlight lang="bash">

1. 1. Restore Matrix data

restic restore latest --target /tmp/restore --include /data/matrix-chat

1. 2. Critical files:
2. - synapse/signing.key — IRREPLACEABLE, needed for federation
3. - synapse/homeserver.yaml — server config
4. - bridges/*/config.yaml — bridge configs
5. - bridges/*/registration.yaml — bridge registrations

1. 3. Restore database

gunzip -c /dumps/matrix-YYYYMMDD.sql.gz | docker exec -i matrix-postgres psql -U synapse

1. 4. Restart stack

cd /library/matrix-chat && docker compose up -d </syntaxhighlight>

---

Verification Commands

<syntaxhighlight lang="bash">

1. List all snapshots

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

1. Check repository integrity

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic check'

1. Check repo size on Hetzner

ssh -p 23 [email protected] "du -sh backups/"

1. List files in a snapshot

docker exec restic-backup sh -c \

 'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
  restic ls latest /data/matrix-chat'

1. Verify a DB dump

gunzip -t /library/backup-stack/dumps/immich-YYYYMMDD.sql.gz && echo "OK" || echo "CORRUPT"

1. Manual backup trigger

sudo /library/backup-stack/scripts/host-dump-databases.sh docker exec restic-backup /scripts/backup-to-hetzner.sh </syntaxhighlight>

---

Automated Monitoring

Check	Frequency	Alert Channel
Backup age (n8n)	Every 6h	ntfy (nc700x-alerts) + Matrix
Integrity check (restic)	Daily 06:00	ntfy if FAIL
Restore test (restic)	Weekly Sunday 07:00	ntfy if FAIL
Disk space (n8n)	Every 4h	ntfy (nc700x-alerts) + Matrix

---

Key Credentials

Keep a copy of these offline:

Credential	Location
Hetzner SSH key	~/.ssh/id_rsa on stronghold
Restic password	/library/backup-stack/.env
Hetzner account	Robot panel, u540853
ntfy token	tk_0fhigfiglbjw9r8sk9jv4hpr7q67f
Authentik admin	https://key.wilsoz.com/if/admin/
VPS SSH	ssh -p 1981 [email protected]
Synapse shared secret	In homeserver.yaml line 50
Matrix signing key	/library/matrix-chat/synapse/signing.key — IRREPLACEABLE

---

Restore Test Log

2026-02-23 — False Failure Alerts + Upload Anomaly Detection

Problem 1 — False failure alert after meta-array removal: validate-backups.sh still contained a restic check block against $METAARRAY_REPO / $METAARRAY_PASSWORD. Both vars were removed from .env when meta-array was dropped on 2026-02-22. Empty repo var → restic failed → ERRORS++ → ntfy nc700x-alerts fired every day.

Fix: Removed the dead meta-array block from validate-backups.sh (lines 44–52).

Problem 2 — Large upload on first run after container recreation: Restic container was recreated during meta-array removal, clearing the local cache. Without cache, restic reported "no parent snapshot found" and re-scanned all 441,884 files (all showed as "new"). Chunk dedup still worked correctly — only 11.5 GiB of 314 GiB was actually uploaded. Confirmed by upload anomaly report: Stalwart ~8k new email files (ongoing Fastmail getmail import) + /data/home (all new to the repo without a parent reference). Normal next run.

Enhancement — Upload anomaly detection: Added detect_upload_anomaly() to run-backup.sh:

• Tracks upload size per run in /dumps/upload-history.txt (rolling 30 entries)
• If upload >2.5× rolling average AND >500 MiB, runs restic diff <prev> <cur> and summarises changes by service path
• POSTs JSON payload to n8n webhook (whO5LBZHQHCXXG9I) which formats and sends to Matrix
• ntfy not used — this is informational, not an emergency

---

2026-02-22 — Recurring SSH Timeout Incident

Problem: Backups failing with subprocess ssh: Timeout / Fatal: unable to save snapshot: context canceled. Hetzner SFTP server drops idle connections (~5 min) during restic's local scan phase (22+ min). Compounded by 96 accumulated index files (documented threshold: 49) because prune was also failing.

Secondary cause removed: meta-array (ma.sdf.org) secondary backup target was unreliable. Since run-backup.sh used set -e, any metaarray failure marked the entire job as failed even when Hetzner backup succeeded. Removed entirely.

Fixes applied:

1. restic repair index — consolidated 96 → 3 index files
2. restic forget --prune — cleaned up retention
3. Added -o sftp.args="-oServerAliveInterval=10 -oServerAliveCountMax=36" to backup-to-hetzner.sh — 10s keepalives, 6-min tolerance
4. Added --retry-lock 10m to backup-to-hetzner.sh
5. Removed backup-to-metaarray.sh and all meta-array config
6. Added /home/mnw → /data/home to backup scope (dotfiles, Documents, source, SSH keys)

If SSH timeouts recur: It means index files are accumulating again (prune is failing silently). Run: <syntaxhighlight lang="bash"> docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic repair index' docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune' </syntaxhighlight>

---

2026-02-16 — First Manual Test

Test: Restore files from latest snapshot (ae89d11d), compare against live originals.

File	Restored	Compared	Result
matrix-chat/synapse/homeserver.yaml	3,022 bytes	vs live	IDENTICAL
matrix-chat/synapse/signing.key	59 bytes	vs live	IDENTICAL
matrix-chat/docker-compose.yml	3,232 bytes	vs live	IDENTICAL
matrix-chat/bridges/whatsapp/config.yaml	33,990 bytes	vs live	IDENTICAL
developer/SystemResiliency/STATUS.md	Restored	pre-edit version	Expected diff (edited today)
DB dump: immich-20260213.sql.gz	463 MB	gunzip -t	VALID
DB dump: matrix-20260215.sql.gz	1 MB	gunzip -t	VALID
DB dump: authentik-20260213.sql.gz	16 MB	gunzip -t	VALID

Issues Found:

• DB dumps from Feb 14-15 were 20 bytes (empty) — container's run-backup.sh was calling dump-databases.sh inside the container where docker CLI doesn't exist, overwriting the good host-side dumps.
• Fixed: Deployed updated run-backup.sh that verifies dumps instead of re-dumping. Fresh dumps created manually and confirmed valid.

Verdict: PASS — restore process works. Files are intact and match live originals.