View source for Identity/AuthentikMatrix

''As of: 2026-07-05 — migrated from INFRASTRUCTURE_INVENTORY.md's Authentication Matrix section''

= How the Backup System Works =

A beginner-friendly guide to understanding your new backup system.

== The Big Picture ==

<syntaxhighlight lang="text">
Your Server (stronghold)
         │
         │ 1. Dump databases to files
         ▼
   ┌─────────────┐
   │ PostgreSQL  │──► .sql.gz files
   │ Containers  │    (in /library/backup-stack/dumps/)
   └─────────────┘
         │
         │ 2. Restic reads all your data
         ▼
   ┌─────────────┐
   │   Restic    │──► Deduplicates, compresses, encrypts
   │  Container  │
   └─────────────┘
         │
         │ 3. Uploads to cloud storage
         ▼
   ┌─────────────┐
   │   Hetzner   │  Your encrypted backups live here
   │ Storage Box │  (in Germany, €3.81/month)
   └─────────────┘
</syntaxhighlight>

== What Gets Backed Up ==

{| class="wikitable"
|-
! What
! Where It Lives
! Size
! Why It Matters
|-
| Authentik DB
| PostgreSQL dump
| 14MB
| Your SSO - lose this, lose all logins
|-
| Vaultwarden
| SQLite file
| 356KB
| Your passwords!
|-
| Immich photos
| /raid1/.../Immich-Photos
| 149GB
| Family photos
|-
| Immich DB
| PostgreSQL dump
| 437MB
| Photo metadata, faces, albums
|-
| Jellyfin
| /var/lib/jellyfin
| 92GB
| Watch history, metadata
|-
| Nextcloud
| Config + PostgreSQL
| ~1GB
| Files, calendar, contacts
|-
| Open-WebUI
| SQLite + uploads
| 1.7GB
| AI chat history
|-
| Memos, Linkwarden, Mailcow
| Various
| Small
| Notes, bookmarks, email
|-
| MediaWiki
| Config bind mount + MariaDB dump + images volume
| ~300KB
| The wilsoz wiki — infra docs, runbooks, ADRs
|}

== The Components ==

=== 1. Restic Container (<code>restic-backup</code>) ===
A small container that knows how to:
* Read your data from mounted directories
* Encrypt everything with your password
* Upload to Hetzner via SFTP
* Deduplicate (only upload changed data)

=== 2. Database Dump Script (<code>host-dump-databases.sh</code>) ===
Runs on the host (not in container) because it needs Docker access.
<syntaxhighlight lang="text">
authentik-postgresql → authentik-20260201.sql.gz
nextcloud-db         → nextcloud-20260201.sql.gz
immich_postgres      → immich-20260201.sql.gz
linkwarden_postgres  → linkwarden-20260201.sql.gz
</syntaxhighlight>

=== 3. Backup Script (<code>backup-to-hetzner.sh</code>) ===
Runs inside the container:
# Connects to Hetzner via SSH (port 23, key auth)
# Reads all your data directories
# Uploads new/changed files
# Prunes old snapshots (keeps 7 daily, 4 weekly, 6 monthly)

=== 4. Systemd Timers ===
Schedules everything to run automatically:
* '''3:00 AM''' - Full backup
* '''6:00 AM''' - Integrity check
* '''Sunday 7:00 AM''' - Restore test

== How Restic Works (The Magic) ==

=== Deduplication ===
Restic breaks files into chunks. If you have 1000 photos and add 1 new one:
* First backup: uploads all 1000 photos (~149GB)
* Second backup: uploads only the 1 new photo (~5MB)

=== Encryption ===
Everything is encrypted with your password before leaving your server.
* Hetzner can't read your data
* Without the password, backups are useless
* '''Password:''' <code>E+mVDSjj8VsB5Feklelsq/BAJ3FmMX3E</code> (stored in .env)

=== Snapshots ===
Each backup creates a "snapshot" - a point-in-time view of your data.
<syntaxhighlight lang="text">
$ restic snapshots
ID        Time                 Host        Tags
─────────────────────────────────────────────────
a1b2c3d4  2026-02-01 23:50    stronghold  full
e5f6g7h8  2026-02-02 03:00    stronghold  full
...
</syntaxhighlight>

You can restore any snapshot, not just the latest.

== How to Check on Backups ==

=== Is the backup running? ===
<syntaxhighlight lang="bash">
docker logs restic-backup
</syntaxhighlight>

=== How big is my backup repo? ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic stats
</syntaxhighlight>

=== List all snapshots ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic snapshots
</syntaxhighlight>

=== Check repo integrity ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic check
</syntaxhighlight>

== How to Restore ==

=== Restore everything (disaster recovery) ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic restore latest --target /tmp/full-restore
</syntaxhighlight>

=== Restore one file ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic restore latest \
  --target /tmp/restore \
  --include "/data/vaultwarden/data/db.sqlite3"
</syntaxhighlight>

=== Restore from specific date ===
<syntaxhighlight lang="bash">
# List snapshots to find the ID
docker exec restic-backup restic snapshots

# Restore that snapshot
docker exec restic-backup restic restore a1b2c3d4 --target /tmp/restore
</syntaxhighlight>

== Important Files ==

<syntaxhighlight lang="text">
/library/backup-stack/
├── docker-compose.yml    # Container definition
├── .env                  # Passwords and settings (SECRET!)
├── ssh-config            # SSH settings for Hetzner
├── dumps/                # Database dumps live here
│   ├── authentik-20260201.sql.gz
│   ├── immich-20260201.sql.gz
│   └── ...
├── scripts/
│   ├── host-dump-databases.sh  # Dumps DBs (runs on host)
│   ├── backup-to-hetzner.sh    # Main backup (runs in container)
│   └── validate-backups.sh     # Tests backups
└── systemd/              # Timer definitions
</syntaxhighlight>

== What Happens If... ==

=== Power goes out during backup? ===
Restic is crash-safe. Next backup continues from where it left off.

=== Hetzner goes down? ===
Your local data is fine. Backups resume when Hetzner is back.

=== You delete a file by accident? ===
Restore it from a previous snapshot.

=== Your server dies completely? ===
# Get new server
# Install Docker
# Connect to Hetzner
# Run <code>restic restore latest</code>
# Rebuild containers with restored data

=== You forget the password? ===
'''Game over.''' The backups are encrypted. No password = no data.
Save it somewhere safe (Vaultwarden, password manager, printed paper in safe).

== Costs ==

{| class="wikitable"
|-
! Item
! Cost
|-
| Hetzner Storage Box 1TB
| €3.81/month
|-
| Everything else
| Free (self-hosted)
|}

== Summary ==

# '''Every night at 3 AM:'''
** Databases get dumped to .sql.gz files
** Restic encrypts and uploads everything to Germany
** Old backups get cleaned up automatically

# '''Every morning at 6 AM:'''
** System checks backup integrity
** Alerts you via ntfy if something's wrong

# '''Every Sunday:'''
** Does a test restore to verify backups actually work

Your data is now protected. Sleep well.

----

= Disaster Recovery Runbook =

= Disaster Recovery Runbook =

'''Last Updated:''' 2026-06-21
'''Print this and store in a safe place.'''

:'''Keep DR current:''' after adding/changing/removing any service, run
:<code>scripts/dr-drift-check.sh</code> and act on whatever it flags. See
:<code>DR_MAINTENANCE.md</code> for the full workflow. Don't skip this — gradual drift
:is how DR plans silently rot until the day you actually need them.

----

== Quick Reference ==

=== Stronghold (primary) ===

{| class="wikitable"
|-
! Item
! Value
|-
| '''Hetzner Storage Box'''
| <code>u540853.your-storagebox.de</code> port '''23'''
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups</code> (port 23 comes from SSH config, '''not''' the URL — see note)
|-
| '''SSH Key'''
| <code>~/.ssh/id_rsa</code> (on stronghold)
|-
| '''Restic Password'''
| In <code>/library/backup-stack/.env</code> (<code>HETZNER_PASSWORD</code>)
|-
| '''Backup Schedule'''
| Daily 03:00 CST
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly
|}

=== VPS (wilsoz.com) ===

{| class="wikitable"
|-
! Item
! Value
|-
| '''Hetzner Storage Box'''
| same box, separate repo
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups-vps</code> (port 23 comes from SSH config, '''not''' the URL — see note)
|-
| '''SSH Key'''
| <code>~/.ssh/id_ed25519</code> (on VPS)
|-
| '''Restic Password'''
| same <code>HETZNER_PASSWORD</code> — stored in <code>/etc/restic-vps.env</code> on VPS
|-
| '''Backup Schedule'''
| Daily 04:30 UTC (offset from stronghold)
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly
|-
| '''Covers'''
| Forgejo data + repos, Atuin (pg_dump), ntfy, nginx, Let's Encrypt certs, <code>/opt/</code> custom scripts
|}

:'''⚠️ SFTP repo URL gotcha (port vs. path):''' In restic's single-colon SFTP form
:(<code>sftp:user@host:path</code>) '''everything after the host colon is the repository path''' —
:there is no port in the URL. The storage box's port '''23''' must be set in SSH config
:(<code>~/.ssh/config</code> → <code>Port 23</code> for host <code>u540853.your-storagebox.de</code>), which both
:stronghold and the <code>restic-backup</code> container already have. Writing the port into the
:URL as <code>:23/backups</code> makes restic look for a repo at the literal path <code>23/backups</code> and
:fail with ''"repository does not exist"''. Always use the relative <code>:./backups</code> /
:<code>:./backups-vps</code> form. (This exact mistake caused false "VPS: no snapshots found"
:alerts from the n8n ''Backup Status Alert (SSH)'' workflow until 2026-06-29.)

----

== What's Backed Up ==

=== Service Data (files) ===

{| class="wikitable"
|-
! Path in Backup
! Source on Host
! Contents
|-
| <code>/data/authentik</code>
| <code>/library/authentik-app/</code>
| SSO/identity provider config
|-
| <code>/data/vaultwarden</code>
| <code>/library/vaultwarden-app/</code>
| Password vault (SQLite DB + attachments)
|-
| <code>/data/nextcloud</code>
| <code>/library/nextcloud-app/</code>
| File sync config + data
|-
| <code>/data/immich-config</code>
| <code>/library/immich-app/</code>
| Photo manager config
|-
| <code>/data/immich-photos</code>
| <code>/raid1/Consolidated/Picture/Immich-Photos/</code>
| All photos (149G)
|-
| <code>/data/jellyfin</code>
| <code>/var/lib/jellyfin/</code>
| Media server metadata (92G)
|-
| <code>/data/jellyfin-etc</code>
| <code>/etc/jellyfin/</code>
| Jellyfin system config
|-
| <code>/data/memos</code>
| <code>/library/memo-app/</code>
| Notes app
|-
| <code>/data/linkwarden</code>
| <code>/library/linkwarden-app/</code>
| Bookmark manager
|-
| <code>/data/open-webui</code>
| <code>/library/open-webui-app/</code>
| AI chat interface
|-
| <code>/data/matrix-chat</code>
| <code>/library/matrix-chat/</code>
| Synapse + bridge configs + signing key
|-
| <code>/data/stalwart</code>
| <code>/library/stalwart-app/data/</code>
| Mail server data
|-
| <code>/data/snappymail</code>
| <code>/library/snappymail-app/data/</code>
| Webmail data
|-
| <code>/data/mailcow</code>
| <code>/opt/mailcow-dockerized/</code>
| Legacy mail (to be archived)
|-
| <code>/data/home</code>
| <code>/home/mnw/</code>
| Home dir: dotfiles, scripts, Documents, source, SSH/GPG keys (selective excludes)
|-
| <code>/data/developer</code>
| <code>/home/mnw/Developer/</code>
| All dev projects (7.9G) — also covered by <code>/data/home</code>
|-
| <code>/data/beets</code>
| <code>/raid1/my-tunes/.beets/</code>
| Music library DB
|-
| <code>/data/openbao</code>
| <code>/library/openbao-app/</code>
| OpenBao secret store — '''incl. <code>.init-output.json</code> (root token + unseal key) and <code>.scripts-token</code> / <code>.claude-token</code>.''' Bootstrap-critical: must be restored and unsealed ''before'' services that depend on secrets.
|-
| <code>/data/ezbookkeeping</code>
| <code>/library/ezbookkeeping-app/data/</code>
| Personal finance tracker (SQLite + uploads)
|-
| <code>/data/omada</code>
| <code>/library/omada-app/</code>
| Omada SDN controller — switch/AP config & topology
|}

=== Database Dumps (in <code>/dumps/</code>) ===

{| class="wikitable"
|-
! File Pattern
! Source Container
! Database
|-
| <code>authentik-YYYYMMDD.sql.gz</code>
| <code>authentik-app_postgresql_1</code>
| Authentik identity DB
|-
| <code>nextcloud-YYYYMMDD.sql.gz</code>
| <code>nextcloud-db</code>
| Nextcloud DB
|-
| <code>immich-YYYYMMDD.sql.gz</code>
| <code>immich_postgres</code>
| Immich photo DB (~460MB compressed)
|-
| <code>linkwarden-YYYYMMDD.sql.gz</code>
| <code>linkwarden_postgres</code>
| Linkwarden bookmark DB
|-
| <code>matrix-YYYYMMDD.sql.gz</code>
| <code>matrix-postgres</code>
| Synapse + bridge DBs (~1MB compressed)
|}

=== NOT Backed Up (re-acquirable) ===

* Jellyfin media files (movies, TV)
* Ollama models (~20G, re-downloadable)
* Steam games
* VirtualBox VMs
* node_modules, .venv, .git/objects
* Wyoming voice models (<code>piper-data-app</code>, <code>whisper-data-app</code>) — redownloadable
* Prometheus metric history, Grafana dashboards data, uptime-kuma — re-buildable from compose

=== ⚠️ Known Gaps — Data Not Currently Backed Up ===

{| class="wikitable"
|-
! Severity
! What
! Where
! Notes
|-
| '''HIGH'''
| AudioBookshelf config + metadata
| <code>/usr/share/audiobookshelf/config/</code> (38M), <code>/usr/share/audiobookshelf/metadata/</code> (936M)
| systemd service, not Docker — not yet bind-mounted into restic container. Lose library DB, progress, and cover art on rebuild.
|}

Previously flagged gaps (all closed as of 2026-06-21): Nextcloud user files, Matrix media, Mealie, n8n named volumes, uptime-kuma, Grafana, AdGuard — all now covered by bind mounts in the restic container compose.

----

== Recovery Scenarios ==

=== Scenario 1: Single Service Failure ===

'''Example:''' Immich database corrupted, need to restore just Immich.

<syntaxhighlight lang="bash">
# 1. Stop the service
cd /library/immich-app && docker compose down

# 2. List available snapshots
docker exec restic-backup sh -c \
  'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

# 3. Restore service data from latest snapshot
docker exec restic-backup sh -c \
  'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
   restic restore latest --target /tmp/restore --include /data/immich-config'

# 4. Copy restored data to host
docker cp restic-backup:/tmp/restore/data/immich-config /tmp/immich-restore

# 5. Replace the broken data
mv /library/immich-app /library/immich-app.broken
mv /tmp/immich-restore /library/immich-app

# 6. Restore the database dump
docker cp restic-backup:/tmp/restore/dumps/immich-YYYYMMDD.sql.gz /tmp/
gunzip /tmp/immich-YYYYMMDD.sql.gz
# Start just the database container
docker compose up -d immich_postgres
# Wait for it to be ready, then restore
docker exec -i immich_postgres psql -U postgres immich < /tmp/immich-YYYYMMDD.sql

# 7. Start the full service
docker compose up -d

# 8. Clean up
docker exec restic-backup rm -rf /tmp/restore
</syntaxhighlight>

=== Scenario 2: Full Server Rebuild ===

'''Starting from:''' Fresh Linux install on new hardware.

==== Bootstrap Order (DO NOT SHUFFLE) ====

Many services depend on OpenBao (secret store) and Authentik (SSO). Bring them
up in this order so each layer has its dependencies in place:

<syntaxhighlight lang="text">
1. Host: Docker, restic, Tailscale, SSH key
2. Restore everything from Hetzner via restic (raw files)
3. OpenBao  ← unseal it first; it holds tokens other services need
4. Authentik (DB dump → start) ← SSO for everything
5. Vaultwarden ← human-facing creds; needed for me-the-operator
6. Restore Claude Code access (bw CLI, MCP wrappers) ← so future-you can drive recovery
7. Postgres-backed services (Immich, Nextcloud, Matrix, Linkwarden): DB dump → start
8. Stalwart + SnappyMail (mail)
9. Everything else (n8n, Grafana, Mealie, etc.)
</syntaxhighlight>

==== Step 1: Install Prerequisites ====

<syntaxhighlight lang="bash">
# Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker mnw

# Install restic
sudo apt install restic

# Set Docker MTU for Tailscale compatibility
# Add to /etc/docker/daemon.json:
# {"mtu": 1280}
sudo systemctl restart docker
</syntaxhighlight>

==== Step 2: Restore SSH Keys ====

You need the SSH key to access Hetzner. If you have a copy:

<syntaxhighlight lang="bash">
mkdir -p ~/.ssh
# Copy id_rsa from your backup/safe location
chmod 600 ~/.ssh/id_rsa
</syntaxhighlight>

If you don't have the key, generate a new one and add it to Hetzner:
<syntaxhighlight lang="bash">
ssh-keygen -t rsa -b 4096
# Add public key via Hetzner Robot panel → Storage Box → SSH Keys
</syntaxhighlight>

==== Step 3: Bootstrap Restic ====

<syntaxhighlight lang="bash">
# Create the backup stack directory
sudo mkdir -p /library/backup-stack
cd /library/backup-stack

# Create .env with Hetzner credentials:
cat > .env << 'EOF'
HETZNER_REPO=sftp:u540853@u540853.your-storagebox.de:23/backups
HETZNER_PASSWORD=<your-restic-password>
NTFY_TOKEN=tk_0fhigfiglbjw9r8sk9jv4hpr7q67f
EOF

# Test connection
export $(grep -v '^#' .env | xargs)
restic -r "$HETZNER_REPO" snapshots
</syntaxhighlight>

==== Step 4: Restore Everything ====

<syntaxhighlight lang="bash">
# Create target directories
sudo mkdir -p /library /raid1/Consolidated/Picture/Immich-Photos

# Restore all service data
restic -r "$HETZNER_REPO" restore latest --target /

# This puts files at their original paths:
#   /data/authentik → needs moving to /library/authentik-app/
#   /data/developer → needs moving to /home/mnw/Developer/
#   etc.

# Move restored data to correct locations
sudo mv /data/authentik /library/authentik-app
sudo mv /data/vaultwarden /library/vaultwarden-app
sudo mv /data/nextcloud /library/nextcloud-app
sudo mv /data/immich-config /library/immich-app
sudo mv /data/immich-photos/* /raid1/Consolidated/Picture/Immich-Photos/
sudo mv /data/jellyfin /var/lib/jellyfin
sudo mv /data/jellyfin-etc /etc/jellyfin
sudo mv /data/memos /library/memo-app
sudo mv /data/linkwarden /library/linkwarden-app
sudo mv /data/open-webui /library/open-webui-app
sudo mv /data/matrix-chat /library/matrix-chat
sudo mv /data/stalwart /library/stalwart-app/data
sudo mv /data/snappymail /library/snappymail-app/data
sudo mv /data/developer /home/mnw/Developer
sudo mv /data/beets /raid1/my-tunes/.beets
sudo chown -R mnw:mnw /home/mnw/Developer
</syntaxhighlight>

==== Step 4b: Bootstrap OpenBao (Secret Store) — REQUIRED before DBs ====

OpenBao holds tokens and secrets that downstream services + scripts read at startup.

<syntaxhighlight lang="bash">
# 1. Move restored data into place
sudo mv /data/openbao /library/openbao-app
sudo chown -R 1000:1000 /library/openbao-app  # OpenBao runs as uid 1000

# 2. Start the container
cd /library/openbao-app && docker compose up -d

# 3. Unseal — uses the unseal key from .init-output.json (restored from backup)
sudo /usr/local/bin/openbao-unseal.sh
# If that script isn't restored yet, manually:
#   ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
#   UNSEAL_KEY=$(jq -r .unseal_keys_b64[0] /library/openbao-app/.init-output.json)
#   docker exec openbao bao operator unseal "$UNSEAL_KEY"

# 4. Install the auto-unseal systemd unit so it survives reboots
sudo install -m 0755 scripts/openbao/openbao-unseal.sh /usr/local/bin/
sudo install -m 0644 scripts/openbao/openbao-unseal.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now openbao-unseal.service

# 5. Verify by reading a known secret
TOKEN=$(cat /library/openbao-app/.scripts-token)
curl -sf http://127.0.0.1:8200/v1/secret/data/ntfy -H "X-Vault-Token: $TOKEN" | jq .data.data
</syntaxhighlight>

'''If <code>.init-output.json</code> is lost:''' OpenBao is unrecoverable without it. Print
it now (or keep it in a sealed envelope) — it contains the root token AND the
unseal key. Without these, you cannot decrypt anything in the Raft store.

==== Step 5: Restore Databases ====

<syntaxhighlight lang="bash">
# For each service, start just the DB container, restore the dump, then start the rest.
# Example for Authentik:
cd /library/authentik-app
docker compose up -d postgresql
sleep 10
gunzip -c /dumps/authentik-YYYYMMDD.sql.gz | docker exec -i authentik-app_postgresql_1 psql -U authentik authentik
docker compose up -d

# Repeat for: nextcloud, immich, linkwarden, matrix
</syntaxhighlight>

==== Step 6: Restore System Services ====

<syntaxhighlight lang="bash">
# Jellyfin (systemd, not Docker)
sudo apt install jellyfin
# Data already restored to /var/lib/jellyfin and /etc/jellyfin

# AudioBookshelf
# Install from package, data at /usr/share/audiobookshelf
</syntaxhighlight>

==== Step 6b: Restore Claude Code Access (Vaultwarden MCP) ====

Once Vaultwarden is up (Step 5 + nginx routed), restore the bw CLI and MCP
wrappers so a Claude session can help you finish recovery from this point on.

<syntaxhighlight lang="bash">
# 1. Install bw CLI (snap is the official Bitwarden source)
sudo snap install bw

# 2. Install Bitwarden MCP server (npm, globally)
npm install -g @bitwarden/mcp-server

# 3. Restore wrappers — these are in /data/home/.config/claude/ from backup
mkdir -p ~/.config/claude
sudo cp /data/home/.config/claude/{vaultwarden.env,bw-mcp-start.sh,bw-get.py} ~/.config/claude/
chmod 600 ~/.config/claude/vaultwarden.env
chmod 700 ~/.config/claude/bw-mcp-start.sh ~/.config/claude/bw-get.py

# 4. Verify bw can talk to Vaultwarden (not the public Bitwarden cloud!)
bw config server  # MUST print https://vault.wilsoz.com — if not, run: bw config server https://vault.wilsoz.com
bw login --apikey  # uses BW_CLIENTID + BW_CLIENTSECRET from env

# 5. Wire MCPs into ~/.claude.json (vault + bitwarden entries)
# See services/VAULTWARDEN.md for the exact JSON structure if it needs to be rebuilt.
</syntaxhighlight>

See <code>services/VAULTWARDEN.md</code> for full setup including the <code>mcp-server-bitwarden</code>
symlink bug workaround.

==== Step 7: Re-establish Networking ====

<syntaxhighlight lang="bash">
# Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# VPS nginx configs point to Tailscale IPs — update if IP changes
# VPS SSH: ssh -p 1981 mnw@wilsoz.com
</syntaxhighlight>

=== Scenario 3: Restore Specific Files ===

<syntaxhighlight lang="bash">
# Restore a single file from a specific snapshot
docker exec restic-backup sh -c \
  'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
   restic restore SNAPSHOT_ID --target /tmp/restore --include /data/developer/SystemResiliency/STATUS.md'

docker cp restic-backup:/tmp/restore/data/developer/SystemResiliency/STATUS.md /tmp/
</syntaxhighlight>

=== Scenario 4: Matrix Chat Recovery ===

Matrix requires special care — the signing key is critical.

<syntaxhighlight lang="bash">
# 1. Restore Matrix data
restic restore latest --target /tmp/restore --include /data/matrix-chat

# 2. Critical files:
#    - synapse/signing.key — IRREPLACEABLE, needed for federation
#    - synapse/homeserver.yaml — server config
#    - bridges/*/config.yaml — bridge configs
#    - bridges/*/registration.yaml — bridge registrations

# 3. Restore database
gunzip -c /dumps/matrix-YYYYMMDD.sql.gz | docker exec -i matrix-postgres psql -U synapse

# 4. Restart stack
cd /library/matrix-chat && docker compose up -d
</syntaxhighlight>

----

== Verification Commands ==

<syntaxhighlight lang="bash">
# List all snapshots
docker exec restic-backup sh -c \
  'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

# Check repository integrity
docker exec restic-backup sh -c \
  'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic check'

# Check repo size on Hetzner
ssh -p 23 u540853@u540853.your-storagebox.de "du -sh backups/"

# List files in a snapshot
docker exec restic-backup sh -c \
  'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
   restic ls latest /data/matrix-chat'

# Verify a DB dump
gunzip -t /library/backup-stack/dumps/immich-YYYYMMDD.sql.gz && echo "OK" || echo "CORRUPT"

# Manual backup trigger
sudo /library/backup-stack/scripts/host-dump-databases.sh
docker exec restic-backup /scripts/backup-to-hetzner.sh
</syntaxhighlight>

----

== Automated Monitoring ==

{| class="wikitable"
|-
! Check
! Frequency
! Alert Channel
|-
| Backup age (n8n)
| Every 6h
| ntfy (nc700x-alerts) + Matrix
|-
| Integrity check (restic)
| Daily 06:00
| ntfy if FAIL
|-
| Restore test (restic)
| Weekly Sunday 07:00
| ntfy if FAIL
|-
| Disk space (n8n)
| Every 4h
| ntfy (nc700x-alerts) + Matrix
|}

----

== Key Credentials ==

'''Keep a copy of these offline:'''

{| class="wikitable"
|-
! Credential
! Location
|-
| Hetzner SSH key
| <code>~/.ssh/id_rsa</code> on stronghold
|-
| Restic password
| <code>/library/backup-stack/.env</code>
|-
| Hetzner account
| Robot panel, u540853
|-
| ntfy token
| <code>tk_0fhigfiglbjw9r8sk9jv4hpr7q67f</code>
|-
| Authentik admin
| <code>https://key.wilsoz.com/if/admin/</code>
|-
| VPS SSH
| <code>ssh -p 1981 mnw@wilsoz.com</code>
|-
| Synapse shared secret
| In <code>homeserver.yaml</code> line 50
|-
| Matrix signing key
| <code>/library/matrix-chat/synapse/signing.key</code> — '''IRREPLACEABLE'''
|}

----

== Restore Test Log ==

=== 2026-02-23 — False Failure Alerts + Upload Anomaly Detection ===

'''Problem 1 — False failure alert after meta-array removal:'''
<code>validate-backups.sh</code> still contained a <code>restic check</code> block against <code>$METAARRAY_REPO</code> / <code>$METAARRAY_PASSWORD</code>. Both vars were removed from <code>.env</code> when meta-array was dropped on 2026-02-22. Empty repo var → restic failed → <code>ERRORS++</code> → ntfy <code>nc700x-alerts</code> fired every day.

'''Fix:''' Removed the dead meta-array block from <code>validate-backups.sh</code> (lines 44–52).

'''Problem 2 — Large upload on first run after container recreation:'''
Restic container was recreated during meta-array removal, clearing the local cache. Without cache, restic reported "no parent snapshot found" and re-scanned all 441,884 files (all showed as "new"). Chunk dedup still worked correctly — only 11.5 GiB of 314 GiB was actually uploaded. Confirmed by upload anomaly report: Stalwart ~8k new email files (ongoing Fastmail getmail import) + <code>/data/home</code> (all new to the repo without a parent reference). Normal next run.

'''Enhancement — Upload anomaly detection:'''
Added <code>detect_upload_anomaly()</code> to <code>run-backup.sh</code>:
* Tracks upload size per run in <code>/dumps/upload-history.txt</code> (rolling 30 entries)
* If upload >2.5× rolling average AND >500 MiB, runs <code>restic diff <prev> <cur></code> and summarises changes by service path
* POSTs JSON payload to n8n webhook (<code>whO5LBZHQHCXXG9I</code>) which formats and sends to Matrix
* ntfy not used — this is informational, not an emergency

----

=== 2026-02-22 — Recurring SSH Timeout Incident ===

'''Problem:''' Backups failing with <code>subprocess ssh: Timeout</code> / <code>Fatal: unable to save snapshot: context canceled</code>. Hetzner SFTP server drops idle connections (~5 min) during restic's local scan phase (22+ min). Compounded by 96 accumulated index files (documented threshold: 49) because prune was also failing.

'''Secondary cause removed:''' meta-array (ma.sdf.org) secondary backup target was unreliable. Since <code>run-backup.sh</code> used <code>set -e</code>, any metaarray failure marked the entire job as failed even when Hetzner backup succeeded. Removed entirely.

'''Fixes applied:'''
# <code>restic repair index</code> — consolidated 96 → 3 index files
# <code>restic forget --prune</code> — cleaned up retention
# Added <code>-o sftp.args="-oServerAliveInterval=10 -oServerAliveCountMax=36"</code> to <code>backup-to-hetzner.sh</code> — 10s keepalives, 6-min tolerance
# Added <code>--retry-lock 10m</code> to <code>backup-to-hetzner.sh</code>
# Removed <code>backup-to-metaarray.sh</code> and all meta-array config
# Added <code>/home/mnw</code> → <code>/data/home</code> to backup scope (dotfiles, Documents, source, SSH keys)

'''If SSH timeouts recur:''' It means index files are accumulating again (prune is failing silently). Run:
<syntaxhighlight lang="bash">
docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic repair index'
docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune'
</syntaxhighlight>

----

=== 2026-02-16 — First Manual Test ===

'''Test:''' Restore files from latest snapshot (ae89d11d), compare against live originals.

{| class="wikitable"
|-
! File
! Restored
! Compared
! Result
|-
| <code>matrix-chat/synapse/homeserver.yaml</code>
| 3,022 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/synapse/signing.key</code>
| 59 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/docker-compose.yml</code>
| 3,232 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/bridges/whatsapp/config.yaml</code>
| 33,990 bytes
| vs live
| IDENTICAL
|-
| <code>developer/SystemResiliency/STATUS.md</code>
| Restored
| pre-edit version
| Expected diff (edited today)
|-
| DB dump: <code>immich-20260213.sql.gz</code>
| 463 MB
| gunzip -t
| VALID
|-
| DB dump: <code>matrix-20260215.sql.gz</code>
| 1 MB
| gunzip -t
| VALID
|-
| DB dump: <code>authentik-20260213.sql.gz</code>
| 16 MB
| gunzip -t
| VALID
|}

'''Issues Found:'''
* DB dumps from Feb 14-15 were 20 bytes (empty) — container's <code>run-backup.sh</code> was calling <code>dump-databases.sh</code> inside the container where <code>docker</code> CLI doesn't exist, overwriting the good host-side dumps.
* '''Fixed:''' Deployed updated <code>run-backup.sh</code> that verifies dumps instead of re-dumping. Fresh dumps created manually and confirmed valid.

'''Verdict:''' PASS — restore process works. Files are intact and match live originals.