Jump to content

Services/BackupSystem

From Stronghold Wiki
Revision as of 18:16, 5 July 2026 by Wikibot (talk | contribs) (Migrated from services/BACKUP_SYSTEM.md)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

As of: 2026-07-05 — migrated from `services/BACKUP_SYSTEM.md`

Backup System — Restic + Timeshift

Complete setup & operations guide for Stronghold's dual-tier backup system: Timeshift (local snapshots) + Restic (remote encrypted backups).

Criticality: CRITICAL — The foundation. All disaster recovery depends on backups. Without them, data loss is permanent. This system enables recovery of every service on Stronghold.

Overview

Stronghold uses a two-tier backup strategy for redundancy and speed:

  1. Timeshift (local, fast)
    • RSYNC-based filesystem snapshots stored on /raid1/timeshift/
    • 8 snapshots retained (mix of daily, weekly, monthly)

Purpose:** Fast recovery of individual files or full system (within days) Speed:** ~5-30 minutes to restore (depends on size) Limitation:** If /raid1 is damaged, all local snapshots are lost

  1. Restic (remote, encrypted)
    • Incremental encrypted backups sent to Hetzner Storage Box
    • Daily backups at 03:00 CST, retention: 7 daily + 4 weekly + 6 monthly

Purpose:** Off-site redundancy; survives hardware failure Coverage:** All service data, configs, media, databases Speed:** ~30 minutes to backup, 1-2 hours per terabyte for restoration Limitation:** Requires Hetzner SSH access and Restic repo password

Recovery decision tree:

  • Drive works, need recent files? → Restore from Timeshift (fast)
  • Drive works, lost some service? → Restore from Timeshift or Restic (both available)
  • Drive failed, building new hardware? → Restore from Restic (only source of truth)

Service Information

Timeshift

Property Value
Storage location /raid1/timeshift/
Snapshot mode RSYNC (not btrfs)
Device /dev/md1 (RAID-1 array)
Current snapshots 8 (mix of daily D, weekly W, monthly M, backfill B)
Retention policy Auto: Last 5 daily, 1 weekly, 1 monthly; manual cleanup for older
Snapshot schedule Daily at 00:00 (D), weekly at 10:00 (W), manual on demand
Excluded from snapshots /var/lib/docker/volumes ⚠️, /proc, /sys, /tmp, /var/tmp

Restic

Property Value
Container name restic-backup
Image restic/restic:latest
Compose file /home/mnw/Developer/SystemResiliency/backup-stack/docker-compose.yml
Network mode host (direct network access)
Backup schedule 03:00 CST daily (systemd timer backup.timer)
Hetzner server u540853.your-storagebox.de:23 (SSH port 23, not 22)
Repo path sftp:[email protected]:./backups
Encryption AES-256, password-based (stored in .env as HETZNER_PASSWORD)
Deduplication Enabled (saves 20-30% space)
Compression zstd (default, good balance)
Retention 7 daily, 4 weekly, 6 monthly snapshots
Database dumps PostgreSQL dumps staged to /library/backup-stack/dumps/ before backup
System-state dump /etc subset + user crontabs + user/system systemd units + dpkg selections tar'd to /library/backup-stack/dumps/system-state-YYYYMMDD.tar.gz before backup (covers schedulers/configs that live outside /data/* and would otherwise be lost if /etc or a user home got wiped)

Initial Setup from Scratch

Timeshift (Local Snapshots)

Timeshift is likely already installed and running on your system. To verify or install:

<syntaxhighlight lang="bash">

  1. Check if Timeshift is installed

which timeshift

  1. If not, install (Ubuntu/Debian)

sudo apt update sudo apt install timeshift

  1. Check current snapshots

sudo timeshift --list

  1. Create first snapshot manually (if none exist)

sudo timeshift --create --comments "Initial snapshot before disaster recovery setup" </syntaxhighlight>

Configuration (in Timeshift GUI or config):

  • Storage location: /raid1/timeshift/
  • Device: /dev/md1 (RAID-1 array)
  • Snapshot mode: RSYNC
  • Auto-backup: Enabled, daily at 00:00
  • Retention: At least 5 daily snapshots kept

Restic (Remote Encrypted Backups)

1. Set Up Hetzner Storage Box

If not already done:

  1. Create Hetzner account at https://www.hetzner.com/storage/storage-box
  2. Order Storage Box (recommended: smallest size starts at €3.81/month)
  3. In Hetzner Console:
    • Go to Storage Box settings
    • Enable "External reachability" (required for SSH/SFTP)
    • Upload your SSH public key (~/.ssh/id_rsa.pub)
  1. Note your credentials:
    • Server: u540853.your-storagebox.de (your number varies)

Port: 23** (not 22!)

    • Username: u540853 (same as your account number)

2. Set Up SSH Access

<syntaxhighlight lang="bash">

  1. Create SSH key if you don't have one

ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa

  1. Test SSH connection to Hetzner (port 23!)

ssh -p 23 [email protected] "echo 'SSH working!'"

  1. Should print: SSH working!

</syntaxhighlight>

3. Copy Backup Stack to /library

<syntaxhighlight lang="bash">

  1. From your project directory

cp -r /home/mnw/Developer/SystemResiliency/backup-stack /library/

cd /library/backup-stack

  1. Create .env from example

cp .env.example .env

  1. Edit with your Hetzner credentials and restic password

nano .env </syntaxhighlight>

.env file contents: <syntaxhighlight lang="bash">

  1. Hetzner Storage Box

HETZNER_REPO="sftp:[email protected]:./backups" HETZNER_PASSWORD="<generate-strong-restic-password-32-chars>"

  1. Alerts

NTFY_TOKEN="<from Vaultwarden if you have one>" NTFY_TOPIC="backup-alerts" </syntaxhighlight>

4. Initialize Restic Repository

<syntaxhighlight lang="bash"> cd /library/backup-stack docker compose up -d

  1. Init Hetzner repository (will encrypt with password)

docker exec -e RESTIC_PASSWORD='<same-as-HETZNER_PASSWORD>' restic-backup \ 

 restic init --repo sftp:[email protected]:./backups
  1. Output should show: "created restic repository" with repository ID

</syntaxhighlight>

5. Install Systemd Timers

<syntaxhighlight lang="bash">

  1. Copy timer definitions

sudo cp /library/backup-stack/systemd/*.service /etc/systemd/system/ sudo cp /library/backup-stack/systemd/*.timer /etc/systemd/system/

  1. Reload systemd

sudo systemctl daemon-reload

  1. Enable and start backup timer

sudo systemctl enable --now backup.timer sudo systemctl enable --now backup-validate-daily.timer sudo systemctl enable --now backup-validate-weekly.timer

  1. Check status

systemctl list-timers | grep backup </syntaxhighlight>

6. Test the Backup

<syntaxhighlight lang="bash">

  1. Run a manual backup to verify everything works

docker exec restic-backup /scripts/run-backup.sh

  1. Watch the logs

docker logs -f restic-backup

  1. Should show:
  2. - Database dumps created
  3. - Restic scanning files
  4. - Upload to Hetzner
  5. - Success message

</syntaxhighlight>

Disaster Mode A: Local Recovery (Fast)

Scenario: Lost a single file or small data

Example: Accidentally deleted a recipe from Mealie 2 days ago.

Recovery Steps:

<syntaxhighlight lang="bash">

  1. 1. List available snapshots

sudo timeshift --list

  1. 2. Mount the snapshot you want to restore from
  2. (Example: restore from 2026-05-11 daily snapshot)

sudo timeshift --mount /run/timeshift/latest

  1. 3. File is now accessible at:
  2. /run/timeshift/latest/library/mealie-app/mealie-data/recipes/
  3. Copy the file back to current system

cp /run/timeshift/latest/library/mealie-app/mealie-data/recipes/recipe.json \ 

  /library/mealie-app/mealie-data/recipes/
  1. 4. Unmount

sudo timeshift --umount </syntaxhighlight>

Scenario: Need to restore entire service from snapshot

Example: Immich database corrupted, want to restore from snapshot.

<syntaxhighlight lang="bash">

  1. 1. Stop the service

cd /library/immich-app docker compose down

  1. 2. Mount snapshot

sudo timeshift --mount

  1. 3. Restore the entire service directory

sudo rm -rf /library/immich-app sudo cp -r /run/timeshift/latest/library/immich-app /library/

  1. 4. Fix permissions (Docker runs as specific user)

sudo chown -R mnw:mnw /library/immich-app

  1. 5. Restart service

docker compose up -d

  1. 6. Unmount snapshot

sudo timeshift --umount </syntaxhighlight>

Scenario: Full system restoration (if OS is still usable)

Example: Stronghold is working but some system files are corrupted.

<syntaxhighlight lang="bash">

  1. List snapshots from a few days ago

sudo timeshift --list

  1. Restore from a known-good snapshot
  2. This restores EVERYTHING except /home, /var/lib/docker, /opt, /etc/hosts, etc.

sudo timeshift --restore --snapshot "2026-05-11_10-00-01" --skip-grub

  1. Reboot after restoration

sudo reboot </syntaxhighlight>

Disaster Mode B: Full Recovery (After Drive Failure)

Scenario: Hard drive failed, new hardware deployed

Prerequisites:

  • New hardware with OS installed (Ubuntu/Debian)
  • Docker and docker-compose installed
  • SSH key for Hetzner auth
  • Restic password (from your secure notes)

Recovery Steps:

<syntaxhighlight lang="bash">

  1. 1. Create directories

mkdir -p /raid1 mkdir -p /library mkdir -p /home/mnw/Developer/SystemResiliency

  1. 2. Copy backup-stack to new system

scp -r username@old-system:/library/backup-stack /library/

  1. 3. Update .env with correct passwords/credentials

nano /library/backup-stack/.env

  1. 4. Start the Restic container

cd /library/backup-stack docker compose up -d

  1. 5. List available backups in Hetzner

docker exec restic-backup restic snapshots --repo sftp:[email protected]:./backups

  1. 6. Restore the full /library directory from latest snapshot

docker exec restic-backup restic restore latest \ 

 --target /tmp/restore \
 --repo sftp:[email protected]:./backups \
 --include "/data/*"
  1. 7. Copy restored data to /library

sudo cp -r /tmp/restore/data/* /library/

  1. 8. Restore /home/mnw

docker exec restic-backup restic restore latest \ 

 --target /tmp/restore \
 --repo sftp:[email protected]:./backups \
 --include "/data/home"

sudo cp -r /tmp/restore/data/home/* /home/

  1. 9. Fix permissions

sudo chown -R mnw:mnw /library sudo chown -R mnw:mnw /home/mnw

  1. 10. Restore PostgreSQL databases
  2. Each service (Authentik, Nextcloud, etc.) has a dump at /library/backup-stack/dumps/
  3. Use the restore procedure from each service's documentation
  1. Example: Restore Authentik PostgreSQL

docker exec authentik-app_postgresql_1 pg_restore \ 

 /library/backup-stack/dumps/authentik-YYYYMMDD.sql < < (gunzip -c /path/to/dump)
  1. 11. Start all services

for dir in /library/*-app/; do 

 cd "$dir"
 docker compose up -d

done

  1. 12. Verify critical services are running

docker ps | grep "server\|worker\|postgres" </syntaxhighlight>

Scenario: Restore specific service from Restic

Example: Only Immich needs recovery, not entire system.

<syntaxhighlight lang="bash"> cd /library/backup-stack docker compose up -d

  1. Restore just Immich config and database dump

docker exec restic-backup restic restore latest \ 

 --target /tmp/restore \
 --repo sftp:[email protected]:./backups \
 --include "/data/immich*" \
 --include "/dumps/immich*"
  1. Move to correct location

mv /tmp/restore/data/immich-config /library/immich-app mv /tmp/restore/data/immich-photos /raid1/Consolidated/Picture/Immich-Photos

  1. Restore the database

cd /library/immich-app docker compose up -d immich_postgres sleep 10 docker exec -i immich_postgres psql -U immich < <(gunzip -c /tmp/restore/dumps/immich-*.sql.gz)

  1. Start full Immich

docker compose up -d </syntaxhighlight>

Data Locations & Backup Coverage

What's Backed Up (via Restic)

<syntaxhighlight lang="text"> Backed up to Hetzner: ├── /data/authentik/ — Authentik OAuth provider config ├── /data/vaultwarden/ — Password vault (SQLite + attachments) ├── /data/nextcloud/ — File sync config ├── /data/immich-config/ — Photo manager config ├── /data/immich-photos/ — All photos (149GB) ├── /data/jellyfin/ — /var/lib/jellyfin (plugins, root, subs; live DBs excluded) ├── /data/jellyfin-etc/ — /etc/jellyfin (system config) ├── /data/jellyfin-server/ — /library/jellyfin-server-app (active metadata 28G — see services/JELLYFIN.md) ├── /data/memos/ — Notes app ├── /data/linkwarden/ — Bookmark manager ├── /data/open-webui/ — AI chat interface ├── /data/matrix-chat/ — Synapse + bridge configs ├── /data/stalwart/ — Mail server data ├── /data/snappymail/ — Webmail data ├── /data/mailcow/ — Legacy mail (to be archived) ├── /data/developer/ — /home/mnw/Developer (7.9G) ├── /data/home/ — /home/mnw (selective: excludes .git/objects, node_modules, .venv) ├── /data/beets/ — Music library DB ├── /dumps/ — PostgreSQL database dumps (daily) │ ├── authentik-YYYYMMDD.sql.gz │ ├── nextcloud-YYYYMMDD.sql.gz │ ├── immich-YYYYMMDD.sql.gz │ ├── linkwarden-YYYYMMDD.sql.gz │ └── matrix-YYYYMMDD.sql.gz </syntaxhighlight>

What's NOT Backed Up (Re-acquirable)

  • /var/lib/docker/volumes ⚠️ (Timeshift excludes this; Restic needs to be updated to include)
  • Jellyfin/Kaleidescape media files (movies, TV)
  • Ollama models (re-downloadable)
  • Steam games
  • VirtualBox VMs
  • node_modules, .venv, .git/objects, build artifacts

Timeshift Snapshots

<syntaxhighlight lang="text"> Stored on /raid1/timeshift/ ├── 2025-05-03_17-15-00 (Backfill from old system) ├── 2026-05-03_11-52-11 (Backfill) ├── 2026-05-05_20-41-31 (Backfill) ├── 2026-05-05_21-36-57 (Backfill) ├── 2026-05-11_10-00-02 (Monthly snapshot) ├── 2026-05-12_00-00-01 (Daily snapshot) ├── 2026-05-12_10-00-01 (Weekly snapshot) └── 2026-05-13_00-00-01 (Daily snapshot) </syntaxhighlight>

Each snapshot contains a full filesystem image (~100-300GB depending on changes).

Critical Gotchas

1. ⚠️ Docker Volumes NOT in Timeshift

Problem: Timeshift is configured to exclude /var/lib/docker/volumes, which contains:

  • PostgreSQL databases for Authentik, Nextcloud, Immich, Linkwarden, Matrix
  • Docker-managed volumes

If you restore from Timeshift, you lose all database data. PostgreSQL will be empty.

Solution: Always use Restic for full recovery (it includes the databases via dumps), or manually restore PostgreSQL dumps.

Status: Known issue, needs fix in Timeshift config or backup-stack dumps.

2. Hetzner Port is 23, NOT 22

Many SSH configs default to port 22. Hetzner requires port 23.

The ssh-config file in backup-stack handles this automatically.

3. Hetzner Relative Paths Required

Hetzner SFTP doesn't support absolute paths for new directories.

  • Wrong: sftp:u540853@...:/backups (fails with SSH_FX_FAILURE)
  • Right: sftp:u540853@....:./backups (relative to home directory)

4. Restic Password != Hetzner Password

Two different passwords:

  • Hetzner account password: Logs into Hetzner web console
  • Restic password: Encrypts your backup repository

Save both securely. If you lose the Restic password, your backups are unrecoverable.

5. SSH Key Permissions in Docker

When mounting SSH keys into the Restic container, SSH is strict about ownership (must be 600 or 400).

The docker-compose entrypoint handles this: <syntaxhighlight lang="yaml"> entrypoint: ["/bin/sh", "-c", "mkdir -p /root/.ssh && cp /tmp/ssh/ /root/.ssh/ && chmod 600 /root/.ssh/ && sleep infinity"] </syntaxhighlight>

Don't modify without understanding SSH permission requirements.

6. PostgreSQL Dumps Staged Before Backup

Database dumps are created at backup time by run-backup.sh: <syntaxhighlight lang="bash"> docker exec <container> pg_dump -U <user> <db> | gzip > /library/backup-stack/dumps/<service>.sql.gz </syntaxhighlight>

If a database is corrupted, the dump will fail, and that database won't be backed up. Check backup logs: <syntaxhighlight lang="bash"> cat /library/backup-stack/dumps/backup-*.log | grep -i "error\|dump" </syntaxhighlight>

7. Backup Window (03:00 CST) Must Not Overlap with Other Heavy Operations

Backup runs daily at 03:00 CST. If you have other cron jobs or maintenance windows:

  • Stagger them by at least 30 minutes
  • Monitor systemd timers: systemctl list-timers

Concurrent heavy I/O can cause slowdowns or backup timeouts.

8. Restic Prune Deletes Old Snapshots Automatically

Restic prune runs after backup to clean up old snapshots per retention policy:

  • Keep 7 daily, 4 weekly, 6 monthly

If you manually delete recent snapshots (for testing), they may be pruned. Don't delete recent snapshots without understanding the implications.

Verification & Testing

Daily Check: Backup Health

<syntaxhighlight lang="bash">

  1. Check if backup completed successfully

docker logs restic-backup | tail -20 | grep -i "success\|error"

  1. Check ntfy for backup alerts

curl https://ntfy.wilsoz.com/backup-alerts

  1. List recent snapshots

docker exec restic-backup restic snapshots --limit 5 </syntaxhighlight>

Weekly Test: Restore a File

<syntaxhighlight lang="bash">

  1. Restore a small non-critical file to /tmp

docker exec restic-backup restic restore latest \ 

 --target /tmp/test-restore \
 --include "/data/memos/*"
  1. Verify files exist

ls -la /tmp/test-restore/data/memos/

  1. Clean up

rm -rf /tmp/test-restore </syntaxhighlight>

Monthly Test: Restore a Service Database

<syntaxhighlight lang="bash">

  1. Restore Linkwarden dump (small DB, ~50MB)

docker exec restic-backup restic restore latest \ 

 --target /tmp/test-restore \
 --include "/dumps/linkwarden*"
  1. Verify dump is valid

gunzip -t /tmp/test-restore/dumps/linkwarden-*.sql.gz

  1. Should report: OK (no errors)
  1. Clean up

rm -rf /tmp/test-restore </syntaxhighlight>

Quarterly: Full System Restore Dry-Run

Once per quarter, do a mock recovery to verify procedures:

<syntaxhighlight lang="bash">

  1. Don't actually restore (too time-consuming), but verify:
  2. 1. Can list snapshots?

docker exec restic-backup restic snapshots --limit 1

  1. 2. Can stat the repository (verify password works)?

docker exec restic-backup restic stats

  1. 3. Can list files in a snapshot?

docker exec restic-backup restic ls latest | head -20

  1. If all three pass, recovery is ready.

</syntaxhighlight>

Troubleshooting

Backup Failed

Check logs: <syntaxhighlight lang="bash">

  1. Container logs

docker logs restic-backup | tail -50

  1. Systemd log

sudo journalctl -u backup.service -n 50

  1. Backup script log

ls -lht /library/backup-stack/dumps/backup-*.log tail -100 /library/backup-stack/dumps/backup-*.log </syntaxhighlight>

Common causes:

  • SSH key permission error → Run entrypoint again: docker compose up -d
  • Hetzner unreachable → Test SSH manually: ssh -p 23 [email protected]
  • Database dump failed → Check Docker container: docker ps | grep postgres
  • Out of disk space → Check /library and /raid1: df -h /library /raid1

Restore Is Slow

Restic restores are network-bound from Hetzner. Typical speeds:

  • Small file (< 100MB): 30 seconds
  • Medium service (1-5GB): 5-15 minutes
  • Large dataset (100+ GB): 1-3 hours

If slower than expected:

  • Check network: ping u540853.your-storagebox.de (port 23 doesn't reply to ping)
  • Try SSH: ssh -p 23 [email protected] echo ok
  • Check Restic cache: du -sh /library/backup-stack/cache

Can't Connect to Hetzner

<syntaxhighlight lang="bash">

  1. Test SSH connectivity

ssh -p 23 [email protected] echo "Connected"

  1. If failed:
  2. 1. Check SSH key exists: ls -la ~/.ssh/id_rsa
  3. 2. Check Hetzner console: "External reachability" enabled?
  4. 3. Check your public key uploaded to Hetzner
  5. 4. Try with password instead: ssh -p 23 -o PubkeyAuthentication=no [email protected]

</syntaxhighlight>

Restic Unlock Stuck

If Restic process was killed, lock file may remain:

<syntaxhighlight lang="bash">

  1. Remove lock

docker exec restic-backup restic unlock --repo sftp:u540853@... </syntaxhighlight>

Maintenance

Monthly: Review Snapshot Count

<syntaxhighlight lang="bash"> sudo timeshift --list

  1. Should show 5-8 snapshots. If too many:

sudo timeshift --delete-snapshot <snapshot_name>

  1. If too few, create one manually:

sudo timeshift --create --comments "Manual snapshot for recovery testing" </syntaxhighlight>

Quarterly: Clean Up Restic Cache

<syntaxhighlight lang="bash">

  1. Cache can grow to 1-2GB over time

du -sh /library/backup-stack/cache

  1. Prune cache (safe, will be rebuilt)

docker exec restic-backup restic cache-dir docker exec restic-backup rm -rf /root/.cache/restic/ </syntaxhighlight>

Annually: Update Restic Image

<syntaxhighlight lang="bash"> cd /library/backup-stack

  1. Update to latest Restic

docker compose pull docker compose up -d

  1. Verify after update

docker exec restic-backup restic version </syntaxhighlight>

Links & References

Related Services

  • All services — depend on this backup system for disaster recovery
  • OpenBao — stores Hetzner SSH key and Restic password
  • ntfy — receives backup alerts (success/failure notifications)
  • PostgreSQL (in each service) — database dumps backed up daily
Retrieved from "https://wiki.wilsoz.com/index.php?title=Services/BackupSystem&oldid=15"