Wikibot: Migrated from services/BACKUP_SYSTEM.md

2026-07-05T18:16:21Z

Migrated from services/BACKUP_SYSTEM.md

New page

''As of: 2026-07-05 — migrated from `services/BACKUP_SYSTEM.md`''

= Backup System — Restic + Timeshift =

Complete setup & operations guide for Stronghold's dual-tier backup system: Timeshift (local snapshots) + Restic (remote encrypted backups).

'''Criticality:''' '''CRITICAL — The foundation.''' All disaster recovery depends on backups. Without them, data loss is permanent. This system enables recovery of every service on Stronghold.

----

== Overview ==

Stronghold uses a '''two-tier backup strategy''' for redundancy and speed:

# '''Timeshift''' (local, fast)
** RSYNC-based filesystem snapshots stored on <code>/raid1/timeshift/</code>
** 8 snapshots retained (mix of daily, weekly, monthly)
''' '''Purpose:** Fast recovery of individual files or full system (within days)
''' '''Speed:** ~5-30 minutes to restore (depends on size)
''' '''Limitation:** If <code>/raid1</code> is damaged, all local snapshots are lost

# '''Restic''' (remote, encrypted)
** Incremental encrypted backups sent to Hetzner Storage Box
** Daily backups at 03:00 CST, retention: 7 daily + 4 weekly + 6 monthly
''' '''Purpose:** Off-site redundancy; survives hardware failure
''' '''Coverage:** All service data, configs, media, databases
''' '''Speed:** ~30 minutes to backup, 1-2 hours per terabyte for restoration
''' '''Limitation:** Requires Hetzner SSH access and Restic repo password

'''Recovery decision tree:'''
* '''Drive works, need recent files?''' → Restore from Timeshift (fast)
* '''Drive works, lost some service?''' → Restore from Timeshift or Restic (both available)
* '''Drive failed, building new hardware?''' → Restore from Restic (only source of truth)

----

== Service Information ==

=== Timeshift ===

{| class="wikitable"
|-
! Property
! Value
|-
| '''Storage location'''
| <code>/raid1/timeshift/</code>
|-
| '''Snapshot mode'''
| RSYNC (not btrfs)
|-
| '''Device'''
| <code>/dev/md1</code> (RAID-1 array)
|-
| '''Current snapshots'''
| 8 (mix of daily D, weekly W, monthly M, backfill B)
|-
| '''Retention policy'''
| Auto: Last 5 daily, 1 weekly, 1 monthly; manual cleanup for older
|-
| '''Snapshot schedule'''
| Daily at 00:00 (D), weekly at 10:00 (W), manual on demand
|-
| '''Excluded from snapshots'''
| <code>/var/lib/docker/volumes</code> ⚠️, <code>/proc</code>, <code>/sys</code>, <code>/tmp</code>, <code>/var/tmp</code>
|}

=== Restic ===

{| class="wikitable"
|-
! Property
! Value
|-
| '''Container name'''
| <code>restic-backup</code>
|-
| '''Image'''
| <code>restic/restic:latest</code>
|-
| '''Compose file'''
| <code>/home/mnw/Developer/SystemResiliency/backup-stack/docker-compose.yml</code>
|-
| '''Network mode'''
| <code>host</code> (direct network access)
|-
| '''Backup schedule'''
| 03:00 CST daily (systemd timer <code>backup.timer</code>)
|-
| '''Hetzner server'''
| <code>u540853.your-storagebox.de:23</code> (SSH port 23, not 22)
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups</code>
|-
| '''Encryption'''
| AES-256, password-based (stored in <code>.env</code> as <code>HETZNER_PASSWORD</code>)
|-
| '''Deduplication'''
| Enabled (saves 20-30% space)
|-
| '''Compression'''
| zstd (default, good balance)
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly snapshots
|-
| '''Database dumps'''
| PostgreSQL dumps staged to <code>/library/backup-stack/dumps/</code> before backup
|-
| '''System-state dump'''
| <code>/etc</code> subset + user crontabs + user/system systemd units + dpkg selections tar'd to <code>/library/backup-stack/dumps/system-state-YYYYMMDD.tar.gz</code> before backup (covers schedulers/configs that live outside <code>/data/*</code> and would otherwise be lost if <code>/etc</code> or a user home got wiped)
|}

----

== Initial Setup from Scratch ==

=== Timeshift (Local Snapshots) ===

'''Timeshift is likely already installed and running on your system.''' To verify or install:

<syntaxhighlight lang="bash">
# Check if Timeshift is installed
which timeshift

# If not, install (Ubuntu/Debian)
sudo apt update
sudo apt install timeshift

# Check current snapshots
sudo timeshift --list

# Create first snapshot manually (if none exist)
sudo timeshift --create --comments "Initial snapshot before disaster recovery setup"
</syntaxhighlight>

'''Configuration''' (in Timeshift GUI or config):
* '''Storage location:''' <code>/raid1/timeshift/</code>
* '''Device:''' <code>/dev/md1</code> (RAID-1 array)
* '''Snapshot mode:''' RSYNC
* '''Auto-backup:''' Enabled, daily at 00:00
* '''Retention:''' At least 5 daily snapshots kept

=== Restic (Remote Encrypted Backups) ===

==== 1. Set Up Hetzner Storage Box ====

If not already done:

# '''Create Hetzner account''' at https://www.hetzner.com/storage/storage-box
# '''Order Storage Box''' (recommended: smallest size starts at €3.81/month)
# '''In Hetzner Console:'''
** Go to Storage Box settings
** Enable "External reachability" (required for SSH/SFTP)
** Upload your SSH public key (<code>~/.ssh/id_rsa.pub</code>)
# '''Note your credentials:'''
** Server: <code>u540853.your-storagebox.de</code> (your number varies)
''' Port: '''23** (not 22!)
** Username: <code>u540853</code> (same as your account number)

==== 2. Set Up SSH Access ====

<syntaxhighlight lang="bash">
# Create SSH key if you don't have one
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa

# Test SSH connection to Hetzner (port 23!)
ssh -p 23 u540853@u540853.your-storagebox.de "echo 'SSH working!'"

# Should print: SSH working!
</syntaxhighlight>

==== 3. Copy Backup Stack to /library ====

<syntaxhighlight lang="bash">
# From your project directory
cp -r /home/mnw/Developer/SystemResiliency/backup-stack /library/

cd /library/backup-stack

# Create .env from example
cp .env.example .env

# Edit with your Hetzner credentials and restic password
nano .env
</syntaxhighlight>

'''.env file contents:'''
<syntaxhighlight lang="bash">
# Hetzner Storage Box
HETZNER_REPO="sftp:u540853@u540853.your-storagebox.de:./backups"
HETZNER_PASSWORD="<generate-strong-restic-password-32-chars>"

# Alerts
NTFY_TOKEN="<from Vaultwarden if you have one>"
NTFY_TOPIC="backup-alerts"
</syntaxhighlight>

==== 4. Initialize Restic Repository ====

<syntaxhighlight lang="bash">
cd /library/backup-stack
docker compose up -d

# Init Hetzner repository (will encrypt with password)
docker exec -e RESTIC_PASSWORD='<same-as-HETZNER_PASSWORD>' restic-backup \
restic init --repo sftp:u540853@u540853.your-storagebox.de:./backups

# Output should show: "created restic repository" with repository ID
</syntaxhighlight>

==== 5. Install Systemd Timers ====

<syntaxhighlight lang="bash">
# Copy timer definitions
sudo cp /library/backup-stack/systemd/*.service /etc/systemd/system/
sudo cp /library/backup-stack/systemd/*.timer /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload

# Enable and start backup timer
sudo systemctl enable --now backup.timer
sudo systemctl enable --now backup-validate-daily.timer
sudo systemctl enable --now backup-validate-weekly.timer

# Check status
systemctl list-timers | grep backup
</syntaxhighlight>

==== 6. Test the Backup ====

<syntaxhighlight lang="bash">
# Run a manual backup to verify everything works
docker exec restic-backup /scripts/run-backup.sh

# Watch the logs
docker logs -f restic-backup

# Should show:
# - Database dumps created
# - Restic scanning files
# - Upload to Hetzner
# - Success message
</syntaxhighlight>

----

== Disaster Mode A: Local Recovery (Fast) ==

=== Scenario: Lost a single file or small data ===

'''Example:''' Accidentally deleted a recipe from Mealie 2 days ago.

==== Recovery Steps: ====

<syntaxhighlight lang="bash">
# 1. List available snapshots
sudo timeshift --list

# 2. Mount the snapshot you want to restore from
# (Example: restore from 2026-05-11 daily snapshot)
sudo timeshift --mount /run/timeshift/latest

# 3. File is now accessible at:
# /run/timeshift/latest/library/mealie-app/mealie-data/recipes/
# Copy the file back to current system
cp /run/timeshift/latest/library/mealie-app/mealie-data/recipes/recipe.json \
/library/mealie-app/mealie-data/recipes/

# 4. Unmount
sudo timeshift --umount
</syntaxhighlight>

=== Scenario: Need to restore entire service from snapshot ===

'''Example:''' Immich database corrupted, want to restore from snapshot.

<syntaxhighlight lang="bash">
# 1. Stop the service
cd /library/immich-app
docker compose down

# 2. Mount snapshot
sudo timeshift --mount

# 3. Restore the entire service directory
sudo rm -rf /library/immich-app
sudo cp -r /run/timeshift/latest/library/immich-app /library/

# 4. Fix permissions (Docker runs as specific user)
sudo chown -R mnw:mnw /library/immich-app

# 5. Restart service
docker compose up -d

# 6. Unmount snapshot
sudo timeshift --umount
</syntaxhighlight>

=== Scenario: Full system restoration (if OS is still usable) ===

'''Example:''' Stronghold is working but some system files are corrupted.

<syntaxhighlight lang="bash">
# List snapshots from a few days ago
sudo timeshift --list

# Restore from a known-good snapshot
# This restores EVERYTHING except /home, /var/lib/docker, /opt, /etc/hosts, etc.
sudo timeshift --restore --snapshot "2026-05-11_10-00-01" --skip-grub

# Reboot after restoration
sudo reboot
</syntaxhighlight>

----

== Disaster Mode B: Full Recovery (After Drive Failure) ==

=== Scenario: Hard drive failed, new hardware deployed ===

'''Prerequisites:'''
* New hardware with OS installed (Ubuntu/Debian)
* Docker and docker-compose installed
* SSH key for Hetzner auth
* Restic password (from your secure notes)

==== Recovery Steps: ====

<syntaxhighlight lang="bash">
# 1. Create directories
mkdir -p /raid1
mkdir -p /library
mkdir -p /home/mnw/Developer/SystemResiliency

# 2. Copy backup-stack to new system
scp -r username@old-system:/library/backup-stack /library/

# 3. Update .env with correct passwords/credentials
nano /library/backup-stack/.env

# 4. Start the Restic container
cd /library/backup-stack
docker compose up -d

# 5. List available backups in Hetzner
docker exec restic-backup restic snapshots --repo sftp:u540853@u540853.your-storagebox.de:./backups

# 6. Restore the full /library directory from latest snapshot
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--repo sftp:u540853@u540853.your-storagebox.de:./backups \
--include "/data/*"

# 7. Copy restored data to /library
sudo cp -r /tmp/restore/data/* /library/

# 8. Restore /home/mnw
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--repo sftp:u540853@u540853.your-storagebox.de:./backups \
--include "/data/home"

sudo cp -r /tmp/restore/data/home/* /home/

# 9. Fix permissions
sudo chown -R mnw:mnw /library
sudo chown -R mnw:mnw /home/mnw

# 10. Restore PostgreSQL databases
# Each service (Authentik, Nextcloud, etc.) has a dump at /library/backup-stack/dumps/
# Use the restore procedure from each service's documentation

# Example: Restore Authentik PostgreSQL
docker exec authentik-app_postgresql_1 pg_restore \
/library/backup-stack/dumps/authentik-YYYYMMDD.sql < < (gunzip -c /path/to/dump)

# 11. Start all services
for dir in /library/*-app/; do
cd "$dir"
docker compose up -d
done

# 12. Verify critical services are running
docker ps | grep "server\|worker\|postgres"
</syntaxhighlight>

=== Scenario: Restore specific service from Restic ===

'''Example:''' Only Immich needs recovery, not entire system.

<syntaxhighlight lang="bash">
cd /library/backup-stack
docker compose up -d

# Restore just Immich config and database dump
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--repo sftp:u540853@u540853.your-storagebox.de:./backups \
--include "/data/immich*" \
--include "/dumps/immich*"

# Move to correct location
mv /tmp/restore/data/immich-config /library/immich-app
mv /tmp/restore/data/immich-photos /raid1/Consolidated/Picture/Immich-Photos

# Restore the database
cd /library/immich-app
docker compose up -d immich_postgres
sleep 10
docker exec -i immich_postgres psql -U immich < <(gunzip -c /tmp/restore/dumps/immich-*.sql.gz)

# Start full Immich
docker compose up -d
</syntaxhighlight>

----

== Data Locations & Backup Coverage ==

=== What's Backed Up (via Restic) ===

<syntaxhighlight lang="text">
Backed up to Hetzner:
├── /data/authentik/ — Authentik OAuth provider config
├── /data/vaultwarden/ — Password vault (SQLite + attachments)
├── /data/nextcloud/ — File sync config
├── /data/immich-config/ — Photo manager config
├── /data/immich-photos/ — All photos (149GB)
├── /data/jellyfin/ — /var/lib/jellyfin (plugins, root, subs; live DBs excluded)
├── /data/jellyfin-etc/ — /etc/jellyfin (system config)
├── /data/jellyfin-server/ — /library/jellyfin-server-app (active metadata 28G — see services/JELLYFIN.md)
├── /data/memos/ — Notes app
├── /data/linkwarden/ — Bookmark manager
├── /data/open-webui/ — AI chat interface
├── /data/matrix-chat/ — Synapse + bridge configs
├── /data/stalwart/ — Mail server data
├── /data/snappymail/ — Webmail data
├── /data/mailcow/ — Legacy mail (to be archived)
├── /data/developer/ — /home/mnw/Developer (7.9G)
├── /data/home/ — /home/mnw (selective: excludes .git/objects, node_modules, .venv)
├── /data/beets/ — Music library DB
├── /dumps/ — PostgreSQL database dumps (daily)
│ ├── authentik-YYYYMMDD.sql.gz
│ ├── nextcloud-YYYYMMDD.sql.gz
│ ├── immich-YYYYMMDD.sql.gz
│ ├── linkwarden-YYYYMMDD.sql.gz
│ └── matrix-YYYYMMDD.sql.gz
</syntaxhighlight>

=== What's NOT Backed Up (Re-acquirable) ===

* <code>/var/lib/docker/volumes</code> ⚠️ (Timeshift excludes this; Restic needs to be updated to include)
* Jellyfin/Kaleidescape media files (movies, TV)
* Ollama models (re-downloadable)
* Steam games
* VirtualBox VMs
* node_modules, .venv, .git/objects, build artifacts

=== Timeshift Snapshots ===

<syntaxhighlight lang="text">
Stored on /raid1/timeshift/
├── 2025-05-03_17-15-00 (Backfill from old system)
├── 2026-05-03_11-52-11 (Backfill)
├── 2026-05-05_20-41-31 (Backfill)
├── 2026-05-05_21-36-57 (Backfill)
├── 2026-05-11_10-00-02 (Monthly snapshot)
├── 2026-05-12_00-00-01 (Daily snapshot)
├── 2026-05-12_10-00-01 (Weekly snapshot)
└── 2026-05-13_00-00-01 (Daily snapshot)
</syntaxhighlight>

Each snapshot contains a full filesystem image (~100-300GB depending on changes).

----

== Critical Gotchas ==

=== 1. ⚠️ Docker Volumes NOT in Timeshift ===

'''Problem:''' Timeshift is configured to exclude <code>/var/lib/docker/volumes</code>, which contains:
* PostgreSQL databases for Authentik, Nextcloud, Immich, Linkwarden, Matrix
* Docker-managed volumes

'''If you restore from Timeshift, you lose all database data.''' PostgreSQL will be empty.

'''Solution:''' Always use Restic for full recovery (it includes the databases via dumps), or manually restore PostgreSQL dumps.

'''Status:''' Known issue, needs fix in Timeshift config or backup-stack dumps.

=== 2. Hetzner Port is 23, NOT 22 ===

Many SSH configs default to port 22. Hetzner requires port 23.

* '''Wrong:''' <code>ssh u540853@u540853.your-storagebox.de</code>
* '''Right:''' <code>ssh -p 23 u540853@u540853.your-storagebox.de</code>

The <code>ssh-config</code> file in backup-stack handles this automatically.

=== 3. Hetzner Relative Paths Required ===

Hetzner SFTP doesn't support absolute paths for new directories.

* '''Wrong:''' <code>sftp:u540853@...:/backups</code> (fails with SSH_FX_FAILURE)
* '''Right:''' <code>sftp:u540853@....:./backups</code> (relative to home directory)

=== 4. Restic Password != Hetzner Password ===

Two different passwords:
* '''Hetzner account password:''' Logs into Hetzner web console
* '''Restic password:''' Encrypts your backup repository

Save both securely. If you lose the Restic password, your backups are unrecoverable.

=== 5. SSH Key Permissions in Docker ===

When mounting SSH keys into the Restic container, SSH is strict about ownership (must be 600 or 400).

The docker-compose entrypoint handles this:
<syntaxhighlight lang="yaml">
entrypoint: ["/bin/sh", "-c", "mkdir -p /root/.ssh && cp /tmp/ssh/'' /root/.ssh/ && chmod 600 /root/.ssh/'' && sleep infinity"]
</syntaxhighlight>

Don't modify without understanding SSH permission requirements.

=== 6. PostgreSQL Dumps Staged Before Backup ===

Database dumps are created at backup time by <code>run-backup.sh</code>:
<syntaxhighlight lang="bash">
docker exec <container> pg_dump -U <user> <db> | gzip > /library/backup-stack/dumps/<service>.sql.gz
</syntaxhighlight>

If a database is corrupted, the dump will fail, and that database won't be backed up. Check backup logs:
<syntaxhighlight lang="bash">
cat /library/backup-stack/dumps/backup-*.log | grep -i "error\|dump"
</syntaxhighlight>

=== 7. Backup Window (03:00 CST) Must Not Overlap with Other Heavy Operations ===

Backup runs daily at 03:00 CST. If you have other cron jobs or maintenance windows:
* Stagger them by at least 30 minutes
* Monitor systemd timers: <code>systemctl list-timers</code>

Concurrent heavy I/O can cause slowdowns or backup timeouts.

=== 8. Restic Prune Deletes Old Snapshots Automatically ===

Restic <code>prune</code> runs after backup to clean up old snapshots per retention policy:
* Keep 7 daily, 4 weekly, 6 monthly

If you manually delete recent snapshots (for testing), they may be pruned. Don't delete recent snapshots without understanding the implications.

----

== Verification & Testing ==

=== Daily Check: Backup Health ===

<syntaxhighlight lang="bash">
# Check if backup completed successfully
docker logs restic-backup | tail -20 | grep -i "success\|error"

# Check ntfy for backup alerts
curl https://ntfy.wilsoz.com/backup-alerts

# List recent snapshots
docker exec restic-backup restic snapshots --limit 5
</syntaxhighlight>

=== Weekly Test: Restore a File ===

<syntaxhighlight lang="bash">
# Restore a small non-critical file to /tmp
docker exec restic-backup restic restore latest \
--target /tmp/test-restore \
--include "/data/memos/*"

# Verify files exist
ls -la /tmp/test-restore/data/memos/

# Clean up
rm -rf /tmp/test-restore
</syntaxhighlight>

=== Monthly Test: Restore a Service Database ===

<syntaxhighlight lang="bash">
# Restore Linkwarden dump (small DB, ~50MB)
docker exec restic-backup restic restore latest \
--target /tmp/test-restore \
--include "/dumps/linkwarden*"

# Verify dump is valid
gunzip -t /tmp/test-restore/dumps/linkwarden-*.sql.gz

# Should report: OK (no errors)

# Clean up
rm -rf /tmp/test-restore
</syntaxhighlight>

=== Quarterly: Full System Restore Dry-Run ===

Once per quarter, do a mock recovery to verify procedures:

<syntaxhighlight lang="bash">
# Don't actually restore (too time-consuming), but verify:
# 1. Can list snapshots?
docker exec restic-backup restic snapshots --limit 1

# 2. Can stat the repository (verify password works)?
docker exec restic-backup restic stats

# 3. Can list files in a snapshot?
docker exec restic-backup restic ls latest | head -20

# If all three pass, recovery is ready.
</syntaxhighlight>

----

== Troubleshooting ==

=== Backup Failed ===

'''Check logs:'''
<syntaxhighlight lang="bash">
# Container logs
docker logs restic-backup | tail -50

# Systemd log
sudo journalctl -u backup.service -n 50

# Backup script log
ls -lht /library/backup-stack/dumps/backup-*.log
tail -100 /library/backup-stack/dumps/backup-*.log
</syntaxhighlight>

'''Common causes:'''
* SSH key permission error → Run entrypoint again: <code>docker compose up -d</code>
* Hetzner unreachable → Test SSH manually: <code>ssh -p 23 u540853@u540853.your-storagebox.de</code>
* Database dump failed → Check Docker container: <code>docker ps | grep postgres</code>
* Out of disk space → Check /library and /raid1: <code>df -h /library /raid1</code>

=== Restore Is Slow ===

Restic restores are network-bound from Hetzner. Typical speeds:
* Small file (< 100MB): 30 seconds
* Medium service (1-5GB): 5-15 minutes
* Large dataset (100+ GB): 1-3 hours

'''If slower than expected:'''
* Check network: <code>ping u540853.your-storagebox.de</code> (port 23 doesn't reply to ping)
* Try SSH: <code>ssh -p 23 u540853@u540853.your-storagebox.de echo ok</code>
* Check Restic cache: <code>du -sh /library/backup-stack/cache</code>

=== Can't Connect to Hetzner ===

<syntaxhighlight lang="bash">
# Test SSH connectivity
ssh -p 23 u540853@u540853.your-storagebox.de echo "Connected"

# If failed:
# 1. Check SSH key exists: ls -la ~/.ssh/id_rsa
# 2. Check Hetzner console: "External reachability" enabled?
# 3. Check your public key uploaded to Hetzner
# 4. Try with password instead: ssh -p 23 -o PubkeyAuthentication=no u540853@u540853.your-storagebox.de
</syntaxhighlight>

=== Restic Unlock Stuck ===

If Restic process was killed, lock file may remain:

<syntaxhighlight lang="bash">
# Remove lock
docker exec restic-backup restic unlock --repo sftp:u540853@...
</syntaxhighlight>

----

== Maintenance ==

=== Monthly: Review Snapshot Count ===

<syntaxhighlight lang="bash">
sudo timeshift --list

# Should show 5-8 snapshots. If too many:
sudo timeshift --delete-snapshot <snapshot_name>

# If too few, create one manually:
sudo timeshift --create --comments "Manual snapshot for recovery testing"
</syntaxhighlight>

=== Quarterly: Clean Up Restic Cache ===

<syntaxhighlight lang="bash">
# Cache can grow to 1-2GB over time
du -sh /library/backup-stack/cache

# Prune cache (safe, will be rebuilt)
docker exec restic-backup restic cache-dir
docker exec restic-backup rm -rf /root/.cache/restic/
</syntaxhighlight>

=== Annually: Update Restic Image ===

<syntaxhighlight lang="bash">
cd /library/backup-stack

# Update to latest Restic
docker compose pull
docker compose up -d

# Verify after update
docker exec restic-backup restic version
</syntaxhighlight>

----

== Links & References ==

* '''Timeshift Docs:''' https://github.com/teejee2008/timeshift
* '''Restic Docs:''' https://restic.readthedocs.io/
* '''Hetzner Storage Box:''' https://www.hetzner.com/storage/storage-box
* '''Backup Strategy Overview:''' <code>/home/mnw/Developer/SystemResiliency/DISASTER_RECOVERY.md</code>
* '''Backup Stack Code:''' <code>/home/mnw/Developer/SystemResiliency/backup-stack/</code>

----

== Related Services ==

* '''All services''' — depend on this backup system for disaster recovery
* '''OpenBao''' — stores Hetzner SSH key and Restic password
* '''ntfy''' — receives backup alerts (success/failure notifications)
* '''PostgreSQL''' (in each service) — database dumps backed up daily

Services/BackupSystem - Revision history

Wikibot: Migrated from services/BACKUP_SYSTEM.md