Stronghold Wiki - User contributions [en]

Services/Authentik

2026-07-05T18:25:21Z

Wikibot: Genericize example password placeholder

''As of: 2026-07-05 — migrated from services/AUTHENTIK.md''

= Authentik — Identity & SSO Provider =

Complete setup & operations guide for Authentik (identity provider, OAuth2/OIDC, LDAP directory service).

'''Criticality:''' '''CRITICAL — The lynchpin.''' Everything on Stronghold depends on Authentik for authentication. Without it, most other services are inaccessible or in an inconsistent state. Must be recovered first in any disaster scenario.

----

== Overview ==

'''Authentik''' is a self-hosted identity provider that serves as the single source of truth for authentication across Stronghold. It provides:
* '''OAuth2 / OIDC''' providers for web apps (Immich, Memos, Linkwarden, Forgejo, Vaultwarden, Grafana, n8n, Matrix, Cloudflare Access)
* '''LDAP directory''' for apps that don't support OAuth (Nextcloud, Jellyfin, Stalwart mail server)
* '''Group membership''' for role-based access (authentik Admins, family-and-friends, email provisioning groups)
* '''User management''' — all users are defined here; other services sync from Authentik
* '''API tokens''' for programmatic access (n8n, CI/CD, automation)

'''Current Users:'''
* <code>marcus</code> (primary admin user, member of authentik Admins)
* <code>akadmin</code> (backup admin user)
* Service accounts: <code>stalwart-svc</code>, <code>nextcloud-svc</code> (for LDAP directory access)

'''Current Version:''' 2025.10.3

'''Access:''' https://key.wilsoz.com/if/admin/

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''Compose file'''
| <code>/library/authentik-app/docker-compose.yml</code>
|-
| '''Containers'''
| <code>authentik-app_server_1</code>, <code>authentik-app_worker_1</code>, <code>authentik-app_postgresql_1</code>
|-
| '''Database'''
| PostgreSQL 16 (in-container, persisted to Docker volume <code>database</code>)
|-
| '''HTTP Port'''
| 9000 (external via 9443 HTTPS via reverse proxy)
|-
| '''HTTPS Port'''
| 9443 (internal, for direct connections)
|-
| '''LDAP Outpost Port'''
| 389 (via LDAP Outpost container; internal only, reaches stronghold via <code>http://172.17.0.1:9000</code>)
|-
| '''Data volumes'''
| <code>/library/authentik-app/media</code> (custom templates, branding)
|-
| '''Database volume'''
| <code>authentik-app_database</code> (PostgreSQL data)
|-
| '''Networks'''
| <code>authentik-app_default</code> (internal Docker network, MTU 1280)
|-
| '''Image versions'''
| server/worker: <code>ghcr.io/goauthentik/server:2025.10.3</code>
|-
| '''Restart policy'''
| <code>unless-stopped</code>
|}

----

== Initial Setup from Scratch ==

=== Prerequisites ===

* Docker and docker-compose installed
* Network access to PostgreSQL (internal container or external)
* Cloudflare DNS configured for <code>key.wilsoz.com</code>
* VPS reverse proxy (nginx) configured to forward to stronghold:9000

=== 1. Create Directory & Files ===

<syntaxhighlight lang="bash">
mkdir -p /library/authentik-app
cd /library/authentik-app
</syntaxhighlight>

=== 2. Create docker-compose.yml ===

<syntaxhighlight lang="yaml">
version: "3.8"

services:
postgresql:
image: docker.io/library/postgres:16-alpine
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
timeout: 5s
retries: 5
start_period: 20s
volumes:
- database:/var/lib/postgresql/data
env_file:
- .env
environment:
POSTGRES_DB: ${PG_DB:-authentik}
POSTGRES_USER: ${PG_USER:-authentik}
POSTGRES_PASSWORD: ${PG_PASS:?database password required}

server:
image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2025.10.3}
restart: unless-stopped
command: server
ports:
- "${COMPOSE_PORT_HTTP:-9000}:9000"
- "${COMPOSE_PORT_HTTPS:-9443}:9443"
env_file:
- .env
environment:
AUTHENTIK_POSTGRESQL__HOST: postgresql
AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required}
volumes:
- ./media:/media
- ./custom-templates:/templates
depends_on:
postgresql:
condition: service_healthy

worker:
image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2025.10.3}
restart: unless-stopped
command: worker
user: root
env_file:
- .env
environment:
AUTHENTIK_POSTGRESQL__HOST: postgresql
AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./media:/media
- ./certs:/certs
- ./custom-templates:/templates
depends_on:
postgresql:
condition: service_healthy

volumes:
database:
</syntaxhighlight>

=== 3. Create .env File ===

<syntaxhighlight lang="bash">
# PostgreSQL
PG_DB=authentik
PG_USER=authentik
PG_PASS=<generate-strong-password-32-chars>

# Authentik
AUTHENTIK_SECRET_KEY=<generate-strong-secret-key-50-chars>
AUTHENTIK_IMAGE=ghcr.io/goauthentik/server
AUTHENTIK_TAG=2025.10.3

# Ports
COMPOSE_PORT_HTTP=9000
COMPOSE_PORT_HTTPS=9443

# Bootstrap (initial admin)
AUTHENTIK_BOOTSTRAP_PASSWORD=<temporary-init-password>
AUTHENTIK_BOOTSTRAP_TOKEN=<temporary-init-token>
</syntaxhighlight>

'''Important:''' Generate secure random values for passwords and secrets:
<syntaxhighlight lang="bash">
openssl rand -base64 32 # For PG_PASS and AUTHENTIK_SECRET_KEY
openssl rand -hex 16 # For AUTHENTIK_BOOTSTRAP_TOKEN
</syntaxhighlight>

=== 4. Start Authentik ===

<syntaxhighlight lang="bash">
cd /library/authentik-app
docker compose up -d
</syntaxhighlight>

Wait for PostgreSQL to be healthy and server to start (~30-60 seconds):

<syntaxhighlight lang="bash">
docker compose logs -f server
# Watch for "Application startup complete"
</syntaxhighlight>

=== 5. Initial Admin User ===

After startup, navigate to <code>https://key.wilsoz.com/if/admin/</code>:
* Username: <code>akadmin</code>
* Password: value of <code>AUTHENTIK_BOOTSTRAP_PASSWORD</code>

'''First steps in Admin UI:'''
# Go to '''Directory → Users → Create'''
** Username: <code>marcus</code>
** Email: <code>marcus@wilsoz.com</code>
** Set password (or let user set on first login)
''' Groups: Add to '''authentik Admins**

# Go to '''Directory → Groups → Create'''
''' '''authentik Admins** (if not already created)
''' '''family-and-friends** (for app access)
''' '''craniumslows-email** (for email provisioning)
''' '''wilsoz-email** (for email provisioning)
''' '''ldap-search** (for LDAP bind accounts)

# Add service users:
''' '''Directory → Users → Create**
** Username: <code>stalwart-svc</code>, Email: <code>stalwart-svc@internal</code>
''' Groups: Add to '''ldap-search**
** Username: <code>nextcloud-svc</code>, Email: <code>nextcloud-svc@internal</code>
''' Groups: Add to '''ldap-search**

=== 6. Set Up LDAP Outpost ===

LDAP is required for Stalwart (mail server), Nextcloud (CalDAV/CardDAV), and Jellyfin (media server) to authenticate.

'''In Authentik Admin UI:'''
# Go to '''Infrastructure → Outposts → Create'''
** Name: <code>authentik-ldap</code>
** Type: LDAP
# In '''Directory → LDAP → Configure LDAP'''
** Base DN: <code>dc=ldap,dc=goauthentik,dc=io</code>
** User base: <code>ou=users,dc=ldap,dc=goauthentik,dc=io</code>
# Assign the LDAP outpost to the LDAP provider

The LDAP server will be reachable at <code>172.17.0.1:389</code> from other containers.

=== 7. Configure OAuth2 Providers ===

For each OAuth application (Immich, Memos, Linkwarden, etc.), create an OAuth2 provider in Authentik:

'''Immich example:'''
# Go to '''Applications → Providers → Create OAuth2/OpenID Provider'''
** Name: <code>immich-oauth</code>
** Authorization flow: <code>default-authentication-flow</code>
** Client type: <code>Confidential</code>
** Redirect URIs:
```
https://photos.wilsoz.com/auth/login/oauth2/callback
https://photos.wilsoz.com/auth/login/oauth2/mobile-redirect
```
# '''Save''' — copy the generated Client ID and Client Secret
# Add property mappings:
** <code>authentik default OAuth Mapping: OpenID 'openid'</code>
** <code>authentik default OAuth Mapping: OpenID 'email'</code>
** <code>authentik default OAuth Mapping: OpenID 'profile'</code>
# Enable: <code>include_claims_in_id_token</code>
# '''Assign a signing key''' (see Critical Gotchas section below)

Repeat for each app.

=== 8. Configure VPS Reverse Proxy ===

On VPS, add nginx server block:

<syntaxhighlight lang="nginx">
server {
listen 443 ssl http2;
server_name key.wilsoz.com;

ssl_certificate /etc/letsencrypt/live/key.wilsoz.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/key.wilsoz.com/privkey.pem;

location / {
proxy_pass http://100.64.207.155:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
</syntaxhighlight>

Reload nginx:
<syntaxhighlight lang="bash">
sudo systemctl reload nginx
</syntaxhighlight>

----

== Disaster Mode A: Configuration Corruption ==

=== Symptoms ===

* Can't log into admin interface
* OAuth providers missing or broken
* LDAP directory returning errors
* Groups or users deleted accidentally
* Provider credentials lost

=== Recovery Steps ===

⚠️ '''This mode resets Authentik to a fresh state. User data will be lost unless you have a PostgreSQL backup (see Disaster Mode B).'''

==== Option 1: Full Reset (if Disaster Mode B not available) ====

<syntaxhighlight lang="bash">
cd /library/authentik-app

# Stop all containers
docker compose down

# Remove the database volume (this deletes all Authentik data)
docker volume rm authentik-app_database

# Start from scratch
docker compose up -d

# Wait for startup
sleep 30
docker logs server | grep "Application startup complete"

# Follow "Initial Setup from Scratch" section above
</syntaxhighlight>

==== Option 2: Selective Reset (if you only need to fix one app) ====

<syntaxhighlight lang="bash">
# Access the Django shell
docker exec authentik-app_server_1 ak shell

# Example: Fix an OAuth provider's signing key
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.crypto.models import CertificateKeyPair

p = OAuth2Provider.objects.get(name='immich-oauth')
jwt_key = CertificateKeyPair.objects.get(name='authentik Internal JWT Certificate')
p.signing_key = jwt_key
p.save()
print(f'Updated {p.name} with JWT signing key')
</syntaxhighlight>

----

== Disaster Mode B: Drive Failure / Full Recovery ==

=== If Backups Accessible ===

Authentik data is backed up in two ways:

# '''PostgreSQL dump''' at Hetzner (in <code>/dumps/authentik-YYYYMMDD.sql.gz</code>)
# '''Media files''' at Hetzner (in <code>/data/authentik/</code>)

==== Recovery procedure: ====

<syntaxhighlight lang="bash">
# 1. Set up fresh Authentik (follow "Initial Setup from Scratch" through step 4)
cd /library/authentik-app
docker compose up -d

# 2. Wait for PostgreSQL to be ready
sleep 30

# 3. Drop the initial empty database
docker exec authentik-app_postgresql_1 psql -U authentik -d postgres -c "DROP DATABASE authentik;"

# 4. Create a fresh empty database
docker exec authentik-app_postgresql_1 psql -U authentik -d postgres -c "CREATE DATABASE authentik;"

# 5. Restore from backup dump (substitute actual date)
# First, extract the backup from Hetzner:
# (You'll need ssh-key access to u540853.your-storagebox.de port 23)
ssh -p 23 u540853@u540853.your-storagebox.de "cat backups/authentik-20260512.sql.gz" | \
gunzip | docker exec -i authentik-app_postgresql_1 psql -U authentik -d authentik

# 6. Restore media files
rsync -av -e "ssh -p 23" u540853@u540853.your-storagebox.de:backups/data/authentik/ /library/authentik-app/media/

# 7. Restart Authentik to pick up restored data
docker compose restart server worker

# 8. Verify
docker logs server | grep "Application startup complete"
curl -s https://key.wilsoz.com/if/admin/ | grep -q "authentik" && echo "Admin UI responding"
</syntaxhighlight>

=== If Backups NOT Accessible ===

Fall back to Mode A (full reset). Note: '''All user data, OAuth provider credentials, and LDAP settings will be lost.'''

After reset, you'll need to:
# Recreate all users and groups
# Recreate all OAuth2 providers and get new Client IDs/Secrets
# Update all dependent services (.env files) with new credentials
# Recreate LDAP configuration and outpost

----

== Data Locations ==

=== Authentik Data on Host ===

<syntaxhighlight lang="text">
/library/authentik-app/
├── docker-compose.yml (service definition)
├── .env (secrets: PG_PASS, AUTHENTIK_SECRET_KEY)
├── media/ (custom templates, branding, uploads)
└── custom-templates/ (HTML templates for login pages, etc.)
</syntaxhighlight>

=== PostgreSQL Database Volume ===

<syntaxhighlight lang="text">
authentik-app_database (Docker volume)
├── postgresql/ (database files)
└── ...
</syntaxhighlight>

Contains:
* All users, groups, group memberships
* All OAuth2 providers, applications, credentials
* All LDAP configuration
* All API tokens
* All authenticator devices (TOTP, WebAuthn)
* All event logs

=== Backup Locations ===

* '''PostgreSQL dump:''' Hetzner <code>/dumps/authentik-YYYYMMDD.sql.gz</code> (daily at 03:00 CST)
* '''Media files:''' Hetzner <code>/data/authentik/</code> (daily Restic snapshot)
* '''Filesystem snapshot:''' Timeshift snapshot on <code>/dev/md1</code> (weekly)

=== What's NOT Backed Up ===

* Container images (re-downloadable from Docker Hub)
* node_modules and build artifacts (not present in Authentik)

----

== Critical Gotchas ==

=== 1. OAuth2 Signing Key Assignment — '''MANDATORY''' ===

⚠️ '''If you don't assign a signing key to an OAuth2 provider, the JWKS endpoint returns an empty object <code>{}</code>, and all clients trying to validate tokens will fail.'''

'''Every OAuth2 provider MUST have:'''
* A signing key assigned: Use '''<code>authentik Internal JWT Certificate</code>''' (HS256 algorithm)
* '''NOT''' <code>authentik Self-signed Certificate</code> (RS256) — some apps like Mealie reject RS256

'''Assignment code:'''
<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.crypto.models import CertificateKeyPair

p = OAuth2Provider.objects.get(name='immich-oauth')
jwt_key = CertificateKeyPair.objects.get(name='authentik Internal JWT Certificate')
p.signing_key = jwt_key
p.save()
print(f'Assigned signing key to {p.name}')
"
</syntaxhighlight>

=== 2. Property Mappings for OAuth Scopes ===

OAuth providers need explicit property mappings for scopes like <code>email</code>, <code>profile</code>. Without them:
* OIDC discovery endpoint returns only <code>["openid"]</code>
* Apps can't get user email or name from token

'''Always add these three mappings to every OAuth provider:'''
* <code>authentik default OAuth Mapping: OpenID 'openid'</code>
* <code>authentik default OAuth Mapping: OpenID 'email'</code>
* <code>authentik default OAuth Mapping: OpenID 'profile'</code>

Also enable: <code>include_claims_in_id_token = True</code>

=== 3. LDAP Outpost Internal-Only ===

The LDAP outpost (port 389) is '''internal to Docker networking only'''. External apps can't reach it directly. Instead:
* Internal Docker containers reach it at <code>172.17.0.1:389</code>
* Other hosts on LAN reach it via Stalwart LDAP passthrough (Stalwart exposes LDAP to port 389 externally)

=== 4. Bootstrap Credentials Not Persisted ===

The <code>AUTHENTIK_BOOTSTRAP_PASSWORD</code> and <code>AUTHENTIK_BOOTSTRAP_TOKEN</code> in <code>.env</code> are only used on first startup. After that, they're ignored. If you lose them:
* You still have the <code>akadmin</code> user created from bootstrap
* If you lose the akadmin password, reset via Django shell:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.core.models import User
u = User.objects.get(username='akadmin')
u.set_password('<choose-a-new-password-here>')
u.save()
print(f'Reset password for {u.username}')
"
</syntaxhighlight>

=== 5. Secret Key Rotation (Rare but Important) ===

The <code>AUTHENTIK_SECRET_KEY</code> is used to encrypt stored secrets (OAuth credentials, etc.). '''If you change this .env value, all encrypted secrets become unreadable.''' Only change if:
* You're migrating to a new server
* You suspect key compromise

When rotating:
# Export all OAuth providers' client secrets before changing
# Update AUTHENTIK_SECRET_KEY
# Restart (secrets will need to be re-entered)

=== 6. Group Membership Propagation Delay ===

Changes to group membership (e.g., adding a user to a group) are propagated asynchronously via the worker. If an app isn't seeing the new group:
* Wait 5-10 seconds
* Restart the worker: <code>docker compose restart worker</code>
* Check the worker logs: <code>docker logs worker | tail -20</code>

=== 7. API Token Expiry ===

API tokens can have an expiry. The current claude-api-token (created 2026-02-16) is set to '''never expire'''. Check token settings:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.core.models import Token
t = Token.objects.get(identifier='claude-api-token')
print(f'Token: {t.identifier}')
print(f'Expires: {t.expires}')
"
</syntaxhighlight>

If a token expires, create a new one in the Admin UI (User → Tokens) and update all scripts/services using it.

=== 8. LDAP Bind Accounts Must Be in ldap-search Group ===

Any user/service that needs to query the LDAP directory must be a member of the '''<code>ldap-search</code>''' group. Currently:
* <code>stalwart-svc</code> — for mail server LDAP lookups
* <code>nextcloud-svc</code> — for CalDAV/CardDAV user sync

Without group membership, LDAP bind fails with "Access Denied".

----

== Verification Checklist ==

After recovery, verify:

* [ ] '''Admin UI accessible:''' <code>https://key.wilsoz.com/if/admin/</code> → login as marcus
* [ ] '''OAuth discovery endpoint working:'''
```bash
curl -s https://key.wilsoz.com/application/o/immich/.well-known/openid-configuration | jq '.scopes_supported'
# Should include: ["openid", "email", "profile"]
```
* [ ] '''JWKS endpoint has keys:'''
```bash
curl -s https://key.wilsoz.com/application/o/immich/jwks/ | jq '.keys | length'
# Should return a number > 0
```
* [ ] '''LDAP directory responding:'''
```bash
ldapsearch -H ldap://172.17.0.1:389 -D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
-w <stalwart-svc-password> -b "ou=users,dc=ldap,dc=goauthentik,dc=io" "uid=*" | grep "numResponses"
# Should return matches (or at least show successful bind)
```
* [ ] '''Users present:'''
```bash
docker exec authentik-app_server_1 ak shell -c "from authentik.core.models import User; print('\n'.join([u.username for u in User.objects.all()]))"
# Should show: akadmin, marcus, stalwart-svc, nextcloud-svc
```
* [ ] '''Groups present:'''
```bash
docker exec authentik-app_server_1 ak shell -c "from authentik.core.models import Group; print('\n'.join([g.name for g in Group.objects.all()]))"
# Should show all configured groups
```
* [ ] '''OAuth providers have signing keys:'''
```bash
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
for p in OAuth2Provider.objects.all():
has_key = p.signing_key is not None
print(f'{p.name}: signing_key={has_key}')
"
# All should show: signing_key=True
```
* [ ] '''Dependent services can authenticate:'''
** [ ] Immich: Try OAuth login at https://photos.wilsoz.com
** [ ] Linkwarden: Try OAuth login at https://bookmarks.wilsoz.com
** [ ] Memos: Try OAuth login at https://notes.wilsoz.com
** [ ] Stalwart IMAP: Test LDAP lookup for mail routing

----

== Dependencies ==

=== What Authentik Depends On ===

* '''PostgreSQL 16''' — must be healthy before server/worker start
* '''Docker''' — for container runtime
* '''Network access to Cloudflare DNS''' — for DNS resolution of <code>key.wilsoz.com</code>
* '''VPS reverse proxy''' — nginx on VPS must forward <code>key.wilsoz.com:443</code> to <code>stronghold:9000</code>

=== What Depends On Authentik ===

* '''All OAuth2 consumers:''' Immich, Memos, Linkwarden, Forgejo, Vaultwarden, Grafana, n8n, Matrix
* '''All LDAP consumers:''' Nextcloud, Jellyfin, Stalwart (mail server)
* '''Cloudflare Access:''' For gating n8n form URLs
* '''Email provisioning workflow (n8n):''' Watches Authentik groups for email user provisioning
* '''Nginx reverse proxy:''' Serves key.wilsoz.com → Authentik

----

== Operations ==

=== Start/Stop/Restart ===

<syntaxhighlight lang="bash">
cd /library/authentik-app

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart server worker

# Full restart (clears caches)
docker compose down && sleep 2 && docker compose up -d
</syntaxhighlight>

=== View Logs ===

<syntaxhighlight lang="bash">
# Server logs
docker logs -f server

# Worker logs (for background tasks)
docker logs -f worker

# PostgreSQL logs (for DB issues)
docker logs postgresql

# All together
docker compose logs -f
</syntaxhighlight>

=== Access Django Shell (Advanced) ===

For one-off admin tasks:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell

# Example: List all users
from authentik.core.models import User
for u in User.objects.all():
print(f"{u.username}: {u.email}, groups: {list(u.groups.values_list('name', flat=True))}")

# Example: Create API token
from authentik.core.models import Token
t = Token.objects.create(user=User.objects.get(username='marcus'), identifier='my-api-key')
print(f"Token: {t.key}")

# Exit: Ctrl+D
</syntaxhighlight>

=== Backup PostgreSQL Manually ===

<syntaxhighlight lang="bash">
# Full database dump
docker exec authentik-app_postgresql_1 pg_dump -U authentik authentik | \
gzip > /home/mnw/backups/authentik-manual-$(date +%Y%m%d-%H%M%S).sql.gz

# Or via Restic (for Hetzner backup)
cd /library/backup-stack && docker compose exec restic-backup bash -c "
export RESTIC_REPOSITORY=\$HETZNER_REPO
export RESTIC_PASSWORD=\$HETZNER_PASSWORD
# Make dump available in /data
docker exec authentik-app_postgresql_1 pg_dump -U authentik authentik | gzip > /data/dumps/authentik-manual.sql.gz
restic backup /data/dumps/
"
</syntaxhighlight>

----

== Troubleshooting ==

=== Can't log into admin UI ===

'''Symptoms:''' 403 Forbidden, or password rejected

'''Fixes:'''
# Check if server is running: <code>docker ps | grep server</code>
# Reset akadmin password via Django shell (see Operations section)
# Check logs: <code>docker logs server | grep -i "error\|auth"</code>

=== OAuth login fails with "invalid_grant" ===

'''Symptoms:''' Client authentication failed, or code exchange error

'''Causes & fixes:'''
* Client secret mismatch: Verify in Authentik provider matches app's .env
* Provider deleted: Recreate OAuth provider and update app config
* Redirect URI mismatch: Ensure app's callback URL matches provider's redirect URIs exactly

'''Debug:'''
<syntaxhighlight lang="bash">
docker logs server | grep -i "oauth\|invalid_grant"
</syntaxhighlight>

=== LDAP bind fails ===

'''Symptoms:''' Stalwart/Nextcloud can't authenticate users

'''Fixes:'''
# Verify LDAP outpost is running: <code>docker ps | grep outposts</code>
# Check user is in <code>ldap-search</code> group: Admin UI → Directory → Groups → ldap-search
# Test bind from stronghold:
```bash
ldapsearch -H ldap://172.17.0.1:389 -D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
-w <password> -b "ou=users" "uid=marcus"
```

=== Worker not picking up changes ===

'''Symptoms:''' Group membership not reflected, or other background tasks stuck

'''Fix:'''
<syntaxhighlight lang="bash">
docker compose restart worker
docker logs worker --tail=50
</syntaxhighlight>

=== Database corruption ===

'''Symptoms:''' PostgreSQL won't start, or "ERROR: relation does not exist"

'''Fix:'''
* Try recovery from backup (Disaster Mode B)
* If no backup: full reset (Disaster Mode A) — all data lost

----

== Links & References ==

* '''Authentik Docs:''' https://goauthentik.io/docs/
* '''Authentik Admin UI:''' https://key.wilsoz.com/if/admin/
* '''PostgreSQL Docs:''' https://www.postgresql.org/docs/16/
* '''OAuth 2.0 Spec:''' https://datatracker.ietf.org/doc/html/rfc6749
* '''OpenID Connect Spec:''' https://openid.net/connect/

----

== Related Services ==

* '''OpenBao''' (secrets store): <code>/library/openbao-app/</code> — stores API tokens
* '''Stalwart''' (mail server): Uses LDAP for user lookups
* '''Nextcloud''' (file sync): Uses LDAP for CalDAV/CardDAV
* '''n8n''' (automation): Uses OAuth for form access control
* '''PostgreSQL''' (database): Internal to Authentik
* '''Nginx''' (reverse proxy): VPS nginx forwards key.wilsoz.com traffic here

Services/Stalwart

2026-07-05T18:23:41Z

Wikibot: Recreated after purging exposed-credential revision

''As of: 2026-07-05 — migrated from services/STALWART.md. Credential values redacted per Rule 0 (see OpenBao secret/stalwart) — the source repo file still has plaintext values and needs a separate scrub + rotation, flagged separately.''

= Stalwart — Mail Server (SMTP/IMAP/LDAP/Sieve/JMAP) =

Complete setup & operations guide for Stalwart mail server with LDAP authentication, relay routing, and getmail integration.

'''Criticality:''' '''HIGH — Mail infrastructure.''' Losing Stalwart means loss of email receive/send capability. Unlike Authentik (SSO), Stalwart failure doesn't break other services, but email is critical infrastructure. Requires careful recovery.

----

== Overview ==

'''Stalwart''' is a modern open-source mail server that provides:
* '''SMTP''' (port 25 inbound, 465/587 outbound) — message transport
* '''IMAP''' (port 143/993) — message retrieval with TLS
* '''POP3''' (port 110/995) — legacy message retrieval
* '''Sieve''' (port 4190/ManageSieve) — mail filtering rules
* '''JMAP''' (port 443) — modern mail protocol (experimental)
* '''LDAP''' (LDAP passthrough to Authentik at port 389) — user authentication
* '''Web Admin''' (port 8181 internal HTTP API) — configuration and management

'''Current setup:'''
* Primary domain: <code>wilsoz.com</code> — marcus@wilsoz.com is main account
* Secondary domain: <code>craniumslows.com</code> — accounts federated to wilsoz via getmail
* Inbound mail sources: Fastmail, MXRoute, Gmail, iCloud, SDF (fetched via getmail)
* Outbound relay routes: Fastmail (wilsoz.com), MXRoute (craniumslows.com), Gmail, SDF (via proxy)
* Mail clients: eM Client (IMAP/SMTP), SnappyMail (web), Thunderbird (config compatible)

'''Current Version:''' Stalwart latest (updates automatically via Watchtower)

'''Access:'''
* Internal admin API: <code>http://localhost:8181/api</code>
* ManageSieve: <code>localhost:4190</code> (STARTTLS required)
* IMAP: <code>localhost:993</code> (TLS) or <code>mymail.wilsoz.com:993</code> (external)
* SMTP: <code>localhost:465</code> or <code>mymail.wilsoz.com:465</code> (external)

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''Compose file'''
| <code>/library/stalwart-app/docker-compose.yml</code>
|-
| '''Container name'''
| <code>stalwart-mail</code>
|-
| '''Image'''
| <code>stalwartlabs/stalwart:latest</code>
|-
| '''Network mode'''
| <code>host</code> (direct network access, not bridged)
|-
| '''Restart policy'''
| <code>unless-stopped</code>
|-
| '''Data volume'''
| <code>/library/stalwart-app/data/</code> (contains <code>/etc/</code>, <code>/queue/</code>, <code>/store/</code>)
|-
| '''TLS certificates'''
| <code>/etc/letsencrypt/live/mymail.wilsoz.com/</code> (mounted readonly)
|-
| '''Admin credentials'''
| <code>admin:<REDACTED - see OpenBao secret/stalwart></code> (plaintext password, hashed in config.toml)
|-
| '''Admin API port'''
| 8181 (internal only)
|-
| '''SMTP port'''
| 25 (inbound), 465 (outbound implicit TLS), 587 (submission STARTTLS)
|-
| '''IMAP ports'''
| 143 (STARTTLS), 993 (implicit TLS)
|-
| '''POP3 ports'''
| 110 (STARTTLS), 995 (implicit TLS)
|-
| '''Sieve port'''
| 4190 (ManageSieve, STARTTLS then PLAIN auth)
|-
| '''JMAP port'''
| 443 (via reverse proxy, not directly exposed)
|-
| '''LDAP passthrough'''
| <code>localhost:389</code> (to Authentik LDAP outpost)
|}

----

== Ports Summary ==

{| class="wikitable"
|-
! Port
! Protocol
! Direction
! Purpose
! TLS
! Notes
|-
| 25
| SMTP
| Inbound
| Receive mail from internet
| None
| MX record points here
|-
| 465
| SMTPS
| Outbound
| Send via Fastmail/MXRoute
| Implicit
| ISP allows only 465, blocks 25/587
|-
| 587
| SMTP
| Outbound
| Send via some relays
| STARTTLS
| ISP blocks this, use 465 instead
|-
| 143
| IMAP
| Internal
| Retrieve mail (legacy)
| STARTTLS
| Plain AUTH disabled, must TLS first
|-
| 993
| IMAPS
| Internal
| Retrieve mail (standard)
| Implicit
| Recommended, supports admin impersonation
|-
| 110/995
| POP3/S
| Internal
| Legacy retrieval
| —
| Available but unused
|-
| 4190
| ManageSieve
| Internal
| Upload sieve rules
| STARTTLS
| Auth only advertised post-TLS
|-
| 8181
| HTTP API
| Internal
| Admin config/settings
| None
| Internal only, no TLS
|}

----

== Initial Setup from Scratch ==

=== Prerequisites ===

* Docker and docker-compose installed
* LDAP access to Authentik LDAP outpost (must be running first)
* TLS certificate for <code>mymail.wilsoz.com</code> at <code>/etc/letsencrypt/live/mymail.wilsoz.com/</code>
* DNS MX records configured to point to Stronghold
* Outbound SMTP to external relays configured (Fastmail, MXRoute, etc.)

=== 1. Create Directory & Files ===

<syntaxhighlight lang="bash">
mkdir -p /library/stalwart-app/data
cd /library/stalwart-app
</syntaxhighlight>

=== 2. Create docker-compose.yml ===

<syntaxhighlight lang="yaml">
version: "3.8"

services:
stalwart:
image: stalwartlabs/stalwart:latest
container_name: stalwart-mail
hostname: stronghold
network_mode: host
restart: unless-stopped
volumes:
- ./data:/opt/stalwart
- /etc/letsencrypt:/opt/letsencrypt:ro
env_file:
- .env
</syntaxhighlight>

=== 3. Create .env File ===

<syntaxhighlight lang="bash">
# Admin credentials (plaintext; password gets hashed in config.toml)
STALWART_ADMIN_USER=admin
STALWART_ADMIN_PASSWORD=<generate-strong-password-20-chars>

# LDAP (Authentik LDAP outpost)
LDAP_HOST=localhost
LDAP_PORT=389
LDAP_BASE_DN="dc=ldap,dc=goauthentik,dc=io"
LDAP_USER_BASE="ou=users,dc=ldap,dc=goauthentik,dc=io"
LDAP_SERVICE_ACCOUNT="cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io"
LDAP_SERVICE_PASSWORD=<from-Authentik-stalwart-svc-user>

# Mail domains
PRIMARY_DOMAIN=wilsoz.com
SECONDARY_DOMAIN=craniumslows.com

# TLS certificate (mounted from host)
TLS_CERT_PATH=/opt/letsencrypt/live/mymail.wilsoz.com/fullchain.pem
TLS_KEY_PATH=/opt/letsencrypt/live/mymail.wilsoz.com/privkey.pem
</syntaxhighlight>

=== 4. Create config.toml (Bootstrap Configuration) ===

This is a complex TOML file. Start with a minimal bootstrap config containing only:
* Listener (SMTP, IMAP, POP3, Sieve ports)
* Storage (database and message store)
* TLS certificate paths
* Authentication (LDAP + fallback admin)
* Admin API

'''Key principle:''' All persistent settings (relay routes, routing strategy, rate limits) must be configured via the Admin HTTP API at port 8181, NOT in config.toml. The reason: <code>GET /api/reload</code> rewrites config.toml in flat key=value format, stripping all TOML sections.

<syntaxhighlight lang="toml">
# Minimal bootstrap config.toml
[server]
hostname = "stronghold"

# Listeners
[[listeners.smtp]]
bind = "0.0.0.0:25"
protocol = "smtp"

[[listeners.smtp]]
bind = "0.0.0.0:465"
protocol = "smtp"
tls.implicit = true

[[listeners.smtp]]
bind = "0.0.0.0:587"
protocol = "smtp"
tls.starttls = true

[[listeners.imap]]
bind = "0.0.0.0:143"
protocol = "imap"
tls.starttls = true

[[listeners.imap]]
bind = "0.0.0.0:993"
protocol = "imap"
tls.implicit = true

[[listeners.pop3]]
bind = "0.0.0.0:110"
protocol = "pop3"
tls.starttls = true

[[listeners.pop3]]
bind = "0.0.0.0:995"
protocol = "pop3"
tls.implicit = true

[[listeners.managesieve]]
bind = "0.0.0.0:4190"
protocol = "sieve"
tls.starttls = true

[[listeners.http]]
bind = "127.0.0.1:8181"
protocol = "http"

# TLS
[certificate.default]
cert = "/opt/letsencrypt/live/mymail.wilsoz.com/fullchain.pem"
private-key = "/opt/letsencrypt/live/mymail.wilsoz.com/privkey.pem"

# Storage (SQLite database + file-based message store)
[storage.data]
type = "sqlite"
path = "/opt/stalwart/store/data.sqlite"

[storage.fts]
type = "sqlite"
path = "/opt/stalwart/store/fts.sqlite"

[storage.directory]
type = "sqlite"
path = "/opt/stalwart/store/directory.sqlite"

[storage.lookup]
type = "sqlite"
path = "/opt/stalwart/store/lookup.sqlite"

[storage.blob]
type = "fs"
path = "/opt/stalwart/store/blobs"

[storage.queue]
type = "fs"
path = "/opt/stalwart/queue"

# Authentication
[auth]
directory = "ldap"

[[directory.ldap]]
name = "ldap"
address = "localhost"
port = 389
base-dn = "dc=ldap,dc=goauthentik,dc=io"
user-base-dn = "ou=users,dc=ldap,dc=goauthentik,dc=io"
bind-dn = "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io"
bind-password = "<from OpenBao: secret/stalwart/ldap-bind-password>" # rotated 2026-05-18
filter = "(&(objectClass=posixAccount)(cn=?))"
mail-attribute = "mail"

# Fallback admin (in case LDAP fails)
[authentication.fallback-admin]
user = "admin"
password = "$6$HFsr55uFqNc31Mw8$rLv/wtAzFCr0yNfY.../...hashed-password" # Will be generated on first startup

# Domains (minimal)
[[domains]]
name = "wilsoz.com"

[[domains]]
name = "craniumslows.com"

# Admin API
[server.admin]
bind = "127.0.0.1"
port = 8181

# Logging
[tracer.log]
level = "info"
</syntaxhighlight>

=== 5. Start Stalwart ===

<syntaxhighlight lang="bash">
cd /library/stalwart-app
docker compose up -d

# Wait for startup (~10 seconds)
sleep 10

# Check logs
docker logs stalwart-mail | tail -20

# Should see: "Stalwart Mail Server started" and port listeners binding
</syntaxhighlight>

=== 6. Configure Via Admin API (port 8181) ===

After startup, all configuration happens via the HTTP API. See "Admin API" section below for how to:
* Create mail accounts (principals)
* Configure relay routes (outbound SMTP)
* Configure routing strategy (which route to use for each sender)
* Upload sieve rules
* Set spam filter thresholds

'''Example: Create a mail account'''

<syntaxhighlight lang="bash">
curl -u admin:<REDACTED - see OpenBao secret/stalwart> -X POST \
http://localhost:8181/api/principal \
-H "Content-Type: application/json" \
-d '{
"type": "individual",
"name": "marcus@wilsoz.com",
"emails": ["marcus@wilsoz.com"],
"secrets": ["marcus-password"]
}'
</syntaxhighlight>

=== 7. Configure LDAP Sync ===

Stalwart can authenticate against Authentik LDAP, but users must exist. The simplest approach:
* Create users in Authentik → Directory → Users
* Add them to <code>wilsoz-email</code> or <code>craniumslows-email</code> groups
* Stalwart reads LDAP attributes: <code>mail</code>, <code>mailAddresses</code>, <code>memberOf</code>

Test LDAP:

<syntaxhighlight lang="bash">
ldapsearch -x -H ldap://localhost:389 \
-D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
-w "<rotated 2026-05-18; see OpenBao secret/stalwart/ldap-bind-password>" \
-b "ou=users,dc=ldap,dc=goauthentik,dc=io" \
"cn=marcus" mail mailAddresses
</syntaxhighlight>

=== 8. Set Up TLS Certificate Renewal ===

Stalwart uses certificates at <code>/opt/letsencrypt/live/mymail.wilsoz.com/</code>. When certificates renew (certbot, daily check):

# '''Create renewal hook''' to reload Stalwart:

<syntaxhighlight lang="bash">
sudo nano /etc/letsencrypt/renewal-hooks/deploy/stalwart-reload.sh
</syntaxhighlight>

<syntaxhighlight lang="bash">
#!/bin/bash
# Reload Stalwart config after certificate renewal
docker exec stalwart-mail curl -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/reload
echo "Stalwart reloaded on $(date)" >> /var/log/stalwart-reload.log
</syntaxhighlight>

<syntaxhighlight lang="bash">
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/stalwart-reload.sh
</syntaxhighlight>

# '''Test renewal:'''

<syntaxhighlight lang="bash">
sudo certbot renew --dry-run
</syntaxhighlight>

=== 9. Configure VPS Reverse Proxy (if external access needed) ===

On VPS, forward <code>mymail.wilsoz.com</code> to Stronghold. Example nginx config:

<syntaxhighlight lang="nginx">
# IMAP/SMTP passthrough (not HTTP)
# This requires HAProxy or similar (not standard nginx)

# Alternatively, use Stronghold's direct port exposure + firewall rules
</syntaxhighlight>

Actually, for mail protocols (SMTP/IMAP), most setups expose ports directly. Configure firewall rules on stronghold to allow:
* Port 25 (inbound SMTP) — public
* Port 465/587 (outbound) — already open by ISP
* Port 993 (IMAP, TLS) — expose to VPS or public

----

== Disaster Mode A: Configuration Corruption ==

=== Symptoms ===

* Can't send/receive mail
* LDAP authentication failing
* Sieve rules not executing
* Relay routes returning errors
* Database corrupted (SQLite integrity errors)
* Admin credentials lost or forgotten

=== Recovery Steps ===

==== Option 1: Reset Configuration (Keep Mail Data) ====

If only config is corrupted but mail data is intact:

<syntaxhighlight lang="bash">
cd /library/stalwart-app

# Stop Stalwart
docker compose down

# Backup old config
cp data/etc/config.toml data/etc/config.toml.corrupted-$(date +%Y%m%d)

# Copy clean bootstrap config (from Initial Setup step 4)
cp config.toml.bootstrap data/etc/config.toml

# Start fresh
docker compose up -d

# Reconfigure via API (routes, strategy, sieve, etc.)
# See Admin API section below
</syntaxhighlight>

==== Option 2: Reset Admin Password (Via Direct API) ====

If you can still reach the API but lost the admin password:

<syntaxhighlight lang="bash">
# Edit config.toml directly to set new password
# Hash password using Stalwart's built-in hash
docker exec stalwart-mail stalwartd hash-password "NewPassword123"

# Copy the hash output, edit config.toml:
[authentication.fallback-admin]
user = "admin"
password = "<paste-hash-here>"

# Restart
docker compose restart stalwart-mail
</syntaxhighlight>

==== Option 3: Full Reset (Lose All Configuration + Users) ====

⚠️ This deletes everything except mail messages themselves.

<syntaxhighlight lang="bash">
cd /library/stalwart-app
docker compose down

# Delete all config and database
rm -rf data/etc/*
rm -rf data/store/*.sqlite
mkdir -p data/etc data/store/blobs data/queue

# Copy bootstrap config
cp config.toml.bootstrap data/etc/config.toml

# Delete Docker volume (optional, if vol became corrupted)
# docker volume rm stalwart-mail_data

# Start from scratch
docker compose up -d

# Reconfigure everything:
# 1. Create relay routes
# 2. Create routing strategy
# 3. Create mail accounts
# 4. Upload sieve rules
# 5. Configure spam filters
</syntaxhighlight>

----

== Disaster Mode B: Drive Failure / Full Recovery ==

=== If Backups Accessible ===

Stalwart data is backed up:
# '''PostgreSQL dumps''' (if using PG backend) — at Hetzner <code>/dumps/stalwart-*.sql.gz</code>
# '''Restic snapshots''' — at Hetzner <code>/data/stalwart/</code>
# '''Timeshift snapshots''' — on <code>/raid1/timeshift/</code> (if /var/lib/docker/volumes issue fixed)

==== Recovery Steps (Using Restic): ====

<syntaxhighlight lang="bash">
# 1. Deploy fresh Stalwart container (follow Initial Setup)
cd /library/stalwart-app
docker compose up -d

# 2. Stop Stalwart
docker compose down

# 3. Restore mail data from Restic
cd /library/backup-stack
docker compose up -d

docker exec restic-backup restic restore latest \
--target /tmp/restore \
--repo sftp:u540853@u540853.your-storagebox.de:./backups \
--include "/data/stalwart"

# 4. Copy restored data to /library/stalwart-app
sudo cp -r /tmp/restore/data/stalwart/* /library/stalwart-app/data/
sudo chown -R 0:0 /library/stalwart-app/data/ # Stalwart runs as root in container

# 5. Start Stalwart
cd /library/stalwart-app
docker compose up -d

# 6. Verify
docker logs stalwart-mail | grep -i "listener\|directory\|ldap"
curl -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/principal | jq '.principals | length'
</syntaxhighlight>

=== If Backups NOT Accessible ===

⚠️ '''Mail data will be permanently lost.''' Recovery options:

# '''Restore from getmail sources:'''
** Getmail configs at <code>/library/email/getmail/*.rc</code> fetch from external sources
** If Hetzner or tape backup includes getmail state files, can re-fetch mail
** Mail will only go back to getmail fetch dates (~7 days, depending on retention)

# '''Request users to re-send important emails'''

# '''Restore from email client backups:'''
** If eM Client/Thunderbird cached mail, can export and reimport

# '''Full rebuild (minimal configuration, zero historical mail):'''
** Create new empty Stalwart instance
** Reconfigure routes, domains, users
** All historical mail is lost; only forward-going mail is received

----

== Data Locations ==

=== Stalwart Data on Host ===

<syntaxhighlight lang="text">
/library/stalwart-app/
├── docker-compose.yml (service definition)
├── .env (LDAP passwords, admin user)
├── config.toml.bootstrap (clean bootstrap config)
└── data/
├── etc/config.toml (runtime config, auto-rewritten by /api/reload)
├── store/
│ ├── data.sqlite (principal directory, accounts, groups)
│ ├── fts.sqlite (full-text search index)
│ ├── directory.sqlite (LDAP cache)
│ ├── lookup.sqlite (MX/DNS lookups)
│ └── blobs/ (message attachments, large objects)
└── queue/ (outbound mail queue)
</syntaxhighlight>

=== Content Storage ===

'''Messages themselves''' are stored differently based on storage backend:
* '''FileSystem (current):''' <code>/library/stalwart-app/data/store/blobs/</code> — one file per message/attachment
* '''Database:''' Could be SQLite or PostgreSQL

=== Backup Locations ===

* '''Hetzner Restic:''' <code>/data/stalwart/</code> (daily snapshots)
'' '''Local Timeshift:''' <code>/raid1/timeshift/''/library/stalwart-app/data/</code> (if /var/lib/docker/volumes included)
* '''Home directory:''' Config snippets, scripts at <code>/home/mnw/Developer/SystemResiliency/</code>

=== What's Backed Up ===

✅ All mail messages (blobs/)
✅ Account directory and settings (data.sqlite)
✅ LDAP cache (directory.sqlite)
✅ Outbound queue (queue/)
✅ Config.toml (auto-rewritten but restored)

❌ '''NOT backed up by Timeshift:''' <code>/var/lib/docker/volumes</code> (includes SQLite WAL files — '''known issue''')
✅ Backed up by Restic: All /library/stalwart-app content

----

== Critical Gotchas ==

=== 1. config.toml Rewriting Gotcha (CRITICAL) ===

'''Problem:''' When you call <code>GET /api/reload</code>, Stalwart rewrites config.toml in flat <code>key=value</code> format, '''stripping all TOML sections.'''

<syntaxhighlight lang="toml">
# Original config.toml
[server.admin]
bind = "127.0.0.1"
port = 8181

# After /api/reload
server.admin.bind = 127.0.0.1
server.admin.port = 8181
</syntaxhighlight>

'''Consequence:''' All relay routes, routing strategy, rate limits, spam settings created via API are LOST when rewriting happens.

'''Solution:'''
* '''Only store bootstrap/listener/storage/auth settings in config.toml'''
* '''All persistent settings (routes, strategy, spam) MUST live in the database via <code>/api/settings</code> API'''
* Never call <code>/api/reload</code> unless you're OK with losing API-configured settings
* If you accidentally called it, restore from backup immediately

=== 2. Routes Live Under <code>queue.route.*</code> Namespace (CRITICAL) ===

'''Wrong:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=route.
# Returns 0 results — namespace is wrong
</syntaxhighlight>

'''Right:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=queue.route.
# Returns all routes
</syntaxhighlight>

'''Lesson:''' Always use full namespace path <code>queue.route.NAME</code> not <code>route.NAME</code>.

=== 3. Settings API Format — Stalwart 0.15.4 Specific ===

'''assert_empty field is REQUIRED (and must be snake_case, NOT camelCase):'''

<syntaxhighlight lang="bash">
# WRONG — camelCase
-d '[{"type":"insert","prefix":"queue.route.fastmail","assertEmpty":false,...}]'
# Returns: 400 JSON deserialization failed

# RIGHT — snake_case
-d '[{"type":"insert","prefix":"queue.route.fastmail","assert_empty":false,...}]'
</syntaxhighlight>

'''Quirk:''' The enum <code>type</code> values are camelCase (<code>insert</code>, <code>delete</code>, <code>clear</code>), but the struct fields use snake_case due to Serde serialization rules. This is '''version-specific''' to 0.15.4. Post-0.16.0 may change.

=== 4. ISP Blocks Port 25 AND 587 Outbound — Port 465 Only ===

'''Fact:''' Your ISP (likely Comcast or similar) blocks:
* ✅ Inbound port 25 (you can receive)
* ❌ Outbound port 25 (can't relay)
* ❌ Outbound port 587 (can't relay, STARTTLS submission)
* ✅ Outbound port 465 (implicit TLS works)

'''Configuration:''' All outbound relay routes must use:
* <code>port = 465</code>
* <code>tls.implicit = true</code>
* <code>protocol = "smtp"</code>

'''Example (Fastmail relay):'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
-d '[{"type":"insert","prefix":"queue.route.fastmail-relay","assert_empty":false,"values":[
["type","relay"],
["address","smtp.fastmail.com"],
["port","465"],
["tls.implicit","true"],
["auth.username","marcus@wilsoz.com"],
["auth.secret","<fastmail-app-password>"]
]}]'
</syntaxhighlight>

=== 5. SDF Relay Requires VPS Python Proxy (SDF CRAM-MD5 auth) ===

'''Problem:''' SDF mail server only supports CRAM-MD5 authentication, which Stalwart doesn't implement.

'''Solution:''' Python proxy on VPS (<code>/opt/sdf-smtp-proxy.py</code>, systemd: <code>sdf-smtp-proxy.service</code>)
* Listens on Tailscale IP <code>100.67.169.50:2587</code> (no auth required)
* Relays to <code>mx.sdf.org:587</code> with CRAM-MD5 auth
* Credentials stored in VPS <code>/opt/secrets.env</code>

'''Stalwart config for SDF:'''
<syntaxhighlight lang="bash">
# Set /etc/hosts mapping on stronghold
100.67.169.50 sdf-relay.internal

# Then add route
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
-d '[{"type":"insert","prefix":"queue.route.sdf-relay","assert_empty":false,"values":[
["type","relay"],
["address","sdf-relay.internal"],
["port","2587"],
["tls.implicit","false"],
["tls.starttls","true"]
]}]'
</syntaxhighlight>

Also, '''config.toml must set <code>resolver.type = "system"</code>''' (not "default") so it reads <code>/etc/hosts</code>:

<syntaxhighlight lang="toml">
[resolver]
type = "system" # NOT "default"
</syntaxhighlight>

=== 6. Routing Strategy: Local Recipients FIRST (Rule 0000) ===

'''Problem:''' If rule 0000 doesn't check for local recipients, mail to <code>marcus@wilsoz.com</code> gets sent through an external relay and fails.

'''Correct strategy (rules 0000-0002):'''

<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
-d '[{"type":"insert","prefix":"queue.strategy.route","assert_empty":false,"values":[
["0000.if","is_local_domain('\'\'\'', rcpt_domain)"],
["0000.then","'\''local'\''"],
["0001.if","sender_domain == '\''wilsoz.com'\''"],
["0001.then","'\''wilsoz-fastmail'\''"],
["0002.else","'\''mx'\''"]
]}]'
</syntaxhighlight>

'''Lesson:''' ALWAYS put local-domain check as rule 0000 (executed first).

=== 7. LDAP Authentication Requires STARTTLS on Port 143 ===

'''Wrong:''' Try to use <code>LOGIN</code> auth on plain port 143
<syntaxhighlight lang="text">
imap.login('marcus', 'password') # Port 143
# Returns: AUTHENTICATIONFAILED
</syntaxhighlight>

'''Right:''' Use STARTTLS on port 143, or implicit TLS on port 993
<syntaxhighlight lang="text">
imap.starttls() # Upgrade to TLS
imap.login('marcus', 'password') # Now LOGIN is available
</syntaxhighlight>

This is a '''Stalwart security feature''' — LOGIN auth is only advertised after TLS negotiation (STARTTLS or implicit).

=== 8. ManageSieve (Sieve Rules) Authentication Quirk ===

'''Problem:''' Python <code>managesieve</code> library calls <code>login()</code> before <code>starttls()</code>. After <code>starttls()</code>, capabilities change to include <code>PLAIN</code>. Library doesn't re-read capabilities, so login fails.

'''Solution:''' Use raw sockets (see "ManageSieve" section below in Operations).

=== 9. Admin Impersonation via IMAP (May Not Work) ===

'''Syntax:''' <code>admin*username@domain</code> with admin password
<syntaxhighlight lang="text">
imap.login('admin*marcus@wilsoz.com', '<REDACTED - see OpenBao secret/stalwart>')
</syntaxhighlight>

'''Status:''' As of 2026-02-22, this returned AUTHENTICATIONFAILED. May need:
* <code>[imap] allow-plain-auth</code> config setting enabled
* Or different impersonation mechanism

'''Workaround:''' Use LDAP directly or set user passwords and log in normally.

=== 10. TLS Certificate Path Must Exist at Startup ===

'''Problem:''' If you mount <code>/etc/letsencrypt:/opt/letsencrypt:ro</code> but cert files don't exist, Stalwart refuses to start.

'''Solution:''' Before starting, verify:
<syntaxhighlight lang="bash">
ls -l /etc/letsencrypt/live/mymail.wilsoz.com/
# fullchain.pem and privkey.pem must exist

# If they don't, run certbot
sudo certbot certonly -d mymail.wilsoz.com --dns-cloudflare
</syntaxhighlight>

=== 11. Outbound Rate Limiter Example (MXRoute 300/hr) ===

MXRoute has a rate limit of ~300 emails/hour. Configure in Stalwart:

<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
-d '[{"type":"insert","prefix":"queue.limiter.outbound.mxroute","assert_empty":false,"values":[
["enable","true"],
["match","mx == '\''blizzard.mxrouting.net'\''"],
["key.0000","mx"],
["rate","300/1h"]
]}]'
</syntaxhighlight>

=== 12. LDAP Password Storage (Plaintext in .env) ===

The <code>LDAP_SERVICE_PASSWORD</code> in <code>.env</code> is stored plaintext. It's the password for <code>cn=stalwart-svc</code> (service account), which:
* Has limited permissions (ldap-search group only)
* Doesn't have login access
* Is used only for directory queries

Store securely; if compromised, change in Authentik and update .env.

----

== Verification Checklist ==

After recovery, verify:

* [ ] '''Container running:''' <code>docker ps | grep stalwart-mail</code>
* [ ] '''Listeners bound:'''
```bash
docker logs stalwart-mail | grep -i "listening\|bound"
# Should show SMTP:25, SMTP:465, IMAP:143, IMAP:993, etc.
```
* [ ] '''Admin API responding:'''
```bash
curl -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/principal | jq '.principals | length'
# Should return number of principals
```
* [ ] '''LDAP connected:'''
```bash
docker logs stalwart-mail | grep -i "ldap\|directory"
# Should show successful connections, not errors
```
* [ ] '''TLS certificate loaded:'''
```bash
docker logs stalwart-mail | grep -i "certificate\|tls"
# Should show cert paths loaded
```
* [ ] '''IMAP login works:'''
```bash
python3 -c "
import imaplib, ssl
ctx = ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
m = imaplib.IMAP4_SSL('localhost', 993)
status, data = m.login('marcus', '<password>')
print('IMAP login:', 'OK' if status == 'OK' else f'FAILED: {status}')
m.close()
"
```
* [ ] '''Relay routes configured:'''
```bash
curl -s -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/settings/list?prefix=queue.route | jq '.settings | length'
# Should return > 0 (number of configured routes)
```
* [ ] '''Routing strategy configured:'''
```bash
curl -s -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/settings/list?prefix=queue.strategy | jq '.settings | length'
# Should return > 0 (routing rules)
```
* [ ] '''Send test email:'''
```bash
# Use eM Client or <code>swaks</code> command-line tool
swaks --to marcus@wilsoz.com --from test@wilsoz.com --server localhost:465 --tls
# Should deliver without errors
```
* [ ] '''Receive test email:'''
** Send email from external account to marcus@wilsoz.com
** Check it appears in IMAP inbox
* [ ] '''Sieve rules executing:'''
```bash
# If you have sieve rules, send test email and verify filtering works
docker logs stalwart-mail | grep -i "sieve\|filter"
```

----

== Dependencies ==

=== What Stalwart Depends On ===

* '''Authentik LDAP outpost''' — for user authentication (localhost:389)
* '''TLS certificate''' — from Let's Encrypt (/etc/letsencrypt)
* '''Network access to external relays:'''
** Fastmail: smtp.fastmail.com:465
** MXRoute: blizzard.mxrouting.net:465
** Gmail: smtp.gmail.com:465
** iCloud: smtp.mail.me.com:465
** SDF: sdf-relay.internal:2587 (via proxy)
* '''DNS resolution''' — for MX lookups and relay host resolution
* '''getmail''' — for inbound mail federation (craniumslows.com forwarding)

=== What Depends On Stalwart ===

* '''SnappyMail''' — web mail interface (frontend to Stalwart IMAP)
* '''eM Client''' — desktop mail client (uses IMAP/SMTP)
* '''getmail''' — mail federation from external accounts
* '''n8n workflows''' — email provisioning/deprovisioning
* '''Mail clients''' — Thunderbird, Outlook, etc.

----

== Operations ==

=== Start/Stop/Restart ===

<syntaxhighlight lang="bash">
cd /library/stalwart-app

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart stalwart-mail

# Full restart (clears cache)
docker compose down && sleep 2 && docker compose up -d
</syntaxhighlight>

=== View Logs ===

<syntaxhighlight lang="bash">
# Real-time logs
docker logs -f stalwart-mail

# Last 50 lines
docker logs stalwart-mail --tail=50

# Filter for errors
docker logs stalwart-mail 2>&1 | grep -i "error\|failed\|fatal"

# Filter for specific component
docker logs stalwart-mail | grep -i "ldap\|imap\|smtp\|queue"
</syntaxhighlight>

=== Admin API Queries ===

<syntaxhighlight lang="bash">
ADMIN="admin:<REDACTED - see OpenBao secret/stalwart>"
BASE="http://localhost:8181/api"

# List all principals (accounts, groups)
curl -s -u "$ADMIN" "$BASE/principal"

# Get specific principal
curl -s -u "$ADMIN" "$BASE/principal/marcus@wilsoz.com"

# Reload config from disk (WARNING: see gotcha #1)
curl -s -u "$ADMIN" "$BASE/reload"

# List routes
curl -s -u "$ADMIN" "$BASE/settings/list?prefix=queue.route"

# List strategy rules
curl -s -u "$ADMIN" "$BASE/settings/list?prefix=queue.strategy"

# List spam settings
curl -s -u "$ADMIN" "$BASE/settings/list?prefix=spam-filter"

# Count messages in queue
curl -s -u "$ADMIN" "$BASE/queue" | jq '.messages | length'

# View a specific message in queue
curl -s -u "$ADMIN" "$BASE/queue/<message-id>"
</syntaxhighlight>

=== Manage Sieve Rules (ManageSieve port 4190) ===

<syntaxhighlight lang="bash">
# Upload sieve script (using helper script)
cd /home/mnw/Developer/SystemResiliency
python3 sieve/upload-sieve.py

# Or use raw socket method (see Disaster Mode B section)
</syntaxhighlight>

----

== Troubleshooting ==

=== Can't Log In via IMAP ===

'''Check LDAP:'''
<syntaxhighlight lang="bash">
ldapsearch -x -H ldap://localhost:389 \
-D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
-w "<rotated 2026-05-18; see OpenBao secret/stalwart/ldap-bind-password>" \
-b "ou=users,dc=ldap,dc=goauthentik,dc=io" \
"cn=marcus" mail
</syntaxhighlight>

'''If LDAP works but IMAP login fails:'''
* Verify using port 993 (TLS) not 143 (plain)
* Or use STARTTLS on port 143
* Check if user is in <code>wilsoz-email</code> group (not required for LDAP, but may be required by Stalwart)

'''Admin impersonation not working:'''
* Try normal login instead: <code>imap.login('marcus', '<password>')</code>
* Impersonation may need config flag changes

=== Relay Routes Not Working ===

'''Debug:'''
<syntaxhighlight lang="bash">
# Check routes exist
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=queue.route

# Check strategy rules
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=queue.strategy

# Check queue for stuck messages
curl -s -u admin:<REDACTED> http://localhost:8181/api/queue | jq '.messages[]'

# Check logs for relay errors
docker logs stalwart-mail | grep -i "relay\|route\|fastmail\|mxroute"
</syntaxhighlight>

'''Common causes:'''
* Route not configured (blank response from API)
* Strategy rule missing or malformed (0000 not checking local domain)
* TLS cert not valid (relay host rejects connection)
* Rate limit exceeded (queue backs up, check MXRoute limit)
* Port 465 blocked (ISP still blocking? test external relay directly)

=== Database Corruption ===

'''Symptoms:''' "SQLite error" in logs, or can't connect to accounts

'''Fix:'''
<syntaxhighlight lang="bash">
cd /library/stalwart-app
docker compose down

# Backup corrupted database
cp data/store/data.sqlite data/store/data.sqlite.corrupted-$(date +%Y%m%d)

# Delete corrupted DBs (queue/messages preserved)
rm data/store/*.sqlite

# Start fresh (will recreate empty DBs)
docker compose up -d

# Reconfigure via API
</syntaxhighlight>

=== Outbound Queue Stuck ===

'''Check queue:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/queue | jq '.messages | length'

# If > 100 messages stuck, investigate:
docker logs stalwart-mail | tail -100 | grep -i "reject\|error\|fail"

# Possible causes:
# - Rate limit hit (check MXRoute limit)
# - TLS cert issue with relay host
# - Firewall blocking outbound to relay
# - LDAP down (affects routing decision)
</syntaxhighlight>

=== Spam Filter Too Aggressive ===

'''Increase spam threshold:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> -X POST http://localhost:8181/api/settings \
-H "Content-Type: application/json" \
-d '[{"type":"insert","prefix":"spam-filter.score","assert_empty":false,"values":[["spam","6.0"]]}]'

# Verify
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=spam-filter
</syntaxhighlight>

----

== Links & References ==

* '''Stalwart Docs:''' https://stalwartlabs.com/docs/
* '''Stalwart Admin API:''' http://localhost:8181/admin (Web UI)
* '''Stalwart Email Blog:''' https://stalwartlabs.com/blog/
* '''LDAP Documentation:''' https://ldapwiki.com/
* '''Sieve Mail Filters:''' https://tools.ietf.org/html/rfc5228
* '''Getmail Integration:''' <code>/home/mnw/Developer/SystemResiliency/services/STALWART.md</code> (reference docs)

----

== Related Services ==

* '''Authentik''' (auth): <code>/library/authentik-app/</code> — LDAP provider, user directory
* '''SnappyMail''' (webmail): <code>/library/snappymail-app/</code> — web mail frontend
* '''getmail''' (mail federation): <code>/library/email/getmail/</code> — inbound mail syncing
* '''Let's Encrypt''' (TLS): <code>/etc/letsencrypt/</code> — certificates for TLS
* '''VPS nginx''' (reverse proxy): Exposes mymail.wilsoz.com:993 externally
* '''Tailscale''' (VPN): Required for SDF relay via proxy

Identity/AuthentikMatrix

2026-07-05T18:19:18Z

Wikibot: Migrated from INFRASTRUCTURE_INVENTORY.md auth matrix

''As of: 2026-07-05 — migrated from INFRASTRUCTURE_INVENTORY.md's Authentication Matrix section''

= How the Backup System Works =

A beginner-friendly guide to understanding your new backup system.

== The Big Picture ==

<syntaxhighlight lang="text">
Your Server (stronghold)
│
│ 1. Dump databases to files
▼
┌─────────────┐
│ PostgreSQL │──► .sql.gz files
│ Containers │ (in /library/backup-stack/dumps/)
└─────────────┘
│
│ 2. Restic reads all your data
▼
┌─────────────┐
│ Restic │──► Deduplicates, compresses, encrypts
│ Container │
└─────────────┘
│
│ 3. Uploads to cloud storage
▼
┌─────────────┐
│ Hetzner │ Your encrypted backups live here
│ Storage Box │ (in Germany, €3.81/month)
└─────────────┘
</syntaxhighlight>

== What Gets Backed Up ==

{| class="wikitable"
|-
! What
! Where It Lives
! Size
! Why It Matters
|-
| Authentik DB
| PostgreSQL dump
| 14MB
| Your SSO - lose this, lose all logins
|-
| Vaultwarden
| SQLite file
| 356KB
| Your passwords!
|-
| Immich photos
| /raid1/.../Immich-Photos
| 149GB
| Family photos
|-
| Immich DB
| PostgreSQL dump
| 437MB
| Photo metadata, faces, albums
|-
| Jellyfin
| /var/lib/jellyfin
| 92GB
| Watch history, metadata
|-
| Nextcloud
| Config + PostgreSQL
| ~1GB
| Files, calendar, contacts
|-
| Open-WebUI
| SQLite + uploads
| 1.7GB
| AI chat history
|-
| Memos, Linkwarden, Mailcow
| Various
| Small
| Notes, bookmarks, email
|-
| MediaWiki
| Config bind mount + MariaDB dump + images volume
| ~300KB
| The wilsoz wiki — infra docs, runbooks, ADRs
|}

== The Components ==

=== 1. Restic Container (<code>restic-backup</code>) ===
A small container that knows how to:
* Read your data from mounted directories
* Encrypt everything with your password
* Upload to Hetzner via SFTP
* Deduplicate (only upload changed data)

=== 2. Database Dump Script (<code>host-dump-databases.sh</code>) ===
Runs on the host (not in container) because it needs Docker access.
<syntaxhighlight lang="text">
authentik-postgresql → authentik-20260201.sql.gz
nextcloud-db → nextcloud-20260201.sql.gz
immich_postgres → immich-20260201.sql.gz
linkwarden_postgres → linkwarden-20260201.sql.gz
</syntaxhighlight>

=== 3. Backup Script (<code>backup-to-hetzner.sh</code>) ===
Runs inside the container:
# Connects to Hetzner via SSH (port 23, key auth)
# Reads all your data directories
# Uploads new/changed files
# Prunes old snapshots (keeps 7 daily, 4 weekly, 6 monthly)

=== 4. Systemd Timers ===
Schedules everything to run automatically:
* '''3:00 AM''' - Full backup
* '''6:00 AM''' - Integrity check
* '''Sunday 7:00 AM''' - Restore test

== How Restic Works (The Magic) ==

=== Deduplication ===
Restic breaks files into chunks. If you have 1000 photos and add 1 new one:
* First backup: uploads all 1000 photos (~149GB)
* Second backup: uploads only the 1 new photo (~5MB)

=== Encryption ===
Everything is encrypted with your password before leaving your server.
* Hetzner can't read your data
* Without the password, backups are useless
* '''Password:''' <code>E+mVDSjj8VsB5Feklelsq/BAJ3FmMX3E</code> (stored in .env)

=== Snapshots ===
Each backup creates a "snapshot" - a point-in-time view of your data.
<syntaxhighlight lang="text">
$ restic snapshots
ID Time Host Tags
─────────────────────────────────────────────────
a1b2c3d4 2026-02-01 23:50 stronghold full
e5f6g7h8 2026-02-02 03:00 stronghold full
...
</syntaxhighlight>

You can restore any snapshot, not just the latest.

== How to Check on Backups ==

=== Is the backup running? ===
<syntaxhighlight lang="bash">
docker logs restic-backup
</syntaxhighlight>

=== How big is my backup repo? ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic stats
</syntaxhighlight>

=== List all snapshots ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic snapshots
</syntaxhighlight>

=== Check repo integrity ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic check
</syntaxhighlight>

== How to Restore ==

=== Restore everything (disaster recovery) ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic restore latest --target /tmp/full-restore
</syntaxhighlight>

=== Restore one file ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--include "/data/vaultwarden/data/db.sqlite3"
</syntaxhighlight>

=== Restore from specific date ===
<syntaxhighlight lang="bash">
# List snapshots to find the ID
docker exec restic-backup restic snapshots

# Restore that snapshot
docker exec restic-backup restic restore a1b2c3d4 --target /tmp/restore
</syntaxhighlight>

== Important Files ==

<syntaxhighlight lang="text">
/library/backup-stack/
├── docker-compose.yml # Container definition
├── .env # Passwords and settings (SECRET!)
├── ssh-config # SSH settings for Hetzner
├── dumps/ # Database dumps live here
│ ├── authentik-20260201.sql.gz
│ ├── immich-20260201.sql.gz
│ └── ...
├── scripts/
│ ├── host-dump-databases.sh # Dumps DBs (runs on host)
│ ├── backup-to-hetzner.sh # Main backup (runs in container)
│ └── validate-backups.sh # Tests backups
└── systemd/ # Timer definitions
</syntaxhighlight>

== What Happens If... ==

=== Power goes out during backup? ===
Restic is crash-safe. Next backup continues from where it left off.

=== Hetzner goes down? ===
Your local data is fine. Backups resume when Hetzner is back.

=== You delete a file by accident? ===
Restore it from a previous snapshot.

=== Your server dies completely? ===
# Get new server
# Install Docker
# Connect to Hetzner
# Run <code>restic restore latest</code>
# Rebuild containers with restored data

=== You forget the password? ===
'''Game over.''' The backups are encrypted. No password = no data.
Save it somewhere safe (Vaultwarden, password manager, printed paper in safe).

== Costs ==

{| class="wikitable"
|-
! Item
! Cost
|-
| Hetzner Storage Box 1TB
| €3.81/month
|-
| Everything else
| Free (self-hosted)
|}

== Summary ==

# '''Every night at 3 AM:'''
** Databases get dumped to .sql.gz files
** Restic encrypts and uploads everything to Germany
** Old backups get cleaned up automatically

# '''Every morning at 6 AM:'''
** System checks backup integrity
** Alerts you via ntfy if something's wrong

# '''Every Sunday:'''
** Does a test restore to verify backups actually work

Your data is now protected. Sleep well.

----

= Disaster Recovery Runbook =

= Disaster Recovery Runbook =

'''Last Updated:''' 2026-06-21
'''Print this and store in a safe place.'''

:'''Keep DR current:''' after adding/changing/removing any service, run
:<code>scripts/dr-drift-check.sh</code> and act on whatever it flags. See
:<code>DR_MAINTENANCE.md</code> for the full workflow. Don't skip this — gradual drift
:is how DR plans silently rot until the day you actually need them.

----

== Quick Reference ==

=== Stronghold (primary) ===

{| class="wikitable"
|-
! Item
! Value
|-
| '''Hetzner Storage Box'''
| <code>u540853.your-storagebox.de</code> port '''23'''
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups</code> (port 23 comes from SSH config, '''not''' the URL — see note)
|-
| '''SSH Key'''
| <code>~/.ssh/id_rsa</code> (on stronghold)
|-
| '''Restic Password'''
| In <code>/library/backup-stack/.env</code> (<code>HETZNER_PASSWORD</code>)
|-
| '''Backup Schedule'''
| Daily 03:00 CST
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly
|}

=== VPS (wilsoz.com) ===

{| class="wikitable"
|-
! Item
! Value
|-
| '''Hetzner Storage Box'''
| same box, separate repo
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups-vps</code> (port 23 comes from SSH config, '''not''' the URL — see note)
|-
| '''SSH Key'''
| <code>~/.ssh/id_ed25519</code> (on VPS)
|-
| '''Restic Password'''
| same <code>HETZNER_PASSWORD</code> — stored in <code>/etc/restic-vps.env</code> on VPS
|-
| '''Backup Schedule'''
| Daily 04:30 UTC (offset from stronghold)
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly
|-
| '''Covers'''
| Forgejo data + repos, Atuin (pg_dump), ntfy, nginx, Let's Encrypt certs, <code>/opt/</code> custom scripts
|}

:'''⚠️ SFTP repo URL gotcha (port vs. path):''' In restic's single-colon SFTP form
:(<code>sftp:user@host:path</code>) '''everything after the host colon is the repository path''' —
:there is no port in the URL. The storage box's port '''23''' must be set in SSH config
:(<code>~/.ssh/config</code> → <code>Port 23</code> for host <code>u540853.your-storagebox.de</code>), which both
:stronghold and the <code>restic-backup</code> container already have. Writing the port into the
:URL as <code>:23/backups</code> makes restic look for a repo at the literal path <code>23/backups</code> and
:fail with ''"repository does not exist"''. Always use the relative <code>:./backups</code> /
:<code>:./backups-vps</code> form. (This exact mistake caused false "VPS: no snapshots found"
:alerts from the n8n ''Backup Status Alert (SSH)'' workflow until 2026-06-29.)

----

== What's Backed Up ==

=== Service Data (files) ===

{| class="wikitable"
|-
! Path in Backup
! Source on Host
! Contents
|-
| <code>/data/authentik</code>
| <code>/library/authentik-app/</code>
| SSO/identity provider config
|-
| <code>/data/vaultwarden</code>
| <code>/library/vaultwarden-app/</code>
| Password vault (SQLite DB + attachments)
|-
| <code>/data/nextcloud</code>
| <code>/library/nextcloud-app/</code>
| File sync config + data
|-
| <code>/data/immich-config</code>
| <code>/library/immich-app/</code>
| Photo manager config
|-
| <code>/data/immich-photos</code>
| <code>/raid1/Consolidated/Picture/Immich-Photos/</code>
| All photos (149G)
|-
| <code>/data/jellyfin</code>
| <code>/var/lib/jellyfin/</code>
| Media server metadata (92G)
|-
| <code>/data/jellyfin-etc</code>
| <code>/etc/jellyfin/</code>
| Jellyfin system config
|-
| <code>/data/memos</code>
| <code>/library/memo-app/</code>
| Notes app
|-
| <code>/data/linkwarden</code>
| <code>/library/linkwarden-app/</code>
| Bookmark manager
|-
| <code>/data/open-webui</code>
| <code>/library/open-webui-app/</code>
| AI chat interface
|-
| <code>/data/matrix-chat</code>
| <code>/library/matrix-chat/</code>
| Synapse + bridge configs + signing key
|-
| <code>/data/stalwart</code>
| <code>/library/stalwart-app/data/</code>
| Mail server data
|-
| <code>/data/snappymail</code>
| <code>/library/snappymail-app/data/</code>
| Webmail data
|-
| <code>/data/mailcow</code>
| <code>/opt/mailcow-dockerized/</code>
| Legacy mail (to be archived)
|-
| <code>/data/home</code>
| <code>/home/mnw/</code>
| Home dir: dotfiles, scripts, Documents, source, SSH/GPG keys (selective excludes)
|-
| <code>/data/developer</code>
| <code>/home/mnw/Developer/</code>
| All dev projects (7.9G) — also covered by <code>/data/home</code>
|-
| <code>/data/beets</code>
| <code>/raid1/my-tunes/.beets/</code>
| Music library DB
|-
| <code>/data/openbao</code>
| <code>/library/openbao-app/</code>
| OpenBao secret store — '''incl. <code>.init-output.json</code> (root token + unseal key) and <code>.scripts-token</code> / <code>.claude-token</code>.''' Bootstrap-critical: must be restored and unsealed ''before'' services that depend on secrets.
|-
| <code>/data/ezbookkeeping</code>
| <code>/library/ezbookkeeping-app/data/</code>
| Personal finance tracker (SQLite + uploads)
|-
| <code>/data/omada</code>
| <code>/library/omada-app/</code>
| Omada SDN controller — switch/AP config & topology
|}

=== Database Dumps (in <code>/dumps/</code>) ===

{| class="wikitable"
|-
! File Pattern
! Source Container
! Database
|-
| <code>authentik-YYYYMMDD.sql.gz</code>
| <code>authentik-app_postgresql_1</code>
| Authentik identity DB
|-
| <code>nextcloud-YYYYMMDD.sql.gz</code>
| <code>nextcloud-db</code>
| Nextcloud DB
|-
| <code>immich-YYYYMMDD.sql.gz</code>
| <code>immich_postgres</code>
| Immich photo DB (~460MB compressed)
|-
| <code>linkwarden-YYYYMMDD.sql.gz</code>
| <code>linkwarden_postgres</code>
| Linkwarden bookmark DB
|-
| <code>matrix-YYYYMMDD.sql.gz</code>
| <code>matrix-postgres</code>
| Synapse + bridge DBs (~1MB compressed)
|}

=== NOT Backed Up (re-acquirable) ===

* Jellyfin media files (movies, TV)
* Ollama models (~20G, re-downloadable)
* Steam games
* VirtualBox VMs
* node_modules, .venv, .git/objects
* Wyoming voice models (<code>piper-data-app</code>, <code>whisper-data-app</code>) — redownloadable
* Prometheus metric history, Grafana dashboards data, uptime-kuma — re-buildable from compose

=== ⚠️ Known Gaps — Data Not Currently Backed Up ===

{| class="wikitable"
|-
! Severity
! What
! Where
! Notes
|-
| '''HIGH'''
| AudioBookshelf config + metadata
| <code>/usr/share/audiobookshelf/config/</code> (38M), <code>/usr/share/audiobookshelf/metadata/</code> (936M)
| systemd service, not Docker — not yet bind-mounted into restic container. Lose library DB, progress, and cover art on rebuild.
|}

Previously flagged gaps (all closed as of 2026-06-21): Nextcloud user files, Matrix media, Mealie, n8n named volumes, uptime-kuma, Grafana, AdGuard — all now covered by bind mounts in the restic container compose.

----

== Recovery Scenarios ==

=== Scenario 1: Single Service Failure ===

'''Example:''' Immich database corrupted, need to restore just Immich.

<syntaxhighlight lang="bash">
# 1. Stop the service
cd /library/immich-app && docker compose down

# 2. List available snapshots
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

# 3. Restore service data from latest snapshot
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
restic restore latest --target /tmp/restore --include /data/immich-config'

# 4. Copy restored data to host
docker cp restic-backup:/tmp/restore/data/immich-config /tmp/immich-restore

# 5. Replace the broken data
mv /library/immich-app /library/immich-app.broken
mv /tmp/immich-restore /library/immich-app

# 6. Restore the database dump
docker cp restic-backup:/tmp/restore/dumps/immich-YYYYMMDD.sql.gz /tmp/
gunzip /tmp/immich-YYYYMMDD.sql.gz
# Start just the database container
docker compose up -d immich_postgres
# Wait for it to be ready, then restore
docker exec -i immich_postgres psql -U postgres immich < /tmp/immich-YYYYMMDD.sql

# 7. Start the full service
docker compose up -d

# 8. Clean up
docker exec restic-backup rm -rf /tmp/restore
</syntaxhighlight>

=== Scenario 2: Full Server Rebuild ===

'''Starting from:''' Fresh Linux install on new hardware.

==== Bootstrap Order (DO NOT SHUFFLE) ====

Many services depend on OpenBao (secret store) and Authentik (SSO). Bring them
up in this order so each layer has its dependencies in place:

<syntaxhighlight lang="text">
1. Host: Docker, restic, Tailscale, SSH key
2. Restore everything from Hetzner via restic (raw files)
3. OpenBao ← unseal it first; it holds tokens other services need
4. Authentik (DB dump → start) ← SSO for everything
5. Vaultwarden ← human-facing creds; needed for me-the-operator
6. Restore Claude Code access (bw CLI, MCP wrappers) ← so future-you can drive recovery
7. Postgres-backed services (Immich, Nextcloud, Matrix, Linkwarden): DB dump → start
8. Stalwart + SnappyMail (mail)
9. Everything else (n8n, Grafana, Mealie, etc.)
</syntaxhighlight>

==== Step 1: Install Prerequisites ====

<syntaxhighlight lang="bash">
# Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker mnw

# Install restic
sudo apt install restic

# Set Docker MTU for Tailscale compatibility
# Add to /etc/docker/daemon.json:
# {"mtu": 1280}
sudo systemctl restart docker
</syntaxhighlight>

==== Step 2: Restore SSH Keys ====

You need the SSH key to access Hetzner. If you have a copy:

<syntaxhighlight lang="bash">
mkdir -p ~/.ssh
# Copy id_rsa from your backup/safe location
chmod 600 ~/.ssh/id_rsa
</syntaxhighlight>

If you don't have the key, generate a new one and add it to Hetzner:
<syntaxhighlight lang="bash">
ssh-keygen -t rsa -b 4096
# Add public key via Hetzner Robot panel → Storage Box → SSH Keys
</syntaxhighlight>

==== Step 3: Bootstrap Restic ====

<syntaxhighlight lang="bash">
# Create the backup stack directory
sudo mkdir -p /library/backup-stack
cd /library/backup-stack

# Create .env with Hetzner credentials:
cat > .env << 'EOF'
HETZNER_REPO=sftp:u540853@u540853.your-storagebox.de:23/backups
HETZNER_PASSWORD=<your-restic-password>
NTFY_TOKEN=tk_0fhigfiglbjw9r8sk9jv4hpr7q67f
EOF

# Test connection
export $(grep -v '^#' .env | xargs)
restic -r "$HETZNER_REPO" snapshots
</syntaxhighlight>

==== Step 4: Restore Everything ====

<syntaxhighlight lang="bash">
# Create target directories
sudo mkdir -p /library /raid1/Consolidated/Picture/Immich-Photos

# Restore all service data
restic -r "$HETZNER_REPO" restore latest --target /

# This puts files at their original paths:
# /data/authentik → needs moving to /library/authentik-app/
# /data/developer → needs moving to /home/mnw/Developer/
# etc.

# Move restored data to correct locations
sudo mv /data/authentik /library/authentik-app
sudo mv /data/vaultwarden /library/vaultwarden-app
sudo mv /data/nextcloud /library/nextcloud-app
sudo mv /data/immich-config /library/immich-app
sudo mv /data/immich-photos/* /raid1/Consolidated/Picture/Immich-Photos/
sudo mv /data/jellyfin /var/lib/jellyfin
sudo mv /data/jellyfin-etc /etc/jellyfin
sudo mv /data/memos /library/memo-app
sudo mv /data/linkwarden /library/linkwarden-app
sudo mv /data/open-webui /library/open-webui-app
sudo mv /data/matrix-chat /library/matrix-chat
sudo mv /data/stalwart /library/stalwart-app/data
sudo mv /data/snappymail /library/snappymail-app/data
sudo mv /data/developer /home/mnw/Developer
sudo mv /data/beets /raid1/my-tunes/.beets
sudo chown -R mnw:mnw /home/mnw/Developer
</syntaxhighlight>

==== Step 4b: Bootstrap OpenBao (Secret Store) — REQUIRED before DBs ====

OpenBao holds tokens and secrets that downstream services + scripts read at startup.

<syntaxhighlight lang="bash">
# 1. Move restored data into place
sudo mv /data/openbao /library/openbao-app
sudo chown -R 1000:1000 /library/openbao-app # OpenBao runs as uid 1000

# 2. Start the container
cd /library/openbao-app && docker compose up -d

# 3. Unseal — uses the unseal key from .init-output.json (restored from backup)
sudo /usr/local/bin/openbao-unseal.sh
# If that script isn't restored yet, manually:
# ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
# UNSEAL_KEY=$(jq -r .unseal_keys_b64[0] /library/openbao-app/.init-output.json)
# docker exec openbao bao operator unseal "$UNSEAL_KEY"

# 4. Install the auto-unseal systemd unit so it survives reboots
sudo install -m 0755 scripts/openbao/openbao-unseal.sh /usr/local/bin/
sudo install -m 0644 scripts/openbao/openbao-unseal.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now openbao-unseal.service

# 5. Verify by reading a known secret
TOKEN=$(cat /library/openbao-app/.scripts-token)
curl -sf http://127.0.0.1:8200/v1/secret/data/ntfy -H "X-Vault-Token: $TOKEN" | jq .data.data
</syntaxhighlight>

'''If <code>.init-output.json</code> is lost:''' OpenBao is unrecoverable without it. Print
it now (or keep it in a sealed envelope) — it contains the root token AND the
unseal key. Without these, you cannot decrypt anything in the Raft store.

==== Step 5: Restore Databases ====

<syntaxhighlight lang="bash">
# For each service, start just the DB container, restore the dump, then start the rest.
# Example for Authentik:
cd /library/authentik-app
docker compose up -d postgresql
sleep 10
gunzip -c /dumps/authentik-YYYYMMDD.sql.gz | docker exec -i authentik-app_postgresql_1 psql -U authentik authentik
docker compose up -d

# Repeat for: nextcloud, immich, linkwarden, matrix
</syntaxhighlight>

==== Step 6: Restore System Services ====

<syntaxhighlight lang="bash">
# Jellyfin (systemd, not Docker)
sudo apt install jellyfin
# Data already restored to /var/lib/jellyfin and /etc/jellyfin

# AudioBookshelf
# Install from package, data at /usr/share/audiobookshelf
</syntaxhighlight>

==== Step 6b: Restore Claude Code Access (Vaultwarden MCP) ====

Once Vaultwarden is up (Step 5 + nginx routed), restore the bw CLI and MCP
wrappers so a Claude session can help you finish recovery from this point on.

<syntaxhighlight lang="bash">
# 1. Install bw CLI (snap is the official Bitwarden source)
sudo snap install bw

# 2. Install Bitwarden MCP server (npm, globally)
npm install -g @bitwarden/mcp-server

# 3. Restore wrappers — these are in /data/home/.config/claude/ from backup
mkdir -p ~/.config/claude
sudo cp /data/home/.config/claude/{vaultwarden.env,bw-mcp-start.sh,bw-get.py} ~/.config/claude/
chmod 600 ~/.config/claude/vaultwarden.env
chmod 700 ~/.config/claude/bw-mcp-start.sh ~/.config/claude/bw-get.py

# 4. Verify bw can talk to Vaultwarden (not the public Bitwarden cloud!)
bw config server # MUST print https://vault.wilsoz.com — if not, run: bw config server https://vault.wilsoz.com
bw login --apikey # uses BW_CLIENTID + BW_CLIENTSECRET from env

# 5. Wire MCPs into ~/.claude.json (vault + bitwarden entries)
# See services/VAULTWARDEN.md for the exact JSON structure if it needs to be rebuilt.
</syntaxhighlight>

See <code>services/VAULTWARDEN.md</code> for full setup including the <code>mcp-server-bitwarden</code>
symlink bug workaround.

==== Step 7: Re-establish Networking ====

<syntaxhighlight lang="bash">
# Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# VPS nginx configs point to Tailscale IPs — update if IP changes
# VPS SSH: ssh -p 1981 mnw@wilsoz.com
</syntaxhighlight>

=== Scenario 3: Restore Specific Files ===

<syntaxhighlight lang="bash">
# Restore a single file from a specific snapshot
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
restic restore SNAPSHOT_ID --target /tmp/restore --include /data/developer/SystemResiliency/STATUS.md'

docker cp restic-backup:/tmp/restore/data/developer/SystemResiliency/STATUS.md /tmp/
</syntaxhighlight>

=== Scenario 4: Matrix Chat Recovery ===

Matrix requires special care — the signing key is critical.

<syntaxhighlight lang="bash">
# 1. Restore Matrix data
restic restore latest --target /tmp/restore --include /data/matrix-chat

# 2. Critical files:
# - synapse/signing.key — IRREPLACEABLE, needed for federation
# - synapse/homeserver.yaml — server config
# - bridges/*/config.yaml — bridge configs
# - bridges/*/registration.yaml — bridge registrations

# 3. Restore database
gunzip -c /dumps/matrix-YYYYMMDD.sql.gz | docker exec -i matrix-postgres psql -U synapse

# 4. Restart stack
cd /library/matrix-chat && docker compose up -d
</syntaxhighlight>

----

== Verification Commands ==

<syntaxhighlight lang="bash">
# List all snapshots
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

# Check repository integrity
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic check'

# Check repo size on Hetzner
ssh -p 23 u540853@u540853.your-storagebox.de "du -sh backups/"

# List files in a snapshot
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
restic ls latest /data/matrix-chat'

# Verify a DB dump
gunzip -t /library/backup-stack/dumps/immich-YYYYMMDD.sql.gz && echo "OK" || echo "CORRUPT"

# Manual backup trigger
sudo /library/backup-stack/scripts/host-dump-databases.sh
docker exec restic-backup /scripts/backup-to-hetzner.sh
</syntaxhighlight>

----

== Automated Monitoring ==

{| class="wikitable"
|-
! Check
! Frequency
! Alert Channel
|-
| Backup age (n8n)
| Every 6h
| ntfy (nc700x-alerts) + Matrix
|-
| Integrity check (restic)
| Daily 06:00
| ntfy if FAIL
|-
| Restore test (restic)
| Weekly Sunday 07:00
| ntfy if FAIL
|-
| Disk space (n8n)
| Every 4h
| ntfy (nc700x-alerts) + Matrix
|}

----

== Key Credentials ==

'''Keep a copy of these offline:'''

{| class="wikitable"
|-
! Credential
! Location
|-
| Hetzner SSH key
| <code>~/.ssh/id_rsa</code> on stronghold
|-
| Restic password
| <code>/library/backup-stack/.env</code>
|-
| Hetzner account
| Robot panel, u540853
|-
| ntfy token
| <code>tk_0fhigfiglbjw9r8sk9jv4hpr7q67f</code>
|-
| Authentik admin
| <code>https://key.wilsoz.com/if/admin/</code>
|-
| VPS SSH
| <code>ssh -p 1981 mnw@wilsoz.com</code>
|-
| Synapse shared secret
| In <code>homeserver.yaml</code> line 50
|-
| Matrix signing key
| <code>/library/matrix-chat/synapse/signing.key</code> — '''IRREPLACEABLE'''
|}

----

== Restore Test Log ==

=== 2026-02-23 — False Failure Alerts + Upload Anomaly Detection ===

'''Problem 1 — False failure alert after meta-array removal:'''
<code>validate-backups.sh</code> still contained a <code>restic check</code> block against <code>$METAARRAY_REPO</code> / <code>$METAARRAY_PASSWORD</code>. Both vars were removed from <code>.env</code> when meta-array was dropped on 2026-02-22. Empty repo var → restic failed → <code>ERRORS++</code> → ntfy <code>nc700x-alerts</code> fired every day.

'''Fix:''' Removed the dead meta-array block from <code>validate-backups.sh</code> (lines 44–52).

'''Problem 2 — Large upload on first run after container recreation:'''
Restic container was recreated during meta-array removal, clearing the local cache. Without cache, restic reported "no parent snapshot found" and re-scanned all 441,884 files (all showed as "new"). Chunk dedup still worked correctly — only 11.5 GiB of 314 GiB was actually uploaded. Confirmed by upload anomaly report: Stalwart ~8k new email files (ongoing Fastmail getmail import) + <code>/data/home</code> (all new to the repo without a parent reference). Normal next run.

'''Enhancement — Upload anomaly detection:'''
Added <code>detect_upload_anomaly()</code> to <code>run-backup.sh</code>:
* Tracks upload size per run in <code>/dumps/upload-history.txt</code> (rolling 30 entries)
* If upload >2.5× rolling average AND >500 MiB, runs <code>restic diff <prev> <cur></code> and summarises changes by service path
* POSTs JSON payload to n8n webhook (<code>whO5LBZHQHCXXG9I</code>) which formats and sends to Matrix
* ntfy not used — this is informational, not an emergency

----

=== 2026-02-22 — Recurring SSH Timeout Incident ===

'''Problem:''' Backups failing with <code>subprocess ssh: Timeout</code> / <code>Fatal: unable to save snapshot: context canceled</code>. Hetzner SFTP server drops idle connections (~5 min) during restic's local scan phase (22+ min). Compounded by 96 accumulated index files (documented threshold: 49) because prune was also failing.

'''Secondary cause removed:''' meta-array (ma.sdf.org) secondary backup target was unreliable. Since <code>run-backup.sh</code> used <code>set -e</code>, any metaarray failure marked the entire job as failed even when Hetzner backup succeeded. Removed entirely.

'''Fixes applied:'''
# <code>restic repair index</code> — consolidated 96 → 3 index files
# <code>restic forget --prune</code> — cleaned up retention
# Added <code>-o sftp.args="-oServerAliveInterval=10 -oServerAliveCountMax=36"</code> to <code>backup-to-hetzner.sh</code> — 10s keepalives, 6-min tolerance
# Added <code>--retry-lock 10m</code> to <code>backup-to-hetzner.sh</code>
# Removed <code>backup-to-metaarray.sh</code> and all meta-array config
# Added <code>/home/mnw</code> → <code>/data/home</code> to backup scope (dotfiles, Documents, source, SSH keys)

'''If SSH timeouts recur:''' It means index files are accumulating again (prune is failing silently). Run:
<syntaxhighlight lang="bash">
docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic repair index'
docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune'
</syntaxhighlight>

----

=== 2026-02-16 — First Manual Test ===

'''Test:''' Restore files from latest snapshot (ae89d11d), compare against live originals.

{| class="wikitable"
|-
! File
! Restored
! Compared
! Result
|-
| <code>matrix-chat/synapse/homeserver.yaml</code>
| 3,022 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/synapse/signing.key</code>
| 59 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/docker-compose.yml</code>
| 3,232 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/bridges/whatsapp/config.yaml</code>
| 33,990 bytes
| vs live
| IDENTICAL
|-
| <code>developer/SystemResiliency/STATUS.md</code>
| Restored
| pre-edit version
| Expected diff (edited today)
|-
| DB dump: <code>immich-20260213.sql.gz</code>
| 463 MB
| gunzip -t
| VALID
|-
| DB dump: <code>matrix-20260215.sql.gz</code>
| 1 MB
| gunzip -t
| VALID
|-
| DB dump: <code>authentik-20260213.sql.gz</code>
| 16 MB
| gunzip -t
| VALID
|}

'''Issues Found:'''
* DB dumps from Feb 14-15 were 20 bytes (empty) — container's <code>run-backup.sh</code> was calling <code>dump-databases.sh</code> inside the container where <code>docker</code> CLI doesn't exist, overwriting the good host-side dumps.
* '''Fixed:''' Deployed updated <code>run-backup.sh</code> that verifies dumps instead of re-dumping. Fresh dumps created manually and confirmed valid.

'''Verdict:''' PASS — restore process works. Files are intact and match live originals.

BackupsDR/Overview

2026-07-05T18:18:36Z

Wikibot: Migrated from backup-stack/HOW_IT_WORKS.md + DISASTER_RECOVERY.md

''As of: 2026-07-05 — migrated from `backup-stack/HOW_IT_WORKS.md` and `DISASTER_RECOVERY.md`. Canonical DR home is `~/Developer/DisasterRecovery/` — this page is the operational summary, not the full cold-start bootstrap guide.''

= How the Backup System Works =

A beginner-friendly guide to understanding your new backup system.

== The Big Picture ==

<syntaxhighlight lang="text">
Your Server (stronghold)
│
│ 1. Dump databases to files
▼
┌─────────────┐
│ PostgreSQL │──► .sql.gz files
│ Containers │ (in /library/backup-stack/dumps/)
└─────────────┘
│
│ 2. Restic reads all your data
▼
┌─────────────┐
│ Restic │──► Deduplicates, compresses, encrypts
│ Container │
└─────────────┘
│
│ 3. Uploads to cloud storage
▼
┌─────────────┐
│ Hetzner │ Your encrypted backups live here
│ Storage Box │ (in Germany, €3.81/month)
└─────────────┘
</syntaxhighlight>

== What Gets Backed Up ==

{| class="wikitable"
|-
! What
! Where It Lives
! Size
! Why It Matters
|-
| Authentik DB
| PostgreSQL dump
| 14MB
| Your SSO - lose this, lose all logins
|-
| Vaultwarden
| SQLite file
| 356KB
| Your passwords!
|-
| Immich photos
| /raid1/.../Immich-Photos
| 149GB
| Family photos
|-
| Immich DB
| PostgreSQL dump
| 437MB
| Photo metadata, faces, albums
|-
| Jellyfin
| /var/lib/jellyfin
| 92GB
| Watch history, metadata
|-
| Nextcloud
| Config + PostgreSQL
| ~1GB
| Files, calendar, contacts
|-
| Open-WebUI
| SQLite + uploads
| 1.7GB
| AI chat history
|-
| Memos, Linkwarden, Mailcow
| Various
| Small
| Notes, bookmarks, email
|-
| MediaWiki
| Config bind mount + MariaDB dump + images volume
| ~300KB
| The wilsoz wiki — infra docs, runbooks, ADRs
|}

== The Components ==

=== 1. Restic Container (<code>restic-backup</code>) ===
A small container that knows how to:
* Read your data from mounted directories
* Encrypt everything with your password
* Upload to Hetzner via SFTP
* Deduplicate (only upload changed data)

=== 2. Database Dump Script (<code>host-dump-databases.sh</code>) ===
Runs on the host (not in container) because it needs Docker access.
<syntaxhighlight lang="text">
authentik-postgresql → authentik-20260201.sql.gz
nextcloud-db → nextcloud-20260201.sql.gz
immich_postgres → immich-20260201.sql.gz
linkwarden_postgres → linkwarden-20260201.sql.gz
</syntaxhighlight>

=== 3. Backup Script (<code>backup-to-hetzner.sh</code>) ===
Runs inside the container:
# Connects to Hetzner via SSH (port 23, key auth)
# Reads all your data directories
# Uploads new/changed files
# Prunes old snapshots (keeps 7 daily, 4 weekly, 6 monthly)

=== 4. Systemd Timers ===
Schedules everything to run automatically:
* '''3:00 AM''' - Full backup
* '''6:00 AM''' - Integrity check
* '''Sunday 7:00 AM''' - Restore test

== How Restic Works (The Magic) ==

=== Deduplication ===
Restic breaks files into chunks. If you have 1000 photos and add 1 new one:
* First backup: uploads all 1000 photos (~149GB)
* Second backup: uploads only the 1 new photo (~5MB)

=== Encryption ===
Everything is encrypted with your password before leaving your server.
* Hetzner can't read your data
* Without the password, backups are useless
* '''Password:''' <code>E+mVDSjj8VsB5Feklelsq/BAJ3FmMX3E</code> (stored in .env)

=== Snapshots ===
Each backup creates a "snapshot" - a point-in-time view of your data.
<syntaxhighlight lang="text">
$ restic snapshots
ID Time Host Tags
─────────────────────────────────────────────────
a1b2c3d4 2026-02-01 23:50 stronghold full
e5f6g7h8 2026-02-02 03:00 stronghold full
...
</syntaxhighlight>

You can restore any snapshot, not just the latest.

== How to Check on Backups ==

=== Is the backup running? ===
<syntaxhighlight lang="bash">
docker logs restic-backup
</syntaxhighlight>

=== How big is my backup repo? ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic stats
</syntaxhighlight>

=== List all snapshots ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic snapshots
</syntaxhighlight>

=== Check repo integrity ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic check
</syntaxhighlight>

== How to Restore ==

=== Restore everything (disaster recovery) ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic restore latest --target /tmp/full-restore
</syntaxhighlight>

=== Restore one file ===
<syntaxhighlight lang="bash">
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--include "/data/vaultwarden/data/db.sqlite3"
</syntaxhighlight>

=== Restore from specific date ===
<syntaxhighlight lang="bash">
# List snapshots to find the ID
docker exec restic-backup restic snapshots

# Restore that snapshot
docker exec restic-backup restic restore a1b2c3d4 --target /tmp/restore
</syntaxhighlight>

== Important Files ==

<syntaxhighlight lang="text">
/library/backup-stack/
├── docker-compose.yml # Container definition
├── .env # Passwords and settings (SECRET!)
├── ssh-config # SSH settings for Hetzner
├── dumps/ # Database dumps live here
│ ├── authentik-20260201.sql.gz
│ ├── immich-20260201.sql.gz
│ └── ...
├── scripts/
│ ├── host-dump-databases.sh # Dumps DBs (runs on host)
│ ├── backup-to-hetzner.sh # Main backup (runs in container)
│ └── validate-backups.sh # Tests backups
└── systemd/ # Timer definitions
</syntaxhighlight>

== What Happens If... ==

=== Power goes out during backup? ===
Restic is crash-safe. Next backup continues from where it left off.

=== Hetzner goes down? ===
Your local data is fine. Backups resume when Hetzner is back.

=== You delete a file by accident? ===
Restore it from a previous snapshot.

=== Your server dies completely? ===
# Get new server
# Install Docker
# Connect to Hetzner
# Run <code>restic restore latest</code>
# Rebuild containers with restored data

=== You forget the password? ===
'''Game over.''' The backups are encrypted. No password = no data.
Save it somewhere safe (Vaultwarden, password manager, printed paper in safe).

== Costs ==

{| class="wikitable"
|-
! Item
! Cost
|-
| Hetzner Storage Box 1TB
| €3.81/month
|-
| Everything else
| Free (self-hosted)
|}

== Summary ==

# '''Every night at 3 AM:'''
** Databases get dumped to .sql.gz files
** Restic encrypts and uploads everything to Germany
** Old backups get cleaned up automatically

# '''Every morning at 6 AM:'''
** System checks backup integrity
** Alerts you via ntfy if something's wrong

# '''Every Sunday:'''
** Does a test restore to verify backups actually work

Your data is now protected. Sleep well.

----

= Disaster Recovery Runbook =

= Disaster Recovery Runbook =

'''Last Updated:''' 2026-06-21
'''Print this and store in a safe place.'''

:'''Keep DR current:''' after adding/changing/removing any service, run
:<code>scripts/dr-drift-check.sh</code> and act on whatever it flags. See
:<code>DR_MAINTENANCE.md</code> for the full workflow. Don't skip this — gradual drift
:is how DR plans silently rot until the day you actually need them.

----

== Quick Reference ==

=== Stronghold (primary) ===

{| class="wikitable"
|-
! Item
! Value
|-
| '''Hetzner Storage Box'''
| <code>u540853.your-storagebox.de</code> port '''23'''
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups</code> (port 23 comes from SSH config, '''not''' the URL — see note)
|-
| '''SSH Key'''
| <code>~/.ssh/id_rsa</code> (on stronghold)
|-
| '''Restic Password'''
| In <code>/library/backup-stack/.env</code> (<code>HETZNER_PASSWORD</code>)
|-
| '''Backup Schedule'''
| Daily 03:00 CST
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly
|}

=== VPS (wilsoz.com) ===

{| class="wikitable"
|-
! Item
! Value
|-
| '''Hetzner Storage Box'''
| same box, separate repo
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups-vps</code> (port 23 comes from SSH config, '''not''' the URL — see note)
|-
| '''SSH Key'''
| <code>~/.ssh/id_ed25519</code> (on VPS)
|-
| '''Restic Password'''
| same <code>HETZNER_PASSWORD</code> — stored in <code>/etc/restic-vps.env</code> on VPS
|-
| '''Backup Schedule'''
| Daily 04:30 UTC (offset from stronghold)
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly
|-
| '''Covers'''
| Forgejo data + repos, Atuin (pg_dump), ntfy, nginx, Let's Encrypt certs, <code>/opt/</code> custom scripts
|}

:'''⚠️ SFTP repo URL gotcha (port vs. path):''' In restic's single-colon SFTP form
:(<code>sftp:user@host:path</code>) '''everything after the host colon is the repository path''' —
:there is no port in the URL. The storage box's port '''23''' must be set in SSH config
:(<code>~/.ssh/config</code> → <code>Port 23</code> for host <code>u540853.your-storagebox.de</code>), which both
:stronghold and the <code>restic-backup</code> container already have. Writing the port into the
:URL as <code>:23/backups</code> makes restic look for a repo at the literal path <code>23/backups</code> and
:fail with ''"repository does not exist"''. Always use the relative <code>:./backups</code> /
:<code>:./backups-vps</code> form. (This exact mistake caused false "VPS: no snapshots found"
:alerts from the n8n ''Backup Status Alert (SSH)'' workflow until 2026-06-29.)

----

== What's Backed Up ==

=== Service Data (files) ===

{| class="wikitable"
|-
! Path in Backup
! Source on Host
! Contents
|-
| <code>/data/authentik</code>
| <code>/library/authentik-app/</code>
| SSO/identity provider config
|-
| <code>/data/vaultwarden</code>
| <code>/library/vaultwarden-app/</code>
| Password vault (SQLite DB + attachments)
|-
| <code>/data/nextcloud</code>
| <code>/library/nextcloud-app/</code>
| File sync config + data
|-
| <code>/data/immich-config</code>
| <code>/library/immich-app/</code>
| Photo manager config
|-
| <code>/data/immich-photos</code>
| <code>/raid1/Consolidated/Picture/Immich-Photos/</code>
| All photos (149G)
|-
| <code>/data/jellyfin</code>
| <code>/var/lib/jellyfin/</code>
| Media server metadata (92G)
|-
| <code>/data/jellyfin-etc</code>
| <code>/etc/jellyfin/</code>
| Jellyfin system config
|-
| <code>/data/memos</code>
| <code>/library/memo-app/</code>
| Notes app
|-
| <code>/data/linkwarden</code>
| <code>/library/linkwarden-app/</code>
| Bookmark manager
|-
| <code>/data/open-webui</code>
| <code>/library/open-webui-app/</code>
| AI chat interface
|-
| <code>/data/matrix-chat</code>
| <code>/library/matrix-chat/</code>
| Synapse + bridge configs + signing key
|-
| <code>/data/stalwart</code>
| <code>/library/stalwart-app/data/</code>
| Mail server data
|-
| <code>/data/snappymail</code>
| <code>/library/snappymail-app/data/</code>
| Webmail data
|-
| <code>/data/mailcow</code>
| <code>/opt/mailcow-dockerized/</code>
| Legacy mail (to be archived)
|-
| <code>/data/home</code>
| <code>/home/mnw/</code>
| Home dir: dotfiles, scripts, Documents, source, SSH/GPG keys (selective excludes)
|-
| <code>/data/developer</code>
| <code>/home/mnw/Developer/</code>
| All dev projects (7.9G) — also covered by <code>/data/home</code>
|-
| <code>/data/beets</code>
| <code>/raid1/my-tunes/.beets/</code>
| Music library DB
|-
| <code>/data/openbao</code>
| <code>/library/openbao-app/</code>
| OpenBao secret store — '''incl. <code>.init-output.json</code> (root token + unseal key) and <code>.scripts-token</code> / <code>.claude-token</code>.''' Bootstrap-critical: must be restored and unsealed ''before'' services that depend on secrets.
|-
| <code>/data/ezbookkeeping</code>
| <code>/library/ezbookkeeping-app/data/</code>
| Personal finance tracker (SQLite + uploads)
|-
| <code>/data/omada</code>
| <code>/library/omada-app/</code>
| Omada SDN controller — switch/AP config & topology
|}

=== Database Dumps (in <code>/dumps/</code>) ===

{| class="wikitable"
|-
! File Pattern
! Source Container
! Database
|-
| <code>authentik-YYYYMMDD.sql.gz</code>
| <code>authentik-app_postgresql_1</code>
| Authentik identity DB
|-
| <code>nextcloud-YYYYMMDD.sql.gz</code>
| <code>nextcloud-db</code>
| Nextcloud DB
|-
| <code>immich-YYYYMMDD.sql.gz</code>
| <code>immich_postgres</code>
| Immich photo DB (~460MB compressed)
|-
| <code>linkwarden-YYYYMMDD.sql.gz</code>
| <code>linkwarden_postgres</code>
| Linkwarden bookmark DB
|-
| <code>matrix-YYYYMMDD.sql.gz</code>
| <code>matrix-postgres</code>
| Synapse + bridge DBs (~1MB compressed)
|}

=== NOT Backed Up (re-acquirable) ===

* Jellyfin media files (movies, TV)
* Ollama models (~20G, re-downloadable)
* Steam games
* VirtualBox VMs
* node_modules, .venv, .git/objects
* Wyoming voice models (<code>piper-data-app</code>, <code>whisper-data-app</code>) — redownloadable
* Prometheus metric history, Grafana dashboards data, uptime-kuma — re-buildable from compose

=== ⚠️ Known Gaps — Data Not Currently Backed Up ===

{| class="wikitable"
|-
! Severity
! What
! Where
! Notes
|-
| '''HIGH'''
| AudioBookshelf config + metadata
| <code>/usr/share/audiobookshelf/config/</code> (38M), <code>/usr/share/audiobookshelf/metadata/</code> (936M)
| systemd service, not Docker — not yet bind-mounted into restic container. Lose library DB, progress, and cover art on rebuild.
|}

Previously flagged gaps (all closed as of 2026-06-21): Nextcloud user files, Matrix media, Mealie, n8n named volumes, uptime-kuma, Grafana, AdGuard — all now covered by bind mounts in the restic container compose.

----

== Recovery Scenarios ==

=== Scenario 1: Single Service Failure ===

'''Example:''' Immich database corrupted, need to restore just Immich.

<syntaxhighlight lang="bash">
# 1. Stop the service
cd /library/immich-app && docker compose down

# 2. List available snapshots
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

# 3. Restore service data from latest snapshot
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
restic restore latest --target /tmp/restore --include /data/immich-config'

# 4. Copy restored data to host
docker cp restic-backup:/tmp/restore/data/immich-config /tmp/immich-restore

# 5. Replace the broken data
mv /library/immich-app /library/immich-app.broken
mv /tmp/immich-restore /library/immich-app

# 6. Restore the database dump
docker cp restic-backup:/tmp/restore/dumps/immich-YYYYMMDD.sql.gz /tmp/
gunzip /tmp/immich-YYYYMMDD.sql.gz
# Start just the database container
docker compose up -d immich_postgres
# Wait for it to be ready, then restore
docker exec -i immich_postgres psql -U postgres immich < /tmp/immich-YYYYMMDD.sql

# 7. Start the full service
docker compose up -d

# 8. Clean up
docker exec restic-backup rm -rf /tmp/restore
</syntaxhighlight>

=== Scenario 2: Full Server Rebuild ===

'''Starting from:''' Fresh Linux install on new hardware.

==== Bootstrap Order (DO NOT SHUFFLE) ====

Many services depend on OpenBao (secret store) and Authentik (SSO). Bring them
up in this order so each layer has its dependencies in place:

<syntaxhighlight lang="text">
1. Host: Docker, restic, Tailscale, SSH key
2. Restore everything from Hetzner via restic (raw files)
3. OpenBao ← unseal it first; it holds tokens other services need
4. Authentik (DB dump → start) ← SSO for everything
5. Vaultwarden ← human-facing creds; needed for me-the-operator
6. Restore Claude Code access (bw CLI, MCP wrappers) ← so future-you can drive recovery
7. Postgres-backed services (Immich, Nextcloud, Matrix, Linkwarden): DB dump → start
8. Stalwart + SnappyMail (mail)
9. Everything else (n8n, Grafana, Mealie, etc.)
</syntaxhighlight>

==== Step 1: Install Prerequisites ====

<syntaxhighlight lang="bash">
# Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker mnw

# Install restic
sudo apt install restic

# Set Docker MTU for Tailscale compatibility
# Add to /etc/docker/daemon.json:
# {"mtu": 1280}
sudo systemctl restart docker
</syntaxhighlight>

==== Step 2: Restore SSH Keys ====

You need the SSH key to access Hetzner. If you have a copy:

<syntaxhighlight lang="bash">
mkdir -p ~/.ssh
# Copy id_rsa from your backup/safe location
chmod 600 ~/.ssh/id_rsa
</syntaxhighlight>

If you don't have the key, generate a new one and add it to Hetzner:
<syntaxhighlight lang="bash">
ssh-keygen -t rsa -b 4096
# Add public key via Hetzner Robot panel → Storage Box → SSH Keys
</syntaxhighlight>

==== Step 3: Bootstrap Restic ====

<syntaxhighlight lang="bash">
# Create the backup stack directory
sudo mkdir -p /library/backup-stack
cd /library/backup-stack

# Create .env with Hetzner credentials:
cat > .env << 'EOF'
HETZNER_REPO=sftp:u540853@u540853.your-storagebox.de:23/backups
HETZNER_PASSWORD=<your-restic-password>
NTFY_TOKEN=tk_0fhigfiglbjw9r8sk9jv4hpr7q67f
EOF

# Test connection
export $(grep -v '^#' .env | xargs)
restic -r "$HETZNER_REPO" snapshots
</syntaxhighlight>

==== Step 4: Restore Everything ====

<syntaxhighlight lang="bash">
# Create target directories
sudo mkdir -p /library /raid1/Consolidated/Picture/Immich-Photos

# Restore all service data
restic -r "$HETZNER_REPO" restore latest --target /

# This puts files at their original paths:
# /data/authentik → needs moving to /library/authentik-app/
# /data/developer → needs moving to /home/mnw/Developer/
# etc.

# Move restored data to correct locations
sudo mv /data/authentik /library/authentik-app
sudo mv /data/vaultwarden /library/vaultwarden-app
sudo mv /data/nextcloud /library/nextcloud-app
sudo mv /data/immich-config /library/immich-app
sudo mv /data/immich-photos/* /raid1/Consolidated/Picture/Immich-Photos/
sudo mv /data/jellyfin /var/lib/jellyfin
sudo mv /data/jellyfin-etc /etc/jellyfin
sudo mv /data/memos /library/memo-app
sudo mv /data/linkwarden /library/linkwarden-app
sudo mv /data/open-webui /library/open-webui-app
sudo mv /data/matrix-chat /library/matrix-chat
sudo mv /data/stalwart /library/stalwart-app/data
sudo mv /data/snappymail /library/snappymail-app/data
sudo mv /data/developer /home/mnw/Developer
sudo mv /data/beets /raid1/my-tunes/.beets
sudo chown -R mnw:mnw /home/mnw/Developer
</syntaxhighlight>

==== Step 4b: Bootstrap OpenBao (Secret Store) — REQUIRED before DBs ====

OpenBao holds tokens and secrets that downstream services + scripts read at startup.

<syntaxhighlight lang="bash">
# 1. Move restored data into place
sudo mv /data/openbao /library/openbao-app
sudo chown -R 1000:1000 /library/openbao-app # OpenBao runs as uid 1000

# 2. Start the container
cd /library/openbao-app && docker compose up -d

# 3. Unseal — uses the unseal key from .init-output.json (restored from backup)
sudo /usr/local/bin/openbao-unseal.sh
# If that script isn't restored yet, manually:
# ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
# UNSEAL_KEY=$(jq -r .unseal_keys_b64[0] /library/openbao-app/.init-output.json)
# docker exec openbao bao operator unseal "$UNSEAL_KEY"

# 4. Install the auto-unseal systemd unit so it survives reboots
sudo install -m 0755 scripts/openbao/openbao-unseal.sh /usr/local/bin/
sudo install -m 0644 scripts/openbao/openbao-unseal.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now openbao-unseal.service

# 5. Verify by reading a known secret
TOKEN=$(cat /library/openbao-app/.scripts-token)
curl -sf http://127.0.0.1:8200/v1/secret/data/ntfy -H "X-Vault-Token: $TOKEN" | jq .data.data
</syntaxhighlight>

'''If <code>.init-output.json</code> is lost:''' OpenBao is unrecoverable without it. Print
it now (or keep it in a sealed envelope) — it contains the root token AND the
unseal key. Without these, you cannot decrypt anything in the Raft store.

==== Step 5: Restore Databases ====

<syntaxhighlight lang="bash">
# For each service, start just the DB container, restore the dump, then start the rest.
# Example for Authentik:
cd /library/authentik-app
docker compose up -d postgresql
sleep 10
gunzip -c /dumps/authentik-YYYYMMDD.sql.gz | docker exec -i authentik-app_postgresql_1 psql -U authentik authentik
docker compose up -d

# Repeat for: nextcloud, immich, linkwarden, matrix
</syntaxhighlight>

==== Step 6: Restore System Services ====

<syntaxhighlight lang="bash">
# Jellyfin (systemd, not Docker)
sudo apt install jellyfin
# Data already restored to /var/lib/jellyfin and /etc/jellyfin

# AudioBookshelf
# Install from package, data at /usr/share/audiobookshelf
</syntaxhighlight>

==== Step 6b: Restore Claude Code Access (Vaultwarden MCP) ====

Once Vaultwarden is up (Step 5 + nginx routed), restore the bw CLI and MCP
wrappers so a Claude session can help you finish recovery from this point on.

<syntaxhighlight lang="bash">
# 1. Install bw CLI (snap is the official Bitwarden source)
sudo snap install bw

# 2. Install Bitwarden MCP server (npm, globally)
npm install -g @bitwarden/mcp-server

# 3. Restore wrappers — these are in /data/home/.config/claude/ from backup
mkdir -p ~/.config/claude
sudo cp /data/home/.config/claude/{vaultwarden.env,bw-mcp-start.sh,bw-get.py} ~/.config/claude/
chmod 600 ~/.config/claude/vaultwarden.env
chmod 700 ~/.config/claude/bw-mcp-start.sh ~/.config/claude/bw-get.py

# 4. Verify bw can talk to Vaultwarden (not the public Bitwarden cloud!)
bw config server # MUST print https://vault.wilsoz.com — if not, run: bw config server https://vault.wilsoz.com
bw login --apikey # uses BW_CLIENTID + BW_CLIENTSECRET from env

# 5. Wire MCPs into ~/.claude.json (vault + bitwarden entries)
# See services/VAULTWARDEN.md for the exact JSON structure if it needs to be rebuilt.
</syntaxhighlight>

See <code>services/VAULTWARDEN.md</code> for full setup including the <code>mcp-server-bitwarden</code>
symlink bug workaround.

==== Step 7: Re-establish Networking ====

<syntaxhighlight lang="bash">
# Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# VPS nginx configs point to Tailscale IPs — update if IP changes
# VPS SSH: ssh -p 1981 mnw@wilsoz.com
</syntaxhighlight>

=== Scenario 3: Restore Specific Files ===

<syntaxhighlight lang="bash">
# Restore a single file from a specific snapshot
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
restic restore SNAPSHOT_ID --target /tmp/restore --include /data/developer/SystemResiliency/STATUS.md'

docker cp restic-backup:/tmp/restore/data/developer/SystemResiliency/STATUS.md /tmp/
</syntaxhighlight>

=== Scenario 4: Matrix Chat Recovery ===

Matrix requires special care — the signing key is critical.

<syntaxhighlight lang="bash">
# 1. Restore Matrix data
restic restore latest --target /tmp/restore --include /data/matrix-chat

# 2. Critical files:
# - synapse/signing.key — IRREPLACEABLE, needed for federation
# - synapse/homeserver.yaml — server config
# - bridges/*/config.yaml — bridge configs
# - bridges/*/registration.yaml — bridge registrations

# 3. Restore database
gunzip -c /dumps/matrix-YYYYMMDD.sql.gz | docker exec -i matrix-postgres psql -U synapse

# 4. Restart stack
cd /library/matrix-chat && docker compose up -d
</syntaxhighlight>

----

== Verification Commands ==

<syntaxhighlight lang="bash">
# List all snapshots
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic snapshots'

# Check repository integrity
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; restic check'

# Check repo size on Hetzner
ssh -p 23 u540853@u540853.your-storagebox.de "du -sh backups/"

# List files in a snapshot
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
restic ls latest /data/matrix-chat'

# Verify a DB dump
gunzip -t /library/backup-stack/dumps/immich-YYYYMMDD.sql.gz && echo "OK" || echo "CORRUPT"

# Manual backup trigger
sudo /library/backup-stack/scripts/host-dump-databases.sh
docker exec restic-backup /scripts/backup-to-hetzner.sh
</syntaxhighlight>

----

== Automated Monitoring ==

{| class="wikitable"
|-
! Check
! Frequency
! Alert Channel
|-
| Backup age (n8n)
| Every 6h
| ntfy (nc700x-alerts) + Matrix
|-
| Integrity check (restic)
| Daily 06:00
| ntfy if FAIL
|-
| Restore test (restic)
| Weekly Sunday 07:00
| ntfy if FAIL
|-
| Disk space (n8n)
| Every 4h
| ntfy (nc700x-alerts) + Matrix
|}

----

== Key Credentials ==

'''Keep a copy of these offline:'''

{| class="wikitable"
|-
! Credential
! Location
|-
| Hetzner SSH key
| <code>~/.ssh/id_rsa</code> on stronghold
|-
| Restic password
| <code>/library/backup-stack/.env</code>
|-
| Hetzner account
| Robot panel, u540853
|-
| ntfy token
| <code>tk_0fhigfiglbjw9r8sk9jv4hpr7q67f</code>
|-
| Authentik admin
| <code>https://key.wilsoz.com/if/admin/</code>
|-
| VPS SSH
| <code>ssh -p 1981 mnw@wilsoz.com</code>
|-
| Synapse shared secret
| In <code>homeserver.yaml</code> line 50
|-
| Matrix signing key
| <code>/library/matrix-chat/synapse/signing.key</code> — '''IRREPLACEABLE'''
|}

----

== Restore Test Log ==

=== 2026-02-23 — False Failure Alerts + Upload Anomaly Detection ===

'''Problem 1 — False failure alert after meta-array removal:'''
<code>validate-backups.sh</code> still contained a <code>restic check</code> block against <code>$METAARRAY_REPO</code> / <code>$METAARRAY_PASSWORD</code>. Both vars were removed from <code>.env</code> when meta-array was dropped on 2026-02-22. Empty repo var → restic failed → <code>ERRORS++</code> → ntfy <code>nc700x-alerts</code> fired every day.

'''Fix:''' Removed the dead meta-array block from <code>validate-backups.sh</code> (lines 44–52).

'''Problem 2 — Large upload on first run after container recreation:'''
Restic container was recreated during meta-array removal, clearing the local cache. Without cache, restic reported "no parent snapshot found" and re-scanned all 441,884 files (all showed as "new"). Chunk dedup still worked correctly — only 11.5 GiB of 314 GiB was actually uploaded. Confirmed by upload anomaly report: Stalwart ~8k new email files (ongoing Fastmail getmail import) + <code>/data/home</code> (all new to the repo without a parent reference). Normal next run.

'''Enhancement — Upload anomaly detection:'''
Added <code>detect_upload_anomaly()</code> to <code>run-backup.sh</code>:
* Tracks upload size per run in <code>/dumps/upload-history.txt</code> (rolling 30 entries)
* If upload >2.5× rolling average AND >500 MiB, runs <code>restic diff <prev> <cur></code> and summarises changes by service path
* POSTs JSON payload to n8n webhook (<code>whO5LBZHQHCXXG9I</code>) which formats and sends to Matrix
* ntfy not used — this is informational, not an emergency

----

=== 2026-02-22 — Recurring SSH Timeout Incident ===

'''Problem:''' Backups failing with <code>subprocess ssh: Timeout</code> / <code>Fatal: unable to save snapshot: context canceled</code>. Hetzner SFTP server drops idle connections (~5 min) during restic's local scan phase (22+ min). Compounded by 96 accumulated index files (documented threshold: 49) because prune was also failing.

'''Secondary cause removed:''' meta-array (ma.sdf.org) secondary backup target was unreliable. Since <code>run-backup.sh</code> used <code>set -e</code>, any metaarray failure marked the entire job as failed even when Hetzner backup succeeded. Removed entirely.

'''Fixes applied:'''
# <code>restic repair index</code> — consolidated 96 → 3 index files
# <code>restic forget --prune</code> — cleaned up retention
# Added <code>-o sftp.args="-oServerAliveInterval=10 -oServerAliveCountMax=36"</code> to <code>backup-to-hetzner.sh</code> — 10s keepalives, 6-min tolerance
# Added <code>--retry-lock 10m</code> to <code>backup-to-hetzner.sh</code>
# Removed <code>backup-to-metaarray.sh</code> and all meta-array config
# Added <code>/home/mnw</code> → <code>/data/home</code> to backup scope (dotfiles, Documents, source, SSH keys)

'''If SSH timeouts recur:''' It means index files are accumulating again (prune is failing silently). Run:
<syntaxhighlight lang="bash">
docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic repair index'
docker exec restic-backup sh -c 'RESTIC_REPOSITORY="$HETZNER_REPO" RESTIC_PASSWORD="$HETZNER_PASSWORD" restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune'
</syntaxhighlight>

----

=== 2026-02-16 — First Manual Test ===

'''Test:''' Restore files from latest snapshot (ae89d11d), compare against live originals.

{| class="wikitable"
|-
! File
! Restored
! Compared
! Result
|-
| <code>matrix-chat/synapse/homeserver.yaml</code>
| 3,022 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/synapse/signing.key</code>
| 59 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/docker-compose.yml</code>
| 3,232 bytes
| vs live
| IDENTICAL
|-
| <code>matrix-chat/bridges/whatsapp/config.yaml</code>
| 33,990 bytes
| vs live
| IDENTICAL
|-
| <code>developer/SystemResiliency/STATUS.md</code>
| Restored
| pre-edit version
| Expected diff (edited today)
|-
| DB dump: <code>immich-20260213.sql.gz</code>
| 463 MB
| gunzip -t
| VALID
|-
| DB dump: <code>matrix-20260215.sql.gz</code>
| 1 MB
| gunzip -t
| VALID
|-
| DB dump: <code>authentik-20260213.sql.gz</code>
| 16 MB
| gunzip -t
| VALID
|}

'''Issues Found:'''
* DB dumps from Feb 14-15 were 20 bytes (empty) — container's <code>run-backup.sh</code> was calling <code>dump-databases.sh</code> inside the container where <code>docker</code> CLI doesn't exist, overwriting the good host-side dumps.
* '''Fixed:''' Deployed updated <code>run-backup.sh</code> that verifies dumps instead of re-dumping. Fresh dumps created manually and confirmed valid.

'''Verdict:''' PASS — restore process works. Files are intact and match live originals.

Email/Architecture

2026-07-05T18:18:17Z

Wikibot: Migrated from TODO.md email section

''As of: 2026-07-05 — migrated from `TODO.md` (Email section) and cross-referenced with [[Services/Stalwart]]''

See also: '''[[Services/Stalwart]]''' for the full mail server operational reference (admin API, ports, sieve rules, gotchas). This page covers the broader mail architecture goals and phase history.

== Email (Stalwart Migration) ==

=== Architecture goal: Multi-user family & friends mail server ===

<syntaxhighlight lang="text">
Inbound: Internet → MXRoute (per-user spool) → getmail → Stalwart (per-user inbox)
Outbound: User authenticates to Stalwart SMTP with own Authentik creds
→ Stalwart relays via user's own MXRoute credentials (per-user relay routes)
→ 400 msg/hr global rate limit to MXRoute enforced by Stalwart queue throttle
Domains: craniumslows.com now. wilsoz.com → MXRoute once craniumslows validated.
Users: Authentik LDAP → Stalwart. Custom attributes drive address provisioning.
Spam: Server-wide Stalwart sieve → spam into each user's Junk
Sieve: Per-user rules via ManageSieve (no rule affects another user)
Webmail: SnappyMail at mymail.wilsoz.com
External: Gmail/iCloud/SDF as SnappyMail additional accounts (direct IMAP, not getmail)
</syntaxhighlight>

=== User provisioning model (Authentik custom LDAP attributes) ===

Each Authentik user gets three custom attributes:
* <code>mail-name</code> — primary local part (e.g. <code>joe</code>)
* <code>mail-alias</code> — secondary local part, optional (e.g. <code>joe.smith</code>)
* <code>default-domain</code> — which domain is the primary SnappyMail identity

Groups control which domains are active:
* <code>craniumslows-email</code> → @craniumslows.com addresses
* <code>wilsoz-email</code> → @wilsoz.com addresses

Address matrix = <code>{mail-name, mail-alias} × {domains from group memberships}</code>
All addresses on a user deliver to one Stalwart inbox. Each address gets its own
MXRoute account + getmail .rc + per-user Stalwart relay route.

Address changes: old address becomes a forwarder to the new one. Inbox never moves.

=== Where the real mail data lives ===
{| class="wikitable"
|-
! Location
! What
! Size
! Action
|-
| <code>/library/email/selfhosted/mail/wilsoz.com/marcus/Maildir/</code>
| Primary mailbox — Mailcow/Dovecot, ~100 nested folders
| ~8.9GB
| '''Import to Stalwart — in progress'''
|-
| <code>/library/email/selfhosted/mail/wilsoz.com/marcus/Orig-Maildir/</code>
| Pre-Mailcow import (iRedMail/Gmail era)
| (subset)
| Verify not unique before discarding
|-
| <code>/library/email/selfhosted/mail/craniumslows.com/marcus/Maildir/</code>
| craniumslows.com mailbox
| 152K
| Import to Stalwart
|-
| <code>/var/mail/mnw</code>
| Old iRedMail mbox (pre-Mailcow era)
| 37MB
| Import after primary migration
|-
| <code>/opt/mailcow-dockerized/</code>
| Mailcow app stack only
| 49MB
| Remove after migration verified
|}

=== Phase 3 (Mail Import) — COMPLETE ✓ ===
* [x] '''Import marcus@wilsoz.com''' — 137,335 messages from Mailcow Maildir, all folders preserved
* [x] '''Import marcus@craniumslows.com''' — done (was tiny, only pre-Stalwart messages)
* [x] '''Verify in SnappyMail''' — folder hierarchy confirmed, messages accessible
* [ ] '''Import old iRedMail mbox''' — <code>/var/mail/mnw</code> (37MB) not yet done (low priority, very old mail)

=== Phase 4 (Getmail) — COMPLETE ✓ (one item remaining) ===
* [x] '''Update delivery targets''' in .rc files → Stalwart LMTP <code>localhost:24</code>
* [x] '''Create craniumslows-marcus.rc''' — fetches marcus@craniumslows.com from MXRoute, 439/439 delivered
* [x] '''Keep <code>delete = false</code>''' — correctly set; flip to true only after validating in daily use
* [x] '''Create systemd timer''' — getmail runs every 10 minutes via systemd
* [x] '''Fastmail import COMPLETE''' — 12,226/12,226 delivered (2026-02-22 ~15:52)
* [ ] ~~Configure SnappyMail additional accounts~~ — '''ARCHITECTURE CHANGED''' (see below): external accounts now use getmail, not SnappyMail connected accounts

=== Architecture decision (2026-02-22) ===
Marcus's external accounts (Gmail x2, iCloud, SDF) use '''getmail''' → Stalwart, NOT SnappyMail connected accounts. This gives a true unified inbox with sieve filtering. Phone/clients connect to one IMAP server.
* [x] '''Add getmail config: thewilson@gmail.com''' — <code>sieve/gmail-thewilson.rc</code> done. Import in progress: ~7,560/58,520 as of 2026-02-23 07:00 (~50 hrs remaining at current rate)
* [x] '''Add getmail config: craniumslows@gmail.com''' — <code>sieve/gmail-craniumslows.rc</code> done. Will start after thewilson finishes.
* [x] '''Add getmail config: marcusthewilson@me.com''' — <code>sieve/icloud-marcusthewilson.rc</code> done.
* [x] '''Add getmail config: mnw@sdf.org''' — <code>sieve/sdf-mnw.rc</code> done. <code>delete=false</code>.
* [x] '''Update systemd getmail.service''' — all 6 rc files active, daemon reloaded 2026-02-23.

=== Action needed — Phase 5 (Spam + Sieve) ===
* [x] '''Fastmail import COMPLETE''' — 12,226/12,226 (done 2026-02-22)
* [x] '''Sieve rules written''' — <code>sieve/marcus.sieve</code> ready. Decade-based numbering, all rules merged. See <code>sieve/README.md</code>.
* [x] '''Mail classification script written''' — <code>sieve/classify-inbox.py</code> done.
'' [x] '''Audit actual Stalwart folder names''' — confirmed INBOX/XX-'' naming in place.
* [x] '''Upload sieve/marcus.sieve''' — uploaded and ACTIVE via ManageSieve (confirmed 2026-02-23). Script: <code>sieve/upload-sieve.py</code>.
* [x] '''Dedup script written''' — <code>sieve/dedup.py</code>. Dry-run confirmed 6,622 dupes. Run <code>--execute</code> to purge.
* [ ] '''Run dedup + retroactive sort + empty .Junk''' — → <code>plans/sieve-dedup.md</code> (covers all three)
* [ ] '''Enable server-wide spam filter''' — after sieve rules confirmed working in daily use

=== Action needed — Phase 6 (n8n provisioner + per-user relay) ===
* [x] '''Create <code>craniumslows-email</code> and <code>wilsoz-email</code> groups''' in Authentik — done 2026-02-22; marcus in both
* [x] '''Marcus's <code>mailAddresses</code> attribute set''' — <code>{"mailAddresses": ["marcus@craniumslows.com"]}</code> — enables dual-domain delivery in Stalwart
* [x] '''Build n8n provisioner workflow''' — Matrix bot IYw0DmCuWEsiV2k9 (deployed 2026-03-01). <code>!provision</code> in #notifications room, walks through username/name/password/domain conversation. Adaptive polling: 15s during conversation, 5-min throttle when idle.
* [x] '''Add custom LDAP attributes''' to Authentik users: <code>mail-name</code>, <code>mail-alias</code>, <code>default-domain</code> (future, when adding more users) — '''done 2026-05-10''': attributes backfilled on marcus and now written by the provisioner for new users; LDAP property mappings + downstream consumers still deferred
* [x] '''Configure global MXRoute rate limit''' — Stalwart queue throttle 400 msg/hr to blizzard.mxrouting.net. → <code>plans/stalwart-helpers-fix.md</code> (Phase 5) — '''done 2026-05-10''': 400/1h limiter set 2026-05-10 via legacy prefix/values API
* [x] '''Migrate marcus to per-user relay routes''' — replace shared craniumslows-mxroute with marcus-specific route — '''done 2026-05-10''': marcus-craniumslows relay + per-sender rule added; shared craniumslows-mxroute kept as fallback

=== Action needed — jkatz@craniumslows.com cleanup ===
* [x] '''jkatz Authentik user''' — Confirmed deleted (pk:18 not found 2026-05-09). MXRoute account was never created. Consider done.

=== Action needed — Matrix deprovisioner (next session) ===
* [x] '''Build Matrix deprovisioner''' — → <code>plans/email-deprovisioner.md</code> (depends on <code>plans/stalwart-helpers-fix.md</code> + <code>plans/email-provisioner.md</code>) — '''done 2026-05-10''': deployed as jbGL9N1vXT8mB92V, end-to-end tested 2026-05-10

=== Action needed — Provisioner cleanup strategy ===
* [x] '''Define orphan .rc file retention policy''' — '''done 2026-05-10''': policy = move .rc into disabled/ with <code>.deprov-YYYY-MM-DD</code> suffix on deprovision; weekly cron (<code>/etc/cron.d/getmail-disabled-purge</code>) purges entries >30 days old. Legacy duplicates + test residue swept.

=== Action needed — Phase 7 (wilsoz.com → MXRoute) ===
* [ ] Validate craniumslows.com in daily use (give it weeks)
* [ ] Move wilsoz.com MX DNS → MXRoute
* [ ] Replace fastmail.rc with wilsoz-marcus.rc (MXRoute fetch)
* [ ] Update Stalwart wilsoz.com relay → MXRoute smarthost
* [ ] '''Flip <code>delete = true</code>''' on all getmail configs once cutover verified

=== Minor ===
* [ ] Investigate unexpected folders in SnappyMail MoveTo dropdown

=== Status (updated 2026-02-22) ===
'''Phases 1–4 complete.''' Stalwart + SnappyMail deployed, LDAP auth working with dual-domain delivery. Maildir import done (137k+ messages). Getmail running: craniumslows 439/439, Fastmail ~5.5k/12.2k in progress. <code>craniumslows-email</code>/<code>wilsoz-email</code> groups created in Authentik. '''Phase 5 (spam/sieve) and Phase 6 (n8n provisioner) are next.''' One manual step remaining: add 4 external accounts to SnappyMail UI.

----

Postmortems/2026-06-28 HA Blue Bootswitch

2026-07-05T18:17:37Z

Wikibot: Migrated from postmortems/2026-06-28-ha-blue-bootswitch.md

''As of: 2026-07-05 — migrated from `postmortems/2026-06-28-ha-blue-bootswitch.md`''

= Postmortem — Home Assistant "offline / won't get an IP" (2026-06-28) =

== Summary ==
The Home Assistant box (shante, <code>192.168.0.140</code>) had been powered off and boxed for
~weeks. On power-up it would not appear on the network — no DHCP lease, unreachable on
LAN and Tailscale. Root cause was '''not''' a network or HAOS-corruption problem: the
device's physical '''boot-mode toggle switch''' had been bumped out of the eMMC position,
so the board booted into its '''Petitboot recovery shell''' instead of loading Home
Assistant OS off the eMMC. HAOS never started, so nothing ever came up on the network.
Flipping the switch back and rebooting restored normal boot.

== Device identification (corrected) ==
The unit is a '''Home Assistant Blue = ODROID-N2+''', not a Green/Yellow as the docs had
it labeled. Identifying tells:
* '''Petitboot''' bootloader screen (Green uses Rockchip/U-Boot; Yellow uses Pi CM4 — neither uses Petitboot).
* A physical '''boot-select toggle switch''' on the front (eMMC ↔ SPI) — an ODROID-N2+ hardware feature.
* Blue translucent case; red (power) + blue (activity) LEDs visible through the sliding door.
* HDMI + single Ethernet on the back.

Docs updated: <code>INFRASTRUCTURE_INVENTORY.md</code>, <code>HOME_ASSISTANT_INTEGRATION.md</code>, <code>VLAN-Setup-Guide.md</code>.

== Timeline ==
* Device boxed unpowered for ~weeks (Tailscale "last seen 26d ago" ≈ 2026-06-02).
* Powered on; never got an IP. LAN ping/UI dead; Tailscale peer <code>offline</code>.
* 8-min network poll for <code>192.168.0.140</code> / <code>homeassistant.local</code> → nothing.
* HDMI attached → '''Petitboot''' menu + "Failed to launch Petitboot, dropping to a shell", <code>#</code> busybox prompt.
* Recognized Petitboot ⇒ bootloader couldn't find a bootable OS ⇒ boot/storage issue, not network.
* User flipped the front toggle switch and rebooted → got a login prompt; reconnected to LAN.
* HA Core first-boot was slow ("Loading data", login "something went wrong" while backend caught up). Settled after a few minutes.
* Confirmed back: Tailscale peer <code>active</code>, direct <code>192.168.0.140:36309</code>, real rx traffic.

== Root cause ==
Physical boot-mode toggle was in the SPI/recovery position (likely bumped while boxed),
so the ODROID-N2+ ran Petitboot's recovery shell instead of chain-loading HAOS from eMMC.

== Why login "spun then failed" right after recovery ==
Not an auth bug. A box stored unpowered for weeks does a slow first start — recorder/DB
catch-up, NTP clock resync, integration re-init. The frontend (port 8123) answers before
Core is ready, so login spins / "something went wrong" until the backend finishes. Fix:
'''wait 5–15 min, don't reboot mid-startup.'''

== Diagnostics that worked ==
<syntaxhighlight lang="bash">
# Is it on the network at all?
ping -c2 192.168.0.140
curl -sS -m8 -o /dev/null -w "%{http_code}\n" http://192.168.0.140:8123/

# Is Tailscale seeing it? ("offline, last seen Nd ago" = box truly down, not just LAN/DHCP)
tailscale status | grep -i homeassistant

# At the Petitboot # shell, check whether OS storage is even detected:
cat /proc/partitions
</syntaxhighlight>

== Prevention / follow-ups ==
* '''Leave the front toggle switch in the eMMC (HAOS-boot) position.''' If the device ever
shows the Petitboot screen again, that switch is the first thing to check.
* If it's stored again, label/tape the switch position so it isn't bumped.
* Consider noting the correct switch orientation physically on the case.

Postmortems/2026-05-24 Ultraplan Exposure

2026-07-05T18:17:37Z

Wikibot: Migrated from postmortems/2026-05-24-ultraplan-exposure.md

''As of: 2026-07-05 — migrated from `postmortems/2026-05-24-ultraplan-exposure.md`''

= Postmortem & Remediation Playbook: Ultraplan upload exposure (2026-05-24) =

'''Status:''' Remediation in progress. This document is BOTH the postmortem
and the step-by-step playbook for executing the remediation manually.

== Incident summary ==

While working on the tsbuffer project plan in Claude Code, the <code>/ultraplan</code>
command was invoked. <code>/ultraplan</code> uploads the current working directory
plus <code>CLAUDE.md</code> and <code>MEMORY.md</code> to an Anthropic-hosted "Claude Code on
the web" session for remote planning by an agent there.

The remote session terminated with <code>error_during_execution</code> before
producing any output. The uploaded content remains in Anthropic cloud
session storage tied to session ID <code>01GUjtsXpaAw1CqHShfePPmq</code>.

'''No warning was surfaced about the upload before it occurred.''' This is
the root cause of the exposure — there is no broken security control to
fix; the tool worked exactly as designed, and we did not know what it did.

== What was in the upload ==

=== Live credentials (rotation required) ===

{| class="wikitable"
|-
! Item
! Where it was
! Why it's bad
|-
| n8n API JWT
| <code>CLAUDE.md</code>, "n8n API" section, pasted in plaintext, no expiry
| Anyone with this can call any n8n API endpoint as admin
|-
| n8n admin UI password (<code>Htdml0fI1aZgtPBVv5oz</code>)
| <code>MEMORY.md</code>, n8n DB Recovery memory entry
| Direct UI login as <code>admin@wilsoz.com</code>
|}

=== Topology / metadata exposed (no rotation possible, hygiene only) ===

* Internal domains: <code>agent.wilsoz.com</code>, <code>vault.wilsoz.com</code>, <code>key.wilsoz.com</code>,
<code>photos.wilsoz.com</code>, <code>ntfy.wilsoz.com</code>, <code>mymail.wilsoz.com</code>,
<code>next.wilsoz.com</code>, <code>wilsoz.cloudflareaccess.com</code>.
* Internal IPs: <code>192.168.0.109</code> (stronghold), <code>192.168.0.144</code> (ollama),
Docker bridges, Tailscale IP <code>100.67.169.50</code> for the SDF relay VPS.
* OpenBao endpoint and secret path layout (paths only, no tokens).
* Vaultwarden item-naming convention ("Stronghold Infrastructure" folder).
* MXRoute server <code>blizzard.mxrouting.net</code>, username <code>craniums</code>.
* Authentik admin URL + admin usernames <code>akadmin</code>, <code>marcus</code>; Cloudflare
Access OIDC client ID.
* Matrix bot @zordon:wilsoz.com, room ID, internal Synapse endpoint.
* All n8n workflow JSONs in <code>n8n-workflows/</code> and every service doc in
<code>services/</code>.
* Email addresses including <code>marcus@wilsoz.com</code>, <code>admin@wilsoz.com</code>,
<code>jkatz@craniumslows.com</code>.

== Top-level constraints for this remediation ==

# '''The credentials store IS the root of trust.''' Vaultwarden + OpenBao
are how we manage every other secret. Compromising either means
compromising everything else.
# '''Never paste a secret into chat, ever.''' Not even temporarily, not
even "just to verify." Once it's in the conversation, it's leaked
exactly the same way the original JWT was.
# '''Verify each new credential works BEFORE deleting the old one.''' A
half-rotated credential where the new one doesn't work is worse than
the original exposure.
# '''No secret values in command-line arguments.''' They show up in
<code>/proc/PID/cmdline</code> and <code>ps auxw</code> output. Use stdin, env vars, or
token files mode 0600.
# '''Two-person rule (where applicable).''' For high-blast-radius
rotations (root tokens, master password) — don't rotate without
confirming the recovery path first.

----

== Remediation steps (manual playbook) ==

Each step has: (a) what to do, (b) how to verify, (c) what to update,
(d) rollback.

=== Step 1. Rotate n8n API JWT ===

'''(a) Action:'''
* Open https://agent.wilsoz.com in a browser; log in as <code>admin@wilsoz.com</code>.
Password: retrieve from Vaultwarden item "n8n UI Login". (This is the
old password — Step 2 will rotate it. The order matters: API key first,
because once we change the admin password we want to verify with the
API that everything's healthy.)
* Settings → n8n API → see the existing key (do NOT copy its value).
* Click "Revoke" on the existing key. Confirm.
* Click "Create an API key". Set "Time to live" to "No expiration".
Copy the new JWT once shown (the UI shows it exactly once).
* '''Paste the new JWT directly into a Vaultwarden item''' — open
Vaultwarden in another tab, edit item "n8n API Key", paste, save. Do
not paste it into a terminal, a text file, or any chat.

'''(b) Verify:'''
<syntaxhighlight lang="bash">
# Retrieve the new key from Vaultwarden via the bw CLI (NOT by pasting):
NEW_KEY=$(bw get item "n8n API Key" | jq -r '.fields[] | select(.name=="apiKey") | .value')
# (Field name may differ — adjust per your existing Vaultwarden item layout.)
curl -sf "https://agent.wilsoz.com/api/v1/workflows" \
-H "X-N8N-API-KEY: $NEW_KEY" \
| jq '.data | length'
# Expect: a number (count of workflows), no errors.
unset NEW_KEY
</syntaxhighlight>

'''(c) Update:'''
* Vaultwarden item "n8n API Key": done above.
* OpenBao at <code>secret/n8n</code> field <code>api-key</code>:
```bash
# Read the new key from Vaultwarden into a shell var (no echo):
NEW_KEY=$(bw get item "n8n API Key" | jq -r '.fields[] | select(.name=="apiKey") | .value')
# Write to OpenBao using stdin (NOT command-line arg):
printf '%s' "$NEW_KEY" | BAO_TOKEN="$(cat /library/openbao-app/.claude-mcp-token)" \
bao kv put -mount=secret n8n api-key=-
unset NEW_KEY
```
* <code>~/Developer/SystemResiliency/CLAUDE.md</code>: replace the JWT-bearing line
with: <code>Retrieve from Vaultwarden item: "n8n API Key"</code>. Remove the
<code>N8N_KEY='...'</code> example line that contains the actual JWT in the
curl examples; replace with a <code># N8N_KEY=$(bw get item "n8n API Key" | ...)</code>
pattern.
* Commit CLAUDE.md change. Verify the diff contains NO JWT value before
committing (<code>git diff CLAUDE.md</code> and visually scan).

'''(d) Rollback:'''
If verification fails, the old JWT was already revoked at n8n — there is
no rollback to the previous value. Instead, immediately mint another new
JWT from the n8n UI and retry. The window of "no working API key" is
acceptable because no automation actively uses it during this rotation.

=== Step 2. Rotate n8n admin UI password ===

'''(a) Action:'''
* Still logged into n8n as <code>admin@wilsoz.com</code>.
* Settings → Personal → Change Password.
* Open Vaultwarden, edit item "n8n UI Login", click the password
regenerate button (32 chars, full charset). Save.
* Copy the new password from Vaultwarden.
* Paste into n8n's "New Password" field. Save.

'''(b) Verify:'''
* Sign out of n8n.
* Sign in with email <code>admin@wilsoz.com</code> and the new password.
* Expect: dashboard loads, workflows visible.

'''(c) Update:'''
* Vaultwarden item "n8n UI Login": done above.
* OpenBao at <code>secret/n8n</code> field <code>admin-password</code> (if used by anything):
check first whether anything reads it. If yes, update via stdin pattern
above. If no, skip.
* <code>MEMORY.md</code> (auto-memory): edit <code>/home/mnw/.claude/projects/-home-mnw-Developer-SystemResiliency/memory/MEMORY.md</code>.
Find the line: <code>Created new owner: admin@wilsoz.com / Htdml0fI1aZgtPBVv5oz</code>
Replace with: <code>Created new owner: admin@wilsoz.com / (rotated 2026-05-24; see Vaultwarden "n8n UI Login")</code>

'''(d) Rollback:'''
Same situation as Step 1 — old password is already invalidated. If the
new password doesn't work, n8n's password reset relies on SMTP email
which is functional. Worst case: re-init the n8n DB (same recovery we
did 2026-03-11).

=== Step 3. Audit CLAUDE.md and MEMORY.md for OTHER secrets ===

This is a one-pass sweep to catch anything I missed when triaging.

'''(a) Action:'''
<syntaxhighlight lang="bash">
cd ~/Developer/SystemResiliency
# Install gitleaks if not present:
which gitleaks || sudo apt install -y gitleaks
# Scan the project + memory dirs:
gitleaks detect --source . --no-git -v
gitleaks detect --source ~/.claude/projects/-home-mnw-Developer-SystemResiliency/memory --no-git -v
</syntaxhighlight>

'''(b) Verify:'''
* Read the findings line by line.
* For each finding: is it actually a secret, or a false positive (e.g.
an ID, a hash of a non-sensitive value)?
* True positives: add to the rotation list, restart from Step 1 for each.

'''(c) Update:'''
* Document any new findings in this file (append a "Found in audit"
subsection below).

'''(d) Rollback:'''
N/A — read-only operation.

=== Step 4. Attempt deletion of the cloud session ===

'''(a) Action:'''
* Open https://claude.ai/code/session_01GUjtsXpaAw1CqHShfePPmq in a
browser. Log in as <code>marcus@wilsoz.com</code>.
* Look for a delete / remove option in the session menu (three-dot icon,
settings gear, or similar).
* If found, click it. Confirm.
* If not found: send an email to <code>privacy@anthropic.com</code> with the session
ID and a request for deletion under data-deletion rights. Reference
https://privacy.anthropic.com. Subject:
<code>Data deletion request — session 01GUjtsXpaAw1CqHShfePPmq</code>.

'''(b) Verify:'''
* If UI delete worked: refresh the session URL, expect 404 or
"session not found".
* If email request: save the sent message; expect a reply within 30 days
per their stated SLA.

'''(c) Update:'''
* Append the outcome to this file: "Cloud session deletion: [UI delete |
email requested on YYYY-MM-DD | response received on YYYY-MM-DD]"

'''(d) Rollback:'''
N/A. Best-effort operation either way.

=== Step 5. Install a pre-commit secret-scanner ===

So a future paste-in is caught before it reaches a commit.

'''(a) Action:'''
<syntaxhighlight lang="bash">
cd ~/Developer/SystemResiliency
# Add a .gitleaks.toml if you want custom rules; defaults are fine.
# Install the pre-commit framework if not already:
which pre-commit || sudo apt install -y pre-commit
cat > .pre-commit-config.yaml <<'EOF'
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.0
hooks:
- id: gitleaks
EOF
pre-commit install
</syntaxhighlight>

Do the same in <code>~/Developer/jellyfin-adhoc/</code> and <code>~/Developer/tsbuffer/</code>.

'''(b) Verify:'''
* Make a trivial change to a file in the repo.
* Stage it. Try to commit.
* Expect: pre-commit runs, gitleaks scans the staged diff, commit
proceeds (because the diff is clean).
* Manually test the catch: stage a file containing <code>EXAMPLE_KEY=eyJabc.def.ghi</code>,
commit; expect gitleaks to block.

'''(c) Update:'''
* Commit the <code>.pre-commit-config.yaml</code> to each repo.

'''(d) Rollback:'''
<code>pre-commit uninstall</code>.

=== Step 6. Adopt a policy on cloud-handoff tools ===

<code>/ultraplan</code> is convenient but its blast radius is the entire working
directory. Several reasonable options:

* '''Block it entirely''' via a Claude Code hook in <code>~/.claude/settings.json</code>.
* '''Allow but require confirmation''' via a pre-tool hook that shows the
user the upload manifest before proceeding.
* '''Sanitize first''' — never invoke from a directory that contains
CLAUDE.md / MEMORY.md.

Decision pending — capture in <code>~/Developer/SystemResiliency/TODO.md</code> for
follow-up after this incident closes.

----

== Outcome (filled in as work happens) ==

* [ ] Step 1 — n8n API JWT rotated
* [ ] Step 2 — n8n admin password rotated
* [ ] Step 3 — Audit complete; findings:
* [ ] Step 4 — Cloud session deletion attempt:
* [ ] Step 5 — Pre-commit hook installed
* [ ] Step 6 — Cloud-handoff policy decision:

== Follow-ups (for TODO.md) ==

* Cloud-handoff tool policy decision (Step 6 above).
* After this exposure, audit existing OpenBao <code>scripts-n8n</code> and
<code>claude-mcp</code> tokens: are they scoped tightly enough? Should they be
rotated on a schedule (e.g. every 90 days) regardless of leaks?
* Consider whether Vaultwarden master password should be rotated on a
cadence (high blast radius if leaked; can't be auto-rotated).

== Lessons ==

# Any Claude Code tool that "uploads context" needs an explicit
user-confirmation prompt with the manifest of what will be sent. The
silent default is a footgun.
# CLAUDE.md and MEMORY.md are read by everything: every Claude Code
session, plus any tool that exports project context. They must never
contain literal secrets. The bar should be "would I publish this to
a public README?" — if no, it doesn't belong in those files.
# Rotation tedium is the actual problem behind "I'll just inline this
credential for now." The exposure-remediation script (see
<code>scripts/exposure-remediation/PLAN.md</code>) is the load-bearing follow-up.

Postmortems/2026-05-07 n8n HA Credential Repair

2026-07-05T18:17:36Z

Wikibot: Migrated from postmortems/2026-05-07-n8n-ha-credential-repair.md

''As of: 2026-07-05 — migrated from `postmortems/2026-05-07-n8n-ha-credential-repair.md`''

= Postmortem: n8n Credential Failures & HA Workflow Repair =

'''Date:''' 2026-05-07
'''Duration:''' ~6 hours of cascading failures (backup alerts since 2026-03-11 DB recovery)
'''Severity:''' Medium — silent failures, no data loss, but monitoring blind spots
'''Status:''' Resolved

----

== Summary ==

After the March 2026 n8n SQLite DB recovery, 12 workflows were silently failing due to stale/placeholder credential IDs. Home Assistant workflows had additional issues: wrong credential format, missing entity IDs, and parallel-merge node connections that n8n can't execute correctly.

----

== Root Causes (in order discovered) ==

=== 1. Stale SSH credential IDs from DB recovery ===
The March 2026 DB recovery recreated credentials with new IDs, but workflow JSONs stored on disk still had old IDs. When workflows were re-imported they referenced credentials that no longer existed.

* <code>CREDENTIAL_ID</code> — literal placeholder, never substituted
* <code>S2muq5nzJFa5ZydC</code> — old SSH credential from pre-recovery DB

'''Current correct SSH credential ID:''' <code>2ugS1ibsZu1LQid8</code>

=== 2. Missing <code>homeAssistantApi</code> credential ===
The <code>homeAssistantApi</code>-type credential (different from <code>httpHeaderAuth</code>) was lost in the DB recovery. Eight workflows referenced ID <code>6CIwjyqyUwnvaezn</code> which no longer existed.

=== 3. Wrong credential <code>host</code> field format ===
n8n's <code>homeAssistantApi</code> credential uses '''separate <code>host</code> and <code>port</code> fields''' — the host is just the IP/hostname with no scheme, no port:

<syntaxhighlight lang="json">
{
"host": "192.168.0.140",
"port": 8123,
"accessToken": "...",
"ssl": false
}
</syntaxhighlight>

'''Wrong formats tried (and why they fail):'''
* <code>"host": "http://homeassistant.local:8123"</code> → Docker can't resolve <code>.local</code> mDNS names
* <code>"host": "http://192.168.0.140:8123"</code> → node builds URL as <code>http://http://192.168.0.140:8123</code> → DNS lookup of literal string <code>"http"</code> → <code>EAI_AGAIN</code>
* <code>"host": "192.168.0.140:8123"</code> → node builds URL as <code>http://192.168.0.140:8123:8123</code> → <code>ERR_INVALID_URL</code>

'''How discovered:''' Read the actual compiled source at:
<syntaxhighlight lang="text">
docker exec n8n find /usr/local/lib/node_modules/n8n -path "*HomeAssistant/GenericFunctions.js"
</syntaxhighlight>
URL construction: `<code> </code>${ssl?'https':'http'}://${credentials.host}:${credentials.port}/api${resource}<code> </code>`

=== 4. Missing entity IDs in HA node parameters ===
The <code>entityId</code> field was never saved in the original workflow JSONs exported from n8n. All HA state-read nodes had <code>resource: "state"</code> but no <code>entityId</code>.

'''Correct entity IDs used:'''

{| class="wikitable"
|-
! Node purpose
! Entity ID
|-
| Living room temp
| <code>sensor.ewelink_snzb_02p_temperature</code>
|-
| Beta sensor temp
| <code>sensor.temp_sensor_beta_temperature</code>
|-
| Closet/Charlie temp
| <code>sensor.temp_sensor_charlie_temperature</code>
|-
| Closet/Charlie humidity
| <code>sensor.temp_sensor_charlie_humidity</code>
|-
| Door sensor
| <code>binary_sensor.shellydw2_075374_door</code>
|-
| Weather forecast
| <code>weather.forecast_ronwood</code>
|-
| Phone next alarm
| <code>sensor.pixel_7_next_alarm</code>
|-
| Announce/TTS
| <code>media_player.home_assistant_voice_098a64</code>
|-
| HA notify
| <code>notify.home_assistant</code>
|}

=== 5. Parallel-merge node connections ===
Workflows had multiple HA nodes connected in '''parallel''' (all from the trigger), all feeding into one downstream code node. n8n runs the downstream node as soon as the first upstream node finishes — so the code crashes with <code>"Node X hasn't been executed"</code> when it tries to read data from the still-running nodes.

'''Pattern causing the bug:'''
<syntaxhighlight lang="text">
Trigger ──┬── Get Temp A ──┐
├── Get Temp B ──┼── Analyze ← runs on first finish, others not done
└── Get Temp C ──┘
</syntaxhighlight>

'''Fix — chain them sequentially:'''
<syntaxhighlight lang="text">
Trigger → Get Temp A → Get Temp B → Get Temp C → Analyze
</syntaxhighlight>

Each node passes its result downstream; the code node then accesses all previous nodes via <code>$('Get Temp A').first()</code> etc.

'''Affected workflows:'''
* Server Closet Monitor (Temp + Humidity)
* Home Temperature Alert (3 temp sensors)
* Morning Briefing (Weather + Energy Usage)
* Bedtime Routine (Door check + Weather)

----

== Diagnostic Approach ==

'''How to find execution errors when the n8n API returns no detail:'''
<syntaxhighlight lang="bash">
# execution_data table has the full run data
sudo sqlite3 /var/lib/docker/volumes/n8n-app_n8n_data/_data/database.sqlite \
"SELECT substr(ed.data,1,5000) FROM execution_entity ee
JOIN execution_data ed ON ee.id = ed.executionId
WHERE ee.id = <EXEC_ID>;"
# Then grep for: EAI_AGAIN, ENOTFOUND, ERR_INVALID_URL, "message", getaddrinfo
</syntaxhighlight>

'''How to find which credentials are broken:'''
<syntaxhighlight lang="bash">
# Valid credential IDs in DB
sudo sqlite3 /var/lib/docker/volumes/n8n-app_n8n_data/_data/database.sqlite \
"SELECT id, name, type FROM credentials_entity ORDER BY name;"

# Scan all workflows for IDs not in that list
# (see the audit script logic used in this session)
</syntaxhighlight>

'''After any credential fix — n8n must be restarted:'''
<syntaxhighlight lang="bash">
sudo -E sh -c 'cd /library/n8n-app && docker compose restart n8n'
</syntaxhighlight>
n8n caches credential validation state. Even after fixing a credential in the DB, workflows show "has issues" until restart.

----

== Current Credential IDs (verified 2026-05-07) ==

{| class="wikitable"
|-
! ID
! Name
! Type
|-
| <code>2ugS1ibsZu1LQid8</code>
| Stronghold SSH (mnw)
| sshPrivateKey
|-
| <code>4RowxM0jrmYDnctK</code>
| Home Assistant API
| homeAssistantApi
|-
| <code>w0gXpM8W793fahAY</code>
| ntfy Auth
| httpHeaderAuth
|-
| <code>x5tv5aXs5N5nd2aK</code>
| Matrix Bot (Zordon)
| httpHeaderAuth
|-
| <code>R9LGxXIhmWCRM5Cn</code>
| Home Assistant API (old httpHeaderAuth)
| httpHeaderAuth
|-
| <code>5MCznsMsxjqOrlhJ</code>
| AudioBookshelf API
| httpHeaderAuth
|-
| <code>XB0V2UwPdWkRJoWj</code>
| OpenAI Bearer
| httpHeaderAuth
|-
| <code>NIQrbnzjhpNzBV1P</code>
| OpenAi account
| openAiApi
|}

'''HA credential data format (correct):'''
* host: <code>192.168.0.140</code> (IP only, no scheme, no port)
* port: <code>8123</code>
* ssl: <code>false</code>
* accessToken: from Vaultwarden → "Home Assistant Long-Lived Token"

----

== Prevention ==

# '''After any n8n DB recovery:''' run the credential audit script before assuming workflows work. Check every active workflow's credential IDs against <code>credentials_entity</code>.
# '''Export workflow JSONs immediately after any credential re-creation''' so the repo stays in sync.
# '''Never use <code>homeassistant.local</code> in n8n''' — Docker containers can't resolve mDNS. Always use <code>192.168.0.140</code>.
# '''n8n parallel-merge pattern is broken''' — if multiple nodes need to read data before a code node, chain them sequentially, not in parallel.
# '''Test workflows after changes''' — check n8n execution history within 30 min of any credential/workflow change.

Postmortems/2026-05-05 Jellyfin XE GPU Hang

2026-07-05T18:17:36Z

Wikibot: Migrated from postmortems/2026-05-05-jellyfin-xe-gpu-hang.md

''As of: 2026-07-05 — migrated from `postmortems/2026-05-05-jellyfin-xe-gpu-hang.md`''

= Postmortem: Jellyfin AV1 Transcoding Causes System Hard Hang =
'''Date:''' 2026-05-05
'''Duration:''' ~2 minutes (20:28 → hard reset → 20:31 back online)
'''Severity:''' High — full system halt, required physical hard reset
'''Status:''' Mitigated (HW transcoding disabled)

----

== Summary ==

Pressing play on a video in Jellyfin triggered hardware-accelerated AV1 encoding via Intel QuickSync (QSV) on the Intel Arc B580 (Battlemage). The xe kernel driver entered a GPU execution queue timeout loop (<code>guc_exec_queue_timedout_job</code>) on both GT0 and GT1, could not recover, and the system hard-locked. Required a physical power reset to recover.

----

== Timeline ==

{| class="wikitable"
|-
! Time (CST)
! Event
|-
| 20:28:53
| Jellyfin starts ffmpeg transcoding <code>Omega Doom (1996).mp4</code> using <code>av1_qsv</code> encoder via VAAPI/QSV
|-
| 20:30:09
| Kernel floods journal with <code>sysrq: HELP</code> messages — system in distress
|-
| 20:30:40
| xe driver reports <code>guc_exec_queue_timedout_job</code> resets on GT0 and GT1 repeatedly (dozens of attempts, all failing)
|-
| ~20:30:45
| System hard-locks. No response to input.
|-
| ~20:31:28
| Hard reset performed. System boots. All services recover normally.
|}

----

== Root Cause ==

Jellyfin was configured to use Intel QuickSync (QSV) hardware acceleration. When transcoding the H.264 source file, ffmpeg selected <code>av1_qsv</code> as the output encoder. AV1 encoding on the Arc B580 via the <code>xe</code> kernel driver is unstable — the GPU execution queues time out, the driver attempts repeated resets, fails to recover, and the kernel hangs.

The <code>xe</code> driver (Intel's replacement for <code>i915</code> for Arc/Battlemage GPUs) is still maturing. AV1 hardware encoding is the newest and least-tested codec path on this driver. The system has no watchdog capable of recovering from this class of GPU hang.

'''ffmpeg command at crash:'''
<syntaxhighlight lang="text">
/usr/lib/jellyfin-ffmpeg/ffmpeg
-init_hw_device vaapi=va:/dev/dri/renderD128,driver=iHD
-init_hw_device qsv=qs@va
-hwaccel vaapi -hwaccel_output_format vaapi
-codec:v:0 av1_qsv
...
</syntaxhighlight>

'''Kernel error (repeated ~50+ times):'''
<syntaxhighlight lang="text">
xe 0000:0b:00.0: [drm] Tile0: GT0: trying reset from guc_exec_queue_timedout_job [xe]
xe 0000:0b:00.0: [drm] Tile0: GT1: trying reset from guc_exec_queue_timedout_job [xe]
</syntaxhighlight>

----

== What Was NOT the Cause ==

* This is unrelated to the previous <code>enable_dc=0</code> fix (that addressed display C-state timeouts on monitor switch, a separate xe instability)
* Not an OOM event — 62GB RAM, only 13GB used
* Not a filesystem or RAID issue

----

== Fix Applied ==

Disabled hardware transcoding in Jellyfin:
* Dashboard → Admin → Playback → Transcoding → Hardware acceleration → '''None'''

ffmpeg now uses software (CPU) encoding. The Ryzen 5 3600X (12 threads) is capable of handling typical transcoding load in software.

----

== Diagnostic Commands ==

<syntaxhighlight lang="bash">
# Check previous boot's GPU errors
journalctl -b -1 | grep "xe\|guc\|drm"

# Check current boot for xe issues
dmesg -T | grep -i "xe\|guc\|drm\|hang\|reset"

# See what Jellyfin's ffmpeg was doing at crash time
journalctl -u jellyfin --since "2026-05-05 20:25" --until "2026-05-05 20:32"
</syntaxhighlight>

----

== Hardware Context ==

{| class="wikitable"
|-
! Component
! Detail
|-
| GPU
| Intel Arc B580 (Battlemage), PCI 0b:00.0, device e20b
|-
| Kernel driver
| xe (replacing i915 for Battlemage)
|-
| Kernel
| 6.18.1-061801-generic
|-
| CPU
| AMD Ryzen 5 3600X (6c/12t)
|-
| RAM
| 62GB
|-
| Existing xe fix
| <code>options xe enable_dc=0</code> in modprobe.d (display C-state fix, unrelated)
|}

----

== Options for Restoring Hardware Transcoding ==

=== Option A: Replace GPU with NVIDIA RTX 3060/4060 ===
* NVENC/NVDEC drivers are rock-solid on Linux
* RTX 3060: AV1 decode only (no AV1 encode)
* RTX 4060: AV1 encode + decode, ~$300
* Drop-in PCIe x16, works with AM4/X570 platform
* No driver instability issues for media workloads

=== Option B: Replace GPU with Intel Arc A770 (Alchemist gen) ===
* Uses <code>i915</code> driver (mature, not <code>xe</code>)
* Full AV1 encode + decode
* ~$200–250
* Lower risk than Battlemage on xe

=== Option C: Wait for xe driver to mature / upgrade kernel ===
* xe driver improves rapidly; a future kernel may handle this gracefully
* No cost, but timeline is unknown
* Risky — hangs are full system halts, not graceful crashes

=== Option D: Keep B580, disable AV1 encode in Jellyfin only ===
* Keep VAAPI HW accel but force output codec to H.264 or H.265
* H.264/H.265 VAAPI paths on xe are more stable (but not guaranteed crash-free)
* Still some risk of xe driver hang on other codec paths

'''Recommendation:''' Option A (RTX 4060) if budget allows — eliminates the entire class of xe driver instability and gives better AV1 support. Option B (Arc A770) is the budget pick if you want to stay Intel.

----

== Follow-up Actions ==

{| class="wikitable"
|-
! Action
! Priority
! Status
|-
| Disable HW transcoding in Jellyfin
| Critical
| Done
|-
| Decide on GPU replacement
| Medium
| Pending
|-
| Add stronghold hardware specs to INFRASTRUCTURE_INVENTORY.md
| Low
| Pending
|}

Postmortems/2026-04-15 Immich Crash

2026-07-05T18:17:36Z

Wikibot: Migrated from postmortems/2026-04-15-immich-crash.md

''As of: 2026-07-05 — migrated from `postmortems/2026-04-15-immich-crash.md`''

= Postmortem: Immich Crash & Service Degradation =
'''Date:''' 2026-04-15
'''Duration:''' Unknown start → resolved ~22:30 CST
'''Severity:''' High — Immich completely unavailable, photos inaccessible
'''Status:''' Resolved

----

== Summary ==

Immich became unavailable due to a Docker bridge networking bug that left the server container unable to reach Redis. When the server was restarted to fix this, a second problem emerged: 11,744 assets in the database had missing source files on disk, causing the thumbnail generation job queue to storm-crash the process with exit code 139 (SIGSEGV) on every boot. The server then entered a persistent crash-restart loop. Root causes were identified, the bad assets were cleaned up in the database, and the server was restored to full health.

----

== Timeline ==

{| class="wikitable"
|-
! Time (CST)
! Event
|-
| Unknown
| Docker bridge veth bug silently breaks immich_server → immich_redis routing
|-
| Unknown
| Server begins crash-restart loop (Redis unreachable, EHOSTUNREACH)
|-
| ~21:00
| User reports Immich down; authenticated successfully but immediately logged out
|-
| ~21:05
| Investigation: <code>docker ps</code> shows immich_server "Up 46 seconds" — crash-looping
|-
| ~21:10
| Root cause 1 identified: <code>EHOSTUNREACH 172.18.0.3:6379</code> — server can't reach Redis despite same bridge network
|-
| ~21:10
| Root cause 2 identified: <code>immich_postgres</code> health check failing — stale checksum failure counter from April 4
|-
| ~21:10
| Fix 1: <code>pg_stat_reset()</code> clears stale checksum counter → postgres becomes healthy
|-
| ~21:15
| Fix 2: <code>docker compose down && docker compose up -d</code> on immich stack — recreates bridge network, resolves veth bug
|-
| ~21:20
| Server comes up healthy. User attempts thumbnail regeneration from admin UI
|-
| ~21:22
| Server crashes immediately with exit code 139 (SIGSEGV)
|-
| ~21:25
| Root cause 3 identified: thumbnail job queue flooded with 11,744+ jobs for missing files; sharp native library segfaults under parallel failure storm
|-
| ~21:30
| Redis thumbnail queue cleared; server stabilises
|-
| ~21:35
| Analysis: 34,397 library assets in DB; only 22,634 files on disk — 11,744 missing
|-
| ~21:40
| sharp tested against all 22,634 on-disk image files — only 4 failures (XMP sidecar files, not real images)
|-
| ~21:45
| Fix 3: 11,744 missing assets marked <code>status=trashed</code> in DB; 4 XMP entries deleted
|-
| ~22:00
| Server confirmed stable, all assets healthy, thumbnail generation safe to re-run
|}

----

== Root Causes ==

=== 1. Docker Bridge veth Bug ===
'''What happened:''' After a prior crash-restart cycle, the immich_server container's virtual Ethernet interface lost routing to other containers on the same Docker bridge network. From the server's perspective, Redis at <code>172.18.0.3</code> was unreachable (<code>EHOSTUNREACH</code>), despite Redis being healthy and all containers nominally on the same <code>immich_default</code> bridge.

'''Why it happened:''' This is a known Docker bug where the kernel's bridge forwarding state for a container's veth pair becomes stale after a crash-restart cycle. <code>docker ps</code> shows the container as "Up" but inter-container routing is broken. The bug does not self-heal.

'''Detection:''' <code>docker logs immich_server</code> showed continuous <code>EHOSTUNREACH 172.18.0.3:6379</code>. Host could ping 172.18.0.3 fine; the server container could not.

'''Fix:''' <code>docker compose down && docker compose up -d</code> — tearing down and recreating the network is the only fix. <code>docker compose restart</code> does '''not''' work as it doesn't recreate the bridge.

----

=== 2. Stale Postgres Checksum Failure Counter ===
'''What happened:''' <code>immich_postgres</code> was marked <code>(unhealthy)</code> by Docker, which prevented <code>immich_server</code> from starting (it has <code>depends_on: database: condition: service_healthy</code>).

'''Why it happened:''' A single page checksum failure occurred on April 4 (11 days prior). The failure count accumulates in <code>pg_stat_database</code> and is never reset automatically. The health check queries <code>SUM(checksum_failures)</code> and exits non-zero if > 0, so the container stayed unhealthy indefinitely even though the database was functioning normally and no data corruption was present.

'''Detection:''' <code>docker inspect immich_postgres --format '{{json .State.Health}}'</code> showed <code>checksum failure count is 1</code> with last failure <code>2026-04-04</code>.

'''Fix:''' <code>docker exec immich_postgres psql -U postgres -d immich -c "SELECT pg_stat_reset();"</code> resets all per-database statistics including the checksum counter. The health check immediately passed on the next interval.

'''Note:''' A real checksum failure is worth investigating — it can indicate a dying disk. In this case the single failure was old and isolated; the database was consistent and Immich was functioning (HTTP 200) right up until the networking broke. Monitor for recurrence.

----

=== 3. Missing Asset Crash Loop (Primary Crash Cause) ===
'''What happened:''' When thumbnail generation was triggered, the BullMQ job queue was flooded with ~11,744 jobs for assets whose source files no longer exist on disk. The Node.js worker processes fired these jobs in high concurrency. The <code>sharp</code> native image library (C++ via N-API) segfaulted under the load of simultaneous "Input file is missing" failure paths, killing the entire Node.js process with exit code 139 (SIGSEGV). Docker's <code>restart: always</code> policy restarted the server, which immediately picked up the queued jobs and crashed again — a persistent crash loop.

'''Why the files were missing:''' 34,397 assets were stored in the "uploaded library" path (<code>/usr/src/app/upload/library/</code>). Only 22,634 files exist on disk. The gap of 11,744 likely accumulated over time as photos were moved, deleted from the host filesystem, or sourced from a device (e.g., iPhone backup migration) that was later reorganised without removing the Immich DB records.

'''Detection:'''
<syntaxhighlight lang="text">
docker logs immich_server → exit code 139 (SIGSEGV)
find /raid1/.../library -type f | wc -l → 22,634
SELECT COUNT(*) FROM asset WHERE originalPath LIKE '/usr/src/app/upload/library/%' → 34,397
</syntaxhighlight>

'''Fix (immediate):''' Clear the Redis thumbnail queue to stop the crash loop:
<syntaxhighlight lang="bash">
docker exec immich_redis redis-cli KEYS "immich_bull:thumbnailGeneration:*" | \
xargs docker exec immich_redis redis-cli DEL
</syntaxhighlight>

'''Fix (permanent):''' Mark missing assets as trashed so Immich never queues thumbnail generation for them again:
<syntaxhighlight lang="bash">
# Pipe missing asset IDs into postgres to bulk-trash
cat /tmp/immich-missing-ids.txt | docker exec -i immich_postgres psql -U postgres -d immich -c "
CREATE TEMP TABLE missing_ids (id uuid);
COPY missing_ids FROM STDIN;
UPDATE asset SET \"deletedAt\" = NOW(), status = 'trashed'
WHERE id IN (SELECT id FROM missing_ids);"
</syntaxhighlight>

Additionally, 4 XMP sidecar files (<code>.jpg.xmp</code>) had been incorrectly indexed as image assets and deleted from the DB:
<syntaxhighlight lang="bash">
docker exec immich_postgres psql -U postgres -d immich -c \
"DELETE FROM asset WHERE \"originalPath\" LIKE '%.xmp';"
</syntaxhighlight>

----

== What Was NOT Broken ==

* '''HEIC files from Mac auto-upload''' — All passed sharp's <code>metadata()</code> test. Pillow (used for the corruption scan) doesn't support HEIC natively; those "failures" were false positives.
* '''External library photos''' (70,840 assets in <code>/app/local-images/</code>) — Library scan confirmed 0 offlined, all intact.
* '''Thumbnails on disk''' — 190,882 thumbnail files were present and correctly structured. No regeneration was needed for most assets.

----

== Impact ==

{| class="wikitable"
|-
! Area
! Impact
|-
| Photo browsing
| Complete outage — all images showed "Error loading image"
|-
| New uploads
| Unavailable during crash loop
|-
| External library
| Intact, no data loss
|-
| Uploaded library
| 11,744 assets with no source files trashed (files were already gone before today)
|-
| Other services
| None — isolated to Immich stack
|}

----

== Diagnostic Commands Reference ==

<syntaxhighlight lang="bash">
# Check all Immich container statuses
docker ps --filter "name=immich" --format "table {{.Names}}\t{{.Status}}"

# Check crash exit code
docker inspect immich_server --format '{{.State.ExitCode}}'
# 139 = SIGSEGV (native crash), 1 = normal error, 0 = clean exit

# Check postgres health detail
docker inspect immich_postgres --format '{{json .State.Health}}' | python3 -m json.tool | tail -20

# Clear thumbnail queue (safe to run anytime — jobs will re-queue from admin UI on demand)
docker exec immich_redis redis-cli KEYS "immich_bull:thumbnailGeneration:*" | \
xargs docker exec immich_redis redis-cli DEL

# Count assets by storage location
docker exec immich_postgres psql -U postgres -d immich -t -c "
SELECT
SUM(CASE WHEN \"originalPath\" LIKE '/usr/src/app/upload/library/%' THEN 1 ELSE 0 END) as uploaded,
SUM(CASE WHEN \"originalPath\" LIKE '/app/local-images/%' THEN 1 ELSE 0 END) as external,
COUNT(*) FILTER (WHERE status = 'trashed') as trashed
FROM asset;"

# Find assets with missing files (run on host, translate paths)
# See: /home/mnw/Developer/SystemResiliency/scripts/immich-find-broken.py

# Reset stale postgres checksum counter
docker exec immich_postgres psql -U postgres -d immich -c "SELECT pg_stat_reset();"
</syntaxhighlight>

----

== Follow-up Actions ==

{| class="wikitable"
|-
! Action
! Priority
! Status
|-
| Run thumbnail generation from Immich admin → Jobs
| High
| Pending
|-
| Empty Immich trash (11,744 trashed assets)
| Medium
| Pending
|-
| Investigate why 11,744 uploaded assets lost their source files
| Low
| Pending
|-
| Monitor postgres checksum failures — recurrence may indicate disk issue
| Medium
| Ongoing
|-
| Monthly rolling restarts workflow (n8n <code>OlzWTPDsCoO9oEo7</code>) now active
| Done
| Complete
|}

----

== Prevention ==

The '''monthly rolling restart workflow''' deployed today (n8n ID: <code>OlzWTPDsCoO9oEo7</code>) performs a full <code>compose down && up -d</code> on all service stacks on the 1st of each month at 2 AM CST. This proactively recreates Docker bridge networks before the veth bug can silently accumulate, and sends per-service status via Matrix (Zordon). The postgres checksum issue and missing asset problem are not recurring risks given the fixes applied.

Services/Mealie

2026-07-05T18:16:42Z

Wikibot: Genericize example password placeholder

''As of: 2026-07-05 — migrated from `services/MEALIE.md`''

= Mealie — Recipe Manager =

Complete setup & operations guide for Mealie (recipe management app) with Authentik OAuth integration.

----

== Overview ==

'''Mealie''' is a self-hosted recipe management app. We run it on Stronghold with:
* Authentik OAuth2 for user authentication
* Group-based access control (admin vs user roles)
* HTTPS via Cloudflare (cook.wilsoz.com)
* Persistent data storage in Docker volume

'''Current Users:''' marcus (admin via Authentik)
'''Access:''' https://cook.wilsoz.com

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''URL'''
| https://cook.wilsoz.com
|-
| '''Container name'''
| mealie
|-
| '''Compose file'''
| <code>/library/mealie-app/docker-compose.yml</code>
|-
| '''Data volume'''
| <code>mealie-app_mealie-data:/app/data</code>
|-
| '''Internal port'''
| 9000
|-
| '''Host port'''
| 9925
|-
| '''Network'''
| <code>mealie-app_mealie</code> (MTU 1280)
|-
| '''Image'''
| <code>ghcr.io/mealie-recipes/mealie:latest</code>
|}

----

== Initial Setup from Scratch ==

=== 1. Create Directory & Files ===

<syntaxhighlight lang="bash">
mkdir -p /library/mealie-app
cd /library/mealie-app
</syntaxhighlight>

=== 2. Create docker-compose.yml ===

<syntaxhighlight lang="yaml">
version: "3.8"
services:
mealie:
image: ghcr.io/mealie-recipes/mealie:latest
container_name: mealie
restart: unless-stopped
env_file:
- .env
volumes:
- mealie-data:/app/data
ports:
- "9925:9000"
networks:
- mealie

volumes:
mealie-data:

networks:
mealie:
driver: bridge
driver_opts:
com.docker.network.driver.mtu: 1280
</syntaxhighlight>

=== 3. Create .env File ===

<syntaxhighlight lang="bash">
# App Configuration
PUID=1000
PGID=1000
TZ=America/Chicago
BASE_URL=https://cook.wilsoz.com
ALLOW_SIGNUP=false

# OIDC Authentication (Authentik)
OIDC_AUTH_ENABLED=true
OIDC_PROVIDER_NAME=Authentik
OIDC_CONFIGURATION_URL=https://key.wilsoz.com/application/o/mealie/.well-known/openid-configuration
OIDC_CLIENT_ID=3fUfT5FALoAqvWHzqiD0eLNYKd3fsyYjdUkEUVy5
OIDC_CLIENT_SECRET=D5IbtoxMgx6Kkmry296bXUz6G3NcvzT70wIBrr7RFu8
OIDC_SIGNUP_ENABLED=true
OIDC_AUTO_REDIRECT=false
OIDC_USER_GROUP=family-and-friends
OIDC_ADMIN_GROUP=authentik Admins
OIDC_GROUPS_CLAIM=groups
OIDC_USER_CLAIM=email
OIDC_NAME_CLAIM=name
</syntaxhighlight>

=== 4. Set Up Authentik OAuth Provider ===

==== Create OAuth2 Provider in Authentik ====

# Go to https://key.wilsoz.com/if/admin/
# '''Applications → Providers → Create OAuth2/OpenID Provider'''
# Fill in:
''' '''Name:** mealie-oauth
''' '''Authorization flow:** (default-authentication-flow)
''' '''Client type:** Confidential
''' '''Redirect URIs:** (one per line)
```
https://cook.wilsoz.com/login
https://cook.wilsoz.com/login?direct=1
```
# '''Save''' — note the '''Client ID''' and '''Client Secret'''

==== Configure Provider: Signing Key ====

⚠️ '''CRITICAL:''' Assign the signing key or JWKS endpoint will return <code>{}</code> and OAuth will fail.

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.crypto.models import CertificateKeyPair

p = OAuth2Provider.objects.get(name='mealie-oauth')
jwt_key = CertificateKeyPair.objects.get(name='authentik Internal JWT Certificate')
p.signing_key = jwt_key
p.save()
print('Assigned JWT Certificate signing key')
"
</syntaxhighlight>

'''Why JWT Certificate?'''
* Uses HS256 algorithm (Mealie's OIDC library supports it)
* Self-signed Certificate uses RS256 (rejected by Mealie)

==== Configure Provider: Property Mappings ====

Add OAuth scope property mappings:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.core.models import PropertyMapping

p = OAuth2Provider.objects.get(name='mealie-oauth')
email_pm = PropertyMapping.objects.get(name='authentik default OAuth Mapping: OpenID \\'email\\'')
profile_pm = PropertyMapping.objects.get(name='authentik default OAuth Mapping: OpenID \\'profile\\'')
openid_pm = PropertyMapping.objects.get(name='authentik default OAuth Mapping: OpenID \\'openid\\'')

p.property_mappings.add(email_pm, profile_pm, openid_pm)
p.include_claims_in_id_token = True
p.save()
print('Added property mappings and enabled claims in ID token')
"
</syntaxhighlight>

==== Create Application in Authentik ====

# Go to '''Applications → Applications → Create'''
# Fill in:
''' '''Name:** Mealie
''' '''Slug:** mealie
''' '''Provider:** mealie-oauth (select from dropdown)
''' '''Authorization flow:** default-authorization-flow
# '''Save'''

=== 5. Start Mealie ===

<syntaxhighlight lang="bash">
cd /library/mealie-app
docker compose up -d
</syntaxhighlight>

Wait ~10 seconds for startup, then check:

<syntaxhighlight lang="bash">
docker logs mealie --tail=20
# Should show: "Application startup complete"
# And: "Enabled: True" under the OIDC section
</syntaxhighlight>

=== 6. Configure VPS Nginx Proxy ===

On VPS, add server block to nginx config (or update if exists):

<syntaxhighlight lang="nginx">
server {
listen 443 ssl http2;
server_name cook.wilsoz.com;

ssl_certificate /etc/letsencrypt/live/cook.wilsoz.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/cook.wilsoz.com/privkey.pem;

location / {
proxy_pass http://100.64.207.155:9925;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
</syntaxhighlight>

Then:

<syntaxhighlight lang="bash">
sudo systemctl reload nginx
</syntaxhighlight>

=== 7. Create First Admin User (In-App) ===

# Navigate to https://cook.wilsoz.com
# You'll see the login page with "Authentik" button
# If Mealie is brand new with no users, it may show an initial setup screen
# Use Authentik to log in with an account in the '''authentik Admins''' group
** This account gets Mealie admin role automatically

----

== Operations ==

=== Start/Stop/Restart ===

<syntaxhighlight lang="bash">
cd /library/mealie-app

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart mealie

# Full restart (clears cached config)
docker compose down && sleep 2 && docker compose up -d
</syntaxhighlight>

=== View Logs ===

<syntaxhighlight lang="bash">
# Live logs
docker logs -f mealie

# Last 50 lines
docker logs mealie --tail=50

# Filter for errors
docker logs mealie 2>&1 | grep -i "error\|exception"

# Filter for OIDC
docker logs mealie 2>&1 | grep -i "oidc"
</syntaxhighlight>

=== Reset Admin Password (Database) ===

If you lose the admin password:

<syntaxhighlight lang="bash">
docker exec mealie python3 -c "
import sqlite3
import bcrypt

db = sqlite3.connect('/app/data/mealie.db')
cursor = db.cursor()

# Set new password for marcus
new_password = '<choose-a-new-password-here>'
salt = bcrypt.gensalt(rounds=10)
hashed = bcrypt.hashpw(new_password.encode('utf-8'), salt).decode('utf-8')

cursor.execute('UPDATE users SET password = ? WHERE username = ?', (hashed, 'marcus'))
db.commit()
print(f'Password reset to: {new_password}')
db.close()
"
</syntaxhighlight>

Then log in and change it in Mealie settings.

=== Manage Users & Groups ===

'''Via Authentik:'''
* Users must be added to Authentik first
* Create/manage groups: '''Directory → Groups'''
* Add users to groups for role assignment

'''Via Mealie:'''
* '''Settings → Users''' — shows Mealie users (created on first OAuth login)
* User roles are determined by Authentik group membership:
''' '''authentik Admins** → Mealie admin
''' '''family-and-friends** → Mealie user

----

== Configuration Reference ==

=== Environment Variables (<code>.env</code>) ===

{| class="wikitable"
|-
! Variable
! Purpose
! Example
|-
| <code>BASE_URL</code>
| External URL for Mealie
| <code>https://cook.wilsoz.com</code>
|-
| <code>ALLOW_SIGNUP</code>
| Allow registration without OIDC
| <code>false</code>
|-
| <code>OIDC_AUTH_ENABLED</code>
| Enable OAuth login
| <code>true</code>
|-
| <code>OIDC_CLIENT_ID</code>
| Authentik provider client ID
| (from Authentik)
|-
| <code>OIDC_CLIENT_SECRET</code>
| Authentik provider secret
| (from Authentik)
|-
| <code>OIDC_CONFIGURATION_URL</code>
| OIDC discovery endpoint
| <code>https://key.wilsoz.com/application/o/mealie/.well-known/openid-configuration</code>
|-
| <code>OIDC_USER_GROUP</code>
| Authentik group for regular users
| <code>family-and-friends</code>
|-
| <code>OIDC_ADMIN_GROUP</code>
| Authentik group for admins
| <code>authentik Admins</code>
|-
| <code>OIDC_GROUPS_CLAIM</code>
| OIDC claim name for groups
| <code>groups</code>
|-
| <code>OIDC_SIGNUP_ENABLED</code>
| Auto-create users on first OIDC login
| <code>true</code>
|-
| <code>OIDC_AUTO_REDIRECT</code>
| Skip login page, go straight to Authentik
| <code>false</code>
|-
| <code>TZ</code>
| Timezone
| <code>America/Chicago</code>
|-
| <code>PUID</code> / <code>PGID</code>
| Linux user/group for file permissions
| <code>1000</code>
|}

----

== Data & Backups ==

=== Data Location ===

<syntaxhighlight lang="text">
/library/mealie-app/mealie-data:/app/data

Contents:
- mealie.db (SQLite database — all recipes, users, settings)
- recipes/ (recipe files)
- users/ (user data)
- backups/ (exported recipe backups)
- templates/ (custom templates)
- groups/ (group/household data)
</syntaxhighlight>

=== Backup Procedure ===

<syntaxhighlight lang="bash">
# Backup the entire data volume
docker run --rm \
-v mealie-app_mealie-data:/data \
-v /home/mnw/backups:/backup \
ubuntu tar czf /backup/mealie-$(date +%Y%m%d).tar.gz /data

# List backups
ls -lh /home/mnw/backups/mealie-*.tar.gz
</syntaxhighlight>

=== Restore from Backup ===

<syntaxhighlight lang="bash">
# Stop Mealie
cd /library/mealie-app && docker compose down

# Restore data
docker run --rm \
-v mealie-app_mealie-data:/data \
-v /home/mnw/backups:/backup \
ubuntu bash -c "rm -rf /data/* && tar xzf /backup/mealie-20260513.tar.gz -C /data"

# Restart
docker compose up -d
</syntaxhighlight>

----

== Troubleshooting ==

=== Login Issues ===

==== "Something went wrong" on OAuth callback ====

'''Symptoms:'''
* Authentik login succeeds, but Mealie shows error when redirected back

'''Common causes & fixes:'''

{| class="wikitable"
|-
! Error
! Cause
! Fix
|-
| <code>KeyError: 'keys'</code>
| JWKS endpoint returns <code>{}</code>
| Assign signing key to OAuth provider (see Setup step 4)
|-
| <code>UnsupportedAlgorithmError: RS256 not allowed</code>
| Wrong signing key algorithm
| Use "authentik Internal JWT Certificate" (HS256), not "Self-signed Certificate" (RS256)
|-
| <code>invalid_client: Client authentication failed</code>
| Client secret mismatch
| Verify <code>OIDC_CLIENT_SECRET</code> in .env matches Authentik provider
|-
| HTTP 500 at <code>/api/auth/oauth/callback</code>
| Mealie can't reach Authentik's token endpoint
| Check network connectivity, SSL certificates, DNS
|}

'''Debug steps:'''

<syntaxhighlight lang="bash">
# Check OIDC config is loaded
docker logs mealie | grep -i "oidc\|enabled"

# Test discovery endpoint
docker exec mealie curl -s "https://key.wilsoz.com/application/o/mealie/.well-known/openid-configuration" -k | jq .

# Check JWKS endpoint has keys
docker exec mealie curl -s "https://key.wilsoz.com/application/o/mealie/jwks/" -k | jq '.keys | length'

# View OAuth errors
docker logs mealie --tail=100 | grep -i "oauth\|error\|exception" | tail -20
</syntaxhighlight>

==== "OIDC not appearing on login page" ====

* Verify <code>OIDC_AUTH_ENABLED=true</code> in <code>.env</code>
* Restart container: <code>docker compose down && docker compose up -d</code>
* Check logs for startup errors

==== "Redirect URI mismatch" error ====

* Ensure Authentik provider redirect URIs match exactly:
** <code>https://cook.wilsoz.com/login</code>
** <code>https://cook.wilsoz.com/login?direct=1</code>
* Verify <code>BASE_URL=https://cook.wilsoz.com</code> in <code>.env</code>
* No trailing slashes or port numbers

==== User not getting admin role ====

* Confirm user is in '''authentik Admins''' group (Directory → Groups)
* Log out and log back in (roles are assigned at login time)
* Check Mealie Settings → Users to see assigned role

=== Performance ===

==== Slow startup ====

Mealie can take 30-60 seconds on first start or after upgrade. Check:

<syntaxhighlight lang="bash">
docker logs mealie | tail -20
# Look for "Application startup complete"
</syntaxhighlight>

==== High memory usage ====

Default image is fine for small recipe collections. For very large:
* Monitor: <code>docker stats mealie</code>
* Limit: Add <code>mem_limit: 2g</code> to compose file if needed

=== Network Issues ===

==== Can't access https://cook.wilsoz.com ====

# Check VPS nginx config (see Setup step 6)
# Verify SSL cert is valid: <code>curl -I https://cook.wilsoz.com</code>
# Check Mealie is running: <code>docker ps | grep mealie</code>
# Check port: <code>netstat -tlnp | grep 9925</code>

==== Mealie can't reach Authentik (key.wilsoz.com) ====

From Mealie container:

<syntaxhighlight lang="bash">
docker exec mealie curl -v https://key.wilsoz.com/api/v3/ -k
# Should return 401 (not accessible, but shouldn't timeout)

docker exec mealie ping -c 1 key.wilsoz.com
# Should succeed
</syntaxhighlight>

----

== Security Notes ==

* '''OIDC_CLIENT_SECRET:''' Store securely in <code>.env</code>, never commit to git
* '''Database:''' <code>/app/data/mealie.db</code> contains all user data — protect with filesystem permissions
* '''Backups:''' Backup volume includes full database — encrypt backups if stored externally
* '''User signup:''' <code>ALLOW_SIGNUP=false</code> means only Authentik users can access (good)
* '''Groups:''' Ensure only trusted users are in <code>authentik Admins</code> group

----

== Links & References ==

* '''Mealie Docs:''' https://nightly.mealie.io/
* '''Mealie GitHub:''' https://github.com/mealie-recipes/mealie
* '''Authentik OIDC:''' https://goauthentik.io/docs/providers/oauth2
* '''Mealie OIDC Integration Guide:''' https://integrations.goauthentik.io/documentation/mealie/

----

== Related Services ==

* '''Authentik''' (auth): <code>/library/authentik-app/</code> — OAuth provider
* '''Nginx''' (proxy): VPS <code>/etc/nginx/sites-available/</code> — external access
* '''Backups:''' <code>/home/mnw/backups/</code> — data snapshots

Services/VpsNginx

2026-07-05T18:16:22Z

Wikibot: Migrated from services/VPS-NGINX.md

''As of: 2026-07-05 — migrated from `services/VPS-NGINX.md`''

= VPS nginx — vhost management scripts =

Reference for the RackNerd VPS (100.67.169.50, public-facing) that reverse-proxies every <code>*.wilsoz.com</code> subdomain back to stronghold over Tailscale (100.64.207.155). Read this before hand-writing an nginx vhost — as of 2026-07-05 there are scripts that do the whole thing in one command.

== Scripts (on the VPS, <code>/home/mnw/</code>) ==

{| class="wikitable"
|-
! Script
! Purpose
|-
| <code>add-vhost.sh</code>
| Create a vhost, symlink it, get a cert, add security headers, reload, verify — one command
|-
| <code>remove-vhost.sh</code>
| Tear a vhost down: remove symlink, delete/revoke cert, delete vhost file, reload
|-
| <code>VHOST-HOWTO.md</code>
| Full usage doc, lives alongside the scripts on the VPS
|}

=== Add a site ===

<syntaxhighlight lang="bash">
sudo /home/mnw/add-vhost.sh <domain> <tailscale-ip:port>
# Example:
sudo /home/mnw/add-vhost.sh wiki.wilsoz.com 100.64.207.155:9926
</syntaxhighlight>

Does, in order: writes <code>/etc/nginx/sites-available/<domain></code> with a plain HTTP proxy block → symlinks into <code>sites-enabled/</code> → <code>nginx -t</code> + reload → <code>certbot --nginx -d <domain></code> (Cloudflare DNS-01, rewrites the file with the SSL block + 80→443 redirect, same as every other cert on this VPS) → injects <code>include /etc/nginx/snippets/security-headers.conf;</code> right after certbot's <code>options-ssl-nginx.conf</code> line → final <code>nginx -t</code> + reload → <code>curl</code> status check.

'''Prerequisites before running:''' DNS A record for the domain already pointing at the VPS (100.67.169.50); the backend already up and reachable on stronghold at the given port. WebSocket backends need <code>proxy_http_version 1.1</code> + <code>Upgrade</code>/<code>Connection</code> headers added manually afterward — the script doesn't detect that case.

=== Remove a site ===

<syntaxhighlight lang="bash">
sudo /home/mnw/remove-vhost.sh <domain>
</syntaxhighlight>

Shows you the backend it's about to remove, asks whether to '''revoke''' the cert (<code>y</code> = revoke + delete, use when the domain is gone for good; default <code>N</code> = delete locally only, use when just moving/rebuilding), removes the <code>sites-enabled</code> symlink, deletes the vhost file, tests + reloads nginx.

== Server topology ==

{| class="wikitable"
|-
! Thing
! Value
|-
| Public VPS
| 100.67.169.50 (RackNerd)
|-
| Home server (stronghold)
| 100.64.207.155 (Tailscale)
|-
| Backend ports (examples)
| 9925 = Mealie, 9926 = MediaWiki
|-
| Cert method
| Let's Encrypt via certbot + Cloudflare DNS-01 (same plugin as <code>mymail.wilsoz.com</code>, see CLAUDE.md)
|-
| HSTS
| <code>max-age=300</code> — staged rollout, expect this to increase over time
|}

== Known nginx warnings (harmless, pre-existing) ==

<code>nginx -t</code> shows an <code>ssl_protocols</code> redefinition warning for <code>pages.git.wilsoz.com</code> and <code>photos.wilsoz.com</code>, and will show the same for any vhost combining <code>options-ssl-nginx.conf</code> (certbot) with <code>security-headers.conf</code>. Cosmetic only — config still loads and serves correctly.

== Gotcha ==

<code>add-vhost.sh</code> aborts rather than overwriting if the vhost file already exists — safe to run repeatedly/accidentally, but means you must <code>remove-vhost.sh</code> first (or hand-edit) to change an existing site's backend port.

Services/ArcB580Driver

2026-07-05T18:16:21Z

Wikibot: Migrated from services/ARC-B580-DRIVER.md

''As of: 2026-07-05 — migrated from `services/ARC-B580-DRIVER.md`''

= Intel Arc B580 (Battlemage) — xe Driver Notes =

== Problem ==

The <code>xe</code> kernel driver for the Arc B580 crashes the system when the monitor is switched
off/on manually. Root cause: the driver uses aggressive display power-saving states (DC states)
that fail to wake properly on Battlemage, causing a cascade of GPU timeouts that brings
the system down.

Symptoms in logs:
'' <code>xe 0000:0b:00.0: [drm] ''ERROR* Tile0: GT1: Force wake domain X failed to ack wake (-ETIMEDOUT)</code>
* <code>xe 0000:0b:00.0: [drm] Tile0: GT1: trying reset from guc_exec_queue_timedout_job</code>
'' <code>xe 0000:0b:00.0: [drm] ''ERROR* PCODE Mailbox failed: 1 Illegal Command</code>
* UBSAN: array-index-out-of-bounds in <code>mtd_intel_dg.c</code>

== Fix Applied (2026-05-03) ==

'''File 1: <code>/etc/modprobe.d/xe-options.conf</code>'''
<syntaxhighlight lang="text">
options xe enable_dc=0
</syntaxhighlight>
Disables aggressive display C-states so the driver doesn't enter a deep sleep it can't
wake from when the monitor is switched off.

'''File 2: <code>/etc/modprobe.d/blacklist.conf</code>''' (appended)
<syntaxhighlight lang="text">
blacklist mtd_intel_dg
</syntaxhighlight>
Blacklists the <code>mtd_intel_dg</code> module which has a confirmed kernel UBSAN bug on this GPU.

Both changes require initramfs rebuild + reboot to take effect.

----

== How to Revert (if the fix breaks boot) ==

SSH in from another host, then:

<syntaxhighlight lang="bash">
# 1. Remove the xe options file
sudo rm /etc/modprobe.d/xe-options.conf

# 2. Remove the two lines added to blacklist.conf
# (the blank line, comment, and blacklist entry at the bottom)
sudo nano /etc/modprobe.d/blacklist.conf
# Delete the last 3 lines:
# (blank line)
# # Intel Arc B580 - mtd_intel_dg has a confirmed kernel UBSAN bug (array out-of-bounds)
# blacklist mtd_intel_dg

# 3. Rebuild initramfs
sudo update-initramfs -u

# 4. Reboot
sudo reboot
</syntaxhighlight>

Or as a one-liner if you want to skip the editor:

<syntaxhighlight lang="bash">
sudo rm /etc/modprobe.d/xe-options.conf && \
sudo head -n -3 /etc/modprobe.d/blacklist.conf | sudo tee /etc/modprobe.d/blacklist.conf.tmp && \
sudo mv /etc/modprobe.d/blacklist.conf.tmp /etc/modprobe.d/blacklist.conf && \
sudo update-initramfs -u && \
sudo reboot
</syntaxhighlight>

----

== If you need to go back to the Radeon ==

# Power off, swap GPU physically
# SSH in (or boot to TTY) and blacklist xe so it doesn't try to load for a missing device:
```bash
echo "blacklist xe" | sudo tee /etc/modprobe.d/blacklist-xe.conf
sudo update-initramfs -u
sudo reboot
```
# The <code>radeon</code> or <code>amdgpu</code> driver will load automatically depending on which card.

----

== Kernel / Firmware versions at time of fix ==

* Kernel: <code>6.18.1-061801-generic</code>
* GuC firmware: <code>xe/bmg_guc_70.bin</code> version <code>70.60.0</code>
* HuC firmware: <code>xe/bmg_huc.bin</code> version <code>8.2.10</code>
* DMC firmware: <code>i915/bmg_dmc.bin</code> version <code>2.6</code>

If you upgrade the kernel and the crashes stop, the fix may no longer be needed — test by
removing <code>xe-options.conf</code> after a kernel upgrade to see if stability holds without it.

Services/BackupSystem

2026-07-05T18:16:21Z

Wikibot: Migrated from services/BACKUP_SYSTEM.md

''As of: 2026-07-05 — migrated from `services/BACKUP_SYSTEM.md`''

= Backup System — Restic + Timeshift =

Complete setup & operations guide for Stronghold's dual-tier backup system: Timeshift (local snapshots) + Restic (remote encrypted backups).

'''Criticality:''' '''CRITICAL — The foundation.''' All disaster recovery depends on backups. Without them, data loss is permanent. This system enables recovery of every service on Stronghold.

----

== Overview ==

Stronghold uses a '''two-tier backup strategy''' for redundancy and speed:

# '''Timeshift''' (local, fast)
** RSYNC-based filesystem snapshots stored on <code>/raid1/timeshift/</code>
** 8 snapshots retained (mix of daily, weekly, monthly)
''' '''Purpose:** Fast recovery of individual files or full system (within days)
''' '''Speed:** ~5-30 minutes to restore (depends on size)
''' '''Limitation:** If <code>/raid1</code> is damaged, all local snapshots are lost

# '''Restic''' (remote, encrypted)
** Incremental encrypted backups sent to Hetzner Storage Box
** Daily backups at 03:00 CST, retention: 7 daily + 4 weekly + 6 monthly
''' '''Purpose:** Off-site redundancy; survives hardware failure
''' '''Coverage:** All service data, configs, media, databases
''' '''Speed:** ~30 minutes to backup, 1-2 hours per terabyte for restoration
''' '''Limitation:** Requires Hetzner SSH access and Restic repo password

'''Recovery decision tree:'''
* '''Drive works, need recent files?''' → Restore from Timeshift (fast)
* '''Drive works, lost some service?''' → Restore from Timeshift or Restic (both available)
* '''Drive failed, building new hardware?''' → Restore from Restic (only source of truth)

----

== Service Information ==

=== Timeshift ===

{| class="wikitable"
|-
! Property
! Value
|-
| '''Storage location'''
| <code>/raid1/timeshift/</code>
|-
| '''Snapshot mode'''
| RSYNC (not btrfs)
|-
| '''Device'''
| <code>/dev/md1</code> (RAID-1 array)
|-
| '''Current snapshots'''
| 8 (mix of daily D, weekly W, monthly M, backfill B)
|-
| '''Retention policy'''
| Auto: Last 5 daily, 1 weekly, 1 monthly; manual cleanup for older
|-
| '''Snapshot schedule'''
| Daily at 00:00 (D), weekly at 10:00 (W), manual on demand
|-
| '''Excluded from snapshots'''
| <code>/var/lib/docker/volumes</code> ⚠️, <code>/proc</code>, <code>/sys</code>, <code>/tmp</code>, <code>/var/tmp</code>
|}

=== Restic ===

{| class="wikitable"
|-
! Property
! Value
|-
| '''Container name'''
| <code>restic-backup</code>
|-
| '''Image'''
| <code>restic/restic:latest</code>
|-
| '''Compose file'''
| <code>/home/mnw/Developer/SystemResiliency/backup-stack/docker-compose.yml</code>
|-
| '''Network mode'''
| <code>host</code> (direct network access)
|-
| '''Backup schedule'''
| 03:00 CST daily (systemd timer <code>backup.timer</code>)
|-
| '''Hetzner server'''
| <code>u540853.your-storagebox.de:23</code> (SSH port 23, not 22)
|-
| '''Repo path'''
| <code>sftp:u540853@u540853.your-storagebox.de:./backups</code>
|-
| '''Encryption'''
| AES-256, password-based (stored in <code>.env</code> as <code>HETZNER_PASSWORD</code>)
|-
| '''Deduplication'''
| Enabled (saves 20-30% space)
|-
| '''Compression'''
| zstd (default, good balance)
|-
| '''Retention'''
| 7 daily, 4 weekly, 6 monthly snapshots
|-
| '''Database dumps'''
| PostgreSQL dumps staged to <code>/library/backup-stack/dumps/</code> before backup
|-
| '''System-state dump'''
| <code>/etc</code> subset + user crontabs + user/system systemd units + dpkg selections tar'd to <code>/library/backup-stack/dumps/system-state-YYYYMMDD.tar.gz</code> before backup (covers schedulers/configs that live outside <code>/data/*</code> and would otherwise be lost if <code>/etc</code> or a user home got wiped)
|}

----

== Initial Setup from Scratch ==

=== Timeshift (Local Snapshots) ===

'''Timeshift is likely already installed and running on your system.''' To verify or install:

<syntaxhighlight lang="bash">
# Check if Timeshift is installed
which timeshift

# If not, install (Ubuntu/Debian)
sudo apt update
sudo apt install timeshift

# Check current snapshots
sudo timeshift --list

# Create first snapshot manually (if none exist)
sudo timeshift --create --comments "Initial snapshot before disaster recovery setup"
</syntaxhighlight>

'''Configuration''' (in Timeshift GUI or config):
* '''Storage location:''' <code>/raid1/timeshift/</code>
* '''Device:''' <code>/dev/md1</code> (RAID-1 array)
* '''Snapshot mode:''' RSYNC
* '''Auto-backup:''' Enabled, daily at 00:00
* '''Retention:''' At least 5 daily snapshots kept

=== Restic (Remote Encrypted Backups) ===

==== 1. Set Up Hetzner Storage Box ====

If not already done:

# '''Create Hetzner account''' at https://www.hetzner.com/storage/storage-box
# '''Order Storage Box''' (recommended: smallest size starts at €3.81/month)
# '''In Hetzner Console:'''
** Go to Storage Box settings
** Enable "External reachability" (required for SSH/SFTP)
** Upload your SSH public key (<code>~/.ssh/id_rsa.pub</code>)
# '''Note your credentials:'''
** Server: <code>u540853.your-storagebox.de</code> (your number varies)
''' Port: '''23** (not 22!)
** Username: <code>u540853</code> (same as your account number)

==== 2. Set Up SSH Access ====

<syntaxhighlight lang="bash">
# Create SSH key if you don't have one
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa

# Test SSH connection to Hetzner (port 23!)
ssh -p 23 u540853@u540853.your-storagebox.de "echo 'SSH working!'"

# Should print: SSH working!
</syntaxhighlight>

==== 3. Copy Backup Stack to /library ====

<syntaxhighlight lang="bash">
# From your project directory
cp -r /home/mnw/Developer/SystemResiliency/backup-stack /library/

cd /library/backup-stack

# Create .env from example
cp .env.example .env

# Edit with your Hetzner credentials and restic password
nano .env
</syntaxhighlight>

'''.env file contents:'''
<syntaxhighlight lang="bash">
# Hetzner Storage Box
HETZNER_REPO="sftp:u540853@u540853.your-storagebox.de:./backups"
HETZNER_PASSWORD="<generate-strong-restic-password-32-chars>"

# Alerts
NTFY_TOKEN="<from Vaultwarden if you have one>"
NTFY_TOPIC="backup-alerts"
</syntaxhighlight>

==== 4. Initialize Restic Repository ====

<syntaxhighlight lang="bash">
cd /library/backup-stack
docker compose up -d

# Init Hetzner repository (will encrypt with password)
docker exec -e RESTIC_PASSWORD='<same-as-HETZNER_PASSWORD>' restic-backup \
restic init --repo sftp:u540853@u540853.your-storagebox.de:./backups

# Output should show: "created restic repository" with repository ID
</syntaxhighlight>

==== 5. Install Systemd Timers ====

<syntaxhighlight lang="bash">
# Copy timer definitions
sudo cp /library/backup-stack/systemd/*.service /etc/systemd/system/
sudo cp /library/backup-stack/systemd/*.timer /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload

# Enable and start backup timer
sudo systemctl enable --now backup.timer
sudo systemctl enable --now backup-validate-daily.timer
sudo systemctl enable --now backup-validate-weekly.timer

# Check status
systemctl list-timers | grep backup
</syntaxhighlight>

==== 6. Test the Backup ====

<syntaxhighlight lang="bash">
# Run a manual backup to verify everything works
docker exec restic-backup /scripts/run-backup.sh

# Watch the logs
docker logs -f restic-backup

# Should show:
# - Database dumps created
# - Restic scanning files
# - Upload to Hetzner
# - Success message
</syntaxhighlight>

----

== Disaster Mode A: Local Recovery (Fast) ==

=== Scenario: Lost a single file or small data ===

'''Example:''' Accidentally deleted a recipe from Mealie 2 days ago.

==== Recovery Steps: ====

<syntaxhighlight lang="bash">
# 1. List available snapshots
sudo timeshift --list

# 2. Mount the snapshot you want to restore from
# (Example: restore from 2026-05-11 daily snapshot)
sudo timeshift --mount /run/timeshift/latest

# 3. File is now accessible at:
# /run/timeshift/latest/library/mealie-app/mealie-data/recipes/
# Copy the file back to current system
cp /run/timeshift/latest/library/mealie-app/mealie-data/recipes/recipe.json \
/library/mealie-app/mealie-data/recipes/

# 4. Unmount
sudo timeshift --umount
</syntaxhighlight>

=== Scenario: Need to restore entire service from snapshot ===

'''Example:''' Immich database corrupted, want to restore from snapshot.

<syntaxhighlight lang="bash">
# 1. Stop the service
cd /library/immich-app
docker compose down

# 2. Mount snapshot
sudo timeshift --mount

# 3. Restore the entire service directory
sudo rm -rf /library/immich-app
sudo cp -r /run/timeshift/latest/library/immich-app /library/

# 4. Fix permissions (Docker runs as specific user)
sudo chown -R mnw:mnw /library/immich-app

# 5. Restart service
docker compose up -d

# 6. Unmount snapshot
sudo timeshift --umount
</syntaxhighlight>

=== Scenario: Full system restoration (if OS is still usable) ===

'''Example:''' Stronghold is working but some system files are corrupted.

<syntaxhighlight lang="bash">
# List snapshots from a few days ago
sudo timeshift --list

# Restore from a known-good snapshot
# This restores EVERYTHING except /home, /var/lib/docker, /opt, /etc/hosts, etc.
sudo timeshift --restore --snapshot "2026-05-11_10-00-01" --skip-grub

# Reboot after restoration
sudo reboot
</syntaxhighlight>

----

== Disaster Mode B: Full Recovery (After Drive Failure) ==

=== Scenario: Hard drive failed, new hardware deployed ===

'''Prerequisites:'''
* New hardware with OS installed (Ubuntu/Debian)
* Docker and docker-compose installed
* SSH key for Hetzner auth
* Restic password (from your secure notes)

==== Recovery Steps: ====

<syntaxhighlight lang="bash">
# 1. Create directories
mkdir -p /raid1
mkdir -p /library
mkdir -p /home/mnw/Developer/SystemResiliency

# 2. Copy backup-stack to new system
scp -r username@old-system:/library/backup-stack /library/

# 3. Update .env with correct passwords/credentials
nano /library/backup-stack/.env

# 4. Start the Restic container
cd /library/backup-stack
docker compose up -d

# 5. List available backups in Hetzner
docker exec restic-backup restic snapshots --repo sftp:u540853@u540853.your-storagebox.de:./backups

# 6. Restore the full /library directory from latest snapshot
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--repo sftp:u540853@u540853.your-storagebox.de:./backups \
--include "/data/*"

# 7. Copy restored data to /library
sudo cp -r /tmp/restore/data/* /library/

# 8. Restore /home/mnw
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--repo sftp:u540853@u540853.your-storagebox.de:./backups \
--include "/data/home"

sudo cp -r /tmp/restore/data/home/* /home/

# 9. Fix permissions
sudo chown -R mnw:mnw /library
sudo chown -R mnw:mnw /home/mnw

# 10. Restore PostgreSQL databases
# Each service (Authentik, Nextcloud, etc.) has a dump at /library/backup-stack/dumps/
# Use the restore procedure from each service's documentation

# Example: Restore Authentik PostgreSQL
docker exec authentik-app_postgresql_1 pg_restore \
/library/backup-stack/dumps/authentik-YYYYMMDD.sql < < (gunzip -c /path/to/dump)

# 11. Start all services
for dir in /library/*-app/; do
cd "$dir"
docker compose up -d
done

# 12. Verify critical services are running
docker ps | grep "server\|worker\|postgres"
</syntaxhighlight>

=== Scenario: Restore specific service from Restic ===

'''Example:''' Only Immich needs recovery, not entire system.

<syntaxhighlight lang="bash">
cd /library/backup-stack
docker compose up -d

# Restore just Immich config and database dump
docker exec restic-backup restic restore latest \
--target /tmp/restore \
--repo sftp:u540853@u540853.your-storagebox.de:./backups \
--include "/data/immich*" \
--include "/dumps/immich*"

# Move to correct location
mv /tmp/restore/data/immich-config /library/immich-app
mv /tmp/restore/data/immich-photos /raid1/Consolidated/Picture/Immich-Photos

# Restore the database
cd /library/immich-app
docker compose up -d immich_postgres
sleep 10
docker exec -i immich_postgres psql -U immich < <(gunzip -c /tmp/restore/dumps/immich-*.sql.gz)

# Start full Immich
docker compose up -d
</syntaxhighlight>

----

== Data Locations & Backup Coverage ==

=== What's Backed Up (via Restic) ===

<syntaxhighlight lang="text">
Backed up to Hetzner:
├── /data/authentik/ — Authentik OAuth provider config
├── /data/vaultwarden/ — Password vault (SQLite + attachments)
├── /data/nextcloud/ — File sync config
├── /data/immich-config/ — Photo manager config
├── /data/immich-photos/ — All photos (149GB)
├── /data/jellyfin/ — /var/lib/jellyfin (plugins, root, subs; live DBs excluded)
├── /data/jellyfin-etc/ — /etc/jellyfin (system config)
├── /data/jellyfin-server/ — /library/jellyfin-server-app (active metadata 28G — see services/JELLYFIN.md)
├── /data/memos/ — Notes app
├── /data/linkwarden/ — Bookmark manager
├── /data/open-webui/ — AI chat interface
├── /data/matrix-chat/ — Synapse + bridge configs
├── /data/stalwart/ — Mail server data
├── /data/snappymail/ — Webmail data
├── /data/mailcow/ — Legacy mail (to be archived)
├── /data/developer/ — /home/mnw/Developer (7.9G)
├── /data/home/ — /home/mnw (selective: excludes .git/objects, node_modules, .venv)
├── /data/beets/ — Music library DB
├── /dumps/ — PostgreSQL database dumps (daily)
│ ├── authentik-YYYYMMDD.sql.gz
│ ├── nextcloud-YYYYMMDD.sql.gz
│ ├── immich-YYYYMMDD.sql.gz
│ ├── linkwarden-YYYYMMDD.sql.gz
│ └── matrix-YYYYMMDD.sql.gz
</syntaxhighlight>

=== What's NOT Backed Up (Re-acquirable) ===

* <code>/var/lib/docker/volumes</code> ⚠️ (Timeshift excludes this; Restic needs to be updated to include)
* Jellyfin/Kaleidescape media files (movies, TV)
* Ollama models (re-downloadable)
* Steam games
* VirtualBox VMs
* node_modules, .venv, .git/objects, build artifacts

=== Timeshift Snapshots ===

<syntaxhighlight lang="text">
Stored on /raid1/timeshift/
├── 2025-05-03_17-15-00 (Backfill from old system)
├── 2026-05-03_11-52-11 (Backfill)
├── 2026-05-05_20-41-31 (Backfill)
├── 2026-05-05_21-36-57 (Backfill)
├── 2026-05-11_10-00-02 (Monthly snapshot)
├── 2026-05-12_00-00-01 (Daily snapshot)
├── 2026-05-12_10-00-01 (Weekly snapshot)
└── 2026-05-13_00-00-01 (Daily snapshot)
</syntaxhighlight>

Each snapshot contains a full filesystem image (~100-300GB depending on changes).

----

== Critical Gotchas ==

=== 1. ⚠️ Docker Volumes NOT in Timeshift ===

'''Problem:''' Timeshift is configured to exclude <code>/var/lib/docker/volumes</code>, which contains:
* PostgreSQL databases for Authentik, Nextcloud, Immich, Linkwarden, Matrix
* Docker-managed volumes

'''If you restore from Timeshift, you lose all database data.''' PostgreSQL will be empty.

'''Solution:''' Always use Restic for full recovery (it includes the databases via dumps), or manually restore PostgreSQL dumps.

'''Status:''' Known issue, needs fix in Timeshift config or backup-stack dumps.

=== 2. Hetzner Port is 23, NOT 22 ===

Many SSH configs default to port 22. Hetzner requires port 23.

* '''Wrong:''' <code>ssh u540853@u540853.your-storagebox.de</code>
* '''Right:''' <code>ssh -p 23 u540853@u540853.your-storagebox.de</code>

The <code>ssh-config</code> file in backup-stack handles this automatically.

=== 3. Hetzner Relative Paths Required ===

Hetzner SFTP doesn't support absolute paths for new directories.

* '''Wrong:''' <code>sftp:u540853@...:/backups</code> (fails with SSH_FX_FAILURE)
* '''Right:''' <code>sftp:u540853@....:./backups</code> (relative to home directory)

=== 4. Restic Password != Hetzner Password ===

Two different passwords:
* '''Hetzner account password:''' Logs into Hetzner web console
* '''Restic password:''' Encrypts your backup repository

Save both securely. If you lose the Restic password, your backups are unrecoverable.

=== 5. SSH Key Permissions in Docker ===

When mounting SSH keys into the Restic container, SSH is strict about ownership (must be 600 or 400).

The docker-compose entrypoint handles this:
<syntaxhighlight lang="yaml">
entrypoint: ["/bin/sh", "-c", "mkdir -p /root/.ssh && cp /tmp/ssh/'' /root/.ssh/ && chmod 600 /root/.ssh/'' && sleep infinity"]
</syntaxhighlight>

Don't modify without understanding SSH permission requirements.

=== 6. PostgreSQL Dumps Staged Before Backup ===

Database dumps are created at backup time by <code>run-backup.sh</code>:
<syntaxhighlight lang="bash">
docker exec <container> pg_dump -U <user> <db> | gzip > /library/backup-stack/dumps/<service>.sql.gz
</syntaxhighlight>

If a database is corrupted, the dump will fail, and that database won't be backed up. Check backup logs:
<syntaxhighlight lang="bash">
cat /library/backup-stack/dumps/backup-*.log | grep -i "error\|dump"
</syntaxhighlight>

=== 7. Backup Window (03:00 CST) Must Not Overlap with Other Heavy Operations ===

Backup runs daily at 03:00 CST. If you have other cron jobs or maintenance windows:
* Stagger them by at least 30 minutes
* Monitor systemd timers: <code>systemctl list-timers</code>

Concurrent heavy I/O can cause slowdowns or backup timeouts.

=== 8. Restic Prune Deletes Old Snapshots Automatically ===

Restic <code>prune</code> runs after backup to clean up old snapshots per retention policy:
* Keep 7 daily, 4 weekly, 6 monthly

If you manually delete recent snapshots (for testing), they may be pruned. Don't delete recent snapshots without understanding the implications.

----

== Verification & Testing ==

=== Daily Check: Backup Health ===

<syntaxhighlight lang="bash">
# Check if backup completed successfully
docker logs restic-backup | tail -20 | grep -i "success\|error"

# Check ntfy for backup alerts
curl https://ntfy.wilsoz.com/backup-alerts

# List recent snapshots
docker exec restic-backup restic snapshots --limit 5
</syntaxhighlight>

=== Weekly Test: Restore a File ===

<syntaxhighlight lang="bash">
# Restore a small non-critical file to /tmp
docker exec restic-backup restic restore latest \
--target /tmp/test-restore \
--include "/data/memos/*"

# Verify files exist
ls -la /tmp/test-restore/data/memos/

# Clean up
rm -rf /tmp/test-restore
</syntaxhighlight>

=== Monthly Test: Restore a Service Database ===

<syntaxhighlight lang="bash">
# Restore Linkwarden dump (small DB, ~50MB)
docker exec restic-backup restic restore latest \
--target /tmp/test-restore \
--include "/dumps/linkwarden*"

# Verify dump is valid
gunzip -t /tmp/test-restore/dumps/linkwarden-*.sql.gz

# Should report: OK (no errors)

# Clean up
rm -rf /tmp/test-restore
</syntaxhighlight>

=== Quarterly: Full System Restore Dry-Run ===

Once per quarter, do a mock recovery to verify procedures:

<syntaxhighlight lang="bash">
# Don't actually restore (too time-consuming), but verify:
# 1. Can list snapshots?
docker exec restic-backup restic snapshots --limit 1

# 2. Can stat the repository (verify password works)?
docker exec restic-backup restic stats

# 3. Can list files in a snapshot?
docker exec restic-backup restic ls latest | head -20

# If all three pass, recovery is ready.
</syntaxhighlight>

----

== Troubleshooting ==

=== Backup Failed ===

'''Check logs:'''
<syntaxhighlight lang="bash">
# Container logs
docker logs restic-backup | tail -50

# Systemd log
sudo journalctl -u backup.service -n 50

# Backup script log
ls -lht /library/backup-stack/dumps/backup-*.log
tail -100 /library/backup-stack/dumps/backup-*.log
</syntaxhighlight>

'''Common causes:'''
* SSH key permission error → Run entrypoint again: <code>docker compose up -d</code>
* Hetzner unreachable → Test SSH manually: <code>ssh -p 23 u540853@u540853.your-storagebox.de</code>
* Database dump failed → Check Docker container: <code>docker ps | grep postgres</code>
* Out of disk space → Check /library and /raid1: <code>df -h /library /raid1</code>

=== Restore Is Slow ===

Restic restores are network-bound from Hetzner. Typical speeds:
* Small file (< 100MB): 30 seconds
* Medium service (1-5GB): 5-15 minutes
* Large dataset (100+ GB): 1-3 hours

'''If slower than expected:'''
* Check network: <code>ping u540853.your-storagebox.de</code> (port 23 doesn't reply to ping)
* Try SSH: <code>ssh -p 23 u540853@u540853.your-storagebox.de echo ok</code>
* Check Restic cache: <code>du -sh /library/backup-stack/cache</code>

=== Can't Connect to Hetzner ===

<syntaxhighlight lang="bash">
# Test SSH connectivity
ssh -p 23 u540853@u540853.your-storagebox.de echo "Connected"

# If failed:
# 1. Check SSH key exists: ls -la ~/.ssh/id_rsa
# 2. Check Hetzner console: "External reachability" enabled?
# 3. Check your public key uploaded to Hetzner
# 4. Try with password instead: ssh -p 23 -o PubkeyAuthentication=no u540853@u540853.your-storagebox.de
</syntaxhighlight>

=== Restic Unlock Stuck ===

If Restic process was killed, lock file may remain:

<syntaxhighlight lang="bash">
# Remove lock
docker exec restic-backup restic unlock --repo sftp:u540853@...
</syntaxhighlight>

----

== Maintenance ==

=== Monthly: Review Snapshot Count ===

<syntaxhighlight lang="bash">
sudo timeshift --list

# Should show 5-8 snapshots. If too many:
sudo timeshift --delete-snapshot <snapshot_name>

# If too few, create one manually:
sudo timeshift --create --comments "Manual snapshot for recovery testing"
</syntaxhighlight>

=== Quarterly: Clean Up Restic Cache ===

<syntaxhighlight lang="bash">
# Cache can grow to 1-2GB over time
du -sh /library/backup-stack/cache

# Prune cache (safe, will be rebuilt)
docker exec restic-backup restic cache-dir
docker exec restic-backup rm -rf /root/.cache/restic/
</syntaxhighlight>

=== Annually: Update Restic Image ===

<syntaxhighlight lang="bash">
cd /library/backup-stack

# Update to latest Restic
docker compose pull
docker compose up -d

# Verify after update
docker exec restic-backup restic version
</syntaxhighlight>

----

== Links & References ==

* '''Timeshift Docs:''' https://github.com/teejee2008/timeshift
* '''Restic Docs:''' https://restic.readthedocs.io/
* '''Hetzner Storage Box:''' https://www.hetzner.com/storage/storage-box
* '''Backup Strategy Overview:''' <code>/home/mnw/Developer/SystemResiliency/DISASTER_RECOVERY.md</code>
* '''Backup Stack Code:''' <code>/home/mnw/Developer/SystemResiliency/backup-stack/</code>

----

== Related Services ==

* '''All services''' — depend on this backup system for disaster recovery
* '''OpenBao''' — stores Hetzner SSH key and Restic password
* '''ntfy''' — receives backup alerts (success/failure notifications)
* '''PostgreSQL''' (in each service) — database dumps backed up daily

Services/Omada

2026-07-05T18:16:20Z

Wikibot: Migrated from services/OMADA.md

''As of: 2026-07-05 — migrated from `services/OMADA.md`''

= TP-Link Omada Controller =

'''Version:''' 6.1 (mbentley/omada-controller)
'''Status:''' ✅ CRITICAL INFRASTRUCTURE — Network controller for all APs
'''Last Updated:''' 2026-05-11

----

== Quick Reference ==

{| class="wikitable"
|-
! Item
! Value
|-
| '''Container'''
| omada-controller (mbentley/omada-controller:6.1)
|-
| '''Compose'''
| <code>/library/omada-app/docker-compose.yml</code>
|-
| '''Data'''
| <code>/library/omada-app/data/</code> (505MB)
|-
| '''Logs'''
| <code>/library/omada-app/logs/</code>
|-
| '''Network'''
| host mode (required for AP discovery via multicast)
|-
| '''UI Port'''
| 8043 (HTTPS), 8188 (HTTP)
|-
| '''Memory'''
| ~1.5GB (expected for MongoDB + Java controller)
|-
| '''Restart Policy'''
| unless-stopped
|-
| '''Health Check'''
| healthy (docker ps shows status)
|}

----

== What It Controls ==

Omada is the '''centralized management platform''' for TP-Link enterprise WiFi:
* '''EAP610''' (or other managed APs) — firmware updates, channel optimization, roaming
* '''Network policies''' — SSID config, authentication, bandwidth limits
* '''Analytics''' — traffic, client health, interference detection
* '''Auto-HA''' (if configured) — automatic failover between controllers

'''Loss of Omada = Manual AP management only''' — devices will continue operating on last-known config, but no remote control.

----

== Backup & Recovery ==

=== Current Backup Status ===
✅ '''Included in daily restic backup'''
* Path: <code>/data/omada</code> in backup scope (docker-compose.yml line 37)
* Data persists in: <code>/library/omada-app/data/</code> (MongoDB + config)
* Size: ~505MB (small, incremental backups efficient)
* Database: embedded MongoDB (not separate container)

=== Restore Procedure ===

'''Single service restore (controller crashed):'''
<syntaxhighlight lang="bash">
# 1. Stop the container
docker compose -f /library/omada-app/docker-compose.yml down

# 2. Restore data from latest snapshot
docker exec restic-backup sh -c \
'export RESTIC_REPOSITORY=$HETZNER_REPO; export RESTIC_PASSWORD=$HETZNER_PASSWORD; \
restic restore latest --target /tmp/restore --include /data/omada'

# 3. Copy restored data
sudo cp -r /tmp/restore/data/omada /library/omada-app/data.restored
sudo rm -rf /library/omada-app/data
sudo mv /library/omada-app/data.restored /library/omada-app/data
sudo chown -R 508:508 /library/omada-app/data

# 4. Restart
docker compose -f /library/omada-app/docker-compose.yml up -d

# 5. Verify health
docker ps | grep omada-controller # should show "healthy"
</syntaxhighlight>

=== Full Disaster Recovery ===
If hardware is lost, Omada config is recoverable via:
# '''MongoDB backup''' in <code>/library/omada-app/data/</code> — restores all device configs, SSIDs, policies
# '''Re-adopt APs''' — once controller is back, press reset button on each AP, it rejoins the controller
# '''Fabric IP recovery''' — if you have Omada Fabric configured, endpoints will auto-rejoin after controller restart

----

== Memory Pressure ==

'''Why 1.5GB?'''
* Omada bundles '''MongoDB 5.0''' for config storage
* Java-based controller (OpenJDK 11+)
* Holds all device telemetry, client sessions, audit logs in memory
* Normal for enterprise controller this size

'''This is expected and required''' — memory is justified by critical infrastructure role.

----

== Key Ports ==

{| class="wikitable"
|-
! Port
! Protocol
! Purpose
|-
| 8043
| HTTPS
| Web UI + API (admin.omada.local)
|-
| 8188
| HTTP
| Portal (auto-redirects to HTTPS)
|-
| 29810-29820
| TCP
| Device communication (APs, switches)
|-
| 27001-27999
| TCP
| MongoDB internal replication
|}

----

== Operational Checks ==

'''Verify controller is healthy:'''
<syntaxhighlight lang="bash">
# Check container status
docker ps | grep omada-controller

# Check web UI (if SSH tunneled to port 8043)
curl -k https://localhost:8043/login # should respond with login page

# Monitor memory (expect 1.5GB)
docker stats omada-controller

# View recent logs
docker logs --tail=50 omada-controller | grep -E "ERROR|WARN|Exception"
</syntaxhighlight>

'''Verify APs are connected:'''
# Open https://admin.omada.local:8043 (or via Tailscale if remote)
# Go to '''Devices''' — should show all EAP610 APs as "Connected"
# If AP shows "Disconnected", check:
** AP has network connectivity (ping it)
** AP is on same LAN or can reach controller IP:29810
** Controller logs for adoption errors: <code>docker logs omada-controller | grep adopt</code>

----

== Upgrade Path ==

Current: '''6.1''' (latest as of 2026-05-11)

Watchtower (if enabled for this image) will auto-update. If manual:
<syntaxhighlight lang="bash">
cd /library/omada-app
docker compose pull && docker compose up -d
</syntaxhighlight>

'''Caution:''' Always backup before upgrading (restic snapshot is enough).

----

== Known Issues / Gotchas ==

# '''Host network mode required''' — Omada needs raw access to network broadcasts for AP discovery. Cannot use bridge networking.
# '''MongoDB locked on restart''' — If controller crashes mid-operation, MongoDB lock file may persist. Container entrypoint handles cleanup, but if stuck: <code>rm /library/omada-app/data/db/mongod.lock</code>
# '''Fabric timeout during big config changes''' — if Fabric mesh is configured, topology recalculation can take 2-5 minutes. Don't panic if APs briefly disconnect.
# '''TZ environment variable''' — set to <code>America/Chicago</code> for CST. Check via UI: Settings → General. If wrong, logs won't align with system time.

----

== Documentation References ==

* '''Omada API:''' https://documenter.getpostman.com/view/6114396 (community Postman docs)
* '''TP-Link Omada SDN:''' https://www.tp-link.com/en/business/omada-sdn-controller/ (official)
* '''Docker image:''' https://hub.docker.com/r/mbentley/omada-controller (custom image, better Linux support than official)

----

== Alerting ==

Should monitor for:
* ✅ Container health check (Uptime Kuma can ping :8043)
* ✅ Memory spike (>2GB = investigate, might indicate memory leak)
* ✅ AP disconnects (if >50% offline, controller may be hung)

Current monitoring: None configured (add to Uptime Kuma if needed)

Suggested: Create http monitor for https://localhost:8043/login returning 200 status

Services/AudioBookshelf

2026-07-05T18:16:20Z

Wikibot: Migrated from services/AUDIOBOOKSHELF.md

''As of: 2026-07-05 — migrated from `services/AUDIOBOOKSHELF.md`''

= AudioBookshelf =

== Service Info ==

* '''Type:''' Native systemd (not Docker)
* '''Version:''' 2.33.2
* '''Port:''' 13378
* '''Public URL:''' https://listen.wilsoz.com (proxied via VPS nginx → Tailscale → stronghold)
* '''Config:''' <code>/etc/default/audiobookshelf</code>
* '''Data:''' <code>/usr/share/audiobookshelf/</code> (config/, metadata/)
* '''DB:''' <code>/usr/share/audiobookshelf/config/absdatabase.sqlite</code> (owned: audiobookshelf:audiobookshelf)
* '''Service user:''' <code>audiobookshelf</code>
* '''Backups:''' <code>/raid1/Consolidated/AudioBookshelfBackup</code>, nightly 1:30 AM, 7 kept

<syntaxhighlight lang="bash">
systemctl status audiobookshelf
systemctl restart audiobookshelf
</syntaxhighlight>

== Authentik OIDC ==

* '''Provider:''' pk 18, name: "AudioBookshelf OAuth", slug: audiobookshelf
* '''Application slug:''' audiobookshelf, URL: https://listen.wilsoz.com
* '''Client ID:''' in OpenBao at <code>secret/audiobookshelf</code> field <code>oidc-client-id</code>
* '''Client Secret:''' in OpenBao at <code>secret/audiobookshelf</code> field <code>oidc-client-secret</code>
* '''Match existing accounts by:''' username
* '''Auto-register new users:''' yes
* '''Auto-launch SSO:''' no (local login still shown as fallback)

=== Redirect URIs registered in Authentik ===
<syntaxhighlight lang="text">
https://listen.wilsoz.com/auth/openid/callback
https://listen.wilsoz.com/audiobookshelf/auth/openid/callback
audiobookshelf://oauth
</syntaxhighlight>

=== OIDC Endpoints (from Authentik discovery) ===
<syntaxhighlight lang="text">
Issuer: https://key.wilsoz.com/application/o/audiobookshelf/
Authorize: https://key.wilsoz.com/application/o/authorize/
Token: https://key.wilsoz.com/application/o/token/
UserInfo: https://key.wilsoz.com/application/o/userinfo/
JWKS: https://key.wilsoz.com/application/o/audiobookshelf/jwks/
Logout: https://key.wilsoz.com/application/o/audiobookshelf/end-session/
</syntaxhighlight>

== Gotchas ==

'''OIDC config lives in SQLite, not a config file.''' Settings are in the <code>settings</code> table under key <code>server-settings</code> as a JSON blob. ABS must be stopped before writing (owned by audiobookshelf user):

<syntaxhighlight lang="bash">
systemctl stop audiobookshelf
sudo -u audiobookshelf sqlite3 /usr/share/audiobookshelf/config/absdatabase.sqlite \
"SELECT value FROM settings WHERE key='server-settings';" | python3 -m json.tool
systemctl start audiobookshelf
</syntaxhighlight>

'''<code>authOpenIDSubfolderForRedirectURLs</code> must be set to <code>""</code>''' (empty string, not absent). If this field is missing from the settings JSON, ABS sends <code>https://listen.wilsoz.com/undefined/auth/openid/callback</code> as the redirect URI — a JS <code>undefined</code> coerced to string. This field is normally set by the v2.17.4 migration, but only if OIDC was already enabled at migration time. If you configure OIDC by writing directly to the DB, add this field manually.

'''Authentik authorize/token/userinfo URLs are shared, not per-provider.''' Even with <code>issuer_mode: per_provider</code>, the actual endpoints are:
* <code>/application/o/authorize/</code> (not <code>/application/o/audiobookshelf/authorize/</code>)
* <code>/application/o/token/</code> (not <code>/application/o/audiobookshelf/token/</code>)
* <code>/application/o/userinfo/</code> (not <code>/application/o/audiobookshelf/userinfo/</code>)

Only the issuer, JWKS, and end-session URLs are per-provider. Use the discovery document to get the correct URLs:
<syntaxhighlight lang="bash">
curl -s https://key.wilsoz.com/application/o/audiobookshelf/.well-known/openid-configuration | python3 -m json.tool
</syntaxhighlight>

'''Authentik API PATCH returns 403 for providers''' — use PUT with the full provider object instead.

'''Python <code>urllib</code> drops Authorization header on HTTPS redirect''' — use <code>subprocess</code> + <code>curl</code> for Authentik API calls from Python scripts.

== Users ==

{| class="wikitable"
|-
! Username
! Email
! Type
|-
| admin
| (none)
| root
|-
| marcus
| (none)
| user
|-
| travis
| (none)
| user
|-
| mehone
| Mehonesleshi1989@gmail.com
| user
|-
| stefany
| (none)
| user
|}

Note: marcus and admin have no email stored — OIDC match is by username, not email.

Services/SnappyMail

2026-07-05T18:16:20Z

Wikibot: Migrated from services/SNAPPYMAIL.md

''As of: 2026-07-05 — migrated from `services/SNAPPYMAIL.md`''

= SnappyMail Webmail =

== Container & Files ==

{| class="wikitable"
|-
! Item
! Value
|-
| Container
| <code>snappymail</code>
|-
| Compose
| <code>/library/snappymail-app/docker-compose.yml</code>
|-
| Data dir
| <code>/library/snappymail-app/data/</code>
|-
| Image
| <code>djmaze/snappymail:latest</code>
|-
| Internal port
| <code>8888</code>
|-
| External port
| <code>8890</code> → proxied via VPS nginx
|}

<syntaxhighlight lang="bash">
cd /library/snappymail-app && docker compose up -d
docker logs --tail=50 snappymail 2>&1 | strings
</syntaxhighlight>

----

== Architecture Note — Unified vs Multi-Account ==

'''SnappyMail connected accounts ≠ unified inbox.'''

Adding external accounts (Gmail, iCloud, SDF) in SnappyMail Settings → Accounts shows them as separate silos side-by-side. There is no combined view.

'''The right approach for a unified inbox''' is to use '''getmail''' to pull all external accounts into Stalwart, then connect SnappyMail (and phones/clients) only to Stalwart. Sieve rules on Stalwart do the filtering. This way:
* One IMAP server to connect to from anywhere
* All mail filtered by sieve on arrival
* SnappyMail just sees one account with well-organized folders

Use SnappyMail's "additional accounts" feature only for occasional read-only access to an external account, not as the primary mail flow mechanism.

----

== Critical: MTU Must Be 1280 ==

The host's surfshark WireGuard VPN (surfshark_wg) has MTU 1280. All Docker bridge networks must match or large TLS packets (Server Hello + cert chain, ~4–6 KB) get silently dropped mid-handshake. '''Without MTU 1280, all external IMAP SSL connections hang for ~2 minutes then reset.'''

The <code>docker-compose.yml</code> already has this fix:

<syntaxhighlight lang="yaml">
networks:
default:
driver: bridge
driver_opts:
com.docker.network.driver.mtu: 1280
</syntaxhighlight>

If SnappyMail is redeployed or the network is recreated, verify MTU with:
<syntaxhighlight lang="bash">
docker network inspect snappymail-app_default | grep -i mtu
</syntaxhighlight>

After changing docker-compose.yml, you must <code>docker compose down && docker compose up -d</code> — <code>restart</code> alone does NOT recreate the network.

----

== Domain Config Files ==

Path: <code>/library/snappymail-app/data/_data_/_default_/domains/</code>

One JSON file per domain. Controls IMAP/SMTP server settings for that domain's accounts.

'''Ownership:''' Files must be owned by <code>uid 82 / gid 82</code> (container's www-data). If created as root, SnappyMail can't read them and shows "Domain is not allowed" or "no domain configuration".

<syntaxhighlight lang="bash">
sudo chown 82:82 /library/snappymail-app/data/_data_/_default_/domains/*.json
</syntaxhighlight>

Active domain configs:
{| class="wikitable"
|-
! File
! Server
! Notes
|-
| <code>wilsoz.com.json</code>
| <code>host.docker.internal</code>
| Stalwart — primary account
|-
| <code>craniumslows.com.json</code>
| <code>host.docker.internal</code>
| Stalwart — secondary address
|-
| <code>gmail.com.json</code>
| <code>imap.gmail.com:993</code>
| Requires App Password
|-
| <code>icloud.com.json</code>
| <code>imap.mail.me.com:993</code>
| Requires App Password
|-
| <code>me.com.json</code>
| <code>imap.mail.me.com:993</code>
| Same server as iCloud
|-
| <code>sdf.org.json</code>
| <code>mx.sdf.org:993</code>
| <code>shortLogin: true, lowerLogin: true</code>
|}

'''Editing domain configs:''' Do NOT bind-mount individual files — the container entrypoint does <code>sed -i</code> substitutions on its config files, which fails with "Resource busy" on bind-mounted files. Bind-mount the whole <code>data/</code> directory (already the case).

----

== Disabled Domains List ==

Path: <code>/library/snappymail-app/data/_data_/_default_/domains/disabled</code>

Plain text, one domain per line. Domains listed here cannot be added as accounts.

Current contents (as of 2026-02-22):
<syntaxhighlight lang="text">
outlook.com
qq.com
yahoo.com
hotmail.com
</syntaxhighlight>

<code>gmail.com</code> was in here by default — removed so users can add Gmail accounts.

----

== Stalwart Connection (wilsoz.com / craniumslows.com) ==

SnappyMail connects to Stalwart via <code>host.docker.internal</code> (resolves to the Docker bridge gateway → host → Stalwart port 993/587).

<syntaxhighlight lang="json">
{
"IMAP": { "host": "host.docker.internal", "port": 993, "type": 1 },
"SMTP": { "host": "host.docker.internal", "port": 587, "type": 2 }
}
</syntaxhighlight>

The <code>extra_hosts: ["host.docker.internal:host-gateway"]</code> in docker-compose.yml enables this.

----

== External Account App Passwords ==

For reference (these are app-specific passwords, not account passwords):

{| class="wikitable"
|-
! Account
! IMAP Server
! App Password
|-
| craniumslows@gmail.com
| imap.gmail.com:993
| <code>jeuwezdxaoaysvni</code>
|-
| thewilson@gmail.com
| imap.gmail.com:993
| <code>wikjuwnmklkeisid</code>
|-
| marcusthewilson@me.com
| imap.mail.me.com:993
| <code>pekx-kvid-iopx-lhyy</code>
|-
| mnw@sdf.org
| mx.sdf.org:993
| (SDF account password)
|}

----

== Troubleshooting ==

'''"Can't connect to host ssl://..."'''
* First check: MTU. Run <code>docker network inspect snappymail-app_default | grep -i mtu</code> — must be 1280.
* Test SSL from inside container: <code>docker exec snappymail php -r "...stream_socket_client('ssl://imap.gmail.com:993'...)"</code>
* If MTU is fine, test from host: <code>curl -v --max-time 10 imaps://imap.gmail.com:993</code>

'''"Domain is not allowed" / "no domain configuration"'''
* Check <code>disabled</code> file — is the domain listed there?
* Check if a <code>.json</code> file exists for that domain
* Check ownership: <code>sudo ls -la /library/snappymail-app/data/_data_/_default_/domains/</code>
* Fix: <code>sudo chown 82:82 /library/snappymail-app/data/_data_/_default_/domains/<domain>.json</code>

'''Admin panel'''
* URL: <code>http://localhost:8890/?admin</code> (or via VPS proxy)
* Default admin password set on first run, stored in <code>data/_data_/_default_/configs/application.ini</code>

Services/Mealie

2026-07-05T18:16:19Z

Wikibot: Migrated from services/MEALIE.md

''As of: 2026-07-05 — migrated from `services/MEALIE.md`''

= Mealie — Recipe Manager =

Complete setup & operations guide for Mealie (recipe management app) with Authentik OAuth integration.

----

== Overview ==

'''Mealie''' is a self-hosted recipe management app. We run it on Stronghold with:
* Authentik OAuth2 for user authentication
* Group-based access control (admin vs user roles)
* HTTPS via Cloudflare (cook.wilsoz.com)
* Persistent data storage in Docker volume

'''Current Users:''' marcus (admin via Authentik)
'''Access:''' https://cook.wilsoz.com

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''URL'''
| https://cook.wilsoz.com
|-
| '''Container name'''
| mealie
|-
| '''Compose file'''
| <code>/library/mealie-app/docker-compose.yml</code>
|-
| '''Data volume'''
| <code>mealie-app_mealie-data:/app/data</code>
|-
| '''Internal port'''
| 9000
|-
| '''Host port'''
| 9925
|-
| '''Network'''
| <code>mealie-app_mealie</code> (MTU 1280)
|-
| '''Image'''
| <code>ghcr.io/mealie-recipes/mealie:latest</code>
|}

----

== Initial Setup from Scratch ==

=== 1. Create Directory & Files ===

<syntaxhighlight lang="bash">
mkdir -p /library/mealie-app
cd /library/mealie-app
</syntaxhighlight>

=== 2. Create docker-compose.yml ===

<syntaxhighlight lang="yaml">
version: "3.8"
services:
mealie:
image: ghcr.io/mealie-recipes/mealie:latest
container_name: mealie
restart: unless-stopped
env_file:
- .env
volumes:
- mealie-data:/app/data
ports:
- "9925:9000"
networks:
- mealie

volumes:
mealie-data:

networks:
mealie:
driver: bridge
driver_opts:
com.docker.network.driver.mtu: 1280
</syntaxhighlight>

=== 3. Create .env File ===

<syntaxhighlight lang="bash">
# App Configuration
PUID=1000
PGID=1000
TZ=America/Chicago
BASE_URL=https://cook.wilsoz.com
ALLOW_SIGNUP=false

# OIDC Authentication (Authentik)
OIDC_AUTH_ENABLED=true
OIDC_PROVIDER_NAME=Authentik
OIDC_CONFIGURATION_URL=https://key.wilsoz.com/application/o/mealie/.well-known/openid-configuration
OIDC_CLIENT_ID=3fUfT5FALoAqvWHzqiD0eLNYKd3fsyYjdUkEUVy5
OIDC_CLIENT_SECRET=D5IbtoxMgx6Kkmry296bXUz6G3NcvzT70wIBrr7RFu8
OIDC_SIGNUP_ENABLED=true
OIDC_AUTO_REDIRECT=false
OIDC_USER_GROUP=family-and-friends
OIDC_ADMIN_GROUP=authentik Admins
OIDC_GROUPS_CLAIM=groups
OIDC_USER_CLAIM=email
OIDC_NAME_CLAIM=name
</syntaxhighlight>

=== 4. Set Up Authentik OAuth Provider ===

==== Create OAuth2 Provider in Authentik ====

# Go to https://key.wilsoz.com/if/admin/
# '''Applications → Providers → Create OAuth2/OpenID Provider'''
# Fill in:
''' '''Name:** mealie-oauth
''' '''Authorization flow:** (default-authentication-flow)
''' '''Client type:** Confidential
''' '''Redirect URIs:** (one per line)
```
https://cook.wilsoz.com/login
https://cook.wilsoz.com/login?direct=1
```
# '''Save''' — note the '''Client ID''' and '''Client Secret'''

==== Configure Provider: Signing Key ====

⚠️ '''CRITICAL:''' Assign the signing key or JWKS endpoint will return <code>{}</code> and OAuth will fail.

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.crypto.models import CertificateKeyPair

p = OAuth2Provider.objects.get(name='mealie-oauth')
jwt_key = CertificateKeyPair.objects.get(name='authentik Internal JWT Certificate')
p.signing_key = jwt_key
p.save()
print('Assigned JWT Certificate signing key')
"
</syntaxhighlight>

'''Why JWT Certificate?'''
* Uses HS256 algorithm (Mealie's OIDC library supports it)
* Self-signed Certificate uses RS256 (rejected by Mealie)

==== Configure Provider: Property Mappings ====

Add OAuth scope property mappings:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.core.models import PropertyMapping

p = OAuth2Provider.objects.get(name='mealie-oauth')
email_pm = PropertyMapping.objects.get(name='authentik default OAuth Mapping: OpenID \\'email\\'')
profile_pm = PropertyMapping.objects.get(name='authentik default OAuth Mapping: OpenID \\'profile\\'')
openid_pm = PropertyMapping.objects.get(name='authentik default OAuth Mapping: OpenID \\'openid\\'')

p.property_mappings.add(email_pm, profile_pm, openid_pm)
p.include_claims_in_id_token = True
p.save()
print('Added property mappings and enabled claims in ID token')
"
</syntaxhighlight>

==== Create Application in Authentik ====

# Go to '''Applications → Applications → Create'''
# Fill in:
''' '''Name:** Mealie
''' '''Slug:** mealie
''' '''Provider:** mealie-oauth (select from dropdown)
''' '''Authorization flow:** default-authorization-flow
# '''Save'''

=== 5. Start Mealie ===

<syntaxhighlight lang="bash">
cd /library/mealie-app
docker compose up -d
</syntaxhighlight>

Wait ~10 seconds for startup, then check:

<syntaxhighlight lang="bash">
docker logs mealie --tail=20
# Should show: "Application startup complete"
# And: "Enabled: True" under the OIDC section
</syntaxhighlight>

=== 6. Configure VPS Nginx Proxy ===

On VPS, add server block to nginx config (or update if exists):

<syntaxhighlight lang="nginx">
server {
listen 443 ssl http2;
server_name cook.wilsoz.com;

ssl_certificate /etc/letsencrypt/live/cook.wilsoz.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/cook.wilsoz.com/privkey.pem;

location / {
proxy_pass http://100.64.207.155:9925;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
</syntaxhighlight>

Then:

<syntaxhighlight lang="bash">
sudo systemctl reload nginx
</syntaxhighlight>

=== 7. Create First Admin User (In-App) ===

# Navigate to https://cook.wilsoz.com
# You'll see the login page with "Authentik" button
# If Mealie is brand new with no users, it may show an initial setup screen
# Use Authentik to log in with an account in the '''authentik Admins''' group
** This account gets Mealie admin role automatically

----

== Operations ==

=== Start/Stop/Restart ===

<syntaxhighlight lang="bash">
cd /library/mealie-app

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart mealie

# Full restart (clears cached config)
docker compose down && sleep 2 && docker compose up -d
</syntaxhighlight>

=== View Logs ===

<syntaxhighlight lang="bash">
# Live logs
docker logs -f mealie

# Last 50 lines
docker logs mealie --tail=50

# Filter for errors
docker logs mealie 2>&1 | grep -i "error\|exception"

# Filter for OIDC
docker logs mealie 2>&1 | grep -i "oidc"
</syntaxhighlight>

=== Reset Admin Password (Database) ===

If you lose the admin password:

<syntaxhighlight lang="bash">
docker exec mealie python3 -c "
import sqlite3
import bcrypt

db = sqlite3.connect('/app/data/mealie.db')
cursor = db.cursor()

# Set new password for marcus
new_password = 'TempPassword123456!'
salt = bcrypt.gensalt(rounds=10)
hashed = bcrypt.hashpw(new_password.encode('utf-8'), salt).decode('utf-8')

cursor.execute('UPDATE users SET password = ? WHERE username = ?', (hashed, 'marcus'))
db.commit()
print(f'Password reset to: {new_password}')
db.close()
"
</syntaxhighlight>

Then log in and change it in Mealie settings.

=== Manage Users & Groups ===

'''Via Authentik:'''
* Users must be added to Authentik first
* Create/manage groups: '''Directory → Groups'''
* Add users to groups for role assignment

'''Via Mealie:'''
* '''Settings → Users''' — shows Mealie users (created on first OAuth login)
* User roles are determined by Authentik group membership:
''' '''authentik Admins** → Mealie admin
''' '''family-and-friends** → Mealie user

----

== Configuration Reference ==

=== Environment Variables (<code>.env</code>) ===

{| class="wikitable"
|-
! Variable
! Purpose
! Example
|-
| <code>BASE_URL</code>
| External URL for Mealie
| <code>https://cook.wilsoz.com</code>
|-
| <code>ALLOW_SIGNUP</code>
| Allow registration without OIDC
| <code>false</code>
|-
| <code>OIDC_AUTH_ENABLED</code>
| Enable OAuth login
| <code>true</code>
|-
| <code>OIDC_CLIENT_ID</code>
| Authentik provider client ID
| (from Authentik)
|-
| <code>OIDC_CLIENT_SECRET</code>
| Authentik provider secret
| (from Authentik)
|-
| <code>OIDC_CONFIGURATION_URL</code>
| OIDC discovery endpoint
| <code>https://key.wilsoz.com/application/o/mealie/.well-known/openid-configuration</code>
|-
| <code>OIDC_USER_GROUP</code>
| Authentik group for regular users
| <code>family-and-friends</code>
|-
| <code>OIDC_ADMIN_GROUP</code>
| Authentik group for admins
| <code>authentik Admins</code>
|-
| <code>OIDC_GROUPS_CLAIM</code>
| OIDC claim name for groups
| <code>groups</code>
|-
| <code>OIDC_SIGNUP_ENABLED</code>
| Auto-create users on first OIDC login
| <code>true</code>
|-
| <code>OIDC_AUTO_REDIRECT</code>
| Skip login page, go straight to Authentik
| <code>false</code>
|-
| <code>TZ</code>
| Timezone
| <code>America/Chicago</code>
|-
| <code>PUID</code> / <code>PGID</code>
| Linux user/group for file permissions
| <code>1000</code>
|}

----

== Data & Backups ==

=== Data Location ===

<syntaxhighlight lang="text">
/library/mealie-app/mealie-data:/app/data

Contents:
- mealie.db (SQLite database — all recipes, users, settings)
- recipes/ (recipe files)
- users/ (user data)
- backups/ (exported recipe backups)
- templates/ (custom templates)
- groups/ (group/household data)
</syntaxhighlight>

=== Backup Procedure ===

<syntaxhighlight lang="bash">
# Backup the entire data volume
docker run --rm \
-v mealie-app_mealie-data:/data \
-v /home/mnw/backups:/backup \
ubuntu tar czf /backup/mealie-$(date +%Y%m%d).tar.gz /data

# List backups
ls -lh /home/mnw/backups/mealie-*.tar.gz
</syntaxhighlight>

=== Restore from Backup ===

<syntaxhighlight lang="bash">
# Stop Mealie
cd /library/mealie-app && docker compose down

# Restore data
docker run --rm \
-v mealie-app_mealie-data:/data \
-v /home/mnw/backups:/backup \
ubuntu bash -c "rm -rf /data/* && tar xzf /backup/mealie-20260513.tar.gz -C /data"

# Restart
docker compose up -d
</syntaxhighlight>

----

== Troubleshooting ==

=== Login Issues ===

==== "Something went wrong" on OAuth callback ====

'''Symptoms:'''
* Authentik login succeeds, but Mealie shows error when redirected back

'''Common causes & fixes:'''

{| class="wikitable"
|-
! Error
! Cause
! Fix
|-
| <code>KeyError: 'keys'</code>
| JWKS endpoint returns <code>{}</code>
| Assign signing key to OAuth provider (see Setup step 4)
|-
| <code>UnsupportedAlgorithmError: RS256 not allowed</code>
| Wrong signing key algorithm
| Use "authentik Internal JWT Certificate" (HS256), not "Self-signed Certificate" (RS256)
|-
| <code>invalid_client: Client authentication failed</code>
| Client secret mismatch
| Verify <code>OIDC_CLIENT_SECRET</code> in .env matches Authentik provider
|-
| HTTP 500 at <code>/api/auth/oauth/callback</code>
| Mealie can't reach Authentik's token endpoint
| Check network connectivity, SSL certificates, DNS
|}

'''Debug steps:'''

<syntaxhighlight lang="bash">
# Check OIDC config is loaded
docker logs mealie | grep -i "oidc\|enabled"

# Test discovery endpoint
docker exec mealie curl -s "https://key.wilsoz.com/application/o/mealie/.well-known/openid-configuration" -k | jq .

# Check JWKS endpoint has keys
docker exec mealie curl -s "https://key.wilsoz.com/application/o/mealie/jwks/" -k | jq '.keys | length'

# View OAuth errors
docker logs mealie --tail=100 | grep -i "oauth\|error\|exception" | tail -20
</syntaxhighlight>

==== "OIDC not appearing on login page" ====

* Verify <code>OIDC_AUTH_ENABLED=true</code> in <code>.env</code>
* Restart container: <code>docker compose down && docker compose up -d</code>
* Check logs for startup errors

==== "Redirect URI mismatch" error ====

* Ensure Authentik provider redirect URIs match exactly:
** <code>https://cook.wilsoz.com/login</code>
** <code>https://cook.wilsoz.com/login?direct=1</code>
* Verify <code>BASE_URL=https://cook.wilsoz.com</code> in <code>.env</code>
* No trailing slashes or port numbers

==== User not getting admin role ====

* Confirm user is in '''authentik Admins''' group (Directory → Groups)
* Log out and log back in (roles are assigned at login time)
* Check Mealie Settings → Users to see assigned role

=== Performance ===

==== Slow startup ====

Mealie can take 30-60 seconds on first start or after upgrade. Check:

<syntaxhighlight lang="bash">
docker logs mealie | tail -20
# Look for "Application startup complete"
</syntaxhighlight>

==== High memory usage ====

Default image is fine for small recipe collections. For very large:
* Monitor: <code>docker stats mealie</code>
* Limit: Add <code>mem_limit: 2g</code> to compose file if needed

=== Network Issues ===

==== Can't access https://cook.wilsoz.com ====

# Check VPS nginx config (see Setup step 6)
# Verify SSL cert is valid: <code>curl -I https://cook.wilsoz.com</code>
# Check Mealie is running: <code>docker ps | grep mealie</code>
# Check port: <code>netstat -tlnp | grep 9925</code>

==== Mealie can't reach Authentik (key.wilsoz.com) ====

From Mealie container:

<syntaxhighlight lang="bash">
docker exec mealie curl -v https://key.wilsoz.com/api/v3/ -k
# Should return 401 (not accessible, but shouldn't timeout)

docker exec mealie ping -c 1 key.wilsoz.com
# Should succeed
</syntaxhighlight>

----

== Security Notes ==

* '''OIDC_CLIENT_SECRET:''' Store securely in <code>.env</code>, never commit to git
* '''Database:''' <code>/app/data/mealie.db</code> contains all user data — protect with filesystem permissions
* '''Backups:''' Backup volume includes full database — encrypt backups if stored externally
* '''User signup:''' <code>ALLOW_SIGNUP=false</code> means only Authentik users can access (good)
* '''Groups:''' Ensure only trusted users are in <code>authentik Admins</code> group

----

== Links & References ==

* '''Mealie Docs:''' https://nightly.mealie.io/
* '''Mealie GitHub:''' https://github.com/mealie-recipes/mealie
* '''Authentik OIDC:''' https://goauthentik.io/docs/providers/oauth2
* '''Mealie OIDC Integration Guide:''' https://integrations.goauthentik.io/documentation/mealie/

----

== Related Services ==

* '''Authentik''' (auth): <code>/library/authentik-app/</code> — OAuth provider
* '''Nginx''' (proxy): VPS <code>/etc/nginx/sites-available/</code> — external access
* '''Backups:''' <code>/home/mnw/backups/</code> — data snapshots

Services/Jellyfin

2026-07-05T18:16:19Z

Wikibot: Migrated from services/JELLYFIN.md

''As of: 2026-07-05 — migrated from `services/JELLYFIN.md`''

= Jellyfin — Media Server =

Operations, backup, and recovery guide for stronghold's Jellyfin install.

'''Criticality:''' '''MEDIUM — Multi-user service.''' Media files themselves are large and not in scope for cloud backup (rerippable from physical media as a last resort). Server state (DB, metadata, plugins, config, subtitles) IS in scope — losing it means rebuilding the library from scratch, redoing all manual metadata fixes, and losing watch history. This is the only stronghold service used by multiple people.

----

== Overview ==

'''Jellyfin''' is the Free Software media server. Stronghold runs it as a native systemd service (not Docker) because of the package's tight integration with the host (ffmpeg, GPU, file permissions).

'''What's special about this install:'''
* Native systemd, not containerised. Updates via <code>apt</code>, not Watchtower.
* Metadata/cache/transcode paths have been '''relocated''' away from <code>/var/lib/jellyfin</code> to <code>/library/jellyfin-server-app/</code>. The old paths still exist on disk but are stale (untouched since Oct 2025).
* ~74G of <code>/var/lib/jellyfin/data</code> is regenerable cache (trickplay, subtitle extraction, internal backups). Only ~16G is genuinely irreplaceable.
* Uses SQLite — Jellyfin does not support PostgreSQL/MySQL in any non-experimental form as of 10.11.

'''Current Version:''' <code>10.11.8+ubu2404</code> (Debian package). Latest is <code>10.11.9</code> (released 2026-05-21) — pending upgrade; see [Maintenance](#maintenance).

'''Users:''' Multi-user — Marcus + family/friends. Auth is local Jellyfin accounts (not Authentik-federated).

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''Package'''
| <code>jellyfin</code> (Debian, Ubuntu 24.04 repo)
|-
| '''Systemd unit'''
| <code>/usr/lib/systemd/system/jellyfin.service</code>
|-
| '''Process user'''
| <code>jellyfin:jellyfin</code>
|-
| '''Working directory'''
| <code>/var/lib/jellyfin</code>
|-
| '''Environment file'''
| <code>/etc/default/jellyfin</code>
|-
| '''Binary'''
| <code>/usr/bin/jellyfin</code>
|-
| '''HTTP port (internal)'''
| 8096
|-
| '''HTTPS port (internal)'''
| 8920 (unused — TLS terminated upstream)
|-
| '''Public URL'''
| (whatever your reverse proxy/Tailscale exposes — internal-only by default)
|}

=== Companion services ===

{| class="wikitable"
|-
! Service
! Purpose
! Where
|-
| <code>logo-cache</code> (Docker, <code>nginx:alpine</code>)
| '''Jellyfin / IPTV support tool.''' Static HTTP server for the IPTV feed used by Jellyfin Live TV and the Apple TV IPTV client app. Serves: channel logos (30-day immutable cache) + <code>filtered.m3u</code> playlist + <code>filtered.xml</code> EPG (no-cache, must-revalidate — updater rewrites every ~3h via atomic rename). Stood up 2026-05-24.
| Compose: <code>/library/logo-cache-app/</code> · Listens: <code>0.0.0.0:8079</code> → container <code>:80</code> · Mounts: <code>/raid1/Consolidated/IPTV/logos → /usr/share/nginx/html:ro</code> (logo files) and <code>/raid1/Consolidated/IPTV → /srv/iptv:ro</code> (parent dir, exposed via nginx <code>alias</code> for <code>/filtered.m3u</code> + <code>/filtered.xml</code> only — directory bind, not single-file, so atomic-rename updates are picked up live) · Watchtower: enabled
|}

=== Data layout (the important table) ===

{| class="wikitable"
|-
! Path
! Purpose
! Size
! Backup?
|-
| <code>/etc/jellyfin/</code>
| System config (<code>system.xml</code>, <code>network.xml</code>, <code>encoding.xml</code>, users)
| <1M
| '''YES''' — mounted as <code>/data/jellyfin-etc</code>
|-
| <code>/var/lib/jellyfin/data/jellyfin.db</code>
| '''Core library DB''' (items, users, playstate)
| 1.3G
| '''YES via SQLite <code>.backup</code> to <code>/dumps/</code>''' — live file excluded from Restic
|-
| <code>/var/lib/jellyfin/data/playback_reporting.db</code>
| Playback history (from plugin)
| small
| '''YES via SQLite <code>.backup</code> to <code>/dumps/</code>'''
|-
| <code>/var/lib/jellyfin/data/subtitles/</code>
| Extracted/cached subs (<code>.srt</code>, <code>.ass</code>, <code>.sup</code>)
| ~40G
| YES — re-extraction is slow + lossy
|-
| <code>/var/lib/jellyfin/data/collections/</code>, <code>playlists/</code>, <code>ScheduledTasks/</code>, <code>livetv/</code>, <code>attachments/</code>
| User-curated state
| <100M
| YES
|-
| <code>/var/lib/jellyfin/plugins/</code>
| Installed plugins + their configs
| 14M
| YES
|-
| <code>/var/lib/jellyfin/root/</code>
| Library definitions
| 7M
| YES
|-
| <code>/var/lib/jellyfin/metadata/</code>
| '''STALE''' (legacy, pre-relocation, last write Oct 2025)
| 14G
| '''NO — excluded'''
|-
| <code>/var/lib/jellyfin/data/trickplay/</code>
| Generated thumbnails for scrub previews
| 16G
| NO — scheduled task regenerates
|-
| <code>/var/lib/jellyfin/data/backups/</code>
| Jellyfin's own internal backup plugin output
| 15G
| NO — we have our own
|-
| <code>/var/lib/jellyfin/data/just-in-case/</code>
| Leftover from some operation
| 2G
| NO — investigate + delete
|-
| <code>/var/lib/jellyfin/data/library.db.old''</code>, <code>library.db.bak''</code>, <code>library.db-wal</code>
| Pre-10.9 schema consolidation leftovers
| ~700M
| NO — safe to delete
|-
| <code>/var/lib/jellyfin/data/SQLiteBackups/</code>
| Jellyfin's auto-DB backups (separate from <code>/data/backups</code>)
| varies
| NO — we have our own
|-
| '''<code>/library/jellyfin-server-app/metadata/</code>'''
| '''ACTIVE metadata''' (images, NFOs, TMDB cache)
| 28G
| '''YES — mounted as <code>/data/jellyfin-server</code>'''
|-
| <code>/library/jellyfin-server-app/cache/</code>
| Image transcodes, web cache
| 739M
| NO — excluded
|-
| <code>/library/jellyfin-server-app/transcodes/</code>
| On-the-fly transcode working dir
| 588K
| NO — excluded
|}

Confirm relocation in <code>/etc/jellyfin/system.xml</code>:
<syntaxhighlight lang="xml">
<CachePath>/library/jellyfin-server-app/cache</CachePath>
<MetadataPath>/library/jellyfin-server-app/metadata</MetadataPath>
</syntaxhighlight>

=== Media files (NOT backed up) ===
* <code>/raid1/Consolidated/Video/Movies/</code>, <code>/raid1/Consolidated/Video/TV/</code>, etc. — many TB
* '''Strategy:''' Not backed up to cloud. Reproducible from physical media (rerip) as last resort. External RAID is the eventual local-only backup plan. Treat the library as effort-recoverable, not byte-recoverable.

----

== Backup ==

=== The cardinal rule ===

'''Never raw-copy a live SQLite WAL DB.''' A <code>cp jellyfin.db</code> (or unmodified file backup) while the server is running can yield an inconsistent file that fails to open, or worse, looks fine but is silently corrupt. SQLite's <code>.backup</code> command is the only safe online snapshot mechanism.

=== How it works on stronghold ===

Two things conspire to make Jellyfin backup safe:

# '''<code>host-dump-databases.sh</code>''' runs as <code>ExecStartPre</code> for <code>backup.service</code> (03:00 CST daily). For each SQLite DB it runs:
```sh
sqlite3 /var/lib/jellyfin/data/jellyfin.db ".backup '$DUMP_DIR/jellyfin-$DATE.db'"
gzip -f "$DUMP_DIR/jellyfin-$DATE.db"
```
The <code>.backup</code> command takes a short shared lock, copies pages atomically, and produces a file that's identical to a <code>VACUUM INTO</code>-style snapshot. Jellyfin keeps running throughout.

# '''<code>backup-to-hetzner.sh</code>''' then runs Restic. The live <code>jellyfin.db</code>, <code>playback_reporting.db</code>, and all their <code>-wal</code>/<code>-shm</code> sidecars are '''excluded''' so the only DB copy in the snapshot is the consistent one from <code>/dumps/</code>.

Restic also excludes the regenerable dirs (trickplay, internal backups, just-in-case, stale metadata path, cache, transcodes).

=== What's actually in a snapshot ===

Per nightly Restic run for Jellyfin:

{| class="wikitable"
|-
! Source mount
! Inside snapshot
! Approx after exclusions
|-
| <code>/etc/jellyfin</code> → <code>/data/jellyfin-etc</code>
| All config XML
| <1M
|-
| <code>/var/lib/jellyfin</code> → <code>/data/jellyfin</code>
| <code>plugins/</code>, <code>root/</code>, <code>data/{subtitles,collections,playlists,ScheduledTasks,livetv,attachments}</code>
| ~40G (subtitles dominate)
|-
| <code>/library/jellyfin-server-app</code> → <code>/data/jellyfin-server</code>
| <code>metadata/</code>
| 28G
|-
| <code>/library/backup-stack/dumps/</code> → <code>/dumps</code>
| <code>jellyfin-YYYYMMDD.db.gz</code>, <code>jellyfin-playback-YYYYMMDD.db.gz</code>
| ~300M compressed
|}

First snapshot ≈ 70G stored. Daily incrementals tiny thanks to Restic dedup/compression.

=== Verifying the backup ===

<syntaxhighlight lang="bash">
# Confirm today's SQLite dump was created
ls -lh /library/backup-stack/dumps/jellyfin-*.db.gz

# Spot-check dump integrity (extract + open)
gunzip -k /library/backup-stack/dumps/jellyfin-$(date +%Y%m%d).db.gz
sqlite3 /tmp/jellyfin-test.db "PRAGMA integrity_check;" # expect "ok"

# Confirm Restic snapshot contains the right paths
docker exec restic-backup restic snapshots --latest 1
docker exec restic-backup restic ls latest | grep -E "^/data/jellyfin-(etc|server)?" | head -20
</syntaxhighlight>

----

== Recovery ==

=== Scenario A: corrupt or lost <code>jellyfin.db</code> ===

The DB is the most fragile and most critical piece. Recovery in order of cost:

# '''Use the live <code>data/SQLiteBackups/</code>''' if it's recent and intact — Jellyfin keeps its own rolling SQLite copies here.
# '''Restore from <code>/library/backup-stack/dumps/jellyfin-YYYYMMDD.db.gz</code>''' (today's dump, lives locally on stronghold).
# '''Restore from Restic''' (Hetzner) if local dumps are gone too.

Procedure (option 2 or 3):
<syntaxhighlight lang="bash">
sudo systemctl stop jellyfin

# Move aside the broken DB
sudo mv /var/lib/jellyfin/data/jellyfin.db{,.broken-$(date +%s)}
sudo rm -f /var/lib/jellyfin/data/jellyfin.db-{wal,shm}

# From local dump:
sudo gunzip -c /library/backup-stack/dumps/jellyfin-YYYYMMDD.db.gz \
| sudo tee /var/lib/jellyfin/data/jellyfin.db >/dev/null
sudo chown jellyfin:jellyfin /var/lib/jellyfin/data/jellyfin.db

# OR from Restic:
docker exec restic-backup restic restore latest \
--target /tmp/jellyfin-restore \
--include /dumps/jellyfin-YYYYMMDD.db.gz
sudo gunzip -c /tmp/jellyfin-restore/dumps/jellyfin-*.db.gz \
| sudo tee /var/lib/jellyfin/data/jellyfin.db >/dev/null
sudo chown jellyfin:jellyfin /var/lib/jellyfin/data/jellyfin.db

sudo systemctl start jellyfin
</syntaxhighlight>

=== Scenario B: full server state loss ===

Rebuild order:
# Install <code>jellyfin</code> package, '''do not start''' the service yet.
# Restic restore <code>/data/jellyfin-etc/</code> → <code>/etc/jellyfin/</code>.
# Restic restore <code>/data/jellyfin/</code> → <code>/var/lib/jellyfin/</code> (minus the excluded paths — they'll regenerate).
# Restic restore <code>/data/jellyfin-server/</code> → <code>/library/jellyfin-server-app/</code>.
# Restore <code>jellyfin.db</code> and <code>playback_reporting.db</code> from <code>/dumps/</code> per Scenario A.
# <code>chown -R jellyfin:jellyfin /var/lib/jellyfin /library/jellyfin-server-app</code>.
# <code>systemctl start jellyfin</code>. First start will regenerate trickplay, transcode dirs, etc.
# Re-scan libraries from web UI (idempotent — won't re-import existing items).

=== Scenario C: media library on /raid1 is gone ===

Server state survives independently. Restore server state per Scenario B; then:
* Media has to come back from physical media (rerip) or external RAID if one exists.
* The library DB still references the original paths, so once the files reappear at the same paths, items light back up. No re-import needed.

----

== Health checks ==

Add as a weekly cron / systemd timer (see [TODO](#known-issues--tracked-work)):

<syntaxhighlight lang="bash">
sqlite3 /var/lib/jellyfin/data/jellyfin.db "PRAGMA integrity_check;"
sqlite3 /var/lib/jellyfin/data/playback_reporting.db "PRAGMA integrity_check;"
</syntaxhighlight>

Any output other than <code>ok</code> → page Matrix.

Also useful, opportunistically:
<syntaxhighlight lang="bash">
# What's holding the DB right now (should be ONLY the jellyfin process)
sudo lsof /var/lib/jellyfin/data/jellyfin.db

# Confirm WAL is on (it should always say "wal")
sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db "PRAGMA journal_mode;"

# Lock-error counts in the last day
sudo grep -c "database table is locked" /var/log/jellyfin/log_$(date +%Y%m%d).log
</syntaxhighlight>

----

== Maintenance ==

=== Apt update path ===
Jellyfin updates via the Ubuntu apt repo, '''not''' Watchtower (it's not a container).

<syntaxhighlight lang="bash">
# Check current vs latest
apt list --installed 2>/dev/null | grep jellyfin
# Latest tag: https://api.github.com/repos/jellyfin/jellyfin/releases/latest

# Upgrade pattern (always snapshot DB first):
sudo systemctl stop jellyfin
sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db ".backup '/library/backup-stack/dumps/jellyfin-pre-upgrade-$(date +%Y%m%d).db'"
sudo apt update && sudo apt install --only-upgrade jellyfin jellyfin-server jellyfin-web
sudo systemctl start jellyfin
</syntaxhighlight>

=== Scheduled tasks worth knowing about ===
Visible in the web UI at <code>Dashboard → Scheduled Tasks</code>. Two that matter:
* '''Extract Subtitles''' — runs daily, can cause <code>UserData</code> lock contention if it overlaps active playback (see [Gotchas](#gotchas)). Schedule defensively (e.g. 04:30, not 01:00).
* '''Refresh Library''' — keeps <code>jellyfin.db</code> warm. Don't run multiple library refreshes in parallel.

----

== Gotchas ==

=== SQLite "database table is locked" errors are internal, not external ===

If you see <code>SQLite Error 6: 'database table is locked: UserData'</code> in <code>/var/log/jellyfin/log_*.log</code>:
* Error code '''6 is <code>SQLITE_LOCKED</code>''' — same-process shared-cache contention. NOT another OS process holding the file. So it is '''not''' a backup/cp problem.
* Code 5 (<code>SQLITE_BUSY</code>) would indicate external process contention. We don't see that.
* Typical cause: '''PlaybackReporting plugin''' is in a tight write loop on <code>UserData</code> for an active session, and '''Extract Subtitles''' scheduled task fires at the same time and tries to write the same table.
* Mitigations:
## Move <code>Extract Subtitles</code> schedule off active-viewing hours.
## Test disabling PlaybackReporting plugin (you lose history but isolate the variable).
## Upgrade — 10.11.9 (PR #15368 + #16845) explicitly touches UserManager EF Core + user session concurrency logging.

These errors abort the failing transaction cleanly. They do '''not''' corrupt the DB.

=== Metadata path relocation ===

<code>system.xml</code> points <code>MetadataPath</code> and <code>CachePath</code> to <code>/library/jellyfin-server-app/</code> — not the package default of <code>/var/lib/jellyfin/</code>. The default-location dirs still exist with stale content from before relocation. '''Don't be fooled''' into thinking <code>/var/lib/jellyfin/metadata</code> is the live one; check <code>system.xml</code> to confirm. Both dirs are listed in the data-layout table above with their actual roles.

=== The <code>.sup</code> subtitle files ===

PGS (bitmap) subs from Blu-rays. Most are 5–30MB, but some legitimately exceed 100MB or 1GB — e.g. '''animated sing-along subtitles''' (Wicked's <code>.sup</code> is 1.1GB and intentional). '''Don't auto-delete by size.''' Look up the title via <code>BaseItems.Id</code> matching the dir hash (dashed-uppercase GUID) before deleting any oversized <code>.sup</code>. Genuine broken extractions can go; Jellyfin re-extracts on demand.

=== DB lookups ===
The on-disk subtitle dir name like <code>da/dab167ae-31c1-aa0a-7b64-dd821c44b879/4.sup</code> maps to a <code>BaseItems.Id</code>. The DB stores it dashed and uppercase:
<syntaxhighlight lang="bash">
sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db \
"SELECT Name, Type, Path FROM BaseItems WHERE Id='DAB167AE-31C1-AA0A-7B64-DD821C44B879';"
</syntaxhighlight>

=== Jellyfin's own backup plugin ===

There's a built-in Backup plugin that writes to <code>data/backups/</code>. We '''don't''' rely on it — our own <code>.backup</code> dumps are simpler to reason about. The <code>data/backups/</code> dir is excluded from Restic for that reason. If you ever want to use the built-in plugin instead, flip the exclusion and update this doc.

=== Stale <code>library.db.*</code> files ===

Jellyfin 10.9 consolidated multiple SQLite DBs into a single <code>jellyfin.db</code>. The leftover <code>library.db.old</code>, <code>library.db.bak1</code>, <code>library.db-wal</code> (~700M total) under <code>/var/lib/jellyfin/data/</code> are safe to delete after confirming Jellyfin starts cleanly without them — Jellyfin will not reach for them again.

----

== Known issues / tracked work ==

See <code>TODO.md</code> § Jellyfin for the live list. Current outstanding items:
* [ ] Investigate + delete oversized <code>.sup</code> files that are genuinely broken (size alone isn't a signal; manual eyeball required).
* [ ] Delete pre-10.9 schema cruft (<code>library.db.old</code>, <code>library.db.bak1</code>, etc.).
* [ ] Weekly <code>PRAGMA integrity_check</code> → Matrix alert on anything other than <code>ok</code>.
* [ ] Upgrade to 10.11.9 (UserManager/EF Core fixes; minor, do alongside next routine maintenance window).
* [ ] Haruhi S01E01 has a nightly subtitle-extraction failure (<code>Unable to get streams for File:...</code>) — investigate file integrity.

----

== References ==

* Backup stack overview: <code>services/BACKUP_SYSTEM.md</code>
* Restic include/exclude patterns: <code>/library/backup-stack/scripts/backup-to-hetzner.sh</code>
* SQLite dump invocations: <code>/library/backup-stack/scripts/host-dump-databases.sh</code>
* Disaster recovery procedure: <code>DISASTER_RECOVERY.md</code> (this service's recovery section above is the authoritative copy)
* Upstream changelog: https://github.com/jellyfin/jellyfin/releases

Services/OpenBao

2026-07-05T18:16:03Z

Wikibot: Migrated from services/OPENBAO.md

''As of: 2026-07-05 — migrated from `services/OPENBAO.md`''

= OpenBao Secret Store =

Self-hosted Vault-API-compatible secret store. The canonical record for credentials used by Claude Code (MCP), shell scripts, and n8n workflows.

'''Vaultwarden remains the human-facing store.''' OpenBao is the machine-facing one. They mirror each other; on rotation, update both.

----

== Container & Files ==

{| class="wikitable"
|-
! Item
! Value
|-
| Container
| <code>openbao</code>
|-
| Compose
| <code>/library/openbao-app/docker-compose.yml</code>
|-
| Config
| <code>/library/openbao-app/config/openbao.hcl</code>
|-
| Storage
| <code>/library/openbao-app/data/</code> (Raft)
|-
| Init output
| <code>/library/openbao-app/.init-output.json</code> (chmod 600 — root token + unseal key)
|-
| Scripts token
| <code>/library/openbao-app/.scripts-token</code> (chmod 600 — used by <code>bao-get.sh</code>)
|-
| Claude token
| <code>/library/openbao-app/.claude-token</code> (chmod 600 — also embedded in <code>~/.claude.json</code>)
|-
| Auto-unseal
| <code>/usr/local/bin/openbao-unseal.sh</code> + <code>openbao-unseal.service</code> (systemd)
|-
| Helper script
| <code>/usr/local/bin/bao-get.sh</code>
|-
| n8n cred sync
| <code>/library/openbao-app/sync-n8n-creds.py</code> (timer: <code>sync-n8n-creds.timer</code>)
|-
| MCP binary
| <code>/usr/local/bin/vault-mcp-server</code> (HashiCorp v0.2.0)
|-
| '''VC sources'''
| <code>scripts/openbao/</code> in this repo — see <code>scripts/openbao/README.md</code> for deploy
|}

<syntaxhighlight lang="bash">
cd /library/openbao-app && docker compose up -d
docker logs --tail=50 openbao 2>&1 | grep -v "^chown\|Could not chown"
</syntaxhighlight>

----

== Ports & Endpoints ==

{| class="wikitable"
|-
! Port
! Purpose
|-
| 8200
| API + UI (HTTP, no TLS — internal only)
|-
| 8201
| Cluster (Raft consensus, single-node here)
|}

{| class="wikitable"
|-
! Endpoint
! URL
|-
| API (host)
| <code>http://localhost:8200</code>
|-
| API (LAN)
| <code>http://192.168.0.109:8200</code>
|-
| API (Tailscale)
| <code>http://100.64.207.155:8200</code>
|-
| API (n8n container)
| <code>http://172.29.0.1:8200</code>
|-
| Web UI
| <code>http://192.168.0.109:8200/ui</code>
|}

'''Not exposed publicly''' — no VPS nginx route, intentionally. UI is LAN/Tailscale only.

----

== Credentials ==

'''All in Vaultwarden → "Stronghold Infrastructure" folder.'''

{| class="wikitable"
|-
! Vaultwarden item
! Contents
|-
| OpenBao Unseal
| root token (<code>s.TqAL...</code>) + unseal key (<code>dHIJ...=</code>) in notes
|-
| OpenBao Tokens
| claude-mcp token (login.password) + scripts-n8n token (custom field)
|}

The init-output.json on stronghold has the same data — that's how <code>openbao-unseal.service</code> reads the unseal key at boot.

----

== Reading Secrets ==

=== From Claude Code (MCP) ===

Just ask: ''"fetch secret/ntfy from vault"'' or ''"what's the mxroute api-key?"'' — the <code>vault</code> MCP server (16 tools) handles it.

=== From shell scripts ===

<syntaxhighlight lang="bash">
bao-get.sh PATH FIELD
# e.g.
bao-get.sh ntfy token
bao-get.sh mxroute api-key
bao-get.sh authentik api-token
</syntaxhighlight>

Reads from <code>/library/openbao-app/.scripts-token</code>. Outputs the field value to stdout, no newline.

=== From curl (any host with LAN access) ===

<syntaxhighlight lang="bash">
TOKEN=$(cat /library/openbao-app/.scripts-token)
curl -sf http://localhost:8200/v1/secret/data/PATH \
-H "X-Vault-Token: $TOKEN" \
| jq -r '.data.data.FIELD'
</syntaxhighlight>

=== From n8n ===

HTTP Request node:
* Method: <code>GET</code>
* URL: <code>http://172.29.0.1:8200/v1/secret/data/PATH</code>
* Headers: <code>X-Vault-Token: <scripts-n8n token></code>
* Response field: <code>data.data.<field></code>

Or in a Code node:
<syntaxhighlight lang="javascript">
const resp = await this.helpers.httpRequest({
method: 'GET',
url: 'http://172.29.0.1:8200/v1/secret/data/mxroute',
headers: { 'X-Vault-Token': '<scripts-n8n token>' },
});
const apiKey = resp.data.data['api-key'];
</syntaxhighlight>

'''Recommended:''' store the scripts-n8n token as an <code>httpHeaderAuth</code> credential in n8n (header name <code>X-Vault-Token</code>) so workflows reference it by credential ID.

----

== Writing Secrets (via root token) ==

<syntaxhighlight lang="bash">
ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)

# kv put REPLACES the entire record — pass all fields you want kept
docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$ROOT_TOKEN" openbao \
bao kv put secret/PATH field1=value1 field2=value2
</syntaxhighlight>

For multiline values (private keys, certs), use the API directly:

<syntaxhighlight lang="bash">
python3 - << 'PY'
import json, urllib.request
payload = {"data": {"private-key": open('/path/to/key').read()}}
req = urllib.request.Request(
"http://localhost:8200/v1/secret/data/PATH",
method="POST", data=json.dumps(payload).encode(),
headers={"X-Vault-Token": "<ROOT_TOKEN>", "Content-Type": "application/json"})
print(urllib.request.urlopen(req).read())
PY
</syntaxhighlight>

----

== Secret Inventory ==

{| class="wikitable"
|-
! Path
! Fields
! Source-of-truth status
|-
| <code>secret/n8n</code>
| <code>api-key</code>
| OpenBao canonical; n8n's own auth is separate
|-
| <code>secret/mxroute</code>
| <code>server</code>, <code>username</code>, <code>api-key</code>
| OpenBao canonical
|-
| <code>secret/homeassistant</code>
| <code>token</code>, <code>host</code>, <code>port</code>
| OpenBao canonical; mirrored in n8n credential <code>R9LGxX...</code> and <code>4RowxM...</code>
|-
| <code>secret/ntfy</code>
| <code>token</code>
| OpenBao canonical; mirrored in n8n credential <code>w0gXpM...</code>
|-
| <code>secret/authentik</code>
| <code>api-token</code>
| OpenBao canonical
|-
| <code>secret/matrix</code>
| <code>zordon-token</code>
| OpenBao canonical; mirrored in n8n credential <code>x5tv5a...</code>
|-
| <code>secret/stalwart</code>
| <code>admin-password</code>
| OpenBao canonical
|-
| <code>secret/immich</code>
| <code>admin-api-key</code>, <code>user-api-key</code>
| OpenBao canonical
|-
| <code>secret/bitwarden</code>
| <code>client-id</code>, <code>client-secret</code>, <code>master-password</code>
| For <code>bw</code> CLI; also in <code>~/.config/claude/vaultwarden.env</code>
|-
| <code>secret/openai</code>
| <code>api-key</code>, <code>bearer-header</code>
| OpenBao canonical; mirrored in n8n credentials <code>NIQrbn...</code>, <code>XB0V2U...</code>
|-
| <code>secret/anthropic</code>
| <code>api-key</code>
| OpenBao canonical; mirrored in n8n credential <code>bEPlqP...</code>
|-
| <code>secret/audiobookshelf</code>
| <code>bearer-header</code>
| OpenBao canonical; mirrored in n8n credential <code>5MCzns...</code>
|-
| <code>secret/ssh-stronghold</code>
| <code>username</code>, <code>host</code>, <code>port</code>, <code>private-key</code>
| OpenBao canonical; mirrored in n8n credential <code>2ugS1i...</code>
|}

To list paths live: <code>bao-get.sh</code> doesn't support list — use the UI or:
<syntaxhighlight lang="bash">
ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$ROOT_TOKEN" openbao bao kv list secret/
</syntaxhighlight>

----

== Operational ==

=== Restart ===

OpenBao re-seals on every restart. The systemd service handles auto-unseal:

<syntaxhighlight lang="bash">
cd /library/openbao-app && docker compose restart
sudo systemctl start openbao-unseal.service
docker exec -e BAO_ADDR=http://127.0.0.1:8200 openbao bao status | grep Sealed
# Expected: Sealed: false
</syntaxhighlight>

Manual unseal (if the systemd unit fails):
<syntaxhighlight lang="bash">
/usr/local/bin/openbao-unseal.sh
</syntaxhighlight>

=== Backups ===

<code>/library/openbao-app/</code> is mounted into <code>restic-backup</code> as <code>/data/openbao:ro</code> and included in the nightly Hetzner restic backup. Both the Raft data '''and''' <code>.init-output.json</code> are backed up — restoring requires both (the unseal key would be lost otherwise).

=== n8n credential sync ===

n8n credentials are mirrored from OpenBao by <code>/library/openbao-app/sync-n8n-creds.py</code>. The script reads each managed credential's spec (n8n ID + OpenBao path + field mapping) from a <code>MAPPINGS</code> table, diffs against <code>n8n export:credentials --decrypted</code>, and runs <code>n8n import:credentials</code> only on drift. '''Match is by credential ID — never creates new credentials.''' Posts to <code>#notifications:wilsoz.com</code> on change.

<syntaxhighlight lang="bash">
/library/openbao-app/sync-n8n-creds.py --dry-run # preview drift
/library/openbao-app/sync-n8n-creds.py # sync all
/library/openbao-app/sync-n8n-creds.py --only ntfy_auth # one mapping
/library/openbao-app/sync-n8n-creds.py --quiet # cron-friendly
</syntaxhighlight>

Daily timer: <code>sync-n8n-creds.timer</code> (04:30 + 0–5 min jitter). Logs in <code>journalctl -u sync-n8n-creds.service</code>.

'''Rotation flow:''' update OpenBao → next nightly sync (or manual run) propagates to n8n. No workflow edits.

'''Adding a new mapping:''' append to <code>MAPPINGS</code> in the script. Must specify <code>n8n_id</code> (existing credential — create the credential in n8n UI first), <code>n8n_name</code>, <code>n8n_type</code>, <code>bao_path</code>, and a <code>build</code> lambda that returns the n8n <code>data</code> dict from the OpenBao field dict.

=== Token rotation ===

<syntaxhighlight lang="bash">
ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
# Revoke old token
docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$ROOT_TOKEN" openbao \
bao token revoke <OLD_TOKEN>
# Mint new with same policy
docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$ROOT_TOKEN" openbao \
bao token create -policy=readonly-secrets -no-default-policy -ttl=0 -display-name="..."
</syntaxhighlight>

Update <code>~/.claude.json</code> (claude-mcp token) and <code>/library/openbao-app/.scripts-token</code> (scripts-n8n token) to the new values, then update Vaultwarden.

=== Adding a new secret path ===

<syntaxhighlight lang="bash">
ROOT_TOKEN=$(jq -r .root_token /library/openbao-app/.init-output.json)
docker exec -e BAO_ADDR=http://127.0.0.1:8200 -e BAO_TOKEN="$ROOT_TOKEN" openbao \
bao kv put secret/NEWPATH field1=value1
</syntaxhighlight>

The <code>readonly-secrets</code> policy uses <code>path "secret/data/*"</code> so new paths are immediately readable by claude-mcp and scripts-n8n tokens — no policy update needed.

----

== Gotchas (hard-won) ==

# '''Container runs as <code>user: "1000:1000"</code> (mnw).''' Without this, files on <code>/library/openbao-app/data</code> end up owned by UID 100 (<code>messagebus</code> on host) which is confusing.
# '''Must set <code>SKIP_CHOWN=1</code> and <code>SKIP_SETCAP=1</code> env vars.''' Without them, the entrypoint's <code>chown /openbao/logs</code> fails under <code>set -e</code> and the container restart-loops.
# '''<code>docker exec</code> to bao CLI needs <code>-e BAO_ADDR=http://127.0.0.1:8200</code>.''' The CLI defaults to https else and fails with "server gave HTTP response to HTTPS client".
# '''Sealed on every restart.''' Raft + Shamir means the box re-seals on container start. Auto-unseal via the systemd service. Don't delete <code>.init-output.json</code>.
# '''<code>bao kv put</code> replaces the whole record.''' Always pass all fields you want kept, not just the changed one.
# '''Multi-line values fail with <code>bao kv put</code> shell escaping.''' Use the HTTP API (curl/python urllib) directly for private keys, certs, etc.
# '''vault-mcp-server release binaries.''' GitHub releases page shows zero assets — download from <code>releases.hashicorp.com</code> instead (<code>https://releases.hashicorp.com/vault-mcp-server/0.2.0/vault-mcp-server_0.2.0_linux_amd64.zip</code>).
# '''<code>bao kv put field=@/path/to/file</code> reads from container fs, not host.''' Either copy the file into the container first or use the HTTP API.

----

== Related ==

* Setup plan (preserved for archaeology): <code>plans/openbao-setup.md</code>
* Memory note: <code>~/.claude/projects/.../memory/project_openbao.md</code>
* CLAUDE.md "OpenBao Secret Store" section

Services/Vaultwarden

2026-07-05T18:16:03Z

Wikibot: Migrated from services/VAULTWARDEN.md

''As of: 2026-07-05 — migrated from `services/VAULTWARDEN.md`''

= Vaultwarden =

Self-hosted Bitwarden-compatible password manager. The '''human-facing''' credential store. (OpenBao is the machine-facing one — see <code>services/OPENBAO.md</code>.)

----

== Container & Endpoints ==

{| class="wikitable"
|-
! Item
! Value
|-
| Container
| <code>vaultwarden</code>
|-
| Compose
| <code>/library/vaultwarden-app/</code>
|-
| Public URL
| <code>https://vault.wilsoz.com</code> (via VPS nginx → Tailscale → stronghold)
|-
| Internal port
| 8080
|-
| Auth
| OAuth2 via Authentik (provider PK 10)
|-
| Master folder
| "Stronghold Infrastructure" (all infrastructure credentials)
|}

----

== Claude Code Integration ==

Claude reads Vaultwarden via the <code>bitwarden</code> MCP server (configured at the user level — works in every Claude session on this machine).

=== Files ===

{| class="wikitable"
|-
! File
! Mode
! Purpose
|-
| <code>~/.config/claude/vaultwarden.env</code>
| 600
| BW_URL, BW_CLIENTID, BW_CLIENTSECRET, BW_EMAIL, BW_PASSWORD
|-
| <code>~/.config/claude/bw-mcp-start.sh</code>
| 700
| Wrapper that unlocks vault and execs the MCP server
|-
| <code>~/.config/claude/bw-get.py</code>
| 700
| One-off shell helper (<code>bw-get.py "Item Name" [field]</code>)
|-
| <code>~/.claude.json</code> → <code>mcpServers.bitwarden</code>
| —
| Registers the MCP with Claude Code
|-
| <code>/snap/bin/bw</code>
| —
| Bitwarden CLI (snap-installed, official)
|-
| <code>/home/mnw/.nvm/versions/node/v22.12.0/lib/node_modules/@bitwarden/mcp-server/</code>
| —
| MCP server (npm-installed globally)
|}

=== How the wrapper works ===

# Reads <code>vaultwarden.env</code> via Python (avoids shell quoting issues with special chars in BW_PASSWORD).
# '''Refuses to start if <code>BW_URL != https://vault.wilsoz.com</code>''' — guard against accidentally authing to the wrong server.
# Runs <code>bw config server</code> (idempotent) so the CLI is pointed at the self-hosted instance.
# Logs in via <code>--apikey</code> (uses BW_CLIENTID/BW_CLIENTSECRET) if not already logged in.
# Unlocks with <code>bw unlock --passwordenv BW_PASSWORD --raw</code> and exports <code>BW_SESSION</code>.
# '''Execs <code>node <path>/dist/index.js</code>''' (not the <code>mcp-server-bitwarden</code> symlink — see Known Bugs below).

=== Available MCP tools ===

58 tools, covering: <code>lock</code>, <code>sync</code>, <code>status</code>, <code>list</code>, <code>get</code>, <code>generate</code>, <code>create_item</code>, <code>create_folder</code>, <code>edit_item</code>, <code>edit_folder</code>, <code>delete</code>, plus organization/group/policy/send management. Used like any other MCP — e.g., "list items matching voip" or "create a new item named X".

----

== Asking the User for Credentials ==

When you need a credential that isn't in Vaultwarden yet (e.g., a new third-party API key the user just got), '''never ask the user to paste it in chat'''. Instead:

=== Pattern: "Save it to Vaultwarden, then I'll fetch it" ===

Ask the user to do this:

:Please add it to Vaultwarden:
:1. Open https://vault.wilsoz.com
:2. New Item → Folder: '''Stronghold Infrastructure'''
:3. Name: <code><service-name></code> (e.g., <code>voip.ms</code>)
:4. Add fields:
: - <code>username</code> = …
: - <code>password</code> = …
: - Custom fields for anything else (e.g., <code>api-key</code>, <code>did</code>)
:5. Save, then tell me "done"
:
:I'll fetch it from Vaultwarden via the MCP — you never need to paste secrets here.

Then on your side:
<syntaxhighlight lang="text">
[use bitwarden MCP] list items matching "<service-name>"
[use bitwarden MCP] get item <id>
[use vault MCP] write secret/<service-name> with the retrieved fields
</syntaxhighlight>

=== Why this matters ===

* Chat transcripts are logged. Secrets in chat = secrets in logs forever.
'' The user has explicitly said: ''never store or surface credentials, use Vaultwarden + OpenBao.*
* If you accidentally print a secret (e.g., via <code>bash -x</code>, <code>set -x</code>, <code>cat env-file</code>, or echoing a fetched variable) — '''stop, tell the user immediately, and ask them to rotate that credential.'''

----

== Hard-Won Gotchas ==

=== 1. <code>mcp-server-bitwarden</code> symlink does not work (v2026.2.0) ===

The npm package installs a symlink at <code>…/bin/mcp-server-bitwarden</code> → <code>…/dist/index.js</code>. The script at the end of <code>index.js</code> checks <code>process.argv[1].endsWith('index.js')</code> to decide whether to start the server. The symlink path doesn't end in <code>index.js</code>, so the server never starts — it just exits silently with code 0.

'''Workaround:''' invoke via <code>node</code> with the full path:
<syntaxhighlight lang="bash">
exec node "$(npm root -g)/@bitwarden/mcp-server/dist/index.js"
</syntaxhighlight>

=== 2. Never use <code>bash -x</code> or <code>set -x</code> near <code>source vaultwarden.env</code> ===

Bash tracing prints every variable assignment to stderr, including BW_PASSWORD. '''This will leak the master password into your terminal scrollback, into logs, and into Claude conversation transcripts.''' Use targeted <code>echo</code> debugging or <code>python3</code> parsing instead.

=== 3. <code>bw login</code> defaults to <code>vault.bitwarden.com</code> ===

If the bw CLI's local config gets reset (<code>~/.config/Bitwarden CLI/data.json</code> deleted), it will default to the public Bitwarden cloud. Running <code>bw login</code> against the wrong server '''sends your credentials to Bitwarden, Inc.''' Always run <code>bw config server "$BW_URL"</code> first. The wrapper script enforces this with a refuse-to-start guard.

=== 4. <code>bw login</code> vs <code>bw unlock</code> ===

* <code>bw login</code> = authenticate the device to the server (once per device, persisted in <code>data.json</code>).
* <code>bw unlock</code> = decrypt the local vault cache (once per session, returns BW_SESSION).
* The MCP server needs both. The wrapper handles login idempotently; unlock runs on every start.

=== 5. <code>bw login</code> with email + password fails on Vaultwarden ===

Vaultwarden / Bitwarden self-hosted requires <code>bw login --apikey</code> (client credentials) rather than <code>bw login email password</code>. The email/password flow expects 2FA prompts and an interactive TTY.

----

== Rotating BW_PASSWORD ==

If the master password leaks (e.g., bash -x mistake):

# '''In Vaultwarden web UI:''' Account Settings → Change Master Password.
# '''On stronghold:''' Update <code>~/.config/claude/vaultwarden.env</code>:
```bash
# Edit the line: BW_PASSWORD='new-password-here'
chmod 600 ~/.config/claude/vaultwarden.env # verify mode still 600
```
# '''Re-login to bw:'''
```bash
bw logout
set -a; source ~/.config/claude/vaultwarden.env; set +a
bw config server "$BW_URL"
bw login --apikey
```
# '''Next Claude session''' will pick up the new password from the env file automatically.

The <code>BW_CLIENTSECRET</code> (API key) is a separate value — only rotate it if it leaks. Rotating it requires regenerating in Account Settings → Security → Keys → View API Key.

----

== Related Docs ==

* <code>services/OPENBAO.md</code> — the machine-facing companion store
* <code>services/OPENBAO.agent.md</code> — agent-focused decision tree for credential access

Services/Authentik

2026-07-05T18:16:02Z

Wikibot: Migrated from services/AUTHENTIK.md

''As of: 2026-07-05 — migrated from `services/AUTHENTIK.md`''

= Authentik — Identity & SSO Provider =

Complete setup & operations guide for Authentik (identity provider, OAuth2/OIDC, LDAP directory service).

'''Criticality:''' '''CRITICAL — The lynchpin.''' Everything on Stronghold depends on Authentik for authentication. Without it, most other services are inaccessible or in an inconsistent state. Must be recovered first in any disaster scenario.

----

== Overview ==

'''Authentik''' is a self-hosted identity provider that serves as the single source of truth for authentication across Stronghold. It provides:
* '''OAuth2 / OIDC''' providers for web apps (Immich, Memos, Linkwarden, Forgejo, Vaultwarden, Grafana, n8n, Matrix, Cloudflare Access)
* '''LDAP directory''' for apps that don't support OAuth (Nextcloud, Jellyfin, Stalwart mail server)
* '''Group membership''' for role-based access (authentik Admins, family-and-friends, email provisioning groups)
* '''User management''' — all users are defined here; other services sync from Authentik
* '''API tokens''' for programmatic access (n8n, CI/CD, automation)

'''Current Users:'''
* <code>marcus</code> (primary admin user, member of authentik Admins)
* <code>akadmin</code> (backup admin user)
* Service accounts: <code>stalwart-svc</code>, <code>nextcloud-svc</code> (for LDAP directory access)

'''Current Version:''' 2025.10.3

'''Access:''' https://key.wilsoz.com/if/admin/

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''Compose file'''
| <code>/library/authentik-app/docker-compose.yml</code>
|-
| '''Containers'''
| <code>authentik-app_server_1</code>, <code>authentik-app_worker_1</code>, <code>authentik-app_postgresql_1</code>
|-
| '''Database'''
| PostgreSQL 16 (in-container, persisted to Docker volume <code>database</code>)
|-
| '''HTTP Port'''
| 9000 (external via 9443 HTTPS via reverse proxy)
|-
| '''HTTPS Port'''
| 9443 (internal, for direct connections)
|-
| '''LDAP Outpost Port'''
| 389 (via LDAP Outpost container; internal only, reaches stronghold via <code>http://172.17.0.1:9000</code>)
|-
| '''Data volumes'''
| <code>/library/authentik-app/media</code> (custom templates, branding)
|-
| '''Database volume'''
| <code>authentik-app_database</code> (PostgreSQL data)
|-
| '''Networks'''
| <code>authentik-app_default</code> (internal Docker network, MTU 1280)
|-
| '''Image versions'''
| server/worker: <code>ghcr.io/goauthentik/server:2025.10.3</code>
|-
| '''Restart policy'''
| <code>unless-stopped</code>
|}

----

== Initial Setup from Scratch ==

=== Prerequisites ===

* Docker and docker-compose installed
* Network access to PostgreSQL (internal container or external)
* Cloudflare DNS configured for <code>key.wilsoz.com</code>
* VPS reverse proxy (nginx) configured to forward to stronghold:9000

=== 1. Create Directory & Files ===

<syntaxhighlight lang="bash">
mkdir -p /library/authentik-app
cd /library/authentik-app
</syntaxhighlight>

=== 2. Create docker-compose.yml ===

<syntaxhighlight lang="yaml">
version: "3.8"

services:
postgresql:
image: docker.io/library/postgres:16-alpine
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 30s
timeout: 5s
retries: 5
start_period: 20s
volumes:
- database:/var/lib/postgresql/data
env_file:
- .env
environment:
POSTGRES_DB: ${PG_DB:-authentik}
POSTGRES_USER: ${PG_USER:-authentik}
POSTGRES_PASSWORD: ${PG_PASS:?database password required}

server:
image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2025.10.3}
restart: unless-stopped
command: server
ports:
- "${COMPOSE_PORT_HTTP:-9000}:9000"
- "${COMPOSE_PORT_HTTPS:-9443}:9443"
env_file:
- .env
environment:
AUTHENTIK_POSTGRESQL__HOST: postgresql
AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required}
volumes:
- ./media:/media
- ./custom-templates:/templates
depends_on:
postgresql:
condition: service_healthy

worker:
image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2025.10.3}
restart: unless-stopped
command: worker
user: root
env_file:
- .env
environment:
AUTHENTIK_POSTGRESQL__HOST: postgresql
AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:?secret key required}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./media:/media
- ./certs:/certs
- ./custom-templates:/templates
depends_on:
postgresql:
condition: service_healthy

volumes:
database:
</syntaxhighlight>

=== 3. Create .env File ===

<syntaxhighlight lang="bash">
# PostgreSQL
PG_DB=authentik
PG_USER=authentik
PG_PASS=<generate-strong-password-32-chars>

# Authentik
AUTHENTIK_SECRET_KEY=<generate-strong-secret-key-50-chars>
AUTHENTIK_IMAGE=ghcr.io/goauthentik/server
AUTHENTIK_TAG=2025.10.3

# Ports
COMPOSE_PORT_HTTP=9000
COMPOSE_PORT_HTTPS=9443

# Bootstrap (initial admin)
AUTHENTIK_BOOTSTRAP_PASSWORD=<temporary-init-password>
AUTHENTIK_BOOTSTRAP_TOKEN=<temporary-init-token>
</syntaxhighlight>

'''Important:''' Generate secure random values for passwords and secrets:
<syntaxhighlight lang="bash">
openssl rand -base64 32 # For PG_PASS and AUTHENTIK_SECRET_KEY
openssl rand -hex 16 # For AUTHENTIK_BOOTSTRAP_TOKEN
</syntaxhighlight>

=== 4. Start Authentik ===

<syntaxhighlight lang="bash">
cd /library/authentik-app
docker compose up -d
</syntaxhighlight>

Wait for PostgreSQL to be healthy and server to start (~30-60 seconds):

<syntaxhighlight lang="bash">
docker compose logs -f server
# Watch for "Application startup complete"
</syntaxhighlight>

=== 5. Initial Admin User ===

After startup, navigate to <code>https://key.wilsoz.com/if/admin/</code>:
* Username: <code>akadmin</code>
* Password: value of <code>AUTHENTIK_BOOTSTRAP_PASSWORD</code>

'''First steps in Admin UI:'''
# Go to '''Directory → Users → Create'''
** Username: <code>marcus</code>
** Email: <code>marcus@wilsoz.com</code>
** Set password (or let user set on first login)
''' Groups: Add to '''authentik Admins**

# Go to '''Directory → Groups → Create'''
''' '''authentik Admins** (if not already created)
''' '''family-and-friends** (for app access)
''' '''craniumslows-email** (for email provisioning)
''' '''wilsoz-email** (for email provisioning)
''' '''ldap-search** (for LDAP bind accounts)

# Add service users:
''' '''Directory → Users → Create**
** Username: <code>stalwart-svc</code>, Email: <code>stalwart-svc@internal</code>
''' Groups: Add to '''ldap-search**
** Username: <code>nextcloud-svc</code>, Email: <code>nextcloud-svc@internal</code>
''' Groups: Add to '''ldap-search**

=== 6. Set Up LDAP Outpost ===

LDAP is required for Stalwart (mail server), Nextcloud (CalDAV/CardDAV), and Jellyfin (media server) to authenticate.

'''In Authentik Admin UI:'''
# Go to '''Infrastructure → Outposts → Create'''
** Name: <code>authentik-ldap</code>
** Type: LDAP
# In '''Directory → LDAP → Configure LDAP'''
** Base DN: <code>dc=ldap,dc=goauthentik,dc=io</code>
** User base: <code>ou=users,dc=ldap,dc=goauthentik,dc=io</code>
# Assign the LDAP outpost to the LDAP provider

The LDAP server will be reachable at <code>172.17.0.1:389</code> from other containers.

=== 7. Configure OAuth2 Providers ===

For each OAuth application (Immich, Memos, Linkwarden, etc.), create an OAuth2 provider in Authentik:

'''Immich example:'''
# Go to '''Applications → Providers → Create OAuth2/OpenID Provider'''
** Name: <code>immich-oauth</code>
** Authorization flow: <code>default-authentication-flow</code>
** Client type: <code>Confidential</code>
** Redirect URIs:
```
https://photos.wilsoz.com/auth/login/oauth2/callback
https://photos.wilsoz.com/auth/login/oauth2/mobile-redirect
```
# '''Save''' — copy the generated Client ID and Client Secret
# Add property mappings:
** <code>authentik default OAuth Mapping: OpenID 'openid'</code>
** <code>authentik default OAuth Mapping: OpenID 'email'</code>
** <code>authentik default OAuth Mapping: OpenID 'profile'</code>
# Enable: <code>include_claims_in_id_token</code>
# '''Assign a signing key''' (see Critical Gotchas section below)

Repeat for each app.

=== 8. Configure VPS Reverse Proxy ===

On VPS, add nginx server block:

<syntaxhighlight lang="nginx">
server {
listen 443 ssl http2;
server_name key.wilsoz.com;

ssl_certificate /etc/letsencrypt/live/key.wilsoz.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/key.wilsoz.com/privkey.pem;

location / {
proxy_pass http://100.64.207.155:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
</syntaxhighlight>

Reload nginx:
<syntaxhighlight lang="bash">
sudo systemctl reload nginx
</syntaxhighlight>

----

== Disaster Mode A: Configuration Corruption ==

=== Symptoms ===

* Can't log into admin interface
* OAuth providers missing or broken
* LDAP directory returning errors
* Groups or users deleted accidentally
* Provider credentials lost

=== Recovery Steps ===

⚠️ '''This mode resets Authentik to a fresh state. User data will be lost unless you have a PostgreSQL backup (see Disaster Mode B).'''

==== Option 1: Full Reset (if Disaster Mode B not available) ====

<syntaxhighlight lang="bash">
cd /library/authentik-app

# Stop all containers
docker compose down

# Remove the database volume (this deletes all Authentik data)
docker volume rm authentik-app_database

# Start from scratch
docker compose up -d

# Wait for startup
sleep 30
docker logs server | grep "Application startup complete"

# Follow "Initial Setup from Scratch" section above
</syntaxhighlight>

==== Option 2: Selective Reset (if you only need to fix one app) ====

<syntaxhighlight lang="bash">
# Access the Django shell
docker exec authentik-app_server_1 ak shell

# Example: Fix an OAuth provider's signing key
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.crypto.models import CertificateKeyPair

p = OAuth2Provider.objects.get(name='immich-oauth')
jwt_key = CertificateKeyPair.objects.get(name='authentik Internal JWT Certificate')
p.signing_key = jwt_key
p.save()
print(f'Updated {p.name} with JWT signing key')
</syntaxhighlight>

----

== Disaster Mode B: Drive Failure / Full Recovery ==

=== If Backups Accessible ===

Authentik data is backed up in two ways:

# '''PostgreSQL dump''' at Hetzner (in <code>/dumps/authentik-YYYYMMDD.sql.gz</code>)
# '''Media files''' at Hetzner (in <code>/data/authentik/</code>)

==== Recovery procedure: ====

<syntaxhighlight lang="bash">
# 1. Set up fresh Authentik (follow "Initial Setup from Scratch" through step 4)
cd /library/authentik-app
docker compose up -d

# 2. Wait for PostgreSQL to be ready
sleep 30

# 3. Drop the initial empty database
docker exec authentik-app_postgresql_1 psql -U authentik -d postgres -c "DROP DATABASE authentik;"

# 4. Create a fresh empty database
docker exec authentik-app_postgresql_1 psql -U authentik -d postgres -c "CREATE DATABASE authentik;"

# 5. Restore from backup dump (substitute actual date)
# First, extract the backup from Hetzner:
# (You'll need ssh-key access to u540853.your-storagebox.de port 23)
ssh -p 23 u540853@u540853.your-storagebox.de "cat backups/authentik-20260512.sql.gz" | \
gunzip | docker exec -i authentik-app_postgresql_1 psql -U authentik -d authentik

# 6. Restore media files
rsync -av -e "ssh -p 23" u540853@u540853.your-storagebox.de:backups/data/authentik/ /library/authentik-app/media/

# 7. Restart Authentik to pick up restored data
docker compose restart server worker

# 8. Verify
docker logs server | grep "Application startup complete"
curl -s https://key.wilsoz.com/if/admin/ | grep -q "authentik" && echo "Admin UI responding"
</syntaxhighlight>

=== If Backups NOT Accessible ===

Fall back to Mode A (full reset). Note: '''All user data, OAuth provider credentials, and LDAP settings will be lost.'''

After reset, you'll need to:
# Recreate all users and groups
# Recreate all OAuth2 providers and get new Client IDs/Secrets
# Update all dependent services (.env files) with new credentials
# Recreate LDAP configuration and outpost

----

== Data Locations ==

=== Authentik Data on Host ===

<syntaxhighlight lang="text">
/library/authentik-app/
├── docker-compose.yml (service definition)
├── .env (secrets: PG_PASS, AUTHENTIK_SECRET_KEY)
├── media/ (custom templates, branding, uploads)
└── custom-templates/ (HTML templates for login pages, etc.)
</syntaxhighlight>

=== PostgreSQL Database Volume ===

<syntaxhighlight lang="text">
authentik-app_database (Docker volume)
├── postgresql/ (database files)
└── ...
</syntaxhighlight>

Contains:
* All users, groups, group memberships
* All OAuth2 providers, applications, credentials
* All LDAP configuration
* All API tokens
* All authenticator devices (TOTP, WebAuthn)
* All event logs

=== Backup Locations ===

* '''PostgreSQL dump:''' Hetzner <code>/dumps/authentik-YYYYMMDD.sql.gz</code> (daily at 03:00 CST)
* '''Media files:''' Hetzner <code>/data/authentik/</code> (daily Restic snapshot)
* '''Filesystem snapshot:''' Timeshift snapshot on <code>/dev/md1</code> (weekly)

=== What's NOT Backed Up ===

* Container images (re-downloadable from Docker Hub)
* node_modules and build artifacts (not present in Authentik)

----

== Critical Gotchas ==

=== 1. OAuth2 Signing Key Assignment — '''MANDATORY''' ===

⚠️ '''If you don't assign a signing key to an OAuth2 provider, the JWKS endpoint returns an empty object <code>{}</code>, and all clients trying to validate tokens will fail.'''

'''Every OAuth2 provider MUST have:'''
* A signing key assigned: Use '''<code>authentik Internal JWT Certificate</code>''' (HS256 algorithm)
* '''NOT''' <code>authentik Self-signed Certificate</code> (RS256) — some apps like Mealie reject RS256

'''Assignment code:'''
<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
from authentik.crypto.models import CertificateKeyPair

p = OAuth2Provider.objects.get(name='immich-oauth')
jwt_key = CertificateKeyPair.objects.get(name='authentik Internal JWT Certificate')
p.signing_key = jwt_key
p.save()
print(f'Assigned signing key to {p.name}')
"
</syntaxhighlight>

=== 2. Property Mappings for OAuth Scopes ===

OAuth providers need explicit property mappings for scopes like <code>email</code>, <code>profile</code>. Without them:
* OIDC discovery endpoint returns only <code>["openid"]</code>
* Apps can't get user email or name from token

'''Always add these three mappings to every OAuth provider:'''
* <code>authentik default OAuth Mapping: OpenID 'openid'</code>
* <code>authentik default OAuth Mapping: OpenID 'email'</code>
* <code>authentik default OAuth Mapping: OpenID 'profile'</code>

Also enable: <code>include_claims_in_id_token = True</code>

=== 3. LDAP Outpost Internal-Only ===

The LDAP outpost (port 389) is '''internal to Docker networking only'''. External apps can't reach it directly. Instead:
* Internal Docker containers reach it at <code>172.17.0.1:389</code>
* Other hosts on LAN reach it via Stalwart LDAP passthrough (Stalwart exposes LDAP to port 389 externally)

=== 4. Bootstrap Credentials Not Persisted ===

The <code>AUTHENTIK_BOOTSTRAP_PASSWORD</code> and <code>AUTHENTIK_BOOTSTRAP_TOKEN</code> in <code>.env</code> are only used on first startup. After that, they're ignored. If you lose them:
* You still have the <code>akadmin</code> user created from bootstrap
* If you lose the akadmin password, reset via Django shell:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.core.models import User
u = User.objects.get(username='akadmin')
u.set_password('NewPassword123')
u.save()
print(f'Reset password for {u.username}')
"
</syntaxhighlight>

=== 5. Secret Key Rotation (Rare but Important) ===

The <code>AUTHENTIK_SECRET_KEY</code> is used to encrypt stored secrets (OAuth credentials, etc.). '''If you change this .env value, all encrypted secrets become unreadable.''' Only change if:
* You're migrating to a new server
* You suspect key compromise

When rotating:
# Export all OAuth providers' client secrets before changing
# Update AUTHENTIK_SECRET_KEY
# Restart (secrets will need to be re-entered)

=== 6. Group Membership Propagation Delay ===

Changes to group membership (e.g., adding a user to a group) are propagated asynchronously via the worker. If an app isn't seeing the new group:
* Wait 5-10 seconds
* Restart the worker: <code>docker compose restart worker</code>
* Check the worker logs: <code>docker logs worker | tail -20</code>

=== 7. API Token Expiry ===

API tokens can have an expiry. The current claude-api-token (created 2026-02-16) is set to '''never expire'''. Check token settings:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell -c "
from authentik.core.models import Token
t = Token.objects.get(identifier='claude-api-token')
print(f'Token: {t.identifier}')
print(f'Expires: {t.expires}')
"
</syntaxhighlight>

If a token expires, create a new one in the Admin UI (User → Tokens) and update all scripts/services using it.

=== 8. LDAP Bind Accounts Must Be in ldap-search Group ===

Any user/service that needs to query the LDAP directory must be a member of the '''<code>ldap-search</code>''' group. Currently:
* <code>stalwart-svc</code> — for mail server LDAP lookups
* <code>nextcloud-svc</code> — for CalDAV/CardDAV user sync

Without group membership, LDAP bind fails with "Access Denied".

----

== Verification Checklist ==

After recovery, verify:

* [ ] '''Admin UI accessible:''' <code>https://key.wilsoz.com/if/admin/</code> → login as marcus
* [ ] '''OAuth discovery endpoint working:'''
```bash
curl -s https://key.wilsoz.com/application/o/immich/.well-known/openid-configuration | jq '.scopes_supported'
# Should include: ["openid", "email", "profile"]
```
* [ ] '''JWKS endpoint has keys:'''
```bash
curl -s https://key.wilsoz.com/application/o/immich/jwks/ | jq '.keys | length'
# Should return a number > 0
```
* [ ] '''LDAP directory responding:'''
```bash
ldapsearch -H ldap://172.17.0.1:389 -D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
-w <stalwart-svc-password> -b "ou=users,dc=ldap,dc=goauthentik,dc=io" "uid=*" | grep "numResponses"
# Should return matches (or at least show successful bind)
```
* [ ] '''Users present:'''
```bash
docker exec authentik-app_server_1 ak shell -c "from authentik.core.models import User; print('\n'.join([u.username for u in User.objects.all()]))"
# Should show: akadmin, marcus, stalwart-svc, nextcloud-svc
```
* [ ] '''Groups present:'''
```bash
docker exec authentik-app_server_1 ak shell -c "from authentik.core.models import Group; print('\n'.join([g.name for g in Group.objects.all()]))"
# Should show all configured groups
```
* [ ] '''OAuth providers have signing keys:'''
```bash
docker exec authentik-app_server_1 ak shell -c "
from authentik.providers.oauth2.models import OAuth2Provider
for p in OAuth2Provider.objects.all():
has_key = p.signing_key is not None
print(f'{p.name}: signing_key={has_key}')
"
# All should show: signing_key=True
```
* [ ] '''Dependent services can authenticate:'''
** [ ] Immich: Try OAuth login at https://photos.wilsoz.com
** [ ] Linkwarden: Try OAuth login at https://bookmarks.wilsoz.com
** [ ] Memos: Try OAuth login at https://notes.wilsoz.com
** [ ] Stalwart IMAP: Test LDAP lookup for mail routing

----

== Dependencies ==

=== What Authentik Depends On ===

* '''PostgreSQL 16''' — must be healthy before server/worker start
* '''Docker''' — for container runtime
* '''Network access to Cloudflare DNS''' — for DNS resolution of <code>key.wilsoz.com</code>
* '''VPS reverse proxy''' — nginx on VPS must forward <code>key.wilsoz.com:443</code> to <code>stronghold:9000</code>

=== What Depends On Authentik ===

* '''All OAuth2 consumers:''' Immich, Memos, Linkwarden, Forgejo, Vaultwarden, Grafana, n8n, Matrix
* '''All LDAP consumers:''' Nextcloud, Jellyfin, Stalwart (mail server)
* '''Cloudflare Access:''' For gating n8n form URLs
* '''Email provisioning workflow (n8n):''' Watches Authentik groups for email user provisioning
* '''Nginx reverse proxy:''' Serves key.wilsoz.com → Authentik

----

== Operations ==

=== Start/Stop/Restart ===

<syntaxhighlight lang="bash">
cd /library/authentik-app

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart server worker

# Full restart (clears caches)
docker compose down && sleep 2 && docker compose up -d
</syntaxhighlight>

=== View Logs ===

<syntaxhighlight lang="bash">
# Server logs
docker logs -f server

# Worker logs (for background tasks)
docker logs -f worker

# PostgreSQL logs (for DB issues)
docker logs postgresql

# All together
docker compose logs -f
</syntaxhighlight>

=== Access Django Shell (Advanced) ===

For one-off admin tasks:

<syntaxhighlight lang="bash">
docker exec authentik-app_server_1 ak shell

# Example: List all users
from authentik.core.models import User
for u in User.objects.all():
print(f"{u.username}: {u.email}, groups: {list(u.groups.values_list('name', flat=True))}")

# Example: Create API token
from authentik.core.models import Token
t = Token.objects.create(user=User.objects.get(username='marcus'), identifier='my-api-key')
print(f"Token: {t.key}")

# Exit: Ctrl+D
</syntaxhighlight>

=== Backup PostgreSQL Manually ===

<syntaxhighlight lang="bash">
# Full database dump
docker exec authentik-app_postgresql_1 pg_dump -U authentik authentik | \
gzip > /home/mnw/backups/authentik-manual-$(date +%Y%m%d-%H%M%S).sql.gz

# Or via Restic (for Hetzner backup)
cd /library/backup-stack && docker compose exec restic-backup bash -c "
export RESTIC_REPOSITORY=\$HETZNER_REPO
export RESTIC_PASSWORD=\$HETZNER_PASSWORD
# Make dump available in /data
docker exec authentik-app_postgresql_1 pg_dump -U authentik authentik | gzip > /data/dumps/authentik-manual.sql.gz
restic backup /data/dumps/
"
</syntaxhighlight>

----

== Troubleshooting ==

=== Can't log into admin UI ===

'''Symptoms:''' 403 Forbidden, or password rejected

'''Fixes:'''
# Check if server is running: <code>docker ps | grep server</code>
# Reset akadmin password via Django shell (see Operations section)
# Check logs: <code>docker logs server | grep -i "error\|auth"</code>

=== OAuth login fails with "invalid_grant" ===

'''Symptoms:''' Client authentication failed, or code exchange error

'''Causes & fixes:'''
* Client secret mismatch: Verify in Authentik provider matches app's .env
* Provider deleted: Recreate OAuth provider and update app config
* Redirect URI mismatch: Ensure app's callback URL matches provider's redirect URIs exactly

'''Debug:'''
<syntaxhighlight lang="bash">
docker logs server | grep -i "oauth\|invalid_grant"
</syntaxhighlight>

=== LDAP bind fails ===

'''Symptoms:''' Stalwart/Nextcloud can't authenticate users

'''Fixes:'''
# Verify LDAP outpost is running: <code>docker ps | grep outposts</code>
# Check user is in <code>ldap-search</code> group: Admin UI → Directory → Groups → ldap-search
# Test bind from stronghold:
```bash
ldapsearch -H ldap://172.17.0.1:389 -D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
-w <password> -b "ou=users" "uid=marcus"
```

=== Worker not picking up changes ===

'''Symptoms:''' Group membership not reflected, or other background tasks stuck

'''Fix:'''
<syntaxhighlight lang="bash">
docker compose restart worker
docker logs worker --tail=50
</syntaxhighlight>

=== Database corruption ===

'''Symptoms:''' PostgreSQL won't start, or "ERROR: relation does not exist"

'''Fix:'''
* Try recovery from backup (Disaster Mode B)
* If no backup: full reset (Disaster Mode A) — all data lost

----

== Links & References ==

* '''Authentik Docs:''' https://goauthentik.io/docs/
* '''Authentik Admin UI:''' https://key.wilsoz.com/if/admin/
* '''PostgreSQL Docs:''' https://www.postgresql.org/docs/16/
* '''OAuth 2.0 Spec:''' https://datatracker.ietf.org/doc/html/rfc6749
* '''OpenID Connect Spec:''' https://openid.net/connect/

----

== Related Services ==

* '''OpenBao''' (secrets store): <code>/library/openbao-app/</code> — stores API tokens
* '''Stalwart''' (mail server): Uses LDAP for user lookups
* '''Nextcloud''' (file sync): Uses LDAP for CalDAV/CardDAV
* '''n8n''' (automation): Uses OAuth for form access control
* '''PostgreSQL''' (database): Internal to Authentik
* '''Nginx''' (reverse proxy): VPS nginx forwards key.wilsoz.com traffic here