View source for Services/Stalwart

''As of: 2026-07-05 — migrated from services/STALWART.md. Credential values redacted per Rule 0 (see OpenBao secret/stalwart) — the source repo file still has plaintext values and needs a separate scrub + rotation, flagged separately.''

= Stalwart — Mail Server (SMTP/IMAP/LDAP/Sieve/JMAP) =

Complete setup & operations guide for Stalwart mail server with LDAP authentication, relay routing, and getmail integration.

'''Criticality:''' '''HIGH — Mail infrastructure.''' Losing Stalwart means loss of email receive/send capability. Unlike Authentik (SSO), Stalwart failure doesn't break other services, but email is critical infrastructure. Requires careful recovery.

----

== Overview ==

'''Stalwart''' is a modern open-source mail server that provides:
* '''SMTP''' (port 25 inbound, 465/587 outbound) — message transport
* '''IMAP''' (port 143/993) — message retrieval with TLS
* '''POP3''' (port 110/995) — legacy message retrieval
* '''Sieve''' (port 4190/ManageSieve) — mail filtering rules
* '''JMAP''' (port 443) — modern mail protocol (experimental)
* '''LDAP''' (LDAP passthrough to Authentik at port 389) — user authentication
* '''Web Admin''' (port 8181 internal HTTP API) — configuration and management

'''Current setup:'''
* Primary domain: <code>wilsoz.com</code> — marcus@wilsoz.com is main account
* Secondary domain: <code>craniumslows.com</code> — accounts federated to wilsoz via getmail
* Inbound mail sources: Fastmail, MXRoute, Gmail, iCloud, SDF (fetched via getmail)
* Outbound relay routes: Fastmail (wilsoz.com), MXRoute (craniumslows.com), Gmail, SDF (via proxy)
* Mail clients: eM Client (IMAP/SMTP), SnappyMail (web), Thunderbird (config compatible)

'''Current Version:''' Stalwart latest (updates automatically via Watchtower)

'''Access:''' 
* Internal admin API: <code>http://localhost:8181/api</code>
* ManageSieve: <code>localhost:4190</code> (STARTTLS required)
* IMAP: <code>localhost:993</code> (TLS) or <code>mymail.wilsoz.com:993</code> (external)
* SMTP: <code>localhost:465</code> or <code>mymail.wilsoz.com:465</code> (external)

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''Compose file'''
| <code>/library/stalwart-app/docker-compose.yml</code>
|-
| '''Container name'''
| <code>stalwart-mail</code>
|-
| '''Image'''
| <code>stalwartlabs/stalwart:latest</code>
|-
| '''Network mode'''
| <code>host</code> (direct network access, not bridged)
|-
| '''Restart policy'''
| <code>unless-stopped</code>
|-
| '''Data volume'''
| <code>/library/stalwart-app/data/</code> (contains <code>/etc/</code>, <code>/queue/</code>, <code>/store/</code>)
|-
| '''TLS certificates'''
| <code>/etc/letsencrypt/live/mymail.wilsoz.com/</code> (mounted readonly)
|-
| '''Admin credentials'''
| <code>admin:<REDACTED - see OpenBao secret/stalwart></code> (plaintext password, hashed in config.toml)
|-
| '''Admin API port'''
| 8181 (internal only)
|-
| '''SMTP port'''
| 25 (inbound), 465 (outbound implicit TLS), 587 (submission STARTTLS)
|-
| '''IMAP ports'''
| 143 (STARTTLS), 993 (implicit TLS)
|-
| '''POP3 ports'''
| 110 (STARTTLS), 995 (implicit TLS)
|-
| '''Sieve port'''
| 4190 (ManageSieve, STARTTLS then PLAIN auth)
|-
| '''JMAP port'''
| 443 (via reverse proxy, not directly exposed)
|-
| '''LDAP passthrough'''
| <code>localhost:389</code> (to Authentik LDAP outpost)
|}

----

== Ports Summary ==

{| class="wikitable"
|-
! Port
! Protocol
! Direction
! Purpose
! TLS
! Notes
|-
| 25
| SMTP
| Inbound
| Receive mail from internet
| None
| MX record points here
|-
| 465
| SMTPS
| Outbound
| Send via Fastmail/MXRoute
| Implicit
| ISP allows only 465, blocks 25/587
|-
| 587
| SMTP
| Outbound
| Send via some relays
| STARTTLS
| ISP blocks this, use 465 instead
|-
| 143
| IMAP
| Internal
| Retrieve mail (legacy)
| STARTTLS
| Plain AUTH disabled, must TLS first
|-
| 993
| IMAPS
| Internal
| Retrieve mail (standard)
| Implicit
| Recommended, supports admin impersonation
|-
| 110/995
| POP3/S
| Internal
| Legacy retrieval
| —
| Available but unused
|-
| 4190
| ManageSieve
| Internal
| Upload sieve rules
| STARTTLS
| Auth only advertised post-TLS
|-
| 8181
| HTTP API
| Internal
| Admin config/settings
| None
| Internal only, no TLS
|}

----

== Initial Setup from Scratch ==

=== Prerequisites ===

* Docker and docker-compose installed
* LDAP access to Authentik LDAP outpost (must be running first)
* TLS certificate for <code>mymail.wilsoz.com</code> at <code>/etc/letsencrypt/live/mymail.wilsoz.com/</code>
* DNS MX records configured to point to Stronghold
* Outbound SMTP to external relays configured (Fastmail, MXRoute, etc.)

=== 1. Create Directory & Files ===

<syntaxhighlight lang="bash">
mkdir -p /library/stalwart-app/data
cd /library/stalwart-app
</syntaxhighlight>

=== 2. Create docker-compose.yml ===

<syntaxhighlight lang="yaml">
version: "3.8"

services:
  stalwart:
    image: stalwartlabs/stalwart:latest
    container_name: stalwart-mail
    hostname: stronghold
    network_mode: host
    restart: unless-stopped
    volumes:
      - ./data:/opt/stalwart
      - /etc/letsencrypt:/opt/letsencrypt:ro
    env_file:
      - .env
</syntaxhighlight>

=== 3. Create .env File ===

<syntaxhighlight lang="bash">
# Admin credentials (plaintext; password gets hashed in config.toml)
STALWART_ADMIN_USER=admin
STALWART_ADMIN_PASSWORD=<generate-strong-password-20-chars>

# LDAP (Authentik LDAP outpost)
LDAP_HOST=localhost
LDAP_PORT=389
LDAP_BASE_DN="dc=ldap,dc=goauthentik,dc=io"
LDAP_USER_BASE="ou=users,dc=ldap,dc=goauthentik,dc=io"
LDAP_SERVICE_ACCOUNT="cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io"
LDAP_SERVICE_PASSWORD=<from-Authentik-stalwart-svc-user>

# Mail domains
PRIMARY_DOMAIN=wilsoz.com
SECONDARY_DOMAIN=craniumslows.com

# TLS certificate (mounted from host)
TLS_CERT_PATH=/opt/letsencrypt/live/mymail.wilsoz.com/fullchain.pem
TLS_KEY_PATH=/opt/letsencrypt/live/mymail.wilsoz.com/privkey.pem
</syntaxhighlight>

=== 4. Create config.toml (Bootstrap Configuration) ===

This is a complex TOML file. Start with a minimal bootstrap config containing only:
* Listener (SMTP, IMAP, POP3, Sieve ports)
* Storage (database and message store)
* TLS certificate paths
* Authentication (LDAP + fallback admin)
* Admin API

'''Key principle:''' All persistent settings (relay routes, routing strategy, rate limits) must be configured via the Admin HTTP API at port 8181, NOT in config.toml. The reason: <code>GET /api/reload</code> rewrites config.toml in flat key=value format, stripping all TOML sections.

<syntaxhighlight lang="toml">
# Minimal bootstrap config.toml
[server]
hostname = "stronghold"

# Listeners
[[listeners.smtp]]
bind = "0.0.0.0:25"
protocol = "smtp"

[[listeners.smtp]]
bind = "0.0.0.0:465"
protocol = "smtp"
tls.implicit = true

[[listeners.smtp]]
bind = "0.0.0.0:587"
protocol = "smtp"
tls.starttls = true

[[listeners.imap]]
bind = "0.0.0.0:143"
protocol = "imap"
tls.starttls = true

[[listeners.imap]]
bind = "0.0.0.0:993"
protocol = "imap"
tls.implicit = true

[[listeners.pop3]]
bind = "0.0.0.0:110"
protocol = "pop3"
tls.starttls = true

[[listeners.pop3]]
bind = "0.0.0.0:995"
protocol = "pop3"
tls.implicit = true

[[listeners.managesieve]]
bind = "0.0.0.0:4190"
protocol = "sieve"
tls.starttls = true

[[listeners.http]]
bind = "127.0.0.1:8181"
protocol = "http"

# TLS
[certificate.default]
cert = "/opt/letsencrypt/live/mymail.wilsoz.com/fullchain.pem"
private-key = "/opt/letsencrypt/live/mymail.wilsoz.com/privkey.pem"

# Storage (SQLite database + file-based message store)
[storage.data]
type = "sqlite"
path = "/opt/stalwart/store/data.sqlite"

[storage.fts]
type = "sqlite"
path = "/opt/stalwart/store/fts.sqlite"

[storage.directory]
type = "sqlite"
path = "/opt/stalwart/store/directory.sqlite"

[storage.lookup]
type = "sqlite"
path = "/opt/stalwart/store/lookup.sqlite"

[storage.blob]
type = "fs"
path = "/opt/stalwart/store/blobs"

[storage.queue]
type = "fs"
path = "/opt/stalwart/queue"

# Authentication
[auth]
directory = "ldap"

[[directory.ldap]]
name = "ldap"
address = "localhost"
port = 389
base-dn = "dc=ldap,dc=goauthentik,dc=io"
user-base-dn = "ou=users,dc=ldap,dc=goauthentik,dc=io"
bind-dn = "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io"
bind-password = "<from OpenBao: secret/stalwart/ldap-bind-password>"  # rotated 2026-05-18
filter = "(&(objectClass=posixAccount)(cn=?))"
mail-attribute = "mail"

# Fallback admin (in case LDAP fails)
[authentication.fallback-admin]
user = "admin"
password = "$6$HFsr55uFqNc31Mw8$rLv/wtAzFCr0yNfY.../...hashed-password"  # Will be generated on first startup

# Domains (minimal)
[[domains]]
name = "wilsoz.com"

[[domains]]
name = "craniumslows.com"

# Admin API
[server.admin]
bind = "127.0.0.1"
port = 8181

# Logging
[tracer.log]
level = "info"
</syntaxhighlight>

=== 5. Start Stalwart ===

<syntaxhighlight lang="bash">
cd /library/stalwart-app
docker compose up -d

# Wait for startup (~10 seconds)
sleep 10

# Check logs
docker logs stalwart-mail | tail -20

# Should see: "Stalwart Mail Server started" and port listeners binding
</syntaxhighlight>

=== 6. Configure Via Admin API (port 8181) ===

After startup, all configuration happens via the HTTP API. See "Admin API" section below for how to:
* Create mail accounts (principals)
* Configure relay routes (outbound SMTP)
* Configure routing strategy (which route to use for each sender)
* Upload sieve rules
* Set spam filter thresholds

'''Example: Create a mail account'''

<syntaxhighlight lang="bash">
curl -u admin:<REDACTED - see OpenBao secret/stalwart> -X POST \
  http://localhost:8181/api/principal \
  -H "Content-Type: application/json" \
  -d '{
    "type": "individual",
    "name": "marcus@wilsoz.com",
    "emails": ["marcus@wilsoz.com"],
    "secrets": ["marcus-password"]
  }'
</syntaxhighlight>

=== 7. Configure LDAP Sync ===

Stalwart can authenticate against Authentik LDAP, but users must exist. The simplest approach:
* Create users in Authentik → Directory → Users
* Add them to <code>wilsoz-email</code> or <code>craniumslows-email</code> groups
* Stalwart reads LDAP attributes: <code>mail</code>, <code>mailAddresses</code>, <code>memberOf</code>

Test LDAP:

<syntaxhighlight lang="bash">
ldapsearch -x -H ldap://localhost:389 \
  -D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
  -w "<rotated 2026-05-18; see OpenBao secret/stalwart/ldap-bind-password>" \
  -b "ou=users,dc=ldap,dc=goauthentik,dc=io" \
  "cn=marcus" mail mailAddresses
</syntaxhighlight>

=== 8. Set Up TLS Certificate Renewal ===

Stalwart uses certificates at <code>/opt/letsencrypt/live/mymail.wilsoz.com/</code>. When certificates renew (certbot, daily check):

# '''Create renewal hook''' to reload Stalwart:

<syntaxhighlight lang="bash">
sudo nano /etc/letsencrypt/renewal-hooks/deploy/stalwart-reload.sh
</syntaxhighlight>

<syntaxhighlight lang="bash">
#!/bin/bash
# Reload Stalwart config after certificate renewal
docker exec stalwart-mail curl -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/reload
echo "Stalwart reloaded on $(date)" >> /var/log/stalwart-reload.log
</syntaxhighlight>

<syntaxhighlight lang="bash">
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/stalwart-reload.sh
</syntaxhighlight>

# '''Test renewal:'''

<syntaxhighlight lang="bash">
sudo certbot renew --dry-run
</syntaxhighlight>

=== 9. Configure VPS Reverse Proxy (if external access needed) ===

On VPS, forward <code>mymail.wilsoz.com</code> to Stronghold. Example nginx config:

<syntaxhighlight lang="nginx">
# IMAP/SMTP passthrough (not HTTP)
# This requires HAProxy or similar (not standard nginx)

# Alternatively, use Stronghold's direct port exposure + firewall rules
</syntaxhighlight>

Actually, for mail protocols (SMTP/IMAP), most setups expose ports directly. Configure firewall rules on stronghold to allow:
* Port 25 (inbound SMTP) — public
* Port 465/587 (outbound) — already open by ISP
* Port 993 (IMAP, TLS) — expose to VPS or public

----

== Disaster Mode A: Configuration Corruption ==

=== Symptoms ===

* Can't send/receive mail
* LDAP authentication failing
* Sieve rules not executing
* Relay routes returning errors
* Database corrupted (SQLite integrity errors)
* Admin credentials lost or forgotten

=== Recovery Steps ===

==== Option 1: Reset Configuration (Keep Mail Data) ====

If only config is corrupted but mail data is intact:

<syntaxhighlight lang="bash">
cd /library/stalwart-app

# Stop Stalwart
docker compose down

# Backup old config
cp data/etc/config.toml data/etc/config.toml.corrupted-$(date +%Y%m%d)

# Copy clean bootstrap config (from Initial Setup step 4)
cp config.toml.bootstrap data/etc/config.toml

# Start fresh
docker compose up -d

# Reconfigure via API (routes, strategy, sieve, etc.)
# See Admin API section below
</syntaxhighlight>

==== Option 2: Reset Admin Password (Via Direct API) ====

If you can still reach the API but lost the admin password:

<syntaxhighlight lang="bash">
# Edit config.toml directly to set new password
# Hash password using Stalwart's built-in hash
docker exec stalwart-mail stalwartd hash-password "NewPassword123"

# Copy the hash output, edit config.toml:
[authentication.fallback-admin]
user = "admin"
password = "<paste-hash-here>"

# Restart
docker compose restart stalwart-mail
</syntaxhighlight>

==== Option 3: Full Reset (Lose All Configuration + Users) ====

⚠️ This deletes everything except mail messages themselves.

<syntaxhighlight lang="bash">
cd /library/stalwart-app
docker compose down

# Delete all config and database
rm -rf data/etc/*
rm -rf data/store/*.sqlite
mkdir -p data/etc data/store/blobs data/queue

# Copy bootstrap config
cp config.toml.bootstrap data/etc/config.toml

# Delete Docker volume (optional, if vol became corrupted)
# docker volume rm stalwart-mail_data

# Start from scratch
docker compose up -d

# Reconfigure everything:
# 1. Create relay routes
# 2. Create routing strategy
# 3. Create mail accounts
# 4. Upload sieve rules
# 5. Configure spam filters
</syntaxhighlight>

----

== Disaster Mode B: Drive Failure / Full Recovery ==

=== If Backups Accessible ===

Stalwart data is backed up:
# '''PostgreSQL dumps''' (if using PG backend) — at Hetzner <code>/dumps/stalwart-*.sql.gz</code>
# '''Restic snapshots''' — at Hetzner <code>/data/stalwart/</code>
# '''Timeshift snapshots''' — on <code>/raid1/timeshift/</code> (if /var/lib/docker/volumes issue fixed)

==== Recovery Steps (Using Restic): ====

<syntaxhighlight lang="bash">
# 1. Deploy fresh Stalwart container (follow Initial Setup)
cd /library/stalwart-app
docker compose up -d

# 2. Stop Stalwart
docker compose down

# 3. Restore mail data from Restic
cd /library/backup-stack
docker compose up -d

docker exec restic-backup restic restore latest \
  --target /tmp/restore \
  --repo sftp:u540853@u540853.your-storagebox.de:./backups \
  --include "/data/stalwart"

# 4. Copy restored data to /library/stalwart-app
sudo cp -r /tmp/restore/data/stalwart/* /library/stalwart-app/data/
sudo chown -R 0:0 /library/stalwart-app/data/  # Stalwart runs as root in container

# 5. Start Stalwart
cd /library/stalwart-app
docker compose up -d

# 6. Verify
docker logs stalwart-mail | grep -i "listener\|directory\|ldap"
curl -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/principal | jq '.principals | length'
</syntaxhighlight>

=== If Backups NOT Accessible ===

⚠️ '''Mail data will be permanently lost.''' Recovery options:

# '''Restore from getmail sources:'''
** Getmail configs at <code>/library/email/getmail/*.rc</code> fetch from external sources
** If Hetzner or tape backup includes getmail state files, can re-fetch mail
** Mail will only go back to getmail fetch dates (~7 days, depending on retention)

# '''Request users to re-send important emails'''

# '''Restore from email client backups:'''
** If eM Client/Thunderbird cached mail, can export and reimport

# '''Full rebuild (minimal configuration, zero historical mail):'''
** Create new empty Stalwart instance
** Reconfigure routes, domains, users
** All historical mail is lost; only forward-going mail is received

----

== Data Locations ==

=== Stalwart Data on Host ===

<syntaxhighlight lang="text">
/library/stalwart-app/
├── docker-compose.yml       (service definition)
├── .env                     (LDAP passwords, admin user)
├── config.toml.bootstrap    (clean bootstrap config)
└── data/
    ├── etc/config.toml      (runtime config, auto-rewritten by /api/reload)
    ├── store/
    │   ├── data.sqlite      (principal directory, accounts, groups)
    │   ├── fts.sqlite       (full-text search index)
    │   ├── directory.sqlite (LDAP cache)
    │   ├── lookup.sqlite    (MX/DNS lookups)
    │   └── blobs/           (message attachments, large objects)
    └── queue/               (outbound mail queue)
</syntaxhighlight>

=== Content Storage ===

'''Messages themselves''' are stored differently based on storage backend:
* '''FileSystem (current):''' <code>/library/stalwart-app/data/store/blobs/</code> — one file per message/attachment
* '''Database:''' Could be SQLite or PostgreSQL

=== Backup Locations ===

* '''Hetzner Restic:''' <code>/data/stalwart/</code> (daily snapshots)
'' '''Local Timeshift:''' <code>/raid1/timeshift/''/library/stalwart-app/data/</code> (if /var/lib/docker/volumes included)
* '''Home directory:''' Config snippets, scripts at <code>/home/mnw/Developer/SystemResiliency/</code>

=== What's Backed Up ===

✅ All mail messages (blobs/)
✅ Account directory and settings (data.sqlite)
✅ LDAP cache (directory.sqlite)
✅ Outbound queue (queue/)
✅ Config.toml (auto-rewritten but restored)

❌ '''NOT backed up by Timeshift:''' <code>/var/lib/docker/volumes</code> (includes SQLite WAL files — '''known issue''')
✅ Backed up by Restic: All /library/stalwart-app content

----

== Critical Gotchas ==

=== 1. config.toml Rewriting Gotcha (CRITICAL) ===

'''Problem:''' When you call <code>GET /api/reload</code>, Stalwart rewrites config.toml in flat <code>key=value</code> format, '''stripping all TOML sections.'''

<syntaxhighlight lang="toml">
# Original config.toml
[server.admin]
bind = "127.0.0.1"
port = 8181

# After /api/reload
server.admin.bind = 127.0.0.1
server.admin.port = 8181
</syntaxhighlight>

'''Consequence:''' All relay routes, routing strategy, rate limits, spam settings created via API are LOST when rewriting happens.

'''Solution:''' 
* '''Only store bootstrap/listener/storage/auth settings in config.toml'''
* '''All persistent settings (routes, strategy, spam) MUST live in the database via <code>/api/settings</code> API'''
* Never call <code>/api/reload</code> unless you're OK with losing API-configured settings
* If you accidentally called it, restore from backup immediately

=== 2. Routes Live Under <code>queue.route.*</code> Namespace (CRITICAL) ===

'''Wrong:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=route.
# Returns 0 results — namespace is wrong
</syntaxhighlight>

'''Right:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=queue.route.
# Returns all routes
</syntaxhighlight>

'''Lesson:''' Always use full namespace path <code>queue.route.NAME</code> not <code>route.NAME</code>.

=== 3. Settings API Format — Stalwart 0.15.4 Specific ===

'''assert_empty field is REQUIRED (and must be snake_case, NOT camelCase):'''

<syntaxhighlight lang="bash">
# WRONG — camelCase
-d '[{"type":"insert","prefix":"queue.route.fastmail","assertEmpty":false,...}]'
# Returns: 400 JSON deserialization failed

# RIGHT — snake_case
-d '[{"type":"insert","prefix":"queue.route.fastmail","assert_empty":false,...}]'
</syntaxhighlight>

'''Quirk:''' The enum <code>type</code> values are camelCase (<code>insert</code>, <code>delete</code>, <code>clear</code>), but the struct fields use snake_case due to Serde serialization rules. This is '''version-specific''' to 0.15.4. Post-0.16.0 may change.

=== 4. ISP Blocks Port 25 AND 587 Outbound — Port 465 Only ===

'''Fact:''' Your ISP (likely Comcast or similar) blocks:
* ✅ Inbound port 25 (you can receive)
* ❌ Outbound port 25 (can't relay)
* ❌ Outbound port 587 (can't relay, STARTTLS submission)
* ✅ Outbound port 465 (implicit TLS works)

'''Configuration:''' All outbound relay routes must use:
* <code>port = 465</code>
* <code>tls.implicit = true</code>
* <code>protocol = "smtp"</code>

'''Example (Fastmail relay):'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
  -d '[{"type":"insert","prefix":"queue.route.fastmail-relay","assert_empty":false,"values":[
    ["type","relay"],
    ["address","smtp.fastmail.com"],
    ["port","465"],
    ["tls.implicit","true"],
    ["auth.username","marcus@wilsoz.com"],
    ["auth.secret","<fastmail-app-password>"]
  ]}]'
</syntaxhighlight>

=== 5. SDF Relay Requires VPS Python Proxy (SDF CRAM-MD5 auth) ===

'''Problem:''' SDF mail server only supports CRAM-MD5 authentication, which Stalwart doesn't implement.

'''Solution:''' Python proxy on VPS (<code>/opt/sdf-smtp-proxy.py</code>, systemd: <code>sdf-smtp-proxy.service</code>)
* Listens on Tailscale IP <code>100.67.169.50:2587</code> (no auth required)
* Relays to <code>mx.sdf.org:587</code> with CRAM-MD5 auth
* Credentials stored in VPS <code>/opt/secrets.env</code>

'''Stalwart config for SDF:'''
<syntaxhighlight lang="bash">
# Set /etc/hosts mapping on stronghold
100.67.169.50 sdf-relay.internal

# Then add route
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
  -d '[{"type":"insert","prefix":"queue.route.sdf-relay","assert_empty":false,"values":[
    ["type","relay"],
    ["address","sdf-relay.internal"],
    ["port","2587"],
    ["tls.implicit","false"],
    ["tls.starttls","true"]
  ]}]'
</syntaxhighlight>

Also, '''config.toml must set <code>resolver.type = "system"</code>''' (not "default") so it reads <code>/etc/hosts</code>:

<syntaxhighlight lang="toml">
[resolver]
type = "system"  # NOT "default"
</syntaxhighlight>

=== 6. Routing Strategy: Local Recipients FIRST (Rule 0000) ===

'''Problem:''' If rule 0000 doesn't check for local recipients, mail to <code>marcus@wilsoz.com</code> gets sent through an external relay and fails.

'''Correct strategy (rules 0000-0002):'''

<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
  -d '[{"type":"insert","prefix":"queue.strategy.route","assert_empty":false,"values":[
    ["0000.if","is_local_domain('\'\'\'', rcpt_domain)"],
    ["0000.then","'\''local'\''"],
    ["0001.if","sender_domain == '\''wilsoz.com'\''"],
    ["0001.then","'\''wilsoz-fastmail'\''"],
    ["0002.else","'\''mx'\''"]
  ]}]'
</syntaxhighlight>

'''Lesson:''' ALWAYS put local-domain check as rule 0000 (executed first).

=== 7. LDAP Authentication Requires STARTTLS on Port 143 ===

'''Wrong:''' Try to use <code>LOGIN</code> auth on plain port 143
<syntaxhighlight lang="text">
imap.login('marcus', 'password')  # Port 143
# Returns: AUTHENTICATIONFAILED
</syntaxhighlight>

'''Right:''' Use STARTTLS on port 143, or implicit TLS on port 993
<syntaxhighlight lang="text">
imap.starttls()  # Upgrade to TLS
imap.login('marcus', 'password')  # Now LOGIN is available
</syntaxhighlight>

This is a '''Stalwart security feature''' — LOGIN auth is only advertised after TLS negotiation (STARTTLS or implicit).

=== 8. ManageSieve (Sieve Rules) Authentication Quirk ===

'''Problem:''' Python <code>managesieve</code> library calls <code>login()</code> before <code>starttls()</code>. After <code>starttls()</code>, capabilities change to include <code>PLAIN</code>. Library doesn't re-read capabilities, so login fails.

'''Solution:''' Use raw sockets (see "ManageSieve" section below in Operations).

=== 9. Admin Impersonation via IMAP (May Not Work) ===

'''Syntax:''' <code>admin*username@domain</code> with admin password
<syntaxhighlight lang="text">
imap.login('admin*marcus@wilsoz.com', '<REDACTED - see OpenBao secret/stalwart>')
</syntaxhighlight>

'''Status:''' As of 2026-02-22, this returned AUTHENTICATIONFAILED. May need:
* <code>[imap] allow-plain-auth</code> config setting enabled
* Or different impersonation mechanism

'''Workaround:''' Use LDAP directly or set user passwords and log in normally.

=== 10. TLS Certificate Path Must Exist at Startup ===

'''Problem:''' If you mount <code>/etc/letsencrypt:/opt/letsencrypt:ro</code> but cert files don't exist, Stalwart refuses to start.

'''Solution:''' Before starting, verify:
<syntaxhighlight lang="bash">
ls -l /etc/letsencrypt/live/mymail.wilsoz.com/
# fullchain.pem and privkey.pem must exist

# If they don't, run certbot
sudo certbot certonly -d mymail.wilsoz.com --dns-cloudflare
</syntaxhighlight>

=== 11. Outbound Rate Limiter Example (MXRoute 300/hr) ===

MXRoute has a rate limit of ~300 emails/hour. Configure in Stalwart:

<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings -H "Content-Type: application/json" \
  -d '[{"type":"insert","prefix":"queue.limiter.outbound.mxroute","assert_empty":false,"values":[
    ["enable","true"],
    ["match","mx == '\''blizzard.mxrouting.net'\''"],
    ["key.0000","mx"],
    ["rate","300/1h"]
  ]}]'
</syntaxhighlight>

=== 12. LDAP Password Storage (Plaintext in .env) ===

The <code>LDAP_SERVICE_PASSWORD</code> in <code>.env</code> is stored plaintext. It's the password for <code>cn=stalwart-svc</code> (service account), which:
* Has limited permissions (ldap-search group only)
* Doesn't have login access
* Is used only for directory queries

Store securely; if compromised, change in Authentik and update .env.

----

== Verification Checklist ==

After recovery, verify:

* [ ] '''Container running:''' <code>docker ps | grep stalwart-mail</code>
* [ ] '''Listeners bound:'''
  ```bash
  docker logs stalwart-mail | grep -i "listening\|bound"
  # Should show SMTP:25, SMTP:465, IMAP:143, IMAP:993, etc.
  ```
* [ ] '''Admin API responding:'''
  ```bash
  curl -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/principal | jq '.principals | length'
  # Should return number of principals
  ```
* [ ] '''LDAP connected:'''
  ```bash
  docker logs stalwart-mail | grep -i "ldap\|directory"
  # Should show successful connections, not errors
  ```
* [ ] '''TLS certificate loaded:'''
  ```bash
  docker logs stalwart-mail | grep -i "certificate\|tls"
  # Should show cert paths loaded
  ```
* [ ] '''IMAP login works:'''
  ```bash
  python3 -c "
  import imaplib, ssl
  ctx = ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
  m = imaplib.IMAP4_SSL('localhost', 993)
  status, data = m.login('marcus', '<password>')
  print('IMAP login:', 'OK' if status == 'OK' else f'FAILED: {status}')
  m.close()
  "
  ```
* [ ] '''Relay routes configured:'''
  ```bash
  curl -s -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/settings/list?prefix=queue.route | jq '.settings | length'
  # Should return > 0 (number of configured routes)
  ```
* [ ] '''Routing strategy configured:'''
  ```bash
  curl -s -u admin:<REDACTED - see OpenBao secret/stalwart> http://localhost:8181/api/settings/list?prefix=queue.strategy | jq '.settings | length'
  # Should return > 0 (routing rules)
  ```
* [ ] '''Send test email:'''
  ```bash
  # Use eM Client or <code>swaks</code> command-line tool
  swaks --to marcus@wilsoz.com --from test@wilsoz.com --server localhost:465 --tls
  # Should deliver without errors
  ```
* [ ] '''Receive test email:'''
** Send email from external account to marcus@wilsoz.com
** Check it appears in IMAP inbox
* [ ] '''Sieve rules executing:'''
  ```bash
  # If you have sieve rules, send test email and verify filtering works
  docker logs stalwart-mail | grep -i "sieve\|filter"
  ```

----

== Dependencies ==

=== What Stalwart Depends On ===

* '''Authentik LDAP outpost''' — for user authentication (localhost:389)
* '''TLS certificate''' — from Let's Encrypt (/etc/letsencrypt)
* '''Network access to external relays:'''
** Fastmail: smtp.fastmail.com:465
** MXRoute: blizzard.mxrouting.net:465
** Gmail: smtp.gmail.com:465
** iCloud: smtp.mail.me.com:465
** SDF: sdf-relay.internal:2587 (via proxy)
* '''DNS resolution''' — for MX lookups and relay host resolution
* '''getmail''' — for inbound mail federation (craniumslows.com forwarding)

=== What Depends On Stalwart ===

* '''SnappyMail''' — web mail interface (frontend to Stalwart IMAP)
* '''eM Client''' — desktop mail client (uses IMAP/SMTP)
* '''getmail''' — mail federation from external accounts
* '''n8n workflows''' — email provisioning/deprovisioning
* '''Mail clients''' — Thunderbird, Outlook, etc.

----

== Operations ==

=== Start/Stop/Restart ===

<syntaxhighlight lang="bash">
cd /library/stalwart-app

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart stalwart-mail

# Full restart (clears cache)
docker compose down && sleep 2 && docker compose up -d
</syntaxhighlight>

=== View Logs ===

<syntaxhighlight lang="bash">
# Real-time logs
docker logs -f stalwart-mail

# Last 50 lines
docker logs stalwart-mail --tail=50

# Filter for errors
docker logs stalwart-mail 2>&1 | grep -i "error\|failed\|fatal"

# Filter for specific component
docker logs stalwart-mail | grep -i "ldap\|imap\|smtp\|queue"
</syntaxhighlight>

=== Admin API Queries ===

<syntaxhighlight lang="bash">
ADMIN="admin:<REDACTED - see OpenBao secret/stalwart>"
BASE="http://localhost:8181/api"

# List all principals (accounts, groups)
curl -s -u "$ADMIN" "$BASE/principal"

# Get specific principal
curl -s -u "$ADMIN" "$BASE/principal/marcus@wilsoz.com"

# Reload config from disk (WARNING: see gotcha #1)
curl -s -u "$ADMIN" "$BASE/reload"

# List routes
curl -s -u "$ADMIN" "$BASE/settings/list?prefix=queue.route"

# List strategy rules
curl -s -u "$ADMIN" "$BASE/settings/list?prefix=queue.strategy"

# List spam settings
curl -s -u "$ADMIN" "$BASE/settings/list?prefix=spam-filter"

# Count messages in queue
curl -s -u "$ADMIN" "$BASE/queue" | jq '.messages | length'

# View a specific message in queue
curl -s -u "$ADMIN" "$BASE/queue/<message-id>"
</syntaxhighlight>

=== Manage Sieve Rules (ManageSieve port 4190) ===

<syntaxhighlight lang="bash">
# Upload sieve script (using helper script)
cd /home/mnw/Developer/SystemResiliency
python3 sieve/upload-sieve.py

# Or use raw socket method (see Disaster Mode B section)
</syntaxhighlight>

----

== Troubleshooting ==

=== Can't Log In via IMAP ===

'''Check LDAP:'''
<syntaxhighlight lang="bash">
ldapsearch -x -H ldap://localhost:389 \
  -D "cn=stalwart-svc,ou=users,dc=ldap,dc=goauthentik,dc=io" \
  -w "<rotated 2026-05-18; see OpenBao secret/stalwart/ldap-bind-password>" \
  -b "ou=users,dc=ldap,dc=goauthentik,dc=io" \
  "cn=marcus" mail
</syntaxhighlight>

'''If LDAP works but IMAP login fails:'''
* Verify using port 993 (TLS) not 143 (plain)
* Or use STARTTLS on port 143
* Check if user is in <code>wilsoz-email</code> group (not required for LDAP, but may be required by Stalwart)

'''Admin impersonation not working:'''
* Try normal login instead: <code>imap.login('marcus', '<password>')</code>
* Impersonation may need config flag changes

=== Relay Routes Not Working ===

'''Debug:'''
<syntaxhighlight lang="bash">
# Check routes exist
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=queue.route

# Check strategy rules
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=queue.strategy

# Check queue for stuck messages
curl -s -u admin:<REDACTED> http://localhost:8181/api/queue | jq '.messages[]'

# Check logs for relay errors
docker logs stalwart-mail | grep -i "relay\|route\|fastmail\|mxroute"
</syntaxhighlight>

'''Common causes:'''
* Route not configured (blank response from API)
* Strategy rule missing or malformed (0000 not checking local domain)
* TLS cert not valid (relay host rejects connection)
* Rate limit exceeded (queue backs up, check MXRoute limit)
* Port 465 blocked (ISP still blocking? test external relay directly)

=== Database Corruption ===

'''Symptoms:''' "SQLite error" in logs, or can't connect to accounts

'''Fix:'''
<syntaxhighlight lang="bash">
cd /library/stalwart-app
docker compose down

# Backup corrupted database
cp data/store/data.sqlite data/store/data.sqlite.corrupted-$(date +%Y%m%d)

# Delete corrupted DBs (queue/messages preserved)
rm data/store/*.sqlite

# Start fresh (will recreate empty DBs)
docker compose up -d

# Reconfigure via API
</syntaxhighlight>

=== Outbound Queue Stuck ===

'''Check queue:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> http://localhost:8181/api/queue | jq '.messages | length'

# If > 100 messages stuck, investigate:
docker logs stalwart-mail | tail -100 | grep -i "reject\|error\|fail"

# Possible causes:
# - Rate limit hit (check MXRoute limit)
# - TLS cert issue with relay host
# - Firewall blocking outbound to relay
# - LDAP down (affects routing decision)
</syntaxhighlight>

=== Spam Filter Too Aggressive ===

'''Increase spam threshold:'''
<syntaxhighlight lang="bash">
curl -s -u admin:<REDACTED> -X POST http://localhost:8181/api/settings \
  -H "Content-Type: application/json" \
  -d '[{"type":"insert","prefix":"spam-filter.score","assert_empty":false,"values":[["spam","6.0"]]}]'

# Verify
curl -s -u admin:<REDACTED> http://localhost:8181/api/settings/list?prefix=spam-filter
</syntaxhighlight>

----

== Links & References ==

* '''Stalwart Docs:''' https://stalwartlabs.com/docs/
* '''Stalwart Admin API:''' http://localhost:8181/admin (Web UI)
* '''Stalwart Email Blog:''' https://stalwartlabs.com/blog/
* '''LDAP Documentation:''' https://ldapwiki.com/
* '''Sieve Mail Filters:''' https://tools.ietf.org/html/rfc5228
* '''Getmail Integration:''' <code>/home/mnw/Developer/SystemResiliency/services/STALWART.md</code> (reference docs)

----

== Related Services ==

* '''Authentik''' (auth): <code>/library/authentik-app/</code> — LDAP provider, user directory
* '''SnappyMail''' (webmail): <code>/library/snappymail-app/</code> — web mail frontend
* '''getmail''' (mail federation): <code>/library/email/getmail/</code> — inbound mail syncing
* '''Let's Encrypt''' (TLS): <code>/etc/letsencrypt/</code> — certificates for TLS
* '''VPS nginx''' (reverse proxy): Exposes mymail.wilsoz.com:993 externally
* '''Tailscale''' (VPN): Required for SDF relay via proxy