View source for Email/Architecture

''As of: 2026-07-05 — migrated from `TODO.md` (Email section) and cross-referenced with [[Services/Stalwart]]''

See also: '''[[Services/Stalwart]]''' for the full mail server operational reference (admin API, ports, sieve rules, gotchas). This page covers the broader mail architecture goals and phase history.

== Email (Stalwart Migration) ==

=== Architecture goal: Multi-user family & friends mail server ===

<syntaxhighlight lang="text">
Inbound:  Internet → MXRoute (per-user spool) → getmail → Stalwart (per-user inbox)
Outbound: User authenticates to Stalwart SMTP with own Authentik creds
          → Stalwart relays via user's own MXRoute credentials (per-user relay routes)
          → 400 msg/hr global rate limit to MXRoute enforced by Stalwart queue throttle
Domains:  craniumslows.com now. wilsoz.com → MXRoute once craniumslows validated.
Users:    Authentik LDAP → Stalwart. Custom attributes drive address provisioning.
Spam:     Server-wide Stalwart sieve → spam into each user's Junk
Sieve:    Per-user rules via ManageSieve (no rule affects another user)
Webmail:  SnappyMail at mymail.wilsoz.com
External: Gmail/iCloud/SDF as SnappyMail additional accounts (direct IMAP, not getmail)
</syntaxhighlight>

=== User provisioning model (Authentik custom LDAP attributes) ===

Each Authentik user gets three custom attributes:
* <code>mail-name</code> — primary local part (e.g. <code>joe</code>)
* <code>mail-alias</code> — secondary local part, optional (e.g. <code>joe.smith</code>)
* <code>default-domain</code> — which domain is the primary SnappyMail identity

Groups control which domains are active:
* <code>craniumslows-email</code> → @craniumslows.com addresses
* <code>wilsoz-email</code> → @wilsoz.com addresses

Address matrix = <code>{mail-name, mail-alias} × {domains from group memberships}</code>
All addresses on a user deliver to one Stalwart inbox. Each address gets its own
MXRoute account + getmail .rc + per-user Stalwart relay route.

Address changes: old address becomes a forwarder to the new one. Inbox never moves.

=== Where the real mail data lives ===
{| class="wikitable"
|-
! Location
! What
! Size
! Action
|-
| <code>/library/email/selfhosted/mail/wilsoz.com/marcus/Maildir/</code>
| Primary mailbox — Mailcow/Dovecot, ~100 nested folders
| ~8.9GB
| '''Import to Stalwart — in progress'''
|-
| <code>/library/email/selfhosted/mail/wilsoz.com/marcus/Orig-Maildir/</code>
| Pre-Mailcow import (iRedMail/Gmail era)
| (subset)
| Verify not unique before discarding
|-
| <code>/library/email/selfhosted/mail/craniumslows.com/marcus/Maildir/</code>
| craniumslows.com mailbox
| 152K
| Import to Stalwart
|-
| <code>/var/mail/mnw</code>
| Old iRedMail mbox (pre-Mailcow era)
| 37MB
| Import after primary migration
|-
| <code>/opt/mailcow-dockerized/</code>
| Mailcow app stack only
| 49MB
| Remove after migration verified
|}

=== Phase 3 (Mail Import) — COMPLETE ✓ ===
* [x] '''Import marcus@wilsoz.com''' — 137,335 messages from Mailcow Maildir, all folders preserved
* [x] '''Import marcus@craniumslows.com''' — done (was tiny, only pre-Stalwart messages)
* [x] '''Verify in SnappyMail''' — folder hierarchy confirmed, messages accessible
* [ ] '''Import old iRedMail mbox''' — <code>/var/mail/mnw</code> (37MB) not yet done (low priority, very old mail)

=== Phase 4 (Getmail) — COMPLETE ✓ (one item remaining) ===
* [x] '''Update delivery targets''' in .rc files → Stalwart LMTP <code>localhost:24</code>
* [x] '''Create craniumslows-marcus.rc''' — fetches marcus@craniumslows.com from MXRoute, 439/439 delivered
* [x] '''Keep <code>delete = false</code>''' — correctly set; flip to true only after validating in daily use
* [x] '''Create systemd timer''' — getmail runs every 10 minutes via systemd
* [x] '''Fastmail import COMPLETE''' — 12,226/12,226 delivered (2026-02-22 ~15:52)
* [ ] ~~Configure SnappyMail additional accounts~~ — '''ARCHITECTURE CHANGED''' (see below): external accounts now use getmail, not SnappyMail connected accounts

=== Architecture decision (2026-02-22) ===
Marcus's external accounts (Gmail x2, iCloud, SDF) use '''getmail''' → Stalwart, NOT SnappyMail connected accounts. This gives a true unified inbox with sieve filtering. Phone/clients connect to one IMAP server.
* [x] '''Add getmail config: thewilson@gmail.com''' — <code>sieve/gmail-thewilson.rc</code> done. Import in progress: ~7,560/58,520 as of 2026-02-23 07:00 (~50 hrs remaining at current rate)
* [x] '''Add getmail config: craniumslows@gmail.com''' — <code>sieve/gmail-craniumslows.rc</code> done. Will start after thewilson finishes.
* [x] '''Add getmail config: marcusthewilson@me.com''' — <code>sieve/icloud-marcusthewilson.rc</code> done.
* [x] '''Add getmail config: mnw@sdf.org''' — <code>sieve/sdf-mnw.rc</code> done. <code>delete=false</code>.
* [x] '''Update systemd getmail.service''' — all 6 rc files active, daemon reloaded 2026-02-23.

=== Action needed — Phase 5 (Spam + Sieve) ===
* [x] '''Fastmail import COMPLETE''' — 12,226/12,226 (done 2026-02-22)
* [x] '''Sieve rules written''' — <code>sieve/marcus.sieve</code> ready. Decade-based numbering, all rules merged. See <code>sieve/README.md</code>.
* [x] '''Mail classification script written''' — <code>sieve/classify-inbox.py</code> done.
'' [x] '''Audit actual Stalwart folder names''' — confirmed INBOX/XX-'' naming in place.
* [x] '''Upload sieve/marcus.sieve''' — uploaded and ACTIVE via ManageSieve (confirmed 2026-02-23). Script: <code>sieve/upload-sieve.py</code>.
* [x] '''Dedup script written''' — <code>sieve/dedup.py</code>. Dry-run confirmed 6,622 dupes. Run <code>--execute</code> to purge.
* [ ] '''Run dedup + retroactive sort + empty .Junk''' — → <code>plans/sieve-dedup.md</code> (covers all three)
* [ ] '''Enable server-wide spam filter''' — after sieve rules confirmed working in daily use

=== Action needed — Phase 6 (n8n provisioner + per-user relay) ===
* [x] '''Create <code>craniumslows-email</code> and <code>wilsoz-email</code> groups''' in Authentik — done 2026-02-22; marcus in both
* [x] '''Marcus's <code>mailAddresses</code> attribute set''' — <code>{"mailAddresses": ["marcus@craniumslows.com"]}</code> — enables dual-domain delivery in Stalwart
* [x] '''Build n8n provisioner workflow''' — Matrix bot IYw0DmCuWEsiV2k9 (deployed 2026-03-01). <code>!provision</code> in #notifications room, walks through username/name/password/domain conversation. Adaptive polling: 15s during conversation, 5-min throttle when idle.
* [x] '''Add custom LDAP attributes''' to Authentik users: <code>mail-name</code>, <code>mail-alias</code>, <code>default-domain</code> (future, when adding more users) — '''done 2026-05-10''': attributes backfilled on marcus and now written by the provisioner for new users; LDAP property mappings + downstream consumers still deferred
* [x] '''Configure global MXRoute rate limit''' — Stalwart queue throttle 400 msg/hr to blizzard.mxrouting.net. → <code>plans/stalwart-helpers-fix.md</code> (Phase 5) — '''done 2026-05-10''': 400/1h limiter set 2026-05-10 via legacy prefix/values API
* [x] '''Migrate marcus to per-user relay routes''' — replace shared craniumslows-mxroute with marcus-specific route — '''done 2026-05-10''': marcus-craniumslows relay + per-sender rule added; shared craniumslows-mxroute kept as fallback

=== Action needed — jkatz@craniumslows.com cleanup ===
* [x] '''jkatz Authentik user''' — Confirmed deleted (pk:18 not found 2026-05-09). MXRoute account was never created. Consider done.

=== Action needed — Matrix deprovisioner (next session) ===
* [x] '''Build Matrix deprovisioner''' — → <code>plans/email-deprovisioner.md</code> (depends on <code>plans/stalwart-helpers-fix.md</code> + <code>plans/email-provisioner.md</code>) — '''done 2026-05-10''': deployed as jbGL9N1vXT8mB92V, end-to-end tested 2026-05-10

=== Action needed — Provisioner cleanup strategy ===
* [x] '''Define orphan .rc file retention policy''' — '''done 2026-05-10''': policy = move .rc into disabled/ with <code>.deprov-YYYY-MM-DD</code> suffix on deprovision; weekly cron (<code>/etc/cron.d/getmail-disabled-purge</code>) purges entries >30 days old. Legacy duplicates + test residue swept.

=== Action needed — Phase 7 (wilsoz.com → MXRoute) ===
* [ ] Validate craniumslows.com in daily use (give it weeks)
* [ ] Move wilsoz.com MX DNS → MXRoute
* [ ] Replace fastmail.rc with wilsoz-marcus.rc (MXRoute fetch)
* [ ] Update Stalwart wilsoz.com relay → MXRoute smarthost
* [ ] '''Flip <code>delete = true</code>''' on all getmail configs once cutover verified

=== Minor ===
* [ ] Investigate unexpected folders in SnappyMail MoveTo dropdown

=== Status (updated 2026-02-22) ===
'''Phases 1–4 complete.''' Stalwart + SnappyMail deployed, LDAP auth working with dual-domain delivery. Maildir import done (137k+ messages). Getmail running: craniumslows 439/439, Fastmail ~5.5k/12.2k in progress. <code>craniumslows-email</code>/<code>wilsoz-email</code> groups created in Authentik. '''Phase 5 (spam/sieve) and Phase 6 (n8n provisioner) are next.''' One manual step remaining: add 4 external accounts to SnappyMail UI.

----