Wikibot: Migrated from services/JELLYFIN.md

2026-07-05T18:16:19Z

Migrated from services/JELLYFIN.md

New page

''As of: 2026-07-05 — migrated from `services/JELLYFIN.md`''

= Jellyfin — Media Server =

Operations, backup, and recovery guide for stronghold's Jellyfin install.

'''Criticality:''' '''MEDIUM — Multi-user service.''' Media files themselves are large and not in scope for cloud backup (rerippable from physical media as a last resort). Server state (DB, metadata, plugins, config, subtitles) IS in scope — losing it means rebuilding the library from scratch, redoing all manual metadata fixes, and losing watch history. This is the only stronghold service used by multiple people.

----

== Overview ==

'''Jellyfin''' is the Free Software media server. Stronghold runs it as a native systemd service (not Docker) because of the package's tight integration with the host (ffmpeg, GPU, file permissions).

'''What's special about this install:'''
* Native systemd, not containerised. Updates via <code>apt</code>, not Watchtower.
* Metadata/cache/transcode paths have been '''relocated''' away from <code>/var/lib/jellyfin</code> to <code>/library/jellyfin-server-app/</code>. The old paths still exist on disk but are stale (untouched since Oct 2025).
* ~74G of <code>/var/lib/jellyfin/data</code> is regenerable cache (trickplay, subtitle extraction, internal backups). Only ~16G is genuinely irreplaceable.
* Uses SQLite — Jellyfin does not support PostgreSQL/MySQL in any non-experimental form as of 10.11.

'''Current Version:''' <code>10.11.8+ubu2404</code> (Debian package). Latest is <code>10.11.9</code> (released 2026-05-21) — pending upgrade; see [Maintenance](#maintenance).

'''Users:''' Multi-user — Marcus + family/friends. Auth is local Jellyfin accounts (not Authentik-federated).

----

== Service Information ==

{| class="wikitable"
|-
! Property
! Value
|-
| '''Package'''
| <code>jellyfin</code> (Debian, Ubuntu 24.04 repo)
|-
| '''Systemd unit'''
| <code>/usr/lib/systemd/system/jellyfin.service</code>
|-
| '''Process user'''
| <code>jellyfin:jellyfin</code>
|-
| '''Working directory'''
| <code>/var/lib/jellyfin</code>
|-
| '''Environment file'''
| <code>/etc/default/jellyfin</code>
|-
| '''Binary'''
| <code>/usr/bin/jellyfin</code>
|-
| '''HTTP port (internal)'''
| 8096
|-
| '''HTTPS port (internal)'''
| 8920 (unused — TLS terminated upstream)
|-
| '''Public URL'''
| (whatever your reverse proxy/Tailscale exposes — internal-only by default)
|}

=== Companion services ===

{| class="wikitable"
|-
! Service
! Purpose
! Where
|-
| <code>logo-cache</code> (Docker, <code>nginx:alpine</code>)
| '''Jellyfin / IPTV support tool.''' Static HTTP server for the IPTV feed used by Jellyfin Live TV and the Apple TV IPTV client app. Serves: channel logos (30-day immutable cache) + <code>filtered.m3u</code> playlist + <code>filtered.xml</code> EPG (no-cache, must-revalidate — updater rewrites every ~3h via atomic rename). Stood up 2026-05-24.
| Compose: <code>/library/logo-cache-app/</code> · Listens: <code>0.0.0.0:8079</code> → container <code>:80</code> · Mounts: <code>/raid1/Consolidated/IPTV/logos → /usr/share/nginx/html:ro</code> (logo files) and <code>/raid1/Consolidated/IPTV → /srv/iptv:ro</code> (parent dir, exposed via nginx <code>alias</code> for <code>/filtered.m3u</code> + <code>/filtered.xml</code> only — directory bind, not single-file, so atomic-rename updates are picked up live) · Watchtower: enabled
|}

=== Data layout (the important table) ===

{| class="wikitable"
|-
! Path
! Purpose
! Size
! Backup?
|-
| <code>/etc/jellyfin/</code>
| System config (<code>system.xml</code>, <code>network.xml</code>, <code>encoding.xml</code>, users)
| <1M
| '''YES''' — mounted as <code>/data/jellyfin-etc</code>
|-
| <code>/var/lib/jellyfin/data/jellyfin.db</code>
| '''Core library DB''' (items, users, playstate)
| 1.3G
| '''YES via SQLite <code>.backup</code> to <code>/dumps/</code>''' — live file excluded from Restic
|-
| <code>/var/lib/jellyfin/data/playback_reporting.db</code>
| Playback history (from plugin)
| small
| '''YES via SQLite <code>.backup</code> to <code>/dumps/</code>'''
|-
| <code>/var/lib/jellyfin/data/subtitles/</code>
| Extracted/cached subs (<code>.srt</code>, <code>.ass</code>, <code>.sup</code>)
| ~40G
| YES — re-extraction is slow + lossy
|-
| <code>/var/lib/jellyfin/data/collections/</code>, <code>playlists/</code>, <code>ScheduledTasks/</code>, <code>livetv/</code>, <code>attachments/</code>
| User-curated state
| <100M
| YES
|-
| <code>/var/lib/jellyfin/plugins/</code>
| Installed plugins + their configs
| 14M
| YES
|-
| <code>/var/lib/jellyfin/root/</code>
| Library definitions
| 7M
| YES
|-
| <code>/var/lib/jellyfin/metadata/</code>
| '''STALE''' (legacy, pre-relocation, last write Oct 2025)
| 14G
| '''NO — excluded'''
|-
| <code>/var/lib/jellyfin/data/trickplay/</code>
| Generated thumbnails for scrub previews
| 16G
| NO — scheduled task regenerates
|-
| <code>/var/lib/jellyfin/data/backups/</code>
| Jellyfin's own internal backup plugin output
| 15G
| NO — we have our own
|-
| <code>/var/lib/jellyfin/data/just-in-case/</code>
| Leftover from some operation
| 2G
| NO — investigate + delete
|-
| <code>/var/lib/jellyfin/data/library.db.old''</code>, <code>library.db.bak''</code>, <code>library.db-wal</code>
| Pre-10.9 schema consolidation leftovers
| ~700M
| NO — safe to delete
|-
| <code>/var/lib/jellyfin/data/SQLiteBackups/</code>
| Jellyfin's auto-DB backups (separate from <code>/data/backups</code>)
| varies
| NO — we have our own
|-
| '''<code>/library/jellyfin-server-app/metadata/</code>'''
| '''ACTIVE metadata''' (images, NFOs, TMDB cache)
| 28G
| '''YES — mounted as <code>/data/jellyfin-server</code>'''
|-
| <code>/library/jellyfin-server-app/cache/</code>
| Image transcodes, web cache
| 739M
| NO — excluded
|-
| <code>/library/jellyfin-server-app/transcodes/</code>
| On-the-fly transcode working dir
| 588K
| NO — excluded
|}

Confirm relocation in <code>/etc/jellyfin/system.xml</code>:
<syntaxhighlight lang="xml">
<CachePath>/library/jellyfin-server-app/cache</CachePath>
<MetadataPath>/library/jellyfin-server-app/metadata</MetadataPath>
</syntaxhighlight>

=== Media files (NOT backed up) ===
* <code>/raid1/Consolidated/Video/Movies/</code>, <code>/raid1/Consolidated/Video/TV/</code>, etc. — many TB
* '''Strategy:''' Not backed up to cloud. Reproducible from physical media (rerip) as last resort. External RAID is the eventual local-only backup plan. Treat the library as effort-recoverable, not byte-recoverable.

----

== Backup ==

=== The cardinal rule ===

'''Never raw-copy a live SQLite WAL DB.''' A <code>cp jellyfin.db</code> (or unmodified file backup) while the server is running can yield an inconsistent file that fails to open, or worse, looks fine but is silently corrupt. SQLite's <code>.backup</code> command is the only safe online snapshot mechanism.

=== How it works on stronghold ===

Two things conspire to make Jellyfin backup safe:

# '''<code>host-dump-databases.sh</code>''' runs as <code>ExecStartPre</code> for <code>backup.service</code> (03:00 CST daily). For each SQLite DB it runs:
```sh
sqlite3 /var/lib/jellyfin/data/jellyfin.db ".backup '$DUMP_DIR/jellyfin-$DATE.db'"
gzip -f "$DUMP_DIR/jellyfin-$DATE.db"
```
The <code>.backup</code> command takes a short shared lock, copies pages atomically, and produces a file that's identical to a <code>VACUUM INTO</code>-style snapshot. Jellyfin keeps running throughout.

# '''<code>backup-to-hetzner.sh</code>''' then runs Restic. The live <code>jellyfin.db</code>, <code>playback_reporting.db</code>, and all their <code>-wal</code>/<code>-shm</code> sidecars are '''excluded''' so the only DB copy in the snapshot is the consistent one from <code>/dumps/</code>.

Restic also excludes the regenerable dirs (trickplay, internal backups, just-in-case, stale metadata path, cache, transcodes).

=== What's actually in a snapshot ===

Per nightly Restic run for Jellyfin:

{| class="wikitable"
|-
! Source mount
! Inside snapshot
! Approx after exclusions
|-
| <code>/etc/jellyfin</code> → <code>/data/jellyfin-etc</code>
| All config XML
| <1M
|-
| <code>/var/lib/jellyfin</code> → <code>/data/jellyfin</code>
| <code>plugins/</code>, <code>root/</code>, <code>data/{subtitles,collections,playlists,ScheduledTasks,livetv,attachments}</code>
| ~40G (subtitles dominate)
|-
| <code>/library/jellyfin-server-app</code> → <code>/data/jellyfin-server</code>
| <code>metadata/</code>
| 28G
|-
| <code>/library/backup-stack/dumps/</code> → <code>/dumps</code>
| <code>jellyfin-YYYYMMDD.db.gz</code>, <code>jellyfin-playback-YYYYMMDD.db.gz</code>
| ~300M compressed
|}

First snapshot ≈ 70G stored. Daily incrementals tiny thanks to Restic dedup/compression.

=== Verifying the backup ===

<syntaxhighlight lang="bash">
# Confirm today's SQLite dump was created
ls -lh /library/backup-stack/dumps/jellyfin-*.db.gz

# Spot-check dump integrity (extract + open)
gunzip -k /library/backup-stack/dumps/jellyfin-$(date +%Y%m%d).db.gz
sqlite3 /tmp/jellyfin-test.db "PRAGMA integrity_check;" # expect "ok"

# Confirm Restic snapshot contains the right paths
docker exec restic-backup restic snapshots --latest 1
docker exec restic-backup restic ls latest | grep -E "^/data/jellyfin-(etc|server)?" | head -20
</syntaxhighlight>

----

== Recovery ==

=== Scenario A: corrupt or lost <code>jellyfin.db</code> ===

The DB is the most fragile and most critical piece. Recovery in order of cost:

# '''Use the live <code>data/SQLiteBackups/</code>''' if it's recent and intact — Jellyfin keeps its own rolling SQLite copies here.
# '''Restore from <code>/library/backup-stack/dumps/jellyfin-YYYYMMDD.db.gz</code>''' (today's dump, lives locally on stronghold).
# '''Restore from Restic''' (Hetzner) if local dumps are gone too.

Procedure (option 2 or 3):
<syntaxhighlight lang="bash">
sudo systemctl stop jellyfin

# Move aside the broken DB
sudo mv /var/lib/jellyfin/data/jellyfin.db{,.broken-$(date +%s)}
sudo rm -f /var/lib/jellyfin/data/jellyfin.db-{wal,shm}

# From local dump:
sudo gunzip -c /library/backup-stack/dumps/jellyfin-YYYYMMDD.db.gz \
| sudo tee /var/lib/jellyfin/data/jellyfin.db >/dev/null
sudo chown jellyfin:jellyfin /var/lib/jellyfin/data/jellyfin.db

# OR from Restic:
docker exec restic-backup restic restore latest \
--target /tmp/jellyfin-restore \
--include /dumps/jellyfin-YYYYMMDD.db.gz
sudo gunzip -c /tmp/jellyfin-restore/dumps/jellyfin-*.db.gz \
| sudo tee /var/lib/jellyfin/data/jellyfin.db >/dev/null
sudo chown jellyfin:jellyfin /var/lib/jellyfin/data/jellyfin.db

sudo systemctl start jellyfin
</syntaxhighlight>

=== Scenario B: full server state loss ===

Rebuild order:
# Install <code>jellyfin</code> package, '''do not start''' the service yet.
# Restic restore <code>/data/jellyfin-etc/</code> → <code>/etc/jellyfin/</code>.
# Restic restore <code>/data/jellyfin/</code> → <code>/var/lib/jellyfin/</code> (minus the excluded paths — they'll regenerate).
# Restic restore <code>/data/jellyfin-server/</code> → <code>/library/jellyfin-server-app/</code>.
# Restore <code>jellyfin.db</code> and <code>playback_reporting.db</code> from <code>/dumps/</code> per Scenario A.
# <code>chown -R jellyfin:jellyfin /var/lib/jellyfin /library/jellyfin-server-app</code>.
# <code>systemctl start jellyfin</code>. First start will regenerate trickplay, transcode dirs, etc.
# Re-scan libraries from web UI (idempotent — won't re-import existing items).

=== Scenario C: media library on /raid1 is gone ===

Server state survives independently. Restore server state per Scenario B; then:
* Media has to come back from physical media (rerip) or external RAID if one exists.
* The library DB still references the original paths, so once the files reappear at the same paths, items light back up. No re-import needed.

----

== Health checks ==

Add as a weekly cron / systemd timer (see [TODO](#known-issues--tracked-work)):

<syntaxhighlight lang="bash">
sqlite3 /var/lib/jellyfin/data/jellyfin.db "PRAGMA integrity_check;"
sqlite3 /var/lib/jellyfin/data/playback_reporting.db "PRAGMA integrity_check;"
</syntaxhighlight>

Any output other than <code>ok</code> → page Matrix.

Also useful, opportunistically:
<syntaxhighlight lang="bash">
# What's holding the DB right now (should be ONLY the jellyfin process)
sudo lsof /var/lib/jellyfin/data/jellyfin.db

# Confirm WAL is on (it should always say "wal")
sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db "PRAGMA journal_mode;"

# Lock-error counts in the last day
sudo grep -c "database table is locked" /var/log/jellyfin/log_$(date +%Y%m%d).log
</syntaxhighlight>

----

== Maintenance ==

=== Apt update path ===
Jellyfin updates via the Ubuntu apt repo, '''not''' Watchtower (it's not a container).

<syntaxhighlight lang="bash">
# Check current vs latest
apt list --installed 2>/dev/null | grep jellyfin
# Latest tag: https://api.github.com/repos/jellyfin/jellyfin/releases/latest

# Upgrade pattern (always snapshot DB first):
sudo systemctl stop jellyfin
sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db ".backup '/library/backup-stack/dumps/jellyfin-pre-upgrade-$(date +%Y%m%d).db'"
sudo apt update && sudo apt install --only-upgrade jellyfin jellyfin-server jellyfin-web
sudo systemctl start jellyfin
</syntaxhighlight>

=== Scheduled tasks worth knowing about ===
Visible in the web UI at <code>Dashboard → Scheduled Tasks</code>. Two that matter:
* '''Extract Subtitles''' — runs daily, can cause <code>UserData</code> lock contention if it overlaps active playback (see [Gotchas](#gotchas)). Schedule defensively (e.g. 04:30, not 01:00).
* '''Refresh Library''' — keeps <code>jellyfin.db</code> warm. Don't run multiple library refreshes in parallel.

----

== Gotchas ==

=== SQLite "database table is locked" errors are internal, not external ===

If you see <code>SQLite Error 6: 'database table is locked: UserData'</code> in <code>/var/log/jellyfin/log_*.log</code>:
* Error code '''6 is <code>SQLITE_LOCKED</code>''' — same-process shared-cache contention. NOT another OS process holding the file. So it is '''not''' a backup/cp problem.
* Code 5 (<code>SQLITE_BUSY</code>) would indicate external process contention. We don't see that.
* Typical cause: '''PlaybackReporting plugin''' is in a tight write loop on <code>UserData</code> for an active session, and '''Extract Subtitles''' scheduled task fires at the same time and tries to write the same table.
* Mitigations:
## Move <code>Extract Subtitles</code> schedule off active-viewing hours.
## Test disabling PlaybackReporting plugin (you lose history but isolate the variable).
## Upgrade — 10.11.9 (PR #15368 + #16845) explicitly touches UserManager EF Core + user session concurrency logging.

These errors abort the failing transaction cleanly. They do '''not''' corrupt the DB.

=== Metadata path relocation ===

<code>system.xml</code> points <code>MetadataPath</code> and <code>CachePath</code> to <code>/library/jellyfin-server-app/</code> — not the package default of <code>/var/lib/jellyfin/</code>. The default-location dirs still exist with stale content from before relocation. '''Don't be fooled''' into thinking <code>/var/lib/jellyfin/metadata</code> is the live one; check <code>system.xml</code> to confirm. Both dirs are listed in the data-layout table above with their actual roles.

=== The <code>.sup</code> subtitle files ===

PGS (bitmap) subs from Blu-rays. Most are 5–30MB, but some legitimately exceed 100MB or 1GB — e.g. '''animated sing-along subtitles''' (Wicked's <code>.sup</code> is 1.1GB and intentional). '''Don't auto-delete by size.''' Look up the title via <code>BaseItems.Id</code> matching the dir hash (dashed-uppercase GUID) before deleting any oversized <code>.sup</code>. Genuine broken extractions can go; Jellyfin re-extracts on demand.

=== DB lookups ===
The on-disk subtitle dir name like <code>da/dab167ae-31c1-aa0a-7b64-dd821c44b879/4.sup</code> maps to a <code>BaseItems.Id</code>. The DB stores it dashed and uppercase:
<syntaxhighlight lang="bash">
sudo sqlite3 /var/lib/jellyfin/data/jellyfin.db \
"SELECT Name, Type, Path FROM BaseItems WHERE Id='DAB167AE-31C1-AA0A-7B64-DD821C44B879';"
</syntaxhighlight>

=== Jellyfin's own backup plugin ===

There's a built-in Backup plugin that writes to <code>data/backups/</code>. We '''don't''' rely on it — our own <code>.backup</code> dumps are simpler to reason about. The <code>data/backups/</code> dir is excluded from Restic for that reason. If you ever want to use the built-in plugin instead, flip the exclusion and update this doc.

=== Stale <code>library.db.*</code> files ===

Jellyfin 10.9 consolidated multiple SQLite DBs into a single <code>jellyfin.db</code>. The leftover <code>library.db.old</code>, <code>library.db.bak1</code>, <code>library.db-wal</code> (~700M total) under <code>/var/lib/jellyfin/data/</code> are safe to delete after confirming Jellyfin starts cleanly without them — Jellyfin will not reach for them again.

----

== Known issues / tracked work ==

See <code>TODO.md</code> § Jellyfin for the live list. Current outstanding items:
* [ ] Investigate + delete oversized <code>.sup</code> files that are genuinely broken (size alone isn't a signal; manual eyeball required).
* [ ] Delete pre-10.9 schema cruft (<code>library.db.old</code>, <code>library.db.bak1</code>, etc.).
* [ ] Weekly <code>PRAGMA integrity_check</code> → Matrix alert on anything other than <code>ok</code>.
* [ ] Upgrade to 10.11.9 (UserManager/EF Core fixes; minor, do alongside next routine maintenance window).
* [ ] Haruhi S01E01 has a nightly subtitle-extraction failure (<code>Unable to get streams for File:...</code>) — investigate file integrity.

----

== References ==

* Backup stack overview: <code>services/BACKUP_SYSTEM.md</code>
* Restic include/exclude patterns: <code>/library/backup-stack/scripts/backup-to-hetzner.sh</code>
* SQLite dump invocations: <code>/library/backup-stack/scripts/host-dump-databases.sh</code>
* Disaster recovery procedure: <code>DISASTER_RECOVERY.md</code> (this service's recovery section above is the authoritative copy)
* Upstream changelog: https://github.com/jellyfin/jellyfin/releases

Services/Jellyfin - Revision history

Wikibot: Migrated from services/JELLYFIN.md