46. The database on an emptyDir, replicated with Litestream#

Date: 2026-11-05

Status#

Accepted. Opt-in (litestream.enabled, off by default); the DLS installer turns it on. With it off the chart renders exactly as before.

Context#

scrap keeps SQLite in WAL mode. On DLS’s argus cluster the only dynamic storage is netapp, which is NFS, and the storage notes say never to put the database on NFS: WAL needs shared memory and locking a network filesystem does not provide reliably. The owner decided that the database moves off NFS rather than testing whether it copes there. This ADR does not claim to explain any past hang: none was traced to NFS.

Block storage, which the storage notes recommend, is not an option here: the argus admins removed it a while back for security reasons, and netapp is the only dynamic class, so it was not pursued.

Decision#

The shape. The database is at /db/scrap.db (SCRAP_DB_PATH) on an emptyDir on the node’s disk (not medium: Memory; argus’s is a local xfs, seen on the cluster). A Litestream sidecar replicates it to /data/litestream on the data volume. files/ and the preview cache stay on the data volume. With SCRAP_DB_PATH unset nothing changes: Store builds dir / "scrap.db", as before. The lock, WAL and scrap restore’s scrap.db.restore follow the database, so its rename never crosses filesystems. gc.lock stays in the data folder, on NFS, as it does today.

The version: Litestream 0.5.18, by tag and digest. The 0.5 line is the maintained one (the docs call 0.3.14 “previous”); its restore checks LTX files and fails on a damaged one, seen below, and it has what is used here: shutdown-sync-timeout, snapshot, and a file replica whose renames are fsynced (0.5.16). 0.5.18 rather than the two-month-old 0.5.17, despite being two days old when chosen, for its fixes on this path: “read the whole WAL when the last-page check fails”, “keep read lock alive after caller context is cancelled” and “cache LTX pages for checkpoint verification”; and it is the version the trial below ran. Moving it is a decision: read the release notes, and repeat the trial’s restore steps.

Two images. The sidecar is Litestream’s own image, not a binary added to scrap’s, so nobody with litestream.enabled off carries it. An init container copies its static binary into a small emptyDir, so the Python below can call it. The Debian image (0.5.18, not -scratch) has the cp and sh this needs. The chart still makes no ConfigMap: the sidecar writes its config at start.

Starting up: scrap db-init, in an init container, so the logic can be unit tested. The first that applies decides:

  1. A database on the emptyDir is kept (the pod restarted on its node), never overwritten.

  2. A staged restore is taken up (below), and the replica renamed aside.

  3. A replica is restored. No -if-replica-exists: if the restore or the integrity and schema check fails, the init container fails and scrap does not start. A retry can work.

  4. No replica but a /data/scrap.db with pages (an install from before this; an empty one counts as absent): brought across once with SQLite’s backup API (it may have a WAL), checked, left where it is.

  5. Neither: a new database, unless something says there should be one.

Telling “really new” from “looks empty because something is wrong”. A new volume with no history looks exactly like a replaced or mistakenly empty one from inside the volume, so the evidence has to include something that outlives it. db-init fails closed on any of:

  • .scrap-db-state.json on the data volume, written the first time a database is put in place (how and at; a later restore of the replica leaves those and sets replica_restored_at), with no replica beside it. So the replica cannot vanish unnoticed, and a stale /data/scrap.db cannot come back by rule 4. This also fires if the pod dies between the first placement and Litestream’s first write, about a second; it costs a person a look, not data.

  • Files in files/.

  • Backup runs (daily or monthly) in the backup folder, which is on other storage (at DLS, /dls/science): the one signal that notices a new, empty data volume.

And any error looking is a failure, not “absent”: os.stat, not Path.exists(), which from Python 3.12 answers False on a stale NFS handle. It cannot catch everything: a data volume and backup folder both new and empty really is a new install, and with backups off only the first two apply. litestream.init.override gets past these refusals, for after a person has looked. Each use is recorded (override_at), and gc only counts until a person clears it (scrap db-hold --clear), so leaving it on cannot make a later loss of the replica silent. A source over half of litestream.sizeLimit is refused regardless, as the WAL and Litestream’s own LTX share that emptyDir.

Off Litestream. With the setting off, or a helm rollback to a chart that has it off, /data/scrap.db is the copy from before, and gc would delete files only the newer database uses. So while the state file exists and SCRAP_DB_PATH is unset, the app and scrap gc refuse. A plain scrap restore removes the state file: that is the way off. Installs that never used Litestream have no state file and are not affected. A rollback to a chart from before this code cannot be guarded, and the how-to warns of it.

Restoring a backup. The old steps wrote /data/scrap.db, which the replica would then silently win over. So scrap restore --stage leaves the checked, migrated backup at /data/scrap.db.restored, which outranks the replica; the replica is renamed litestream.retired-<time>, not deleted, and Litestream starts a new one. Guards: a plain scrap restore is refused where SCRAP_LITESTREAM_REPLICA is set (inside the pod), --stage where SCRAP_DB_PATH exists, and --stage on a folder with no sign of Litestream, where nothing would take it up. A /data/scrap.db whose size or modified time differ from the state file stops the start, saying the replica may be the newer copy. A staged restore beside a live emptyDir database stops the pod until it is scaled to 0, and one older than any replica file (scrap kept running after it) is refused, naming how to choose. A retired replica can bring back rows by hand, never files: files added since the backup are unreferenced after a restore, so gc only counts for 14 days afterwards, or until cleared.

Backups stay CronJobs and work from the replica. scrap backup --from-replica restores to a temporary file, then runs the existing backup() on that plus /data/files: same integrity check and COMPLETE marker, both jobs. The data volume is mounted read only. The restore is at or behind the live database and blobs are written before the row that names them, so the files it names are there, but for a gc in the last second, and gc and backups share a lock and run an hour apart.

A stale replica must not make a good-looking backup, because gc trusts COMPLETE. Idle databases write nothing: Litestream’s files do not change, and a snapshot is skipped when nothing changed (seen locally), so file times prove nothing. So scrap schedule stamps the time in the database header (PRAGMA application_id, unused and needing no schema change) every 15 minutes, and the backup refuses a restore whose stamp is over 3 hours old. If Litestream is failing, or the jobs container, the backup fails loudly and then gc only counts.

Writes must not be accepted while replication is broken, then rolled back at the next start. Litestream that cannot write logs monitor error and stays up. Nightly backups would notice hours later, and everything since would go at the next restart. So the jobs container looks each minute for a new file under the replica: a working one has one within about 15 minutes, from the stamp alone. At 20 minutes (watch.py; nothing is judged until 20 minutes after it starts, so a start cannot flap) it logs, notifies and removes the file its readiness probe wants young, so the pod leaves its Service. It recovers by itself. It also notes each stamp on the data volume, with a last one at shutdown. The next start compares: behind is a warning (a node that died, or a last sync that did not land, which nothing else can detect: Litestream’s only sign of that is a log line in a container that has gone); over 30 minutes behind is a refusal. The note is on the same volume, so a whole volume failing does not show here, only as the stall.

gc’s safety limit is the backstop for what all that misses, since deleted originals cannot come back. On a Litestream install gc --delete counts only, and fails, when the database has no captures but files/ has content, or when the files to delete are over MAX_UNUSED_SHARE (10 %) of them and at least MIN_UNUSED (5). Low on purpose: a healthy night deletes few, a false alarm costs one --allow-large by hand, and a miss deletes originals. It cannot catch a small rollback, whose few files fall under it; that is what the state holds are for.

gc and the Claude passes must see the live database, so they move into the pod. A jobs container runs scrap schedule, on the existing image with no new dependency: a few hundred lines of cron (cron.py, following robfig/cron v3 as a CronJob’s schedule is read, but refusing @every and signed numbers, and stopping the rollout on a bad one) and threads (schedule.py). The same values set the same schedules and deadlines. It is close to a CronJob, not the same: an overlapping run is skipped, and after a gap one run stands for those missed, if one is within the hour, as with concurrencyPolicy: Forbid and startingDeadlineSeconds: 3600; but an empty timeZone is UTC rather than the controller’s zone; a time missed during a rollout is not made up and a rollout stops a running pass, as there is deliberately no state to resume from; the jobs share one set of resources, not one each; and there is no Job object to look at afterwards. gc’s script is the same one, shared through a Helm template, so it still takes the backup lock, checks for a COMPLETE run under 26 hours old and otherwise only counts and fails. So the backup volume is mounted in the scrap pod.

Not fighting Litestream. scrap never checkpoints or sets wal_autocheckpoint, locking_mode or synchronous; its one journal_mode = WAL is what Litestream needs; it sets the busy_timeout = 5000 Litestream’s docs ask for; a test checks all of that. SQLite’s automatic passive checkpoint stays: Litestream’s docs recommend turning it off only at very high write rates, and Litestream’s read lock holds it back anyway. A restart of Litestream alone, with writes during the gap, caught up with no loss on the cluster.

Shutdown and probes. Litestream is a native sidecar (an init container with restartPolicy: Always, Kubernetes 1.29), so it starts before scrap and stops after it, and scrap’s last writes reach it. Its final sync has up to 25 s, and terminationGracePeriodSeconds grows by 30. Probes are unchanged: init containers run before the startup probe counts, so a slow restore cannot fail it, and readiness still means /healthz.

Consequences#

  • What can be lost. Replication is asynchronous: while it is healthy, about a second of writes if the node dies. While it has stalled, up to the 20 minutes before scrap leaves its Service. A clean stop loses none. Files are written straight to NFS, so the file of a lost capture stays until gc removes it.

  • Backup jobs still run beside the scrap pod. It was hoped they would not need to. They do on argus: the data claim is ReadWriteOnce, and a pod on another node, even mounting it read only, waits with FailedAttachVolume: Waiting for detach ... Volume is already used by pod(s) (seen on the cluster). The affinity stays. A new ReadWriteMany claim (netapp offers it) might allow dropping it, but accessModes cannot change on an existing claim, and that was not tried.

  • A failed gc or Claude pass no longer shows as a failed Job: it is in the jobs container’s log, and pushed to SCRAP_NOTIFY_URL if set. A pass stopped by a queue pause or a spent budget is not reported as failed: it says so on stderr, and scrap schedule reads that last line.

  • A restart of litestream or jobs that keeps failing takes the pod out of its Service. For both that is right: jobs is what watches replication, so with it down nothing does.

  • The scrap pod now needs the backup folder mounted: a hostPath that a node lacks, or an RWO backup claim on another node, keeps the pod from starting.

  • The database must fit half of litestream.sizeLimit (2 Gi), and at DLS a backup’s restored copy must fit the 2 Gi per container argus allows. Past that the start is refused or the backup fails. jobs has a /tmp of its own, so a job cannot fill scrap’s.

  • As with SQLite on any RWO volume, one writer relies on Recreate and the attach check; two Litestreams replicating into one place can corrupt it.

  • Turning it off again does not bring the database back: /data/scrap.db is the old one. Take a backup, turn it off, and scrap restore --force it (see the how-to).

Tested on argus on 2026-11-05, with Litestream 0.5.18 from Docker Hub, in a throwaway release, as the DLS uid, and this branch’s Python laid over the 1.6.0 image with PYTHONPATH (there is no published image with it yet). Writes reached the replica; a deleted pod came back on another node with its data, including a write just before the delete; truncated LTX files and a missing replica each failed the init container, and scrap did not start; a /data/scrap.db with a -wal came across once; both backup jobs made COMPLETE runs; a staged restore of one came up with its file, without the later write; a plain restore was caught; a stale stamp failed a backup; gc ran in the pod, by hand and on its schedule, and refused to delete with no backup. Not tried there: the Claude passes doing real work (no API key: only that scrap enrich --related runs in the container), a real node death, the backups on a hostPath, install-dls.sh itself, and a database of real size. The rule that keeps a database on the emptyDir has only unit tests: a killed container does not rerun the init container, and a pod restarting on its node with its emptyDir could not be forced. The override was tried, starting a fresh database past a refusal.