Keep a long render running

Rendering a book takes hours and somnia does one at a time. The thing that does them is somnia worker: a supervisor that watches the queue, starts one child per book, and waits for it. Run it as a systemd user unit and you can forget about it — books asked for from the page, from the agent or from a terminal all land in the same line, and it empties it.

Before starting one, check the machine can actually finish it with somnia-doctor.sh (in the repo’s scripts/, or curl it as Installation does):

bash somnia-doctor.sh

ffmpeg, espeak-ng and a CPU torch all have to be there, and a book needs roughly 30MB of disk an hour at the default bitrate — double that for a book you actually listen to, because opening one in the page joins its chapters into a second copy under SOMNIA_DATA_DIR/streams, which is how a chapter boundary stopped taking the lock screen down with it (ADR 7). That copy is a cache and can be deleted at any time the page is not open; it costs a second or two of ffmpeg -c copy to make again.

Listening to a book while it renders costs more than double, and how much more depends entirely on how fast the box is. A join is named by how many chapters it holds, so every time the listener catches up with the render a new one is written and the old one is left behind — on a box that renders barely faster than it is read, that is a join of the first chapter, of the first two, of the first three, all night. Nothing reaps them yet. If a data directory is larger than you can account for, that is where it went, and rm -r on SOMNIA_DATA_DIR/streams with the page shut is the whole of the cure.

The worker unit

# ~/.config/systemd/user/somnia-worker.service
[Unit]
Description=somnia render worker
After=network-online.target

[Service]
Type=simple
EnvironmentFile=%h/somnia.env
ExecStart=%h/somnia-venv/bin/somnia worker
Restart=always
RestartSec=30
# A stop has to wait for the sentence being spoken, not the chapter being
# rendered: the child finishes its sentence, puts the book back in the queue
# and exits. Ninety seconds is far more than that needs and far less than a
# chapter, which is the point — a chapter killed part way through is the one
# death that can leave the index holding words the player cannot see.
TimeoutStopSec=90
# The page must never wait behind Kokoro on two cores. This is why the renderer
# is a separate unit at all.
Nice=10
CPUWeight=20

[Install]
WantedBy=default.target

loginctl enable-linger $USER            # survive logging out
systemctl --user daemon-reload
systemctl --user enable --now somnia-worker
journalctl --user -u somnia-worker -f

Restart=always is new, and it used to be forbidden in bold. What earned it: re-rendering a chapter now replaces that chapter’s passages in the index rather than adding a second copy of every one of them, a render resumes at the first chapter that has no row rather than at chapter one, and a job that is interrupted three times is given up on with a sentence rather than retried all night. Those three together are #11 closed, and until they landed an automatic restart would have industrialised it.

Deploying is now two restarts — somnia-serve and somnia-worker — and they are independent on purpose: restarting the page’s unit no longer touches a render, which it silently killed every time before.

Watching it

somnia queue
journalctl --user -u somnia-worker -f
somnia find 271 "any phrase from an early chapter"

somnia queue is the quick answer: what is rendering, what is behind it, what died overnight, and which chapter of how many. The journal is the child’s own output — rendering chapter 3/34 per chapter, and the whole traceback of anything that broke. The third is the real end-to-end check, and the more interesting one: it only answers once a chapter has been rendered and indexed, so it says the pipeline works rather than that a process is alive. The page will already be playing chapter one by then — a book is listenable while the rest of it is still being read.

What that looks like on the phone if the listener catches up with the renderer is a pause, not an ending: the sound stops a fraction before the end of what has been rendered, the lock screen card stays up with a play button on it, and the book carries on by itself when the next chapter lands. It is deliberate, and the reason is that letting the audio actually run out is what takes the card down and ends the night — ADR 7. So a silent phone with a paused card at 1am usually means the renderer, not the network, and somnia queue is the thing to look at.

Stopping one

somnia queue stop 4

It stops at the end of the sentence it is on, about twenty seconds, and everything already rendered stays playable.

systemctl --user stop somnia-worker is the other kind of stop, and means something different: nobody stopped wanting the book, so it goes back into the queue rather than being cancelled, and the chapter in flight is lost — minutes of Kokoro. The detail of both is in queue stop and worker.

Renders the agent starts

When serve is running the agent can ask for a book itself, and since it goes through the same queue as everything else, two things that used to be true no longer are.

It no longer spawns anything, so its renders are not invisible any more. They run under somnia-worker like every other render, which means journalctl --user -u somnia-worker has the chapter lines and the tracebacks for books nobody started by hand — the first time an agent-started render has left a trace anywhere.

And it no longer refuses a book somnia already has in any state. A book rendered in full is still refused, and so is one already in the queue, but a render that died, was stopped, or was killed by a reboot can be asked for again — which is the retry that was impossible before, and which is safe now only because a resume adds no second copy of anything.

By hand, without the unit

somnia add 271

This puts the book in the queue and then renders the head of the line itself, under exactly the same claim the worker’s child takes. So it refuses in the first second — with a sentence pointing at somnia queue — if the worker is already rendering something, rather than becoming a second renderer on two cores. somnia worker --once is the same thing without submitting anything: take the next book in the line, render it, exit.

Both run in the foreground and die with the terminal, so use tmux if you are going to close it:

tmux new -s render 'somnia add 271'
tmux attach -t render

Fine for one book you are watching. The unit is better for anything you want a journal of afterwards, and it is the only thing that empties a queue by itself.