lllm3090.engine

Lifecycle for the llama.cpp server.

One GPU means one engine, so this module owns a single process tracked by a pidfile. Start blocks until the server actually answers – the HTTP port binds long before the weights finish uploading, so “the port is open” is not the same as “the model is ready”, and reporting the former as the latter is the most common way to make a slow load look like a failure.

lllm3090.engine.ANSI = re.compile('\\x1b\\[[0-9;]*[A-Za-z]')

Colour and cursor escapes llama.cpp writes when it thinks it has a terminal.

lllm3090.engine.GGUF_MAGIC = b'GGUF'

First four bytes of every GGUF file. The engine reads no other container – the older GGML and GGJT magics were dropped from llama.cpp long before the builds this installs – so this is the whole test.

lllm3090.engine.clean(line: str) → str

Strip ANSI colour and collapse a progress-bar redraw to its last frame.

lllm3090.engine.TAIL_BYTES = 262144

How much of the end of the log is worth reading to find its last lines. An engine that has been up for a day has logged every request it served.

lllm3090.engine.tail(lines: int = 200) → list[str]

The last lines readable lines of the engine log.

This lives with the code that writes the log rather than with either front end, because both read it: the panel streams it to a browser over SSE, and the terminal UI – which is on the same machine by construction, since the panel binds loopback – simply re-reads the end of the file.

Split on newlines alone, deliberately. Python’s universal newlines turn a carriage return into a line ending, which silently makes a progress-bar redraw into hundreds of near-identical lines – the exact thing clean() exists to collapse.

lllm3090.engine.server_binary() → Path

lllm3090.engine.alive(target: int) → bool

Is this PID still a process?

/proc rather than kill(pid, 0): the panel and the CLI both call this, and only one of them is the engine’s parent, so a check that depends on signal permissions would answer differently depending on who asked.

lllm3090.engine.pid() → int | None

PID of the running engine, or None. Stale pidfiles are ignored.

lllm3090.engine.served_model() → str | None

Model id the engine reports, or None if it is not answering yet.

lllm3090.engine.served_slots() → int | None

How many conversations the running engine can hold at once, or None.

Asked rather than derived from the --parallel it was started with: a start can override that, and this project is not necessarily the thing that started the engine answering on this port. /props reports what it is really serving.

lllm3090.engine.healthy() → bool

lllm3090.engine.not_a_gguf(path: Path) → str | None

Why path is not a GGUF, or None if it is one.

A pre-flight rather than a formality. llama.cpp rejects a non-GGUF itself, but it does so a second or two after launch, in the log – and the panel starts the engine with wait=0, so by then it has already told the user “starting”, written a pidfile, and gone back to polling a process that is about to exit. Reading four bytes here turns that into a refusal with a reason, before anything is launched.

lllm3090.engine.stop(timeout: int = 40) → tuple[bool, str]

Terminate the engine and wait for the VRAM to actually come back.

lllm3090.engine.start(model_path: str, name: str, ctx: int, parallel: int = 1, wait: int = 300, chat_template: str | None = None, mmproj: str | None = None) → tuple[bool, str]

Launch llama-server and block until it answers.

wait is how many seconds to poll for readiness; 0 returns as soon as the process is launched.

ctx is the whole KV pool and parallel is how many conversations share it, so each slot gets ctx // parallel tokens. Sizing the pool for one conversation is what makes an agent’s subagents queue behind their parent, so the default is two.

The cache is quantised to q8_0, which halves its cost per token for close to no quality loss and is what makes long context affordable on 24 GB.

lllm3090.engine.status() → dict

Everything the panel needs to render engine state in one call.