lllm3090.engine

Lifecycle for the llama.cpp server.

One GPU means one engine, so this module owns a single process tracked by a pidfile. Start blocks until the server actually answers – the HTTP port binds long before the weights finish uploading, so “the port is open” is not the same as “the model is ready”, and reporting the former as the latter is the most common way to make a slow load look like a failure.

lllm3090.engine.ANSI = re.compile('\\x1b\\[[0-9;]*[A-Za-z]')

Colour and cursor escapes llama.cpp writes when it thinks it has a terminal.

lllm3090.engine.GGUF_MAGIC = b'GGUF'

First four bytes of every GGUF file. The engine reads no other container – the older GGML and GGJT magics were dropped from llama.cpp long before the builds this installs – so this is the whole test.

lllm3090.engine.clean(line: str) → str

Strip ANSI colour and collapse a progress-bar redraw to its last frame.

lllm3090.engine.TAIL_BYTES = 262144

How much of the end of the log is worth reading to find its last lines. An engine that has been up for a day has logged every request it served.

lllm3090.engine.tail(lines: int = 200) → list[str]

The last lines readable lines of the engine log.

This lives with the code that writes the log rather than with the panel, because the log is the engine’s rather than any front end’s: the panel streams it to a browser over SSE, and lllm3090 status and a shell on the same box read the same file.

Split on newlines alone, deliberately. Python’s universal newlines turn a carriage return into a line ending, which silently makes a progress-bar redraw into hundreds of near-identical lines – the exact thing clean() exists to collapse.

lllm3090.engine.server_binary() → Path

The llama-server that will serve, which is not always the installed one.

Resolved per call rather than held in a constant: the panel is a long-running process, and an engine chosen in the browser has to reach the next start without a restart of the thing the browser is talking to.

lllm3090.engine.supports(flag: str) → bool

Whether the installed llama-server accepts flag.

The engine build is pinned, but the installed build is whatever is on disk: install-engine skips replacement when a binary is already there, and setup calls it that way, so upgrading lllm3090 does not upgrade llama.cpp. A flag this project learned about in one release can therefore meet a binary from an earlier one.

Passing an unknown flag is not a soft failure – llama-server exits, and the panel reports “starting” over a process that is already gone. Asking first costs one --help per binary per process: the text is what is remembered, not the answer, because the key used to include the flag and a start asking about five of them spawned five processes to read one page.

lllm3090.engine.CACHE_TYPE = 'q8_0'

What every KV cache this engine allocates is quantised to.

One constant for both caches, because there are two and the bug this fixes was that they disagreed. --cache-type-k/v sizes the main cache; --spec-draft-type-k/v sizes the draft model’s, which llama.cpp defaults to f16 and does not inherit from the main one. So for as long as only the first pair was passed, the model’s own draft context was held at full precision beside a main cache at half – and that gap was the whole of what multi-token prediction cost in memory.

q8_0 halves the cost per token against f16 for close to no quality loss, and is what makes long context affordable on 24 GB. q4_0 would halve it again but degrades long-context reasoning, so it is deliberately not offered.

lllm3090.engine.fixed_flags() → dict[str, Any]

What every model is started with, whichever model it is.

Three settings are identical for all nine catalogue entries and are not visible anywhere: flash attention, the main KV cache quantisation, and the draft model’s own cache. The third is the one worth stating – quantising it gave back 2.45 of the 4.80 KiB/token that MTP costs on the dense 27B, which is most of a percent of the card, and until now nothing said it had happened.

Stated here rather than on each row precisely because it does not vary. A fact repeated nine times reads as nine facts, and the reader has to compare them to discover it is one.

draft_cache_type is None on a binary too old for the flag, because then it is not what happens: the draft cache stays at llama.cpp’s f16 default and the window costs what it used to.

lllm3090.engine.spec_flags(profile: Profile, model_path: str) → list[str]

The speculation arguments for this profile against this checkpoint.

Two things can silently remove a flag here, and both have to be checked on an install that works:

• The binary. The engine build is pinned but not upgraded in place – install-engine skips replacement when a binary is already there – so a binary older than --spec-type survives an lllm3090 upgrade and would exit on an argument it has never heard of.

• The checkpoint. Only the file on disk decides whether the MTP head is there, and llama.cpp refuses to start with --spec-type draft-mtp against one that lacks it. See lllm3090.gguf.

Dropping draft-mtp from a checkpoint that has no head is right for the default profile and wrong for a named one, which is why a named profile is refused earlier, in start(), rather than quietly reshaped here.

lllm3090.engine.REASONING_EFFORTS = ('minimal', 'low', 'medium', 'high', 'xhigh', 'max')

Levels llama-server --reasoning-effort accepts, less default – which means “pass nothing”, and is spelled here by leaving --effort off.

The engine takes the level and hands it to the chat template. What the template does with it is the template’s business: Qwen3.8-27B’s implements low, medium and xhigh, folds high into xhigh, and raises on everything else. So this list is what the engine will accept, not what any given model will render. template_refuses_effort() is what closes that gap.

lllm3090.engine.template_refuses_effort(effort: str, timeout: float = 5.0) → str | None

The chat template’s complaint about effort, or None if it renders.

A template that does not implement a level raises while rendering the prompt, which is a 500 on every request. The engine starts, loads the weights, and answers /health regardless – so without this the start reports success and the failure arrives later, per request, as a Jinja traceback that does not obviously point back at a flag typed minutes ago.

/apply-template renders a prompt and generates nothing, so asking costs a round trip. Anything other than a rendering error – a build without the endpoint, a connection that fails – returns None: not being able to check is not evidence of a problem.

lllm3090.engine.alive(target: int) → bool

Is this PID still a process?

/proc rather than kill(pid, 0): the panel and the CLI both call this, and only one of them is the engine’s parent, so a check that depends on signal permissions would answer differently depending on who asked.

lllm3090.engine.pid() → int | None

PID of the running engine, or None. Stale pidfiles are ignored.

lllm3090.engine.served_model() → str | None

Model id the engine reports, or None if it is not answering yet.

lllm3090.engine.served_slots() → int | None

How many conversations the running engine can hold at once, or None.

Asked rather than derived from the --parallel it was started with: a start can override that, and this project is not necessarily the thing that started the engine answering on this port. /props reports what it is really serving.

lllm3090.engine.healthy() → bool

lllm3090.engine.not_a_gguf(path: Path) → str | None

Why path is not a GGUF, or None if it is one.

A pre-flight rather than a formality. llama.cpp rejects a non-GGUF itself, but it does so a second or two after launch, in the log – and the panel starts the engine with wait=0, so by then it has already told the user “starting”, written a pidfile, and gone back to polling a process that is about to exit. Reading four bytes here turns that into a refusal with a reason, before anything is launched.

lllm3090.engine.stop(timeout: int = 40) → tuple[bool, str]

Terminate the engine and wait for the VRAM to actually come back.

lllm3090.engine.start(model_path: str, name: str, ctx: int, parallel: int = 1, wait: int = 300, chat_template: str | None = None, mmproj: str | None = None, spec: Profile | None = None, effort: str | None = None) → tuple[bool, str]

Launch llama-server and block until it answers.

wait is how many seconds to poll for readiness; 0 returns as soon as the process is launched.

spec is what the engine guesses ahead with; None is lllm3090.speculation.DEFAULT, which is what every start got before there was a choice. Whether a profile is allowed on the installed backend is settled by lllm3090.speculation.resolve() before we get here – this only has to refuse the one thing that cannot be known until the checkpoint is in hand.

ctx is the whole KV pool and parallel is how many conversations share it, so each slot gets ctx // parallel tokens. What to pass is decided by lllm3090.catalog.plan(), which fills one conversation to the model’s ceiling before opening a second; an agent that wants room for a subagent has to ask for it.

Both caches are quantised to CACHE_TYPE – the main one and the draft model’s, which is a separate cache with a separate flag and does not inherit the setting. See spec_flags().

effort is a reasoning level from REASONING_EFFORTS, given to the chat template for the life of the process. It is the only way to reach a model’s thinking-length control from a harness that has no idea it exists: Claude Code’s /effort travels as output_config.effort, a field llama.cpp does not implement and does not reject, so it is dropped in silence and the template’s own default – xhigh on Qwen3.8-27B – stands.

lllm3090.engine.status() → dict

Everything the panel needs to render engine state in one call.