lllm3090.engines

The llama.cpp build this project runs, and the ones it measures against.

llama.cpp tags almost every commit that passes CI and attaches prebuilt binaries to each tag, so a build number is a complete identifier: tag, commit and asset all follow from it. Nothing here tracks a branch and nothing resolves “latest” – that would silently change the thing every figure in the catalogue was measured against.

Two digests, and they are not the same claim. LLAMA_SHA256 below is recorded in the repository: reviewed in a diff, outside the serving party’s control, and therefore the only one that can notice a tag whose bytes changed after the fact. The digest the release API publishes cannot – it would move with them – but it does say what GitHub is serving right now, which is what lets a build nobody has pinned yet be verified as strictly as one that is.

So a fetch verifies against both where both exist, and promoting a candidate build to the pin is the act of committing the digest it printed.

lllm3090.engines.LLAMA_BUILD = 'b10715'

The build the product runs.

lllm3090.engines.LLAMA_SHA256 = '1246c764f630f1cfc7c0921353a9719603c5d9ccfa7ced621bac216ffd9b2d87'

Its digest. See the module docstring for why this is written down rather than only asked for at fetch time.

lllm3090.engines.FLAVOUR = 'bin-ubuntu-vulkan-x64'

Vulkan rather than CUDA, for the reason in docs/tutorials/installation.md: upstream publishes no CUDA archive for Linux, only for Windows.

exception lllm3090.engines.BuildError

Bases: RuntimeError

A build could not be fetched, or could not be trusted once it was.

lllm3090.engines.asset(build: str) → str

lllm3090.engines.url(build: str) → str

lllm3090.engines.bench_dir(build: str) → Path

Where a build lives when it is here to be measured rather than to be run.

Kept apart from config.LLAMA_DIR so that benchmarking a candidate never disturbs the engine the panel launches – which matters because deciding whether to move the pin means running the candidate and the incumbent.

lllm3090.engines.published_digest(build: str, timeout: float = 30.0) → str

The sha256 GitHub publishes for this build’s asset, read before fetching.

Available for any tag, which is what makes “always verify” achievable for a candidate build and not only for the pinned one.

lllm3090.engines.fetch(build: str, target: Path, expect: str | None = None) → str

Download, verify and unpack one build. Returns the digest of what landed.

expect is a digest recorded in this repository. When it is given and the API disagrees with it, the fetch stops: the asset behind the tag has been replaced, which is the one thing a published digest can never tell you on its own. When it is given and the API cannot be reached – it is rate limited unauthenticated – the recorded digest stands alone and the fetch continues, because it is the stronger of the two anyway. Without it, an unreachable API is fatal: an unverified build is not worth measuring.

lllm3090.engines.DEVICE_BACKENDS = {'cuda': 'cuda', 'metal': 'metal', 'rocm': 'rocm', 'sycl': 'sycl', 'vulkan': 'vulkan'}

The device-name prefixes llama.cpp prints, and the backend each one means.

--list-devices names devices, not backends: a Vulkan build reports Vulkan0, a CUDA build CUDA0. That is the only place the distinction is visible from outside – both ship a binary called llama-server, neither writes a backend banner to its log on this build, and --version reports the commit, which for two builds of the same commit is the same.

lllm3090.engines.CPU = 'cpu'

What a build reports when it found no accelerator at all.

A real answer about a real binary, and therefore not what an engine that could not be asked is called – see UNKNOWN. The two were one value until a CPU-only build would have had the catalogue label Vulkan GPU speeds “measured” on an engine incapable of producing them.

lllm3090.engines.UNKNOWN = 'unknown'

What an engine that could not be asked is called – nothing installed yet, a probe that timed out, a probe that exited non-zero.

Distinct from CPU because the two carry opposite claims. cpu is a fact about a binary – no accelerator, so no measured speed here applies. unknown is the absence of a fact, and it has to stay the benign fallback: lllm3090 models runs before install-engine on a fresh machine, and qualifying every row there would be noise about a backend that is about to be Vulkan.

Never cached, for the same reason a timeout is not: it is a statement about the moment, and the binary is still whatever it was compiled as.

lllm3090.engines.backend(directory: Path | None = None) → str

Which compute backend the engine in directory was built against.

Answers "cuda", "vulkan" or "cpu" – and names ROCm, SYCL or Metal if it ever meets one. This is not cosmetic. The KV cache costs about 10% more per token on CUDA than on Vulkan and the backend carries its own fixed overhead, so a planner that does not know which one it is planning for over-promises context on the more expensive of the two; and the speculation settings that win on one lose on the other. A package with two backends on disk and no way to tell them apart gets both wrong.

Cached, because catalog.fit runs once per catalogue entry per lllm3090 models and a subprocess apiece would be felt.

lllm3090.engines.VULKAN_CEILING = 200704

The longest pool of the dense 27B each backend will actually hold, found by loading until it dies, with a desktop session running. Vulkan dies at 204800 and CUDA at 176128.

These are what makes “CUDA costs context” a number rather than an impression: it is ~10% more KV per token plus ~230 MiB more fixed overhead, and it lands as about a seventh of the window. Recorded here, together, so that the figure setup quotes and the figure the planner would use cannot drift apart – and so re-measuring is one edit.

lllm3090.engines.cuda_window_cost() → int

What choosing CUDA costs in window, as a whole percent.

lllm3090.engines.MIN_CUDA = (13, 3)

The oldest CUDA that can compile this against a current glibc.

Not a preference. Ubuntu 26.04 ships 13.1 in multiverse, and it declares rsqrt/rsqrtf without an exception specifier while glibc 2.43 declares them noexcept(true); nvcc then refuses to compile anything that includes <math.h>, which is everything. 13.3 tests for glibc >= 2.42 and matches it. So the toolkit has to come from NVIDIA’s own repository, and a machine with both installed must not be allowed to pick the older one – which is exactly what /usr/local/cuda does when alternatives points there.

lllm3090.engines.SOURCE_REPO = 'https://github.com/ggml-org/llama.cpp.git'

Where llama.cpp is cloned from to build it. The same project the pinned archives come from, checked out at the same tag, so a locally built engine is the pinned build compiled differently rather than a different engine.

lllm3090.engines.BUILD_TARGETS = ('llama-server', 'llama-bench', 'llama-cli')

What is worth compiling. The server is the product; bench and cli are what make a locally built engine measurable against a downloaded one, which is the only way anyone can check the claim that it is faster.

lllm3090.engines.TOOLKIT_INSTRUCTIONS = "  wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb\n  sudo dpkg -i cuda-keyring_1.1-1_all.deb\n  sudo apt-get update\n  sudo apt-get install -y cuda-toolkit-13-3\n\nNot `apt install nvidia-cuda-toolkit`. The distribution's 13.1 cannot compile\nthis against glibc 2.43 -- it declares rsqrt/rsqrtf without an exception\nspecifier where glibc declares them noexcept(true), and nvcc then refuses\nevery file that includes <math.h>."

What to tell someone who has no usable toolkit.

Printed rather than executed. This is 4-6 GB of packages from a third-party repository, and installing that behind someone’s back – during a command whose job is to get a Vulkan engine working – is not a reasonable reading of “setup”.

class lllm3090.engines.Toolkit(nvcc: Path, version: tuple[int, ...])

Bases: object

An nvcc on this machine, and whether it is new enough to be used.

nvcc: Path

version: tuple[int, ...]

property text: str

property usable: bool

lllm3090.engines.toolkits() → list[Toolkit]

Every CUDA toolkit findable on this machine, newest first.

Both the PATH and /usr/local are searched, and /usr/local/cuda is deliberately not trusted as the answer: it is a symlink managed by alternatives, and on a machine with 13.1 and 13.3 installed it points at whichever one was configured last. Picking the newest that works is the only reading of “a CUDA toolkit is present” that cannot pick the one this cannot be built with.

lllm3090.engines.toolkit() → Toolkit | None

The newest toolkit that can actually compile this, or None.

lllm3090.engines.sm(compute_capability: str) → str | None

A compute capability as CUDA spells an architecture: 8.6 -> 86.

None for a card that would not report one. Derived from what nvidia-smi says rather than typed anywhere, because a hand-written architecture is a number that goes stale in a machine the moment its card is replaced – and the binary it produces does not fail, it runs wrong.

lllm3090.engines.cuda_dir(build: str, arch: str) → Path

Where a CUDA build for one architecture lives.

The architecture is in the name on purpose. The build is compiled sm_<arch>-real, which will not run on a different architecture at all, so naming the directory after the tag alone would let a card swap turn into an engine that dies at load for no stated reason. Named this way, the absence is legible: the directory for the new card simply is not there.

lllm3090.engines.cuda_builds() → list[Path]

Every locally compiled CUDA engine on this machine.

lllm3090.engines.stale_cuda_builds(build: str | None = None) → list[Path]

Locally compiled engines that are no longer the pinned build.

The most likely way this feature rots. A CUDA engine is tied to the commit it was compiled from, and moving LLAMA_BUILD does not move it – so a user who chose CUDA is silently left on an older engine than the one every figure in the catalogue was measured against, with nothing to say so.

lllm3090.engines.build_cuda(target: Path, arch: str, kit: Toolkit, build: str | None = None, log: Path | None = None, on_progress: Callable[[str], None] | None = None) → Path

Compile a CUDA llama-server for one architecture. Returns target.

Checked out at the pinned tag, so this is the shipped build compiled differently rather than a different engine. Built into a temporary directory and copied into place only once the binary exists, because a half-populated engine directory is indistinguishable from a complete one and would be launched as though it were.

class lllm3090.engines.Choice(path: Path, backend: str, installed: bool, stale: bool)

Bases: object

One engine this machine could serve with.

path: Path

backend: str

installed: bool

True for the engine install-engine manages. The others are compiled.

stale: bool

Compiled against a build that is no longer the pinned one. Offered anyway – it works – but every catalogue figure was measured on the pin.

property id: str

What a front end sends back to choose this one.

The directory name, never a path. The panel is a loopback HTTP server that starts processes, and accepting a path from it would let a request name any binary on the disk; a name is looked up in the list this function produced, so nothing outside it can be selected.

lllm3090.engines.choices() → list[Choice]

Every engine on this machine that could serve, best-known first.

The managed install first because it is what the catalogue’s speeds were measured on and what a fresh machine has.

lllm3090.engines.chosen() → Path | None

The engine directory recorded as chosen, or None.

None covers every way of not having an answer – never chosen, the file corrupted, the directory deleted since. A stored choice that no longer names a runnable engine is not an error to raise at the caller: the engine it named can be removed by a build-cuda --force that failed halfway, and the right behaviour then is to fall back to the install that is still there rather than to refuse to start anything.

lllm3090.engines.active_dir() → Path

The engine directory that will actually serve the next start.

Precedence, highest first:

• LLLM3090_LLAMA_DIR. An explicit override from the environment, and the documented way to run an engine this code knows nothing about – a build under test, a distribution’s own. It outranks a stored choice because it is the more specific statement, and because a variable set in a shell is visibly about this invocation.

• What select() recorded.

• The managed install.

lllm3090.engines.select(choice_id: str | None) → Path

Record which engine serves from now on. Returns the directory chosen.

None clears the choice, which is how a front end says “back to the managed install” without having to know its name.

Nothing is started, stopped or copied. A running engine is a process that already has its binary open and keeps serving from it; this decides what the next start loads, and the front ends say so rather than implying a switch that has not happened.