lllm3090.catalog#

The curated model list, and the arithmetic that decides what fits.

Two sources feed the panel’s model list: data/models.yaml (things you could download) and whatever GGUF files are already on disk (things you can run). A catalogue entry carries the numbers needed to answer “will this fit, and how much context do I get” before anything is downloaded – see fit() for the arithmetic and the reasoning behind it.

lllm3090.catalog.MIB = 1048576#: Bytes per KiB/MiB, spelled out so the arithmetic below reads unambiguously.

class lllm3090.catalog.Model(id: str, name: str, repo: str, file: str, size_gb: float, kind: str, params: str, kv_kib_per_token: float, max_ctx: int, mtp: bool = False, full_attention_layers: int | None = None, expected_tok_s: int | None = None, chat_template: str | None = None, mmproj: str | None = None, mmproj_gb: float = 0.0, min_compute_capability: str | None = None, verified: bool = False, notes: str = '', tags: list[str] = <factory>)[source]#

Bases: object

One entry in the curated catalogue.

id: str#

name: str#

repo: str#

file: str#

size_gb: float#

kind: str#

params: str#

kv_kib_per_token: float#: KV cache cost per token at f16, in KiB. Derived from the architecture: full_attention_layers x 2 (K,V) x kv_heads x head_dim x 2 bytes. Linear-attention layers contribute nothing per token; sliding-window layers are counted at the engine’s provisioning ratio, not the window.

max_ctx: int#: The model’s own RoPE ceiling. Context beyond this is incoherent, not merely expensive, so it caps every calculation here.

mtp: bool = False#

Whether the checkpoint carries a multi-token prediction head.

Declared here because fit has to price a model before it is downloaded, and the head costs a second KV cache. What actually launches with --spec-type draft-mtp is still decided by reading the file – see lllm3090.gguf.has_mtp() – because being wrong there produces an engine that will not start, where being wrong here produces a window off by a few percent. Different costs, so different sources.

full_attention_layers: int | None = None#

How many of the model’s layers keep a KV cache.

Only needed where mtp is set, because the head is one more of these and that is how its cost is priced. Derived from the GGUF header rather than counted by hand: (block_count - 1) / full_attention_interval, where the - 1 is the head itself. See lllm3090.gguf.full_attention_layers(), and the test that checks every entry here against the real checkpoint.

expected_tok_s: int | None = None#

chat_template: str | None = None#: Optional replacement chat template shipped in lllm3090.data. Used where a model’s own template rejects something a client legitimately sends; see docs/explanations and the file’s own comment for why.

mmproj: str | None = None#: Multimodal projector shipped alongside the weights in the same repo. Present means the model can see images: the engine is given –mmproj, and the projector is downloaded with the weights.

mmproj_gb: float = 0.0#: The projector’s size. It occupies VRAM like any other weights, so it is counted against the budget – otherwise the panel promises context that the projector has already spent.

min_compute_capability: str | None = None#

The oldest GPU generation that can run this entry at all, as an nvidia-smi compute capability – "12.0" for a model whose quantisation only has Blackwell kernels, say.

Deliberately the only hardware requirement that is typed by hand. Everything about memory is arithmetic (min_vram_mib()) and travels to any card on its own; a kernel that does not exist for an older architecture cannot be derived from a file size, so it has to be recorded. Left unset means “no known floor”, which is the common case.

verified: bool = False#

notes: str = ''#

tags: list[str]#

property weights_mib: float#

property vision: bool#

property min_vram_gb: float | None#

Smallest card that runs this model at a window an agent can use.

Computed with a desktop session held back, because that is the machine most people are actually sitting at, and over-promising is the failure this arithmetic exists to prevent. None means no card is enough – see min_vram_mib().

class lllm3090.catalog.Fit(fits: bool, pool_f16: int, pool_q8: int, spare_mib: int)[source]#

Bases: object

Whether a model fits, and how many tokens of cache it leaves room for.

pool_* are total tokens across all concurrent conversations, bounded by VRAM alone. A single conversation is additionally bounded by the model’s RoPE ceiling – see plan().

fits: bool#

pool_f16: int#

pool_q8: int#

spare_mib: int#

property headline: str#

class lllm3090.catalog.Plan(pool: int, parallel: int, per_session: int, capped_by: str)[source]#

Bases: object

How to actually start a model: pool size, slots, and what each gets.

pool: int#

parallel: int#

per_session: int#

capped_by: str#: What stopped the window growing – the card, the architecture, or nothing at all. default is a GGUF the catalogue has never seen, whose KV cost is unknown, so its window is a fixed conservative figure.

property agent_ready: bool#

Whether one conversation clears the harness’s own system prompt.

Fitting and being usable are different questions, and the gap between them is wide: a model can load with room to spare and still leave less context than Claude Code spends before your first word. See config.AGENT_PROMPT_FLOOR.

property summary: str#

lllm3090.catalog.kv_cost(model: Model, backend: str = 'vulkan') → float[source]#

What one token of KV cache really costs this model, in KiB.

Four terms, and the point of separating them is that they move independently – a single constant covering all four was wrong in a different direction for every combination of backend and speculation:

The tensors, at the cache’s real precision. kv_kib_per_token is the f16 figure and q8_0 is 0.53125 of it, not half: 34 bytes per 32 values. Every call site used to halve it, under-counting by 6%.
Multi-token prediction’s own cache, where the checkpoint has the head. It is one more full-attention layer, and it is not free – it was the whole of the difference between this model’s nominal and resident cost. See config.MTP_LAYER_OVERHEAD.
The backend. CUDA costs 1.100-1.136x per token of the same cache.
The allocator, which is what is left once the other three are named.

Deliberately not a fifth, per-model term. Resident VRAM is not linear in context on a hybrid MoE, so a measured per-token figure for one is an average over a curve rather than a property of the model – see config.ALLOCATOR_OVERHEAD. Everything above is arithmetic or a ratio.

lllm3090.catalog.fit(model: Model, desktop: bool = True, profile: Profile | None = None, backend: str | None = None) → Fit[source]#

Compute the context a model leaves room for on the target card.

The KV cache is what actually decides context, and it is compressible: the engine is run with q8_0 key and value caches, which cost about half per token for close to no quality loss. q4_0 would halve it again but degrades long-context reasoning, so it is deliberately not offered here.

backend is which engine would serve. It is not cosmetic: CUDA costs ~10% more per token of the same cache and holds ~230 MiB more before any of it is allocated, so a planner blind to it promises a window the card cannot hold – which reads to a user as the model crashing, not as the plan being too big.

lllm3090.catalog.plan(model: Model, parallel: int | None = None, desktop: bool = True, profile: Profile | None = None, backend: str | None = None) → Plan[source]#

Decide the pool size and per-conversation window for a model.

Two limits apply and they are not the same limit:

VRAM bounds the whole pool, shared across concurrent conversations.
RoPE bounds one conversation. Past the model’s ceiling, output becomes incoherent rather than merely expensive, so extra pool beyond parallel x max_ctx buys nothing and is not requested.

Spare capacity therefore goes to concurrency rather than to a window the model cannot use – which is what leaves room for an agent’s subagents.

lllm3090.catalog.min_vram_mib(model: Model, desktop: bool = True, parallel: int | None = None) → float | None[source]#

The smallest card that leaves this model a window an agent can use.

The inverse of fit(). Where fit asks what context a given card leaves, this asks what card the agent floor demands – the question someone shopping for hardware, or reading this catalogue on a card it was not curated for, is actually asking.

It is derived rather than declared, and that is deliberate. A hand-typed figure in models.yaml would be a second source of truth beside this arithmetic, free to disagree with it and silently invalidated by every change to a reserve – VISION_WORKSPACE_RESERVE_MIB was introduced after the catalogue shipped and moved the answer for every vision entry.

None means no amount of memory is enough: the model’s RoPE ceiling is already at or below the floor, so its window is bounded by the architecture and a bigger card cannot move it. Qwen3-8B is that case, at 32k.

lllm3090.catalog.capability_ok(model: Model, profile: Profile) → bool[source]#

Whether this card’s architecture is new enough to run this model at all.

Unknown in either direction is no objection. A missing min_compute_capability is the common case and means nothing is known to be required; a card that will not report its own capability cannot be proven inadequate, and hiding half the catalogue on that basis is a worse answer than letting a download fail.

lllm3090.catalog.UNKNOWN_MODEL_CTX = 32768#

Per-slot window for a GGUF that is not in the catalogue.

Nothing is known about its KV cost per token, so there is no arithmetic to do. Guessing high produces an engine that loads and then fails every request out of device memory, which is the expensive mistake; this is the cheap one.

lllm3090.catalog.launch_plan(name: str, parallel: int | None = None) → Plan[source]#

How to start an installed model, whether or not the catalogue knows it.

Both front ends – the CLI and the panel – have to answer the same question before they can launch anything, and they must answer it identically: a model started from the console and the same model started from the panel are the same engine on the same card.

lllm3090.catalog.load_catalog() → list[Model][source]#: The curated list shipped with the package.

lllm3090.catalog.installed(models_dir: Path | None = None) → list[dict[str, Any]][source]#: GGUF checkpoints present on disk, one entry per directory.

lllm3090.catalog.vram_needed_mib(model: Model, ctx: int, backend: str | None = None) → float[source]#

Roughly what a pool of ctx tokens costs, weights and projector included.

Priced on the engine that would actually serve, because this is compared against free VRAM on this machine rather than used to give advice.

lllm3090.catalog.startup_vram_mib(model: Model, ctx: int, backend: str | None = None) → float[source]#

What the card must have free for this plan to load and keep serving.

vram_needed_mib is what the load itself allocates. The engine then needs room to work in on top of that – the compute buffers and fragmentation headroom fit() holds back out of the budget, and the vision tower’s own buffers when a projector is loaded. Both are already subtracted when the plan is computed, so a check that compares only the load against free VRAM passes plans that load and then fail every request, which is the exact failure the check exists to catch.

The desktop reserve is deliberately not added: free VRAM is a measurement, and a compositor that is running has already taken its share out of it.

lllm3090.catalog.free_vram_warning(model: Model | None, ctx: int) → str | None[source]#

Warn if the card cannot hold this plan now, or None if it can.

The plan is computed against fixed reserves, which describe a machine at rest. This is the measurement, and it is what catches the estimate being wrong: a model sized on a text console and started under a desktop, or started beside anything else holding VRAM, loads and reports itself healthy before failing every request out of device memory.

Both front ends have to make this check – an engine started from the panel is the same engine on the same card as one started from the console – so the comparison and its wording live here rather than at each call site. Call it after engine.stop(), or the outgoing engine’s VRAM is counted as used against its own replacement.

lllm3090.catalog.STATUS_OK = 'ok'#

What a catalogue entry is, on the card in this machine.

Three states rather than two, because “fits” and “usable” are not the same claim. Ornith-1.5-35B-A3B loads on a 24 GB card and leaves 5k of context; DeepSeek-R1-Distill-Qwen-32B loads and leaves 8k. Both are widely recommended for this card, and neither can run an agent harness whose system prompt alone is 40k. A UI with one “fits” flag has to show them as successes.

lllm3090.catalog.status(model: Model, plan_: Plan, fit_: Fit, profile: Profile) → tuple[str, str][source]#

This model’s state on this card, and the phrase every front end shows.

The wording lives here rather than in the panel and the CLI separately, because both answer the same question about the same card and disagreeing about it is how a promise gets made in one place and withdrawn in another.

lllm3090.catalog.speed_qualifier(profile: Profile, backend: str) → tuple[bool, str][source]#

Whether the catalogue’s speeds describe this machine, and what to say.

Returns (applies, note), where note is the word every front end puts after the figure. There are two independent ways a measured speed can fail to describe what you would get, and collapsing them into one boolean was fine while there was only one of them:

The card. Speeds are never scaled to a different one – a bandwidth ratio produces a guess, and the UI would show it in the same typeface as a measurement.
The backend. New. The same dense 27B serves 54.8 tok/s under Vulkan and 84.9 under CUDA, so a figure labelled only “measured” on a CUDA engine understates by a third while looking like a fact about this machine.

An engine that could not be asked which backend it is – nothing installed yet, a probe that timed out – is no objection, the same way an unknown compute capability is in capability_ok(). lllm3090 models runs before install-engine on a fresh machine, and qualifying every row on that basis would be pure noise about a backend that is about to be Vulkan.

That licence belongs to lllm3090.engines.UNKNOWN alone. A build that reports cpu is an answer, and the wrong one to wave through: the catalogue’s speeds are GPU speeds, and an engine with no accelerator cannot produce them on any card, measured or not.

lllm3090.catalog.mtp_state(model: Model, path: str | None) → tuple[bool, str][source]#

Whether this row gets a multi-token prediction head, and any caveat.

Returns (mtp, note). mtp is deliberately not Model.mtp: it is the answer to the question a reader is actually asking, which is what happens if Start is pressed now. For a checkpoint on the disk that is decided by the file, because the file is what engine.start reads; for one that has not been downloaded there is nothing to read and the catalogue’s declaration is the best available answer.

note is empty unless the two disagree, and both directions matter:

Declared but absent. It starts without speculation, and its context figures were priced for a second KV cache it will not allocate – so the window shown is smaller than the one it will get.
Present but undeclared. It starts with speculation, and the figures under-price the draft cache – so the window shown is one the card may not hold.

Nothing here picks a winner between the two sources. A disagreement is a fact about a checkpoint that is not what the catalogue thinks it is, and quietly preferring either one would hide it. Verified against all nine installed checkpoints on 2026-09-03: they agree, which is why this path has a synthetic test rather than a real one.

lllm3090.catalog.catalog_for_panel(desktop: bool | None = None) → list[dict[str, Any]][source]#

Catalogue entries decorated with fit, plan and installed-state, for the UI.

speed_applies is false when the GPU in this machine is not the one the speeds were measured on. Fit and context are computed for the real card; speeds are never scaled to it, because a bandwidth ratio produces a guess and the UI would show it in the same typeface as a measurement.