lllm3090.catalog#

The curated model list, and the arithmetic that decides what fits.

Two sources feed the panel’s model list: data/models.yaml (things you could download) and whatever GGUF files are already on disk (things you can run). A catalogue entry carries the numbers needed to answer “will this fit, and how much context do I get” before anything is downloaded – see fit() for the arithmetic and the reasoning behind it.

lllm3090.catalog.MIB = 1048576#: Bytes per KiB/MiB, spelled out so the arithmetic below reads unambiguously.

class lllm3090.catalog.Model(id: str, name: str, repo: str, file: str, size_gb: float, kind: str, params: str, kv_kib_per_token: float, max_ctx: int, expected_tok_s: int | None = None, chat_template: str | None = None, mmproj: str | None = None, mmproj_gb: float = 0.0, verified: bool = False, notes: str = '', tags: list[str] = <factory>)[source]#

Bases: object

One entry in the curated catalogue.

id: str#

name: str#

repo: str#

file: str#

size_gb: float#

kind: str#

params: str#

kv_kib_per_token: float#: KV cache cost per token at f16, in KiB. Derived from the architecture: full_attention_layers x 2 (K,V) x kv_heads x head_dim x 2 bytes. Linear-attention layers contribute nothing per token; sliding-window layers are counted at the engine’s provisioning ratio, not the window.

max_ctx: int#: The model’s own RoPE ceiling. Context beyond this is incoherent, not merely expensive, so it caps every calculation here.

expected_tok_s: int | None = None#

chat_template: str | None = None#: Optional replacement chat template shipped in lllm3090.data. Used where a model’s own template rejects something a client legitimately sends; see docs/explanations and the file’s own comment for why.

mmproj: str | None = None#: Multimodal projector shipped alongside the weights in the same repo. Present means the model can see images: the engine is given –mmproj, and the projector is downloaded with the weights.

mmproj_gb: float = 0.0#: The projector’s size. It occupies VRAM like any other weights, so it is counted against the budget – otherwise the panel promises context that the projector has already spent.

verified: bool = False#

notes: str = ''#

tags: list[str]#

property weights_mib: float#

property vision: bool#

class lllm3090.catalog.Fit(fits: bool, pool_f16: int, pool_q8: int, spare_mib: int)[source]#

Bases: object

Whether a model fits, and how many tokens of cache it leaves room for.

pool_* are total tokens across all concurrent conversations, bounded by VRAM alone. A single conversation is additionally bounded by the model’s RoPE ceiling – see plan().

fits: bool#

pool_f16: int#

pool_q8: int#

spare_mib: int#

property headline: str#

class lllm3090.catalog.Plan(pool: int, parallel: int, per_session: int, capped_by: str)[source]#

Bases: object

How to actually start a model: pool size, slots, and what each gets.

pool: int#

parallel: int#

per_session: int#

capped_by: str#: What stopped the window growing – the card, the architecture, or nothing at all. default is a GGUF the catalogue has never seen, whose KV cost is unknown, so its window is a fixed conservative figure.

property summary: str#

lllm3090.catalog.fit(model: Model, desktop: bool = True, profile: Profile | None = None) → Fit[source]#

Compute the context a model leaves room for on the target card.

The KV cache is what actually decides context, and it is compressible: the engine is run with q8_0 key and value caches, which halve the per-token cost and are close to lossless. q4_0 would halve it again but degrades long-context reasoning, so it is deliberately not offered here.

lllm3090.catalog.plan(model: Model, parallel: int | None = None, desktop: bool = True, profile: Profile | None = None) → Plan[source]#

Decide the pool size and per-conversation window for a model.

Two limits apply and they are not the same limit:

VRAM bounds the whole pool, shared across concurrent conversations.
RoPE bounds one conversation. Past the model’s ceiling, output becomes incoherent rather than merely expensive, so extra pool beyond parallel x max_ctx buys nothing and is not requested.

Spare capacity therefore goes to concurrency rather than to a window the model cannot use – which is what leaves room for an agent’s subagents.

lllm3090.catalog.UNKNOWN_MODEL_CTX = 32768#

Per-slot window for a GGUF that is not in the catalogue.

Nothing is known about its KV cost per token, so there is no arithmetic to do. Guessing high produces an engine that loads and then fails every request out of device memory, which is the expensive mistake; this is the cheap one.

lllm3090.catalog.launch_plan(name: str, parallel: int | None = None) → Plan[source]#

How to start an installed model, whether or not the catalogue knows it.

Every front end – the CLI, the panel and the terminal UI – has to answer the same question before it can launch anything, and they must answer it identically: a model started from the console and the same model started from the panel are the same engine on the same card.

lllm3090.catalog.load_catalog() → list[Model][source]#: The curated list shipped with the package.

lllm3090.catalog.installed(models_dir: Path | None = None) → list[dict[str, Any]][source]#: GGUF checkpoints present on disk, one entry per directory.

lllm3090.catalog.vram_needed_mib(model: Model, ctx: int) → float[source]#

Roughly what a pool of ctx tokens costs, weights and projector included.

The q8 cache halves the per-token figure, which is stored at f16, and config.KV_OVERHEAD_FACTOR restores what the engine holds on top of it.

lllm3090.catalog.startup_vram_mib(model: Model, ctx: int) → float[source]#

What the card must have free for this plan to load and keep serving.

vram_needed_mib is what the load itself allocates. The engine then needs room to work in on top of that – the compute buffers and fragmentation headroom fit() holds back out of the budget, and the vision tower’s own buffers when a projector is loaded. Both are already subtracted when the plan is computed, so a check that compares only the load against free VRAM passes plans that load and then fail every request, which is the exact failure the check exists to catch.

The desktop reserve is deliberately not added: free VRAM is a measurement, and a compositor that is running has already taken its share out of it.

lllm3090.catalog.free_vram_warning(model: Model | None, ctx: int) → str | None[source]#

Warn if the card cannot hold this plan now, or None if it can.

The plan is computed against fixed reserves, which describe a machine at rest. This is the measurement, and it is what catches the estimate being wrong: a model sized on a text console and started under a desktop, or started beside anything else holding VRAM, loads and reports itself healthy before failing every request out of device memory.

Both front ends have to make this check – an engine started from the panel is the same engine on the same card as one started from the console – so the comparison and its wording live here rather than at each call site. Call it after engine.stop(), or the outgoing engine’s VRAM is counted as used against its own replacement.

lllm3090.catalog.catalog_for_panel(desktop: bool | None = None) → list[dict[str, Any]][source]#

Catalogue entries decorated with fit, plan and installed-state, for the UI.

speed_applies is false when the GPU in this machine is not the one the speeds were measured on. Fit and context are computed for the real card; speeds are never scaled to it, because a bandwidth ratio produces a guess and the UI would show it in the same typeface as a measurement.