lllm3090.config

Paths, ports and the hardware envelope this project targets.

Everything is overridable by environment variable so the test suite and a second machine do not have to agree with the defaults.

lllm3090.config.MODELS_DIR = PosixPath('/home/runner/models')

Where GGUF checkpoints live. One directory per model.

lllm3090.config.STATE_DIR = PosixPath('/home/runner/.local/state/lllm3090')

Pidfiles and engine logs.

lllm3090.config.LLAMA_DIR = PosixPath('/home/runner/.local/share/lllm3090/llama.cpp')

The unpacked llama.cpp build.

lllm3090.config.PANEL_URL = 'http://127.0.0.1:8080'

Where a front end that is not a browser goes looking for the panel.

lllm3090.config.REFERENCE_PROFILE = 'rtx-3090'

The profile whose card the catalogue’s speeds were measured on.

lllm3090.config.MIN_DRIVER_VERSION = 550

Minimum driver that carries a working Vulkan ICD for this stack.

lllm3090.config.DESKTOP_RESERVE_MIB = 2400

VRAM held by a typical desktop session (compositor, browser). Subtracted so “will it fit” is honest for a machine someone is also sitting at.

lllm3090.config.WORKSPACE_RESERVE_MIB = 1024

Compute buffers, CUDA/Vulkan graphs and fragmentation headroom.

lllm3090.config.DRIVER_RESERVE_MIB = 512

VRAM the driver holds back before any process allocates a byte – page tables, the console framebuffer, and the card’s own bookkeeping. nvidia-smi reports it as memory.reserved and it is not part of what a process can claim, so a budget computed from the nameplate capacity overstates the card by exactly this much.

Measured at 451 MiB on the 3090 here, on a text console with nothing else running. Leaving it out is what let a plan of 2 x 262144 tokens be issued with 52 MiB of margin against a card that had already given 451 away: the engine loaded, prefill degraded from 88 to 21 tok/s over three batches as compute buffers fought for room that was not there, and the run ended in vk::DeviceLostError with the GPU spinning at 100% and zero memory traffic. This is the fallback for a profile that is not the running card; hardware.detect substitutes the live figure when nvidia-smi reports one.

lllm3090.config.KV_OVERHEAD_FACTOR = 1.12

What a token of KV cache really costs, against the nominal kv_kib_per_token. The nominal figure is the tensor arithmetic; llama.cpp also carries per-cell bookkeeping and allocates the pool whole at load, so resident cost runs above it.

Measured on Gemma-4-26B-A4B at two pool sizes 344k tokens apart: solving the two peaks for a fixed cost plus a per-token cost gives 11.2 KiB/token against a nominal 10, and the implied fixed cost agreed between the two runs to within 1 MiB. Without this, a plan sized to the last byte of the nominal cache overruns the card by 12% of the pool.

lllm3090.config.VISION_WORKSPACE_RESERVE_MIB = 1024

Extra VRAM held back when a multimodal projector is loaded, on top of the workspace reserve above. The vision tower needs its own compute buffers, and they are not the projector file’s size: measured on a 3090, Gemma-4-26B-A4B’s 1.19 GB projector cost 1376 MiB resident, and at a full KV pool the engine then loaded happily and failed every request with vk::Device::allocateMemory: ErrorOutOfDeviceMemory. Counting only the file promises context the card cannot serve.

lllm3090.config.AGENT_PROMPT_FLOOR = 40000

Tokens an agent harness spends on system prompt and tool definitions before any of your work, every turn. Claude Code sits around 40k. A model whose per-conversation window is below this cannot run it at all – the first message fails – and a window only slightly above it leaves no room to work.

lllm3090.config.DEFAULT_PARALLEL = 2

How many conversations must fit at once.

The KV cache is a single pool shared by every concurrent request, not a per-conversation budget. An agent that spawns subagents therefore needs room for more than one: with a pool sized for exactly one session, a parent holding most of it leaves nowhere to admit a subagent, so the scheduler serialises them – and the subagent’s prefill evicts the parent’s cached prefix, so the parent then pays a full cold prefill on its next turn. Two is the minimum that keeps a parent and one subagent resident together.

lllm3090.config.MAX_AUTO_PARALLEL = 4

Ceiling on slots handed out automatically.

A model that reaches its RoPE ceiling before it exhausts VRAM can have extra slots for free – the spare cache cannot become context, so it may as well become admission. But llama.cpp sizes some compute buffers per slot, and the value of a fifth concurrent conversation is speculative on a single-GPU box, so the automatic grant stops here. Ask for more explicitly if you want it.