Command line#
lllm3090 <command> [options]
command |
what it does |
|---|---|
|
Check the machine can run the stack; exits non-zero on any failure |
|
Prepare the machine; choose where checkpoints live |
|
Fetch and checksum-verify the pinned llama.cpp build |
|
Fetch a build to measure against, beside the install rather than over it |
|
Compile a CUDA engine for this card, beside the installed one. Never switches to it |
|
The catalogue: size, kind, achievable context, state, expected speed |
|
Stop any running engine and start this model |
|
Terminate the engine and wait for the VRAM to be released |
|
Benchmark a model and print a profile block to contribute |
|
Survey published GGUF models and price them against a card |
|
Whether the engine is running, on what, and whether it answers yet |
|
Run the control panel in the foreground |
|
Launch Claude Code against the local engine |
|
Print that environment instead of launching it |
See Choose a context window and slot count for choosing between them, and Find models that fit your card for widening the list.
sweep downloads no weights: it derives each candidate’s KV cost from that
model’s own config.json and runs it through the same arithmetic the panel
uses. --gpu prices against any profile in profiles.yaml instead of the
detected card. It never produces a speed — bench is what does that.
start computes the KV pool from what fits, splits it into --parallel slots
(default 2, so an agent’s subagents have somewhere to go), and gives each slot
as much as it can up to the model’s RoPE ceiling. --ctx overrides the whole
pool; an unknown GGUF gets a conservative 32768 per slot. Anything after claude is passed
through to Claude Code unchanged.
--profile picks what the engine guesses ahead with. The default is
multi-token prediction alone at the engine’s own draft width, which is what
serves when nothing is asked for. --profile copy adds prompt-lookup drafting
and widens the draft to 7, taking the dense 27B from 94.0 to 115.1 tok/s on
copy-heavy work — and it is refused on Vulkan, where the same configuration
loses. The verdicts genuinely invert between backends, because verifying a
draft is a batched forward pass and Vulkan gets nothing from a wider batch; see
Speed levers, measured.
The panel’s engine row does the same selection in a browser and remembers it
across restarts; LLLM3090_LLAMA_DIR still wins over both.
build-cuda compiles llama.cpp at the pinned tag with CUDA, into
engines/<tag>-cuda-sm<arch> where the architecture comes from nvidia-smi
rather than from anything typed. It needs CUDA 13.3 or newer — not the
distribution’s 13.1 — and prints NVIDIA’s repository instructions when there is
none. It never replaces the installed engine: point LLLM3090_LLAMA_DIR at the
result to use it. setup offers the same build when it finds a usable toolkit,
and never builds unprompted, --yes included.
--effort sets how long the model thinks, for the life of the engine:
minimal, low, medium, high, xhigh or max, handed to the chat
template as llama.cpp’s --reasoning-effort. Omit it and the model’s own
default applies — xhigh on Qwen3.8-27B. It is a launch argument because
there is nowhere else to put it: a harness cannot set it per request, and
Claude Code’s /effort does not reach the engine at all
(see Use Claude Code against a local model). A level the model’s template does not
implement fails the start rather than producing an engine that 500s on every
request.
claude --print-env writes the environment as export lines and nothing else
to stdout, so eval "$(lllm3090 claude --print-env)" sets up a shell for a
harness this project does not know about. Everything else it has to say —
warnings, refusals — goes to stderr.
Environment#
variable |
default |
meaning |
|---|---|---|
|
|
Where GGUF checkpoints live |
|
|
Pidfile and engine log |
|
|
The engine build |
|
|
Engine port |
|
|
Panel port |
|
|
Install prefix (installer only) |