Installation#

What you need#

  • An RTX 3090 (24 GB, compute capability 8.6). The installer checks and warns loudly on anything else — see Why this is scoped to one GPU.

  • Debian 13 or a derivative — Ubuntu 24.04 and 26.04 are tested.

  • A working NVIDIA driver, 550 or newer. nvidia-smi must run.

  • ~20 GB free disk for a first model, more if you collect them.

Install#

uv manages the tool and brings its own Python, so there is nothing to build and no virtualenv to think about:

# uv, if you do not have it: https://docs.astral.sh/uv/getting-started/installation/
curl -LsSf https://astral.sh/uv/install.sh | sh

uv tool install lllm3090
lllm3090 setup

The uv installer itself needs curl, which a desktop install will have but a minimal Debian image will not (apt install curl). See uv’s installation docs for other methods, including pipx and a standalone binary.

uv tool install puts lllm3090 on your PATH; lllm3090 setup does everything uv cannot do for itself:

  1. checks the OS, GPU, driver and Vulkan ICD, and stops with a specific message rather than a stack trace if something is missing;

  2. installs libvulkan1 if absent — the only thing it touches outside $HOME, and the only apt package the engine needs;

  3. downloads a pinned llama.cpp build and verifies its SHA-256;

  4. writes a systemd user unit for the panel and starts it.

It downloads no model weights. That is deliberate: the right first model depends on what you want to do, and a 15 GB surprise during setup is rude.

setup is safe to re-run — every step is skipped when already done — so it doubles as the repair command after an upgrade.

Installing something other than the release#

To run a branch, a fork, or a local checkout, point uv at it:

uv tool install "git+https://github.com/gilesknap/lllm3090.git@my-branch"
uv tool install --editable .        # from a checkout, for development

Then lllm3090 setup as before.

Check it worked#

lllm3090 doctor

  [ ok ] os           Ubuntu 26.04 LTS
  [ ok ] gpu          NVIDIA GeForce RTX 3090 (24576 MiB, compute 8.6)
  [ ok ] driver       driver 595.84
  [ ok ] vulkan       Vulkan ICD at /usr/share/vulkan/icd.d/nvidia_icd.json
  [ ok ] engine       ~/.local/share/lllm3090/llama.cpp/llama-server
  [ ok ] models dir   /home/giles/models (766 GB free)

Then open http://127.0.0.1:8080.

Note

The panel binds loopback only, by design: its endpoints start processes and write to disk with no authentication. To reach it from another machine use an SSH tunnel — see Reach the panel from another machine.

Why Vulkan and not CUDA#

llama.cpp publishes prebuilt CUDA binaries for Windows only. Building the CUDA backend on Linux needs a host compiler CUDA accepts (GCC ≤ 14), which recent Debian and Ubuntu releases no longer default to. The prebuilt Vulkan binary needs no compiler, works on any NVIDIA driver with the Vulkan ICD installed, and decodes at close to CUDA speed.

The trade-off is prompt processing: Vulkan prefill is roughly 3–4× slower than CUDA. That shows up as a slow first turn on a very long prompt (about two minutes at 80k tokens) and not at all afterwards, because the prefix cache carries the prompt between turns.