lllm2

A local NVIDIA LLM workbench: download GGUF models, run llama.cpp, compare settings and save measured results from a browser panel.

Install and run

On Linux with Python 3.11+ and a working NVIDIA driver, install uv, then:

uv tool install --upgrade lllm2
lllm2 engines install cuda
lllm2

The engine is downloaded from the matching lllm2 release, with CUDA runtime libraries included. No host compiler or CUDA toolkit is needed. Existing builds can also be discovered. Open http://127.0.0.1:8082, download a model and click Start. The model API listens at http://127.0.0.1:1920/v1.

How the documentation is structured

Documentation is split into four categories, also accessible from links in the top bar.

• Tutorials
  • From a model to Pi
  • Run the panel as a service
  • Run a model on a Modal GPU
  • Upgrade lllm2
  • Run a model without GNOME

Tutorials for installation and typical usage. New users start here.

• How-to Guides
  • Access over SSH
  • Compare and save settings
  • Connect a client
  • Contribute to the project
  • Development and releases
  • Find and download models
  • Install a model engine
  • Run Pi in a container
  • Prepare your machine
  • Serve a model from Modal
  • Serve from the terminal
  • Troubleshoot
  • Let a coding agent find your best settings
  • Use your own GGUF or engine

Practical step-by-step guides for the more experienced user.

• Explanations
  • Architectural Decision Records
  • Defaults and measurements
  • Experiment and model settings
  • Remote engines

Explanations of how it works and why it works that way.

• Reference
  • Commands
  • Paths and ports
  • Modal GPU types
  • Packaged data
  • Historical plans and research
  • Release notes

Technical reference material including APIs and release notes.