lllm3090.hardware

Which GPU is in the machine, and what may be claimed for it.

The catalogue makes two kinds of claim and they travel differently. Whether a model fits, and what context it leaves, is arithmetic over capacity – it holds on any card once you know how much memory it has. How fast it runs is a measurement, true of the card it was taken on and nowhere else.

So a profile carries capacity (used) and bandwidth (recorded, never used to scale a measured speed).

class lllm3090.hardware.Profile(id: str, name: str, compute_capability: str, vram_mib: int, bandwidth_gbs: int, measured: bool = False, notes: str = '', detected: bool = False, present: bool = True, driver_reserve_mib: int = 512)

Bases: object

A GPU this project knows how to reason about.

id: str

name: str

compute_capability: str

vram_mib: int

bandwidth_gbs: int

measured: bool = False

True for the one card the catalogue’s speeds were measured on.

notes: str = ''

detected: bool = False

True when this was synthesised from an unrecognised GPU rather than matched against the shipped list.

present: bool = True

False when no GPU could be found at all and the capacity is borrowed so the catalogue can still be inspected. Nothing about such a profile describes a card in this machine.

driver_reserve_mib: int = 512

What the driver has already taken out of vram_mib. Read from the running card where possible, otherwise the documented default. See config.DRIVER_RESERVE_MIB.

usable_vram_mib(desktop: bool = True) → int

VRAM available for weights plus KV cache.

vram_mib is the nameplate capacity, which is not what a process may allocate: the driver has already reserved part of it. Subtracting that first is what keeps the rest of this arithmetic honest – everything below is spent against a number the card can actually hand out.

lllm3090.hardware.load_profiles() → list[Profile]

lllm3090.hardware.detect() → Profile

The profile for the GPU in this machine.

An unrecognised card gets a profile built from what nvidia-smi reports, marked detected and never measured: its capacity is known, so fit and context are computed correctly, while nothing claims its speed.

With no GPU at all – CI, a container – capacity is borrowed from the reference profile so the catalogue can still be inspected, but the profile is marked present=False and never measured: there is no card here, so nothing may be said about how fast one would be.

lllm3090.hardware.graphical() → bool

Whether a desktop session is holding VRAM on this machine.

The catalogue reserves DESKTOP_RESERVE_MIB for a compositor and its clients. On a text console that memory is free, and it is worth a great deal: on a 24 GB card it is most of a 35B model’s cache. Rather than make the user remember a flag, ask systemd what target is running.

Unknown means assume the desktop is there. Guessing the other way would hand out context the card does not have – the failure this module exists to prevent – so the safe default is the pessimistic one.

lllm3090.hardware.free_vram_mib() → int | None

VRAM not currently spoken for, as nvidia-smi sees it.

The catalogue plans against a fixed reserve, which is an estimate. This is the measurement, and it is what catches the estimate being wrong – a model sized for a text console and started under a desktop loads, reports itself healthy, and then fails every request out of device memory.

lllm3090.hardware.reference() → Profile

The profile the catalogue’s speeds were measured on.