Serve a model from Modal#

lllm2 can run a catalogue model on a Modal GPU instead of a local card. The remote server appears on the usual engine port, http://127.0.0.1:1920/v1, so clients, harness wrappers and claude-sandbox need no changes.

Modal bills for every second a GPU container runs. Read Estimate the cost and Clean up before your first launch.

Install the Modal extra#

The Modal client is an optional dependency:

uv tool install --upgrade 'lllm2[modal]'

With pip, use pip install 'lllm2[modal]'. In a checkout, use uv sync --extra modal. Without the extra, a Modal launch stops with a message that names it.

Set up Modal#

  1. Create a Modal account and sign in from the terminal:

    uv tool install modal
modal token new

    The extra puts the Modal client inside lllm2, but it does not put the modal command on your path, so install the command line as its own tool. modal token new writes ~/.modal.toml, which lllm2 reads. The Modal client also reads MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, and MODAL_CONFIG_PATH for a credentials file elsewhere.

  2. Add a payment method in the Modal dashboard. Modal refuses to run GPU functions for a workspace without one.

  3. Check the credentials and deploy the lllm2 app:

    lllm2 modal setup

    This deploys an app named lllm2 to your workspace. lllm2 deploys it again automatically when its version changes, so you run setup only once. The first image build downloads the lllm2 CUDA engine release and takes a few minutes.

  4. Optionally check a GPU type:

    lllm2 modal probe --gpu L4

    A short, billed container prints the GPU name, its memory, the engine’s device list, the engine build it ran and the engine sha256. The build line names three different versions: the llama.cpp release, the CUDA track and the compiler. lllm2 saves the result, and later launches and validation use it. Add --json to print the record as JSON. The container installs the same engine release as lllm2 engines install cuda for the same lllm2 version.

Download a model#

Launch downloads a missing model automatically, but you can store it ahead of time:

lllm2 modal download qwen3-8b

Pass a catalogue id, such as qwen3-8b. The download runs inside Modal and fetches the model and its companion files from Hugging Face into the lllm2-models Volume, so no weights pass through your connection. The command prints progress and downloads nothing if the model is already stored. Press Ctrl-C to cancel; the partial download stays in the Volume and a rerun resumes it.

Launch a model#

lllm2 launch --backend modal --gpu L40S --model qwen3.8-27b

--model accepts a catalogue id or name. Without --model, lllm2 serves the top-ranked catalogue recommendation. Only catalogue models can be downloaded; lllm2 does not upload local files. --engine and --device do not apply, because every Modal launch runs the lllm2 CUDA engine release for the installed lllm2 version.

See Modal GPU types for the --gpu values and their memory. lllm2 sizes context and slots for the chosen GPU, as it does for a local card. Saved workbench settings for the same model, backend and GPU type take precedence.

lllm2 ships no measured recommendation profiles for Modal GPUs yet, so the starting settings are inherited estimates. Measured profiles arrive after a later paid measurement run. Until then, run an experiment on your GPU type and save the settings you choose.

The launch goes through these steps:

  1. Probe. On the first launch with a GPU type, a short container reports the GPU’s real memory and the engine’s capabilities. lllm2 saves the result in remote-probes.json in the state directory and does not probe again until lllm2 or its engine release changes.

  2. Download. If the model is not in the lllm2-models Volume, Modal downloads it from Hugging Face into the Volume. The download runs in Modal, not over your connection, and later launches reuse the file.

  3. Cold start. A GPU container starts, runs llama-server and opens an encrypted Modal tunnel to it. Loading a large model from the Volume takes tens of seconds to a few minutes. --timeout (default 180 seconds) bounds this step; raise it for large models.

  4. Serve. lllm2 listens on 127.0.0.1:1920 and forwards each request to the tunnel. It prints the API URL, the Modal call ID, the estimated hourly cost and the idle timeout.

Press Ctrl-C to stop the container. Billing stops when Modal ends the call.

In the panel#

Start the panel with lllm2 as usual. You can switch between a local and a Modal backend without restarting it. For the short path, follow Run a model on a Modal GPU.

  1. Open Launch → Customize settings and choose the Modal backend in Backend. The engine and device fields are hidden, because every Modal launch uses the lllm2 CUDA engine release.

  2. Choose a Remote GPU type. Each option shows the GPU’s memory and its estimated hourly price. Starting settings are sized for the chosen type.

  3. Optionally set Idle stop (minutes). Blank or 0 disables the idle stop.

  4. Choose a model. With the Modal backend, the model list shows catalogue entries with their presence in the Volume: In Modal storage starts without a download, and Not in Modal storage downloads on the first start. Download to Modal fetches a model ahead of time, and Remove from Modal… deletes it. Download progress also appears in the Downloads card on Launch model. No weights pass through your workstation.

  5. Start the model. The status line names the cold-start phase: checking the GPU type, downloading the model into remote storage, starting the GPU container, loading the model into GPU memory, then ready.

While the container runs, the launch status shows the elapsed time, the estimated cost so far at the hourly price, and the time left until the idle stop, with the pricing caveat underneath. To change the idle stop for the running model, edit Idle stop after (minutes) and click Apply to running model; blank disables it. When the idle stop fires, the status says that the model stopped after that many minutes without requests and no longer bills.

Engine logs & exact command shows the llama-server log lines that the container sends back. Stop model cancels the call. Closing the panel with Ctrl-C, or a stopped service, cancels every call the panel owns.

The Modal storage & running calls card, near the bottom of Launch, lists the stored models with their sizes and the running lllm2 calls with elapsed time and estimated cost. Remove… is disabled for a model that a running call serves. Refresh storage and calls reads the account again.

The panel checks for calls from earlier sessions when it loads, after Refresh storage and calls, and after you adopt or stop a call. It asks a provider only when that provider is the selected backend, the panel already runs its engine, or the call records in the state directory name it. The banner lists only calls with no live owner: a call another session still heartbeats to never appears there, however this workstation’s records look. If a call from an earlier session is still running, a banner at the top of the panel shows its GPU type, model, elapsed time and estimated cost:

  • Adopt and serve takes over the call and serves it on the usual engine port. If a model is already running, the button reads Stop current model and adopt. Adopting needs the call’s saved key, so only calls started from this workstation can be adopted.

  • Stop call cancels it.

Connect claude-sandbox#

A Modal model serves on the same local port as a local one, so Connect a client applies unchanged. Keep lllm2 running on the host while the model serves, and start the client as usual, for example:

uv tool install claude-sandbox && claude-sandbox pi

The client never sees the tunnel or the API key. If the idle stop ends the call, the client’s next request fails until you start the model again.

Estimate the cost#

lllm2 estimates cost from a bundled table of Modal’s per-GPU hourly prices. The estimate excludes CPU, memory and Volume storage charges, and Modal can change its prices, so check current Modal pricing.

Some examples from the bundled table:

Session

Estimate

One hour on a T4

$0.59

One hour on an L40S

$1.95

An L40S left running until the default 30-minute idle timeout

$0.98 after the last request

The probe and download steps also run containers, but for much less time than a serving session.

Set the idle timeout#

A running container stops after 30 minutes without requests. The timer does not run while a request is in progress, and it restarts when a request passes through the local proxy.

lllm2 launch --backend modal --gpu T4 --model qwen3-8b --idle-timeout 10

--idle-timeout accepts whole minutes from 0 to 1440, or off. 0 and off disable the timeout. lllm2 launch takes the first of these that is set:

  1. --idle-timeout.

  2. LLLM2_IDLE_TIMEOUT_MINUTES.

  3. The saved launch settings’ idle_timeout_minutes.

  4. 30 minutes.

The panel fills its Idle stop (minutes) field from LLLM2_IDLE_TIMEOUT_MINUTES, or 30, and sends the field with each start. lllm2 reads the variable once at startup, so set it before starting lllm2:

LLLM2_IDLE_TIMEOUT_MINUTES=off lllm2

With the timeout disabled, the GPU bills until you stop the launch. A serve call ends after 12 hours regardless.

Disable the timeout only for unattended runs that leave long gaps between requests.

Understand the safety net#

While the lllm2 process that starts a container drives it, it sends the container heartbeats, at most every 10 seconds. This applies to serve and download containers. About 3 minutes after the owning process goes silent, the container stops its work and ends the call. A crash, lost network or powered-off machine therefore stops billing within a few minutes. The same rule applies when the owner exits cleanly without cancelling. As a hard limit, Modal ends a serve call after 12 hours and a download call after 2 hours.

Those heartbeats also say who owns a call. A container belongs to the session that keeps heartbeating to it, on whichever machine or container that session runs. Another lllm2 session sees the call as in use until the heartbeat goes silent, and only then as an orphan it may adopt or stop.

Clean up#

List running lllm2 serve calls with their owner, elapsed time and estimated cost:

lllm2 modal list

Each call is owned by this process, in use by another live lllm2 session, or an orphan that no live session owns. A call counts as in use while its owner still heartbeats, even when this workstation has no record of it, so a CLI in a container never mistakes the panel’s call for an orphan. Stop calls:

lllm2 modal stop CALL_ID
lllm2 modal stop --all

--all stops every orphan and never touches a call in use. stop CALL_ID skips a call in use and tells you where it runs; adding --force stops it anyway, which leaves that session without its model, so stop it from that session where you can. --force works on one call id, not with --all.

Stored models incur Modal storage charges. List and remove them:

lllm2 modal models
lllm2 modal remove qwen3-8b

Pass a catalogue id, or a name exactly as models prints it. remove deletes the model with its companion files, and refuses while a running call serves that model.

To remove everything lllm2 created, stop all calls first, then delete the lllm2 app, the lllm2-models Volume, the lllm2-state Dict and the lllm2-logs Queue from the Modal dashboard or with the modal CLI.

See Remote engines for how the proxy, tunnel and API key fit together.