Serve remote engines through a tunnel and a local proxy#

Status#

Accepted, 15 September 2026.

Context#

lllm2 can run llama-server on a remote GPU, starting with Modal. Bench, warm, harness wrappers, the panel and claude-sandbox all reach the engine at a loopback URL on the engine port, and bench depends on llama-server’s raw completion stream.

Modal offers two ways to reach a server in a container. A web endpoint gives a stable HTTPS URL but applies Modal’s request timeout limits, which a long prompt can exceed. A tunnel exposes a container port directly for as long as the function runs.

Another provider must be able to plug in without changes to consumers.

Decision#

The serve function opens a Modal tunnel to llama-server’s port and publishes the tunnel address. llama-server requires an API key that lllm2 generates for each launch.

lllm2 runs a streaming HTTP proxy on 127.0.0.1 at the usual engine port. It forwards requests to the tunnel without buffering and adds the API key. Engine.base stays a loopback URL for both local and remote engines.

The proxy and the engine lifecycle live in a provider-neutral RemoteEngine. Each provider implements the RemoteProvider interface and registers a factory and a GPU table.

Consequences#

  • Consumers need no changes, and clients keep sending placeholder tokens.

  • llama-server’s full native API is available, with no provider request timeout on long prompts.

  • Anyone who learns the tunnel address can reach llama-server’s endpoints that need no key: /health, /v1/health and the web UI files. Completions and the model list need the key.

  • The local lllm2 process must stay running while the model serves. The proxy also sees every request, which drives the idle timer.

  • The tunnel address changes on every launch, so a client cannot connect to the remote server without lllm2.

  • A new provider needs only a RemoteProvider implementation and a GPU table.