S2 — vscode-server inside an ephemeral container#

Verdict: PASS Date: 2026-08-15 (cluster: k3s v1.34, mixed arm64/amd64)

The VS Code remote server installs, starts, listens, installs extensions and survives client disconnect inside a debian:bookworm-slim ephemeral container attached to a running pod, on both amd64 (nuc2) and arm64 (node02/RK3588), reached only through kubectl exec (no port-forward, no pod IP). Cold path — download + extract + start + port discovery — is 5.8 s on amd64, ~10 s on arm64.

Several brief assumptions turned out wrong; see Deviations. The three that change the plan: the version lock is a hard handshake rejection, not a soft fallback; the ~1 GB budget is already blown by the stock server + one extension; and an OOM inside the debug container destroys the whole session unrecoverably because ephemeral containers cannot be restarted.

What was tested#

#

Item

Result

1

target pod + debian:bookworm-slim ephemeral container, sshd re-setup

PASS

2

glibc >= 2.28 verified empirically from ELF symbol versions; Alpine/musl claim tested on a real Alpine ephemeral container

PASS (claim confirmed)

3

Download + extract + run the server (bin/code-server --start-server)

PASS

4

Measured download size / on-disk size / RSS on both arches

PASS

5

Headless --install-extension ms-vscode.cpptools (+ ms-python, redhat.vscode-yaml, local .vsix)

PASS

6

Disconnect / reconnect survival + idempotent re-bootstrap (run 3x)

PASS, after fixing a real liveness bug

7

Egress endpoints captured by strace + IP→hostname matching

PASS

8

Repeat everything on arm64 (node02)

PASS

—

Extra: commit-mismatch handshake probed at the wire protocol level

Hard rejection observed

—

Extra: pod memory limit vs ephemeral container; in-place pod resize

Important findings

Client was the OpenSSH CLI plus a hand-written VS Code remote-protocol probe — no real VS Code GUI client connected, so no extension host / language server was ever spawned. All RSS numbers below are therefore idle server numbers (lower bound). This is the main limitation of the spike.

Exact commands that worked#

1. Namespace + target pod#

kubectl create namespace podbench-s2

# target-amd64.yaml  (swap nodeSelector to kubernetes.io/hostname: node02 for arm64)
apiVersion: v1
kind: Pod
metadata:
  name: target-amd64
  namespace: podbench-s2
spec:
  nodeSelector:
    kubernetes.io/arch: amd64
  containers:
  - name: app
    image: python:3.12-slim
    imagePullPolicy: IfNotPresent
    command: ["python3","-c","import time\nwhile True: time.sleep(5)"]
    resources:
      requests: {cpu: 50m, memory: 64Mi}
      limits: {cpu: "2", memory: 3Gi}

kubectl apply -f target-amd64.yaml
timeout 240 kubectl -n podbench-s2 wait --for=condition=Ready pod/target-amd64 --timeout=230s

2. Ephemeral container (must have a long-running command)#

kubectl -n podbench-s2 debug target-amd64 \
  --image=debian:bookworm-slim --image-pull-policy=IfNotPresent \
  --target=app --container=code --profile=general --attach=false -- sleep infinity

timeout 180 kubectl -n podbench-s2 wait \
  --for=jsonpath='{.status.ephemeralContainerStatuses[?(@.name=="code")].state.running.startedAt}' \
  pod/target-amd64 --timeout=170s

3. sshd, and ssh without any listening socket (inetd mode over kubectl exec)#

PUB=$(cat podbench_key.pub)
kubectl -n podbench-s2 exec target-amd64 -c code -- bash -lc "
  export DEBIAN_FRONTEND=noninteractive
  apt-get update -qq && apt-get install -y -qq openssh-server curl ca-certificates procps
  ssh-keygen -A; mkdir -p /run/sshd /root/.ssh; chmod 700 /root/.ssh
  echo '$PUB' > /root/.ssh/authorized_keys; chmod 600 /root/.ssh/authorized_keys
  printf 'PermitRootLogin prohibit-password\nPasswordAuthentication no\nUsePAM no\n' \
    > /etc/ssh/sshd_config.d/podbench.conf
  /usr/sbin/sshd -t && echo sshd_ok"

# ssh_config
Host podbench-s2
  HostName target-amd64
  User root
  IdentityFile ./podbench_key
  IdentitiesOnly yes
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ProxyCommand kubectl -n podbench-s2 exec -i target-amd64 -c code -- /usr/sbin/sshd -i

$ ssh -F ssh_config podbench-s2 'echo SSH_OK; id; uname -m'
SSH_OK
uid=0(root) gid=0(root) groups=0(root)
x86_64

sshd -i (inetd mode) means no TCP listener anywhere in the pod — the SSH transport is the exec channel itself. Session setup 0.28–0.32 s; bulk throughput 27 MB/s (200 MiB in 7.4 s). Note: do not pass -e to sshd — it writes the auth log to stderr, which kubectl exec muxes back onto the client’s stderr and pollutes every session.

4. Server URLs (both arches exist; commit is the identity)#

$ curl -sSL -o /dev/null -w '%{url_effective}\n' \
    https://update.code.visualstudio.com/latest/server-linux-x64/stable
https://vscode.download.prss.microsoft.com/dbazure/download/stable/a5b500951314efd502d07465bd138dfbd714a960/vscode-server-linux-x64.tar.gz

# arm64 and the small CLI tarball resolve at the same commit, both HTTP 200:
#   .../a5b5009.../vscode-server-linux-arm64.tar.gz
#   .../a5b5009.../vscode_cli_linux_x64_cli.tar.gz

Enumerate every published server commit:

$ curl -sS https://update.code.visualstudio.com/api/commits/stable/server-linux-x64 | jq length
200
$ curl -sS https://update.code.visualstudio.com/api/update/linux-x64/stable/latest | jq -c '{name,version}'
{"name":"1.133.0","version":"a5b500951314efd502d07465bd138dfbd714a960"}

5. The bootstrap script that actually works (idempotent, zombie-safe)#

This is the artifact to steal for the implementation. v1 (pid-file based) had a real bug — see Deviations #4.

#!/bin/sh
# bootstrap-server-v2.sh — run over ssh:  ssh podbench 'sh -s' < bootstrap-server-v2.sh
set -e
COMMIT="${VSCODE_COMMIT:-a5b500951314efd502d07465bd138dfbd714a960}"
case "$(uname -m)" in
  x86_64)  VSARCH=x64 ;;
  aarch64) VSARCH=arm64 ;;
  *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;;
esac
ROOT="$HOME/.vscode-server"
SRV="$ROOT/cli/servers/Stable-$COMMIT/server"
DATA="$ROOT/data"
LOG="$ROOT/.$COMMIT.log"
PORTFILE="$ROOT/.$COMMIT.port"
TOKENFILE="$ROOT/.$COMMIT.token"
mkdir -p "$SRV" "$DATA"

# --- install: stage then move, so a half-extracted tree is never treated as installed
if [ ! -x "$SRV/bin/code-server" ]; then
  echo "BOOTSTRAP: downloading server $COMMIT ($VSARCH)"
  STAGE="$ROOT/.stage.$COMMIT.$$"; mkdir -p "$STAGE"
  curl -fsSL "https://update.code.visualstudio.com/commit:$COMMIT/server-linux-$VSARCH/stable" \
    | tar -xz -C "$STAGE" --strip-components=1
  cp -a "$STAGE/." "$SRV/"; rm -rf "$STAGE"
else
  echo "BOOTSTRAP: server present"
fi

[ -s "$TOKENFILE" ] || cat /proc/sys/kernel/random/uuid > "$TOKENFILE"
TOKEN=$(cat "$TOKENFILE")

# --- liveness by HTTP probe, NOT by pid: pids go zombie under a non-reaping pid-1
alive() {
  [ -s "$PORTFILE" ] || return 1
  curl -fsS -m 3 "http://127.0.0.1:$(cat "$PORTFILE")/version" 2>/dev/null | grep -q "^$COMMIT$"
}

if alive; then
  echo "BOOTSTRAP: reusing running server on port $(cat "$PORTFILE")"
else
  rm -f "$PORTFILE"; : > "$LOG"
  # deliberately NOT passing --enable-remote-auto-shutdown (5-min idle kill, see findings)
  setsid nohup "$SRV/bin/code-server" \
      --accept-server-license-terms --start-server \
      --host=127.0.0.1 --port=0 \
      --connection-token-file "$TOKENFILE" \
      --telemetry-level off \
      --server-data-dir "$DATA" >>"$LOG" 2>&1 < /dev/null &
  i=0
  while [ $i -lt 60 ]; do
    P=$(sed -n 's/^Extension host agent listening on \([0-9]*\).*/\1/p' "$LOG" | head -1)
    [ -n "$P" ] && { echo "$P" > "$PORTFILE"; break; }
    i=$((i+1)); sleep 1
  done
  [ -s "$PORTFILE" ] || { echo "BOOTSTRAP: FAILED"; tail -20 "$LOG"; exit 1; }
  echo "BOOTSTRAP: started on port $(cat "$PORTFILE")"
fi
echo "listeningOn==$(cat "$PORTFILE")=="
echo "connectionToken==$TOKEN=="

Cold run and two warm runs (amd64):

$ time ssh -F ssh_config podbench-s2 'sh -s' < bootstrap-server-v2.sh
BOOTSTRAP: downloading server a5b500951314efd502d07465bd138dfbd714a960 (x64)
BOOTSTRAP: started on port 38155
listeningOn==38155==
connectionToken==8ff4cbb3-7ef3-4cb5-a05d-e63cf27b00b7==
cold_bootstrap_ms=5764

$ ssh -F ssh_config podbench-s2 'sh -s' < bootstrap-server-v2.sh
BOOTSTRAP: server present
BOOTSTRAP: reusing running server on port 38155
...
$ ssh -F ssh_config podbench-s2 'sh -s' < bootstrap-server-v2.sh   # 3rd time, same output

6. Proving the client data path (HTTP + WebSocket over the exec channel)#

$ ssh -F ssh_config -f -N -L 19999:127.0.0.1:38327 -o ExitOnForwardFailure=yes podbench-s2
$ curl -sS http://127.0.0.1:19999/version
a5b500951314efd502d07465bd138dfbd714a960

$ curl -i -H 'Connection: Upgrade' -H 'Upgrade: websocket' -H 'Sec-WebSocket-Version: 13' \
    -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
    "http://127.0.0.1:19999/?reconnectionToken=$(uuidgen)&reconnection=false&skipWebSocketFrames=false&connectionToken=$TOKEN"
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

7. Extensions, headless#

$ $SRV/bin/code-server --accept-server-license-terms \
    --server-data-dir /root/.vscode-server/data --install-extension ms-vscode.cpptools --force
Installing extension 'ms-vscode.cpptools'...
Extension 'ms-vscode.cpptools' v1.32.2 was successfully installed.
install_ms=8354         # amd64;  8068 ms on arm64

$ du -sh /root/.vscode-server/data/extensions/*
330M  ms-vscode.cpptools-1.32.2-linux-x64        # arm64 auto-resolved to -linux-arm64, 261M

Dependency resolution and local .vsix install both work:

$ ... --install-extension ms-python.python --force
Extension 'ms-python.vscode-python-envs' v1.36.0 was successfully installed.
Extension 'ms-python.debugpy' v2026.6.0 was successfully installed.
Extension 'ms-python.python' v2026.4.0 was successfully installed.
Extension 'ms-python.vscode-pylance' v2026.3.1 was successfully installed.

$ ... --install-extension /tmp/cached.vsix --force
Extension 'cached.vsix' was successfully installed.

8. Version-lock probe (raw VS Code remote protocol over the WebSocket)#

Handshake is: WS upgrade with skipWebSocketFrames=true → raw 13-byte framed messages [type:u8][id:u32][ack:u32][len:u32], type=2 = Control. Message 0 is {"type":"auth","auth":"<connection token>","data":...}; message 1 is {"type":"connectionType","commit":"<client commit>",...}.

### A: WRONG commit (deadbeef...)
after auth:            {"type":"sign","data":"fh2yIw...","signedData":"8691Gx..."}
after connectionType:  {"type":"error","reason":"Client refused: version mismatch"}   <<CLOSED>>

### B: CORRECT commit (a5b5009...)
after auth:            {"type":"sign","data":"oYAoMm...","signedData":"915dQl..."}
after connectionType:  {"type":"error","reason":"Unauthorized client refused"}        <<CLOSED>>

(B fails only on the next check — the signed-data challenge, which the probe fakes. The commit check is passed.) The matching source in out/server-main.js:

let S=v.commit, T=this._productService.commit;
if (S && T && S !== T) return m("Client refused: version mismatch");

9. glibc / musl#

# Debian bookworm ephemeral container
$ ldd --version | head -1
ldd (Debian GLIBC 2.36-9+deb12u14) 2.36
$ objdump -T $SRV/node | grep -oE 'GLIBC_[0-9.]+' | sort -uV | tail -3
GLIBC_2.25
GLIBC_2.27
GLIBC_2.28                         # <- highest required symbol version, brief confirmed
$ objdump -T $SRV/node | grep -oE 'GLIBCXX_[0-9.]+' | sort -uV | tail -1
GLIBCXX_3.4.21                     # libstdc++ from GCC 5.1+
$ readelf -p .interp $SRV/node | tail -2
  [     0]  /lib64/ld-linux-x86-64.so.2
$ find $SRV -name '*.node' | xargs -n1 objdump -T | grep -oE 'GLIBC_[0-9.]+' | sort -uV | tail -1
GLIBC_2.28                         # native addons need no more than node itself

# alpine:3.20 ephemeral container, same pod
$ /srv/node --version
sh: /srv/node: not found                 # <- missing ELF interpreter, not missing file
$ ls /lib/ld-musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
/lib/ld-musl-x86_64.so.1
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory
$ /srv/bin/code-server --version; echo "exit=$?"
/srv/bin/code-server: line 22: /srv/node: not found
exit=0                                   # <- WRAPPER SWALLOWS THE FAILURE

10. Pod memory limit vs the debug container#

# pod limit is 3Gi (app container). Allocate 3.6 GiB inside the EPHEMERAL container:
$ kubectl -n podbench-s2 exec target-amd64 -c code -- python3 -c '...allocate 3600 MiB...'
allocated MiB: 2800
command terminated with exit code 137

$ kubectl -n podbench-s2 get pod target-amd64 -o jsonpath='{.status.ephemeralContainerStatuses[0].state}'
{"terminated":{"exitCode":137,"reason":"OOMKilled",...}}     # the whole debug container died
$ kubectl -n podbench-s2 get pod target-amd64 \
    -o jsonpath='{.status.containerStatuses[0].restartCount}'
0                                                            # app container untouched

In-place resize rescues it, and does not disturb a running ephemeral container:

$ kubectl -n podbench-s2 patch pod target-amd64 --subresource resize --patch \
   '{"spec":{"containers":[{"name":"app","resources":{"limits":{"cpu":"2","memory":"6Gi"},
     "requests":{"cpu":"50m","memory":"64Mi"}}}]}}'
pod/target-amd64 patched
# event: ResizeCompleted
$ kubectl -n podbench-s2 exec target-amd64 -c code2 -- python3 -c '...allocate 4200 MiB...'
REACHED 4200 MiB alive
# resizing again 6Gi -> 7Gi while code2 was running: code2 state stayed {"running":{...}}

11. Trimming the server (candidate optimisation)#

$ du -sh $SRV/*
264M  extensions
259M  node_modules
117M  node
7.7M  out
$ du -sh $SRV/extensions/* | sort -rh | head -2
160M  copilot
59M   mermaid-markdown-features

$ rm -rf $SRV/extensions/copilot $SRV/extensions/copilot-chat \
         $SRV/extensions/mermaid-markdown-features
$ du -sh $SRV
428M                       # was 646M on arm64 — 34% saved
$ curl -sS http://127.0.0.1:$PORT/version
a5b500951314efd502d07465bd138dfbd714a960     # still starts and serves; log clean

Findings#

Measurements (VS Code 1.133.0, commit a5b5009…)#

Metric

amd64 (nuc2)

arm64 (node02, RK3588)

Server tarball download

224,027,728 B (213.6 MiB)

215,331,901 B (205.4 MiB)

Download time

2.17 s

2.26 s

Extract time

5.62 s

5.56 s

Extracted server on disk

713,893,077 B (680.8 MiB / 689 M du -sh)

669,263,321 B (638.3 MiB / 646 M)

Idle server RSS

~97 MiB (99,308 KiB fresh; 98,108 KiB with a WS client)

~92 MiB (93,844 KiB)

Second server, same container

85,920 KiB RSS, coexists fine

—

ms-vscode.cpptools on disk

330 M (-linux-x64)

261 M (-linux-arm64)

cpptools install time

8.35 s

8.07 s

Cold bootstrap end-to-end over ssh

5.76 s

~10 s

apt-get install openssh-server curl ca-certificates procps

14–21 s

24 s

Ephemeral container ready (image cached)

~2 s

~5 s

ssh session setup through kubectl exec

0.28–0.32 s

—

Bulk throughput through kubectl exec

27 MB/s

—

Aggregate on arm64 with exactly one extension (cpptools): ~/.vscode-server = 1,043,404,085 B = 995 MiB. Plus the Debian base layer + apt packages, a realistic “Observe mode” debug container is 1.1–1.3 GB of node disk.

Extra disk you will also pay for, once each:

  • data/data/CachedExtensionVSIXs — 190 M after 6 extensions. Safe to delete after install.

  • A second server version (client upgrade) — another full ~690 MiB, coexisting. My test container reached 2.2 GB of ~/.vscode-server with two server versions and six extensions.

Mechanism findings#

  1. sshd -i over kubectl exec is the right transport. No listener, no port-forward, no pod IP. 0.3 s setup, 27 MB/s, and ssh -L forwards work through it — the VS Code remote protocol’s WebSocket upgrade completed (HTTP 101) end-to-end over that path.

  2. code-server --port 0 picks a free port and prints it; the parseable line is Extension host agent listening on <port>. /version returns the bare commit hash and is the cheapest liveness/identity probe.

  3. The server survives client disconnect. After pkill-ing every local ssh process, the server was still listening on the same port and the bootstrap re-attached to it. Default reconnection-grace-time is 10,800,000 ms (3 h).

  4. Two commit-pinned servers run side by side happily in the same container (cli/servers/Stable-<commit>/server), ~86 MiB and ~98 MiB RSS.

  5. All 200 published server commits are still downloadable via https://update.code.visualstudio.com/commit:<sha>/server-linux-<arch>/stable (spot-checked indices 1, 5, 12, 30, 60, 100 — all HTTP 200). Older builds are much smaller: index 100 is 59.5 MB vs today’s 224 MB.

  6. Extension install works headlessly, resolves extensionDependencies, and picks the right platform-specific build per arch automatically.

Egress endpoints (empirically observed, via strace -e connect + IP matching)#

Host

Purpose

Confirmed

update.code.visualstudio.com

redirector + /api/commits, /api/update

yes (150.171.109.216)

vscode.download.prss.microsoft.com

actual server tarball (Fastly, 199.232.x)

yes (redirect target)

marketplace.visualstudio.com

gallery API /‌_apis/public/gallery

yes, matched 150.171.74.16

www.vscode-unpkg.net, <publisher>.vscode-unpkg.net

VSIX asset download + NLS

yes, matched 150.171.109.216

main.vscode-cdn.net

extensions/marketplace.json control list, mcp/servers.json

yes, matched 150.171.109.216

crl.microsoft.com, www.microsoft.com

CRL / signature verification during VSIX install

yes, Akamai 2.19.117.x / 23.214.210.91

default.exp-tas.com, mobile.events.data.microsoft.com

A/B experiments + telemetry

resolved, not contacted with --telemetry-level off

Note crl.microsoft.com — extension signature verification reaches out to Microsoft CRL/OCSP. Any air-gap story must account for it, not just the marketplace.

Deviations from the brief#

1. The version lock is a HARD rejection, not a soft fallback. (biggest finding) The brief says pre-baking is “version-locked to the client”. It is worse than that word suggests. The server compares the client’s commit against its own product.json commit at handshake message 1, before signature validation, and closes the socket with {"type":"error","reason":"Client refused: version mismatch"}. There is no negotiation, no minor-version tolerance, no override flag on the server side. A user on VS Code 1.134 cannot use a baked 1.133 server, full stop. The only escape hatch visible in the source is if (S && T && ...) — a build whose product.json has no commit accepts anything, which is why OSS forks are lax, but the official tarball always sets it. Consequence: a baked image is correct for at most ~4 weeks (VS Code’s monthly cadence), and users on Insiders or a stale client are broken immediately.

2. The ~1 GB soft budget for Observe mode is already exceeded by the stock server. 680 MiB extracted before a single extension; cpptools alone adds 330 MiB. ~/.vscode-server hit 995 MiB on arm64 with one extension and 2.2 GB on amd64 with two server versions and six extensions. The budget needs restating as ~1.5 GB, or the server needs trimming — 34% comes off by deleting extensions/copilot (160 M) and extensions/mermaid-markdown-features (59 M), which still starts and serves /version.

3. An OOM inside the debug container is unrecoverable, and the debug container has no resource limits of its own. Inside the ephemeral container /sys/fs/cgroup/memory.max reads max and cpu.max reads max 100000 — an ephemeral container’s spec cannot carry resources. But it is still confined by the pod-level cgroup: allocating 3.6 GiB in a pod whose only container is limited to 3Gi OOM-killed the ephemeral container itself (exit 137, reason: OOMKilled), while the app container survived with 0 restarts. Because ephemeral containers cannot be restarted, that session was gone permanently — and a replacement ephemeral container comes up with a completely fresh rootfs (ls: cannot access '/root/.vscode-server'), so the 690 MiB server, the extensions and the sshd host keys all have to be reinstalled. Verified directly.

4. kill -0 $pid is not a valid liveness check in this environment. With --target, the shared PID namespace’s pid 1 is the target container’s app process (python3 -c ... here), which does not reap. Every orphaned helper becomes a permanent zombie — I accumulated four. code-server’s sh wrapper went zombie, kill -0 on it returned success, and my first bootstrap script confidently reported “already running, port 38327” for a server that had exited. A client following that would connect to a dead port. Liveness must be an HTTP probe of /version, matched against the expected commit.

5. Remote-SSH’s own --enable-remote-auto-shutdown kills the server after 5 minutes idle. Observed exactly: started 10:44:32, log line at 10:49:32 ServerLifetime: all consumers inactive, shutting down. A concurrently-running server started without the flag stayed alive indefinitely. This interacts badly with “ephemeral containers can’t restart” — but only if the bootstrap is not re-runnable. Recommendation: omit the flag for long-lived debug sessions, or keep it deliberately and make re-bootstrap the documented reconnect path.

6. bin/code-server exits 0 when the interpreter is missing. On Alpine, code-server --version printed /srv/node: not found and returned exit code 0. Any bootstrap that trusts the wrapper’s exit status will report success on a totally broken install. Check for output/port, never the exit code.

7. The Alpine/musl claim is correct but for a more basic reason than glibc symbols. The failure is not a missing GLIBC_2.28 symbol — it is a missing ELF interpreter: node is linked against /lib64/ld-linux-x86-64.so.2, which Alpine does not have, so the kernel returns ENOENT and the shell says not found. (Symbol analysis independently confirms max requirement is exactly GLIBC_2.28 + GLIBCXX_3.4.21.) So the minimum base is glibc ≥ 2.28 → Debian ≥ 10, Ubuntu ≥ 18.04, RHEL/UBI ≥ 8. gcompat was not tested.

8. In-place pod resize works on this cluster and is non-disruptive to ephemeral containers. kubectl patch pod --subresource resize raised the app container’s memory limit 3Gi→6Gi→7Gi with a ResizeCompleted event, no restarts, and the ephemeral container stayed running throughout — afterwards it could allocate 4.2 GiB. This is a genuine new lever the brief does not mention: Podbench can make room for itself.

9. Cold-start is far cheaper than the brief’s framing implies, on both arches. 5.8 s amd64 / ~10 s arm64 for the full download-extract-start cycle. “arm64 is slow” was true for image pulls, not for this: the RK3588 extracted 646 MiB in 5.6 s and downloaded 205 MiB in 2.3 s — statistically identical to the x86 NUC. The dominant cost is apt-get install openssh-server (14–24 s), not the VS Code server.

10. Minor mechanics. kubectl debug ... -- true leaves a Completed ephemeral container that can never be reused, and the name is burnt for the pod’s lifetime — always pass a long-running command. kubectl debug --attach=false plus a wait --for=jsonpath=...state.running.startedAt is the reliable ready gate. And beware pkill -f server-main.js run through kubectl exec -- bash -lc '...': the pattern matches the exec’s own command line and kills the session (exit 143).

Recommendations for implementation#

Download on first connect. Do not bake the server. The version lock is absolute (Deviation 1) and the download is 2.2 s / 5.8 s end-to-end (Finding 9), so baking buys ~6 seconds and costs correctness within a month. Bake the base (Debian ≥ 11 slim + openssh-server + curl + gdb + your tooling) — that removes the 14–24 s apt step, which is the actual bottleneck. If you must reduce cold-start further, run an in-cluster mirror of vscode-server-linux-{x64,arm64}.tar.gz keyed by commit and point the client at it with the remote.SSH.serverDownloadUrlTemplate setting (client-side, not verified in this spike), with update.code.visualstudio.com as fallback.

Ship the v2 bootstrap script verbatim. Its three non-obvious properties are all load-bearing: HTTP /version liveness (never kill -0), stage-then-move extraction, and setsid nohup … </dev/null. Take $COMMIT from the client rather than hard-coding it — Remote-SSH already supplies it; a Podbench-native client should read it from the local VS Code’s product.json.

Omit --enable-remote-auto-shutdown and instead reclaim the whole ephemeral container when the session ends. If you keep it, document that reconnect after 5 min idle re-runs the bootstrap (which is safe — verified 3× in a row).

Size the pod before attaching. Compute required headroom (server ~100 MiB idle + your language servers; Pylance alone is a 117 MiB install and its RSS was not measured here) and call kubectl patch pod --subresource resize to raise the target container’s memory limit before starting the server, restoring it on detach. Without this, Podbench will OOM-kill itself inside tightly-limited production pods and lose all session state (Deviation 3). Fall back to a loud pre-flight warning when the resize subresource is unavailable or the pod is owned by a controller that would fight it.

Superseded on the controller half — see report R13. This spike only ever resized a standalone pod, so “a controller that would fight it” was a guess, and it was wrong: a ReplicaSet reconciles pod existence, not pod spec, and leaves an in-place resize alone. The real hazard is the opposite one — the raised limit lives on the pod while the controller’s template does not change, so the next rollout regenerates the pod without it. The recommendation above stands; only its fallback condition was mis-stated.

Treat the ephemeral container as strictly disposable. Nothing may live only in its writable layer. Put ~/.vscode-server on an emptyDir-backed path if the target pod has one, or accept a full re-bootstrap on every attach (6 s, which is acceptable). Name containers podbench-<n> with an incrementing suffix, because a dead name is burnt for the pod’s lifetime.

Trim the built-in extensions (copilot, copilot-chat, mermaid-markdown-features) post-extract: −218 MiB, 34%, server still healthy. Also rm -rf $DATA/data/CachedExtensionVSIXs after extension installs (−190 MiB). That brings a one-extension Observe-mode container back inside the ~1 GB budget. Re-validate with a real GUI client before shipping — this spike only confirmed the server starts and serves /version after trimming.

Base image must be glibc ≥ 2.28 — debian:bookworm-slim (2.36) is a good default and what was tested. Document Alpine as unsupported, and never trust code-server’s exit code as an install check.

Air-gap requires four host groups, not two: the tarball origin (update.code.visualstudio.com → vscode.download.prss.microsoft.com), the gallery (marketplace.visualstudio.com), the asset CDN (*.vscode-unpkg.net, main.vscode-cdn.net), and crl.microsoft.com / www.microsoft.com for VSIX signature verification. The offline path that works today is --install-extension /path/to.vsix, but extensions with extensionDependencies (e.g. ms-python.debugpy) still reach the marketplace for their deps, so an air-gapped bundle must ship the full dependency closure.

Follow-up work this spike did not cover (none of it blocking):

  • RSS under a real GUI client — extension host + Pylance/cpptools language servers are the memory that actually matters, and were never spawned. This is the single biggest open number for the memory budget.

  • Whether remote.SSH.serverDownloadUrlTemplate genuinely redirects the download.

  • Whether gcompat makes Alpine viable.

  • The small vscode_cli_linux_{x64,arm64}_cli.tar.gz bootstrap path (exists at the same commit, HTTP 200 verified) as an alternative to fetching the full server tarball.

What was left behind#

  • Namespace podbench-s2 — left in place and empty (Active, No resources found).

  • Pods target-amd64 and target-arm64 deleted; all ephemeral containers went with them.

  • Node-side leftovers: the debian:bookworm-slim, alpine:3.20 and python:3.12-slim images are now in containerd’s image cache on nuc2 and node02. Nothing was written to any host path; all state lived in container writable layers, which were reclaimed with the pods.

  • Nothing outside podbench-s2 was created, patched or deleted. The in-place resizes were applied only to podbench-s2/target-amd64, which no longer exists.

  • Local scratch artifacts (not in the repo): /tmp/claude-0/-workspaces-tpi-k3s-ansible/2c9abbf4-6c26-436d-9237-a03194fe5977/scratchpad/spikes/ — bootstrap-server.sh (v1, buggy, kept for reference), bootstrap-server-v2.sh, vsx_probe.py, ssh_config*, podbench_key{,.pub}, target-*.yaml, helper *.sh.