Your first session#

By the end of this you will have a debug container running inside a pod, an ssh session into it, and VS Code editing files that live in your cluster. Nothing you do here touches a real workload — you will create a throwaway namespace and delete it at the end.

Budget about fifteen minutes, most of it waiting for image pulls.

You need uv and kubectl on your machine, the one-time ssh Include line, and a cluster you are allowed to create pods in. A local kind cluster is ideal. The launcher itself is not installed — uvx fetches and runs it.

uvx podbench doctor --fix checks all of that and adds the Include for you; it exits 0 when nothing is in the way.

Note

The measurements quoted throughout these docs were taken on a 6-node k3s cluster, not on kind. The behaviour is the same; the timings will not be.

1. Something to debug#

$ kubectl create namespace podbench-demo
$ kubectl -n podbench-demo apply -f - <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}
    spec:
      containers:
        - name: web
          image: python:3.12-slim
          command: ["python", "-m", "http.server", "8080"]
          workingDir: /tmp
          ports:
            - containerPort: 8080
          resources:
            limits:
              memory: 3Gi
              ephemeral-storage: 4Gi
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector: {app: web}
  ports: [{port: 80, targetPort: 8080}]
YAML
$ kubectl -n podbench-demo rollout status deploy/web

The limits are deliberate and generous. A VS Code session is a 1.1–1.3 GB working set on node disk, and in Observe mode it is spent from this pod’s budget — see step 4 below.

Get the pod name:

$ kubectl -n podbench-demo get pods
NAME                   READY   STATUS    RESTARTS   AGE
web-6c9d7f4b8b-hq2vn   1/1     Running   0          25s

2. Attach#

$ uvx podbench attach web -n podbench-demo
'web' matched pod web-6c9d7f4b8b-hq2vn

web rather than the whole name: podbench matches what you type against the pods in the namespace and says what it resolved to. The full name and pod/NAME still work, and a substring matching several pods gets you a list to choose from (see Naming the pod).

Nothing is installed to run that: uvx fetches the launcher, runs it against your kubeconfig and leaves nothing behind. It does five things:

  1. reads the target pod’s spec, so it knows the workload container’s UID and whether the pod insists on runAsNonRoot;

  2. walks the capability ladder — root + CAP_SYS_PTRACE first, the target’s own UID with no capabilities if admission refuses that — posting each attempt to the ephemeralcontainers subresource;

  3. waits for the container to be genuinely running, not merely accepted;

  4. runs podbench capreport inside the container it just landed, on that node;

  5. writes an ssh stanza to ~/.podbench/config.d/ and tells you the host alias.

3. Wait — what did it just do to my pod?#

It appended an ephemeral container to the pod spec, permanently. Ephemeral containers cannot be removed, restarted or edited; the name podbench-1 is now burnt for the rest of that pod’s life. Running attach again reconnects to it rather than adding a second one.

That is why the demo pod is disposable. On a real pod, read Read this before you attach to a live pod on the front page first.

4. Read the report before you connect#

The point of the capability report is that it is measured, not inferred from the spec podbench asked for:

seat        podbench-demo/web-6c9d7f4b8b-hq2vn[podbench-1]  (new)
target      web
version     0.4.0b1, the same build as this launcher
owner       kubernetes-admin
rung        full - uid 0, gid 0, CapEff 00000000a80c25fb
ladder
  full      landed   running since 2026-08-21T09:14:02Z
supports
  [x] live attach (gdb -p <pid>)
  [x] read-only inspect (/proc/<pid>/root, maps, environ)
      root, maps and environ readable
  [x] debug launched processes (podbench dbg --launch ./prog)
  [ ] iterate (edit, relaunch, verify through the Service)
  [x] ssh seat (Remote-SSH: editor, shell, git, sftp)
  [x] exec seat (kubectl exec -- podbench capreport, pids, dbg)
measured    --no-probe skips this block
  verdict     live attach available
  blocker     none
  node        kind-worker
  yama        1
  ids         seat 0:0, target 0:0
  pause       none - PTRACE_SEIZE does not stop the tracee
  memory      2986Mi free of 3Gi (86Mi in use)

Four lines are worth learning to read:

  • version — which build of podbench answered, measured by running podbench --version inside the seat. The launcher and the image are one release in two places and can drift apart, most easily on a tag that moves: the node serves the copy it already has, and a fix that is in the launcher but not in the seat reads exactly like a fix that does not work. When the two differ the report says so, on one WARNING line.

  • rung — what the seat is, read from its own /proc/self/status, which is why the line cites the numbers behind it. degraded means the seat runs at the target’s own UID with nothing effective — a normal outcome, not a failure, and the command still exits 0. What podbench asked admission for is on the ladder lines below, and the two can differ: a policy that rewrites a request rather than refusing it leaves a container the spec no longer describes.

  • blocker — what actually stops ptrace, if anything. Several unrelated mechanisms refuse with the same EPERM — a missing capability, Yama, seccomp, an LSM label mismatch (SELinux or AppArmor), or a uid/gid mismatch; this line names which.

  • yama and node — both are per-node. Attach working on one pod and being denied on the next, in the same cluster, is expected: kernel flavours differ. podbench never caches a cluster-wide answer.

  • memory — the headroom in this pod, read with kubectl top pod. An ample margin is a number and not a caution. Where there is no metrics API the row reads in use not measured (no metrics API here), which says unmeasured and not fine.

The indented line under a tick is the measurement the tick was taken from — here, which of the target’s /proc paths actually opened. Read it rather than the box: cmdline, status and fd only under an empty box means the seat is launch-only, and podbench dbg --launch is where to go next.

5. Connect with ssh#

The last lines of the attach output tell you the alias:

ssh config written to /home/you/.podbench/config.d/podbench-demo-web-6c9d7f4b8b-hq2vn-1.conf
add this to ~/.ssh/config once:  Include /home/you/.podbench/config.d/*.conf
or let podbench check and add it:  podbench doctor --fix
then:  ssh podbench-podbench-demo-web-6c9d7f4b8b-hq2vn-1   (or Remote-SSH: Connect to Host -> podbench-podbench-demo-web-6c9d7f4b8b-hq2vn-1)
to debug in VS Code, run `podbench debug-config` in the seat (writes .vscode/launch.json)

If you have not added the Include line yet, run uvx podbench doctor --fix now — it adds the line above any Host * block, which is where it has to be (see Setup). Then:

$ ssh podbench-podbench-demo-web-6c9d7f4b8b-hq2vn-1
root@web-6c9d7f4b8b-hq2vn:~# podbench pids
container web: the processes in its PID namespace
    PID  UID  TARGET  ST  THR  PTRACE  CONTAINER     COMM      CMDLINE
 >  1    0    yes     S   1    ok      87d20e23a1b4  python    python -m http.server 8080
    42   0    -       S   1    ok      7206c89bf0e1  podbench  podbench agent
> is the process `podbench dbg` attaches to with no `--pid`. 2 shown, 1 in the
target container.

> marks the pid podbench dbg picks on its own, so the usual case needs no argument at all. Rows outside the target container are dimmed — that second one is this seat’s own agent — and CMDLINE is cut to the width the other columns leave, since one long argv would otherwise be wrapped by the terminal and put the next row’s PID under this row’s PTRACE. The cut takes the middle: on a pod whose processes are all one interpreter every cmdline opens /app/.venv/bin/python …, so the end is the only part that tells one row from the next. podbench pids --json has the untruncated form.

There is no listening socket in that pod, no port-forward and no pod IP involved. ssh’s ProxyCommand is a kubectl exec running sshd -i -e on the far side; your kubeconfig is the outer authentication and your ssh key is the inner one. (-e is mandatory and is not about logging — see VS Code Remote-SSH.)

Look around the workload’s filesystem — this works even against a distroless target with no shell of its own, because you are reading it from outside through the shared PID namespace:

# ls /proc/1/root/tmp
# cat /proc/1/environ | tr '\0' '\n'

6. Connect VS Code#

In VS Code, run Remote-SSH: Connect to Host… and choose the same alias. On first connect the server downloads and extracts itself into the container (about 2 s to download, 6 s to extract, ~680 MiB on disk).

Warning

A real VS Code GUI client has now connected — and the numbers still have not been taken. On 2026-08-17 a Remote-SSH client reached a seat, started an extension host, unpacked ms-vscode.cpptools and drove gdb through the C++ adapter into a live IOC. The transport was verified at the protocol level besides, and the server was driven headlessly, so the memory figures in these docs are lower bounds — no extension host or language server has been measured. Expect the connection to work and the footprint to be larger than quoted.

Open the seat’s home in the remote window — /root, or /home/podbench on a podbench-home volume — and you are editing inside the cluster. Do not open /: it points the file watcher and the search indexer at /proc, where the walk has no bottom. See VS Code Remote-SSH for sizing, extensions and the settings that matter.

7. Look at what you have running#

$ uvx podbench status pod/web-6c9d7f4b8b-hq2vn -n podbench-demo
$ uvx podbench list -n podbench-demo

status lists every podbench container in a pod, including dead ones whose names are burnt, and re-runs the probe in each running one so that the verdict beside a seat is measured today rather than inferred from the rung it landed on a week ago. list does the same across the namespace, minus the probe — every verdict there reads not probed, because a fleet listing execs into nothing. Each pod’s block ends with the ssh line to reconnect with, taken from the stanza attach wrote, so the alias survives the attach output scrolling away.

8. Clean up#

An ephemeral container dies with its pod, which is the only way to remove one:

$ kubectl delete namespace podbench-demo

Also drop the generated stanza if you want a tidy config directory:

$ rm ~/.podbench/config.d/podbench-demo-web-6c9d7f4b8b-hq2vn-1.conf

Where next#

  • Attach to a pod — reconnecting, restricted namespaces, making memory headroom, and what the failures look like.

  • Debug with gdb — a distroless target to a breakpoint with source, in under ten minutes.

  • Iterate on Python — the mode that has its own resource limits, and where you should do anything heavier than looking.