Debug with gdb#

Take a distroless container — no shell, no gdb, no libc headers, nothing to exec into — and land on a breakpoint with source shown and locals readable. Ten minutes, most of it image pulls.

Note

Commands here are written podbench <verb> — the only spelling there is. If you have not installed the launcher, run each as uvx podbench <verb>. See Setup.

Warning

A breakpoint on a probed pod is on a timer. A process stopped in a debugger does not answer its probes, and the kubelet cannot tell that from a hang. Two deadlines follow, and the quiet one is the one that will catch you out. Both are (failureThreshold - 1) x max(periodSeconds, timeoutSeconds) + timeoutSeconds after the pause begins, plus up to one more period depending on where in the cycle it began:

deadline

what happens

how visible

readinessProbe

the pod goes not-ready and stops taking Service traffic — the address stays in the EndpointSlice with conditions.ready: false

quiet — nothing restarts, and it recovers a probe period after you continue, so afterwards nothing points at the debugger

livenessProbe

the container is killed and restarted, and the seat — which shares its namespaces — is killed with it

loud — an event, a bumped restart count, and a burnt seat name: an ephemeral container cannot be restarted, so coming back needs attach --new

Measured against tests/e2e/apps/python-service.yaml (readiness every 5 s, liveness every 10 s, both failureThreshold: 3, both timeoutSeconds: 1, so 11–16 s and 21–31 s): a gdb attach held 18 s took the pod out of the Service after ~12 s and put it back 5 s after detach, with no restart. Held 45 s, it also produced Container app failed liveness probe, will be restarted at ~25 s and exitCode: 137 on both the workload and the seat.

podbench attach prints these numbers for the pod you name, computed from its own spec — including the opposite answer, “no probes, no deadline”, when the target has none.

Probes cannot be turned off on a running pod. A pod update may change only containers[*].image, initContainers[*].image, activeDeadlineSeconds, tolerations and terminationGracePeriodSeconds; unlike resources — the asymmetry that makes --resize possible — probes have no resize-style subresource. So live attach on a probed pod is a short-visit tool: break, look, continue. Logpoints and conditional breakpoints never stop the process. For an unlimited pause use podbench dev, which strips all three probes by construction.

Everything here is driven by podbench dbg, run from a terminal in the seat. podbench dbg is not gdb -p: it fixes seven commands in one order, and the order is a correctness property rather than a preference. Setting the sysroot after attaching gives you a backtrace that looks entirely believable and is wrong.

1. A distroless target#

An initContainer compiles the program into an emptyDir; the distroless container runs it. No local build, no registry.

$ kubectl create namespace podbench-gdb
$ kubectl -n podbench-gdb apply -f - <<'YAML'
apiVersion: v1
kind: ConfigMap
metadata:
  name: victim-src
data:
  victim.c: |
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <unistd.h>
    #include <math.h>

    struct work { int id; double value; char label[32]; };

    static double transform(struct work *w) {
      double r = sqrt((double)w->id) * w->value;
      return r;
    }
    static int compute(int n, struct work *w) {
      w->id = n;
      w->value = (double)n * 1.5;
      snprintf(w->label, sizeof(w->label), "item-%d", n);
      double t = transform(w);
      return (int)t;
    }
    static void outer_loop(void) {
      struct work w; int i = 0;
      for (;;) {
        memset(&w, 0, sizeof(w));
        int v = compute(i, &w);
        printf("tick %d -> %d (%s)\n", i, v, w.label); fflush(stdout);
        i++; sleep(2);
      }
    }
    int main(int argc, char **argv) {
      printf("victim starting, pid=%d\n", (int)getpid()); fflush(stdout);
      outer_loop(); return 0;
    }
---
apiVersion: v1
kind: Pod
metadata:
  name: victim
  labels: {app: victim}
spec:
  restartPolicy: Never
  volumes:
    - {name: app, emptyDir: {}}
    - {name: src, configMap: {name: victim-src}}
  initContainers:
    - name: build
      image: debian:bookworm-slim
      command: ["/bin/sh", "-c"]
      args:
        - |
          set -ex
          apt-get update -qq
          apt-get install -y -qq --no-install-recommends gcc libc6-dev
          mkdir -p /app/src
          cp /src/victim.c /app/src/victim.c
          cd /app/src && gcc -g -O0 -o /app/victim victim.c -lm
      volumeMounts:
        - {name: app, mountPath: /app}
        - {name: src, mountPath: /src}
  containers:
    - name: victim
      image: gcr.io/distroless/cc-debian12
      command: ["/app/victim"]
      volumeMounts: [{name: app, mountPath: /app}]
YAML
$ kubectl -n podbench-gdb wait --for=condition=Ready pod/victim --timeout=300s

Confirm there is genuinely no shell in there:

$ kubectl -n podbench-gdb exec victim -c victim -- /bin/sh -c 'echo hi'
error: Internal error occurred: error executing command in container:
failed to exec in container: ... exec: "/bin/sh": stat /bin/sh: no such file or directory

Note the source is left at /app/src/victim.c inside the target’s rootfs. That is not incidental — see Where source text actually comes from below.

2. Attach the seat#

$ podbench attach pod/victim -n podbench-gdb

Check the report says [x] live attach. If it does not, skip to Without SYS_PTRACE below — you still get source-level debugging, just not of the already-running process.

Then ssh in with the alias it printed.

3. Find the process#

$ ssh podbench-podbench-gdb-victim-1
root@victim:~# podbench pids
container victim: the processes in its PID namespace
PID  UID  TARGET  ST  THR  PTRACE  CONTAINER      COMM    CMDLINE
1    0    yes     S   3    ok      87d20e23a1b4   victim  /app/victim
38   0    -       S   1    ok      7206c89bf0e1   podbench  podbench agent

podbench pids is not ps. Under a shared PID namespace every process in the pod is visible — including other podbench sessions’ — so attribution keys off the target’s container runtime ID, which the launcher injected as PODBENCH_TARGET_CID. The rules that look obvious are all wrong: “the target is PID 1” breaks under shareProcessNamespace: true (PID 1 is /pause), and matching mount namespaces breaks there too.

If the TARGET column is a guess rather than a fact, podbench pids says so.

ST, THR and PTRACE are the columns to read when you disagree with the pid dbg or debug-config chose, because they are what it chose on. ST is the kernel’s process state: a Z there is a zombie, which no seat can attach to however good its credentials — it is the entry you should never pass as podbench dbg <pid>. PTRACE says whether this seat may read that process at all; a DENIED beside a live process is usually a uid the seat does not share (see Attach to a pod for the rung that fixes it). THR is the thread count, which is what separates a workload from the short-lived helper it forks.

4. Attach gdb#

root@victim:~# podbench dbg 1

With no argument at all, podbench dbg discovers the pid from the target container ID. Before starting gdb it runs the capability probe, so if attach is going to be denied you are told which mechanism denies it rather than being handed an EPERM.

What it feeds gdb, in this order — see it without starting gdb using podbench dbg --dry-run 1:

set pagination off
handle SIGURG nostop noprint pass
set sysroot /proc/1/root
directory /proc/1/root
add-auto-load-safe-path /proc/1/root
set debuginfod enabled on
file /proc/1/root/app/victim
attach 1

Every line earns its place:

Command

Why

handle SIGURG nostop noprint pass before attach

This pins gdb’s own default rather than repairing anything: the image’s own gdb 13.1, with no startup commands and no attach, already reports SIGURG No No Yes — as it does for SIGCHLD and SIGWINCH, so those three words are gdb’s class for a routine signal rather than a decision about this one. It is there so a gdb configured otherwise cannot cost you the session — Go preempts goroutines by sending SIGURG at up to a few hundred a second, and a gdb that stops and announces each one does not make an attached Go session slow, it makes it unusable, with nothing in the wall of Program received signal SIGURG saying why. pass, not ignore: the signal is still delivered, because swallowing it changes the runtime’s scheduling. Issued for every target, since podbench dbg attaches to whatever pid it is handed

set sysroot /proc/<pid>/root before attach

gdb resolves the target’s loader and shared libraries, not the debug image’s. gdb 13’s default sysroot is target:, which needs CAP_SYS_ADMIN and fails loudly without it

directory /proc/<pid>/root

sysroot does not cover source lookup. This is what turns victim.c: No such file or directory into real source text

add-auto-load-safe-path /proc/<pid>/root

setting a sysroot makes gdb decline to auto-load the target’s libthread_db.so.1 — no info threads, no per-thread backtraces. Narrow, never set auto-load safe-path /

set debuginfod enabled on

symbols for stripped binaries and system libraries. Symbols only — see below. A library’s are fetched after the attach, with the target stopped, so the image bounds each fetch at DEBUGINFOD_TIMEOUT=2 and --no-debuginfod turns the whole thing off for one run

file /proc/<pid>/root$(readlink /proc/<pid>/exe) before attach

this is what recovers the user frames. A trailing (deleted) is stripped

source /opt/podbench/gdb/rust_printers.py

Rust targets only. A Rust binary asks for gdb_load_rust_pretty_printers.py, which ships with a rustup toolchain that a production container does not have; without it Vec, String and Option print as the RawVecInner/Unique/NonNull nest they are made of. Sourced only where the binary was identified as Rust, because source of a missing path is an error on every other attach

When that file line names /tmp instead#

gdb canonicalises the exec file’s name before BFD opens it, and the kernel resolves /proc/<pid>/root to /. The sysroot prefix is therefore erased for this one command, and gdb reads whatever this seat keeps at the target’s own path. Where the two files differ you get

BFD: /python/…/python3.11: .gnu.version_r invalid entry
warning: Can't read symbols from /python/…/python3.11: bad value
Error reading attached process's symbol file.

— a complaint that names the target’s path while describing the seat’s file — and where they differ only slightly, no complaint at all and symbols from the wrong build. It is not rare: podbench’s image and any uv-managed workload both install an interpreter at /python/cpython-<version>-<triple>/, so a Python target collides by construction (issue #90).

So podbench dbg checks whether this container has a file at the target’s exe path, and if it does, copies the target’s binary to /tmp/podbench-exe/<pid>/ and points file there. It says so in one line, and --dry-run prints the command it will actually run. Nothing else in the sequence changes — shared libraries are resolved by a different path in gdb and stay sysrooted. podbench dbg <pid> --print-exec-file prints just that path. The image’s gdb-podbench wrapper asks for more than the path — podbench dbg <pid> --print-startup-commands prints every line above except the attach, and the wrapper passes each as -iex before a third-party gdb --pid <n> attaches, so gdb -p, cpptools and debugpy all get the sysroot, the staged exec file, the auto-load path and the SIGURG handling that podbench dbg gets.

The staged copy is gdb’s fix, and lldb has no equivalent#

lldb has the same bug and cannot be given the same cure, which was measured rather than assumed. gdb canonicalises the path you hand it, so a path nothing shadows is enough. lldb ignores the path you hand it: once it has attached it re-resolves the executable from the process and overrides the target with the name as resolved in the seat’s own mount namespace —

warning: Executable binary changed from "/tmp/podbench-exe/1/victim" to "/app/victim".
Executable binary set to "/app/victim".

— so the staged copy is loaded and then dropped. settings set target.exec-search-paths does not help either; moving the seat’s own file aside does, which is the same mechanism as gdb’s under a remedy nobody has in a pod. It is at least loud, unlike gdb’s second failure mode, but the warning lands in the debug console and the session carries the wrong symbols regardless.

So debug-config emits no lldb entry at all where this seat has a file at the target’s exe path, and says which file refused it. The gdb entry beside it is unaffected — that one gets the staged copy — and so is podbench dbg.

Measured 2026-08-19 with a standalone lldb 21.1.8 on a test bed, against a real mount namespace built with podman - not inside a podbench seat. CodeLLDB’s own bundled lldb, running in a remote extension host, has not been observed doing this; the refusal assumes it behaves as the lldb it ships. Untried, and the shape that would actually be correct: platform select remote-linux against an lldb-server started inside the target’s namespace, which never resolves anything in the seat’s.

5. Breakpoint, source, step#

victim declares no probes, so this pause is unlimited and you can take as long over it as you like — podbench attach said as much under supports. On a pod that does carry probes, read the timer warning at the top of this page before you break anywhere.

(gdb) break compute
Breakpoint 1 at 0x575fbad351f4: file victim.c, line 19.
(gdb) continue
Continuing.

Breakpoint 1, compute (n=23, w=0x7ffc633f7600) at victim.c:19
19	      w->id = n;
(gdb) bt
#0  compute (n=23, w=0x7ffc633f7600) at victim.c:19
#1  0x0000575fbad35297 in outer_loop () at victim.c:31
#2  0x0000575fbad3531b in main (argc=1, argv=0x7ffc633f7778) at victim.c:42
(gdb) next
20	      w->value = (double)n * 1.5;
(gdb) print *w
$1 = {id = 23, value = 0, label = '\000' <repeats 31 times>}
(gdb) info source
Current source file is victim.c
Compilation directory is /app/src
Located in /proc/1/root/app/src/victim.c

That last block is the check that matters. Located in /proc/1/root/app/src/victim.c is a single, clean sysroot prefix — the field a DAP client hands to the editor.

Detach and leave the workload running:

(gdb) detach
(gdb) quit

Where source text actually comes from#

Be clear-eyed about this: debuginfod delivers symbols, not sources, on Debian and Ubuntu targets. Both halves of that were measured.

  • Symbols work, and they are cheap: a fully symbolised, source-line-annotated backtrace across coreutils and glibc for a stripped binary cost 4.7 MB of ~/.cache/debuginfod_client, and it works through the sysroot including for distroless libraries.

  • Sources fail, twice over, and both causes are fatal on their own:

    Download failed: Invalid argument.  Continuing without source file ./src/sleep.c.
142	src/sleep.c: Inappropriate ioctl for device.

    Debian’s -dbgsym packages carry DW_AT_comp_dir : . from reproducible-builds normalisation, and the debuginfod protocol requires an absolute path, so gdb’s ./src/sleep.c is rejected client-side with EINVAL. And the server has no sources anyway: /buildid/<id>/debuginfo returns 200, /buildid/<id>/source/src/sleep.c returns 404.

Fedora/RHEL debuginfod is known to serve sources. Debian and Ubuntu targets will not. So source text has to come from one of these, and you have to pick one deliberately:

  1. Source in the target image — what the demo above does. Ship /app/src in the workload image (or on a volume) and directory /proc/<pid>/root finds it with no path mapping at all. Simplest, and the only option that needs nothing on the client.

  2. A checkout in the debug container. Clone the source into the seat and point podbench dbg at it:

    root@victim:~# git clone https://github.com/you/app /workspace/src
root@victim:~# podbench dbg --source-dir /workspace/src 1

    --source-dir is repeatable and is wired with gdb’s directory. gdb searches the most recently added directory first, so your checkout wins over the target’s rootfs.

  3. Client-side mapping to a checkout on your laptop, via sourceFileMap in launch.json — see below. This is the only one that keeps a full clone off the pod, and it is also the least proven.

None of these is a finished design. Source provisioning for Observe mode is an open problem, not a solved one.

VS Code attach templates#

These go in .vscode/launch.json in the remote window — the debug adapter runs inside the debug container, next to gdb, so every path below is a path in that container.

Let podbench write it#

Do not hand-copy the templates below unless you have to. In the seat:

root@victim:~# podbench debug-config
debug-config: native target, observe mode, x86_64
debug-config: emitting gdb: native target, observe mode
debug-config: emitting lldb: native target; CodeLLDB brings its own lldb to the seat
debug-config written to /root/.vscode/launch.json
  Run and Debug -> "podbench: attach to victim [pid 1 victim] (gdb)"
  Run and Debug -> "podbench: attach to victim [pid 1 victim] (lldb)"

The pid and the kernel’s name for the process are in every entry name because up to five candidates are offered at once, and an entrypoint script’s children are routinely three processes all called python. Anything the ranking left out is named in a debug-config: line, and podbench pids lists the rest.

It fills in the pid, the sysroot-prefixed program, the setup ordering, the architecture, miDebuggerPath and the mode’s path mappings from what it can already see, which is the whole point: every one of those fails silently when wrong.

Every flavour that applies is emitted, named for its debugger, because launch.json holds a list and VS Code’s dropdown is a better chooser than a guess. A flavour that cannot be emitted gets a sentence naming the mechanism instead — --flavour gdb|lldb|delve|debugpy asks for one by name and makes it say why if it cannot. --print-config emits instead of writing, --output puts it beside the folder you actually opened, and re-running replaces its own entries and leaves any hand-written configuration alone.

For a Python target the answer is debugpy rather than gdb, and it depends on the architecture as well as the language — the CLI reference has the three axes and what each one changes. A stock Python image has no debugpy for the injection to bootstrap with, and debug-config then prints the uv pip install that puts one inside the target; --provision runs it, at the cost of ~15 MB of the pod’s ephemeral storage, egress from the pod, and nothing that survives a restart.

VS Code reads .vscode/launch.json from the folder that is open, not from $HOME. A config written to ~ when you opened / never appears in the Run and Debug list, and nothing reports an error.

C/C++ with ms-vscode.cpptools (gdb)#

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "podbench: attach to the workload",
      "type": "cppdbg",
      "request": "attach",
      "processId": "1",
      "program": "/proc/1/root/app/victim",
      "cwd": "/root",
      "MIMode": "gdb",
      "miDebuggerPath": "/usr/local/bin/gdb-podbench",
      "targetArchitecture": "arm64",
      "setupCommands": [
        { "text": "set sysroot /proc/1/root" },
        { "text": "directory /proc/1/root" },
        { "text": "add-auto-load-safe-path /proc/1/root" },
        { "text": "set debuginfod enabled on" }
      ],
      "sourceFileMap": {
        "/app/src": "/proc/1/root/app/src"
      }
    }
  ]
}

Four things are load-bearing.

program must be the sysroot-prefixed path, or gdb reads the debug image’s idea of the binary. setupCommands run before the attach, which is the ordering the CLI sequence depends on — do not move the sysroot line into a post-attach hook.

miDebuggerPath must be /usr/local/bin/gdb-podbench, the image’s wrapper, and never /usr/bin/gdb. cpptools launches gdb as a child, so gdb inherits cpptools’ own working directory — its extension directory, which VS Code replaces wholesale on extension update. gdb links libpython; -enable-pretty-printing (which cpptools always sends) initialises CPython; CPython calls getcwd(); and gdb dies during startup:

gdb: warning: error finding working directory: No such file or directory
Fatal signal:
A fatal error internal to GDB has been detected, further
debugging is not possible.  GDB will now terminate.

No signal name, no backtrace — it crashes before it can format either — and VS Code surfaces only ERROR: Unable to start debugging. GDB exited unexpectedly, which points at the attach rather than at startup. podbench dbg never hits this because it never enables pretty-printing, so the CLI works perfectly on a seat where the VS Code debugger cannot start at all. Reproduce it in any seat with:

mkdir -p /tmp/gone && cd /tmp/gone && rmdir /tmp/gone
printf -- "-enable-pretty-printing\n-gdb-exit\n" | gdb --interpreter=mi

cwd must be set, and must be a directory this seat can enter. On a developer’s machine ${workspaceFolder} always exists so nobody sets it; in a seat it can resolve to nothing, and the result is that same unformattable crash. debug-config fills it in from the seat’s own $HOME: /root above, because that example is a root seat, but /tmp/podbench-home on a seat pinned to the target’s uid — /root is mode 0700 there and belongs to an account the image has no record of, so naming it emitted a cwd the debugger could not chdir into. Where the seat has no $HOME at all it is /tmp, which is 1777 in every image. Measured inside a seat with test -d and test -x; no VS Code client has driven the adapter.

Note

targetArchitecture is worth setting on arm64 — without it cpptools logs Debuggee TargetArchitecture not detected, assuming x86_64 — but it is not what causes the crash above. It is a plausible-looking red herring that sits directly next to the real symptom in the log.

Warning

The VS Code C++ extension consumes info source’s fullname, which is exactly the field the wrong source-mapping approach corrupts, so if paths come out doubled (/proc/1/root/proc/1/root/…) that is the failure to look for. Map the compilation directory, never /.

sourceFileMap keys are the compilation directories recorded in the DWARF (info source prints them as Compilation directory); values are where those directories can be read from in the debug container.

Rust with vadimcn.vscode-lldb (CodeLLDB)#

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "podbench: attach to the Rust workload",
      "type": "lldb",
      "request": "attach",
      "pid": "1",
      "program": "/proc/1/root/app/myapp",
      "sourceMap": {
        "/app/src": "/proc/1/root/app/src",
        "/rustc/<commit-hash>": "/workspace/rust-src/library"
      },
      "initCommands": [
        "settings set target.exec-search-paths /proc/1/root/usr/lib /proc/1/root/lib"
      ]
    }
  ]
}

CodeLLDB spells the key sourceMap, not sourceFileMap. lldb has no exact analogue of gdb’s set sysroot for /proc/<pid>/root, so the executable path must be sysroot-prefixed explicitly and the library search paths set by hand. The /rustc/<commit-hash> key is what rustc bakes into standard-library debug info; rustup component add rust-src provides the right-hand side.

That program holds only while this container has no file of its own at /app/myapp. Where it does, lldb discards the path after attaching and reads the seat’s copy — “The staged copy is gdb’s fix, and lldb has no equivalent”, under section 4 above, has the measurement and the warning it prints. debug-config withdraws the entry in that case rather than writing one, so a template copied by hand is the only way to get it back, and it will be wrong.

The wrong-sysroot failure, verbatim#

This is the most dangerous failure in this whole workflow, because it does not look like a failure. set sysroot / against a target whose glibc differs from the debug image’s (here Ubuntu 24.04’s 2.39 against the image’s 2.36):

### WRONG: set sysroot /
0x000077f6377dcb7a in wcsxfrm_l () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00005c459789e35f in outer_loop () at victim.c:29        <- wrong line, too
#5  0x00005c45978a0d80 in __frame_dummy_init_array_entry ()

### RIGHT: set sysroot /proc/1/root
0x000077f6377dcb7a in clock_nanosleep () from /proc/1/root/lib/x86_64-linux-gnu/libc.so.6
#3  0x00005c459789e35c in outer_loop () at victim.c:35

clock_nanosleep reported as wcsxfrm_l, interleaved frames, and even the user-code line number wrong. Against a Debian-12 distroless target the bug is invisible, because both glibcs share a build ID and the backtrace comes out correct — a matched debug image hides this rather than fixing it. podbench dbg is what makes it impossible.

Three anti-patterns#

  • CAP_SYS_ADMIN. It makes gdb’s default target: sysroot work with zero configuration, which is exactly why someone rediscovers it every year. It also breaks libthread_db (Expected absolute pathname for libpthread in the inferior, but got target:/lib/…), is container-escape-adjacent and is rejected by any restricted Pod Security Standard. podbench never asks for it.

  • set substitute-path / /proc/<pid>/root/. It functions, but gdb re-applies the substitution on display and emits /proc/1/root/proc/1/root/proc/1/root/… — which is the string a DAP client hands to your editor. Use directory.

  • set sysroot after attach. Libraries get fixed up on the fly, so it looks like it worked; the main executable does not, and the frames above libc come back as ?? ().

Without SYS_PTRACE#

Losing the capability costs exactly one thing: attach to an already running process. Full source-level debugging survives, because gdb starting the program itself needs no capability and is exempt from Yama:

root@victim:/workspace# podbench dbg --launch ./myprog --some-flag

--launch consumes the rest of the command line, so put any other podbench dbg flags before it. Add --run to start the program immediately.

podbench dbg will tell you this itself when attach is denied — it names the mechanism and points at the alternative:

podbench dbg: cannot attach to pid 1: yama-scope
  Yama's ptrace_scope forbids attaching to a non-descendant...
  verdict: read-only inspection of the target; no live attach
  the target's rootfs, maps and environ are still readable, so `podbench pids`
  and read-only inspection work.
  ptrace-free alternative: `podbench dbg --launch ./yourprog [args]`. gdb forks the
  inferior itself, which needs no capability and is not subject to Yama below
  ptrace_scope=2.
  to keep attaching to this process, the target can opt in with one line:
  prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY).

That last line is worth knowing: the natural workflow myprog & ; gdb -p $! makes gdb a sibling, which Yama denies at ptrace_scope=1. A sibling that called prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY) is attachable with zero capabilities. It is a one-line, capability-free, node-change-free opt-in for a program you control.

On a pod where the ptrace-gated reads went with the attach — the launch-only verdict — the consolation line says that instead, because offering a sysroot that will not open costs a second afternoon:

  verdict: launch-only: `podbench dbg --launch` works; no read-only inspection
  the reads that take PTRACE_MODE_READ went with it (cmdline, status and fd
  only; root, maps and environ denied), so a sysroot, `environ` or `maps` is not
  the fallback here.

The matrix is in the line because it is the honest form of the answer: cmdline, status and fd are still readable — they always are — so podbench pids works and “the target is closed” would be the same overclaim as issue #51, pointing the other way.

--launch survives all of that: gdb forks the inferior and traces its own descendant, which needs no capability and no Yama exemption. It is still measured rather than assumed, and the tick beside debug launched processes is the measurement — podbench capreport attaches to a child it forked itself before it claims the rung. Three things take it away with everything else, none of which cares whose descendant the inferior is: a seccomp filter that rejects ptrace(2) outright, ptrace_scope 2 or 3 — scope 2 is the one Yama setting with no descendant exemption, and it demands CAP_SYS_PTRACE of PTRACE_TRACEME too — and an LSM policy that denies ptrace — differing AppArmor profiles, or differing SELinux MCS categories, which capreport reports as lsm-mismatch. podbench capreport names whichever of the three it finds.

The offer is conditional for that reason: where the scratch attach was measured and failed, podbench dbg says so instead of pointing at --launch, because a second identical denial is what costs the afternoon.

Gotchas#

  • ASLR cannot be disabled. RuntimeDefault seccomp permits ptrace(2) but blocks personality(ADDR_NO_RANDOMIZE), so gdb warns Error disabling address space randomization and addresses vary run to run. Set breakpoints by symbol.

  • Attach works on one node and not the next. Yama’s ptrace_scope is a node-level knob and differs by kernel flavour, not by architecture — two arm64 nodes in the same cluster disagreed in testing. podbench probes per node and prints the node name for this reason.

  • A target already being traced refuses with the same EPERM as a policy denial, because a tracee has exactly one tracer. podbench capreport reports already-traced when it can tell.

  • Multithreaded targets are unproven. libthread_db loaded and info threads listed the single LWP in testing; a genuinely multithreaded target was never tried. Nor was a target in a user namespace.

  • elfutils/binutils are in the image — readelf --debug-dump=info and eu-readelf are how a build-ID miss gets diagnosed.

Clean up#

$ kubectl delete namespace podbench-gdb