Job queue and Claude features#

Queued Claude work in scrap runs on one background worker thread that takes jobs from SQLite, one at a time; the scrap enrich passes call Claude from their own process, one page at a time, under the same pause and budget. Either way what comes back is stored as derived rows beside the page, never in the owner’s text. This page shows the shape of that queue; the prompts, limits and operating detail are in Claude ingest and What Claude is asked, per page.

Colours in the diagrams: blue is stored data, orange is Claude, green is the user.

What feeds the worker and what it writes#

Most triggers end as a row in a jobs table, which the worker passes through the budget gate to Claude; the scheduled scrap enrich passes go through the same gate from their own process. Each result lands in a derived table.

        flowchart LR
    subgraph IN["Triggers"]
        photo["Photo capture<br/>or Reprocess"]:::user
        save["Note save<br/>or link capture"]:::user
        ask["Ask Claude"]:::user
        night["Nightly timer"]:::user
        week["Weekly timer"]:::user
    end
    subgraph Q["Queue (SQLite)"]
        job[("job<br/>kind ingest")]:::store
        ej[("enrich_job<br/>capture, save, ask,<br/>related, writeup, topic")]:::store
    end
    W["Worker thread<br/>one job at a time"]
    P["scrap enrich pass<br/>its own process,<br/>one page at a time"]
    G{"Budget gate<br/>and pause"}
    C(["Claude<br/>API or claude -p"]):::claude
    subgraph OUT["Derived, apart from the text"]
        d[("derived<br/>ocr, description,<br/>excalidraw")]:::store
        e[("enrichment, model_tag<br/>summary, tags")]:::store
        r[("related")]:::store
        w[("writeup<br/>boards and topics")]:::store
    end
    photo --> job
    save -->|"after debounce"| ej
    ask -->|"now"| ej
    night -->|"enrich --related"| P
    week -->|"enrich --topics"| P
    P -->|"queues topic syntheses"| ej
    job --> W
    ej --> W
    W --> G --> C
    P --> G
    C --> d & e & r & w
    job -. "read: queues<br/>write-up or related" .-> ej
    classDef store fill:#e8f1ff,stroke:#4a7bd0;
    classDef claude fill:#fff3e0,stroke:#e08a00;
    classDef user fill:#e9f7ef,stroke:#2e9d5b;

Job

Table and reason

Queued by

Calls

Result

Photo ingest

job, kind ingest

A photo capture, Reprocess, Ask Claude on one image

Vision model, one per distinct image

derived ocr, description, excalidraw; page title, tags and kind

Enrich

enrich_job capture, save, ask, backfill

A text page save or link capture, a file page’s files read (SCRAP_FILE_ENRICH=auto: the start of their text is sent), Ask Claude, scrap enrich --backfill

Text model

enrichment summary, model_tag tags, a link’s or file page’s title

Related

enrich_job related

A page’s first read, Ask Claude (the nightly pass calls Claude in its own process)

Text model, candidates from tags and search

related rows (at most five per page)

Write-up

enrich_job writeup

A board page whose photos are all read, Ask Claude (scrap enrich --writeups calls Claude in its own process)

Vision model with the board photos

writeup row of the page

Topic

enrich_job topic

The weekly topics pass

Vision model

writeup row of a topic page

The nightly related pass and the weekly topics pass are Helm CronJobs (enrichRelated, default 41 3 * * *; enrichTopics, default 11 4 * * 0) that run scrap enrich --related and --topics. Related picks run in the CronJob’s own process, under the same budget and pause, as do the write-ups of scrap enrich --writeups run by hand; topic discovery runs there too and queues one topic job per topic due, which the service’s worker runs.

How the jobs are chosen and ordered:

  • Photos first. The worker claims a photo job; only when none is ready does it claim an enrich job (Worker.step). Both claims fail while the queue is paused.

  • One queued enrich job per page. A later trigger moves the existing job (a save later, Ask now) instead of adding one. Ask and a write-up or topic reason are kept when a weaker trigger arrives.

  • Debounce. A note save sets run_after to SCRAP_ENRICH_DEBOUNCE seconds ahead (default 60), and each save moves it later, so a burst of edits is one call. A saved link runs at once.

  • Chaining. When a photo is read or goes dead, queue_after_read queues the board’s write-up or the page’s first related call. After a write-up (done or dead) the first related call follows, so it sees the write-up.

  • Jobs are per page or per photo, and never for a trashed page or the Pinned page (32. Pins are an ordinary page of links).

Job lifecycle: pause instead of fail#

A failure the owner can fix, or that is not the job’s fault, pauses the whole queue and refunds the job; any other failure is charged to the job and it dies on the third.

        stateDiagram-v2
    [*] --> queued
    queued --> running : claimed (queue not paused)
    running --> done : result stored
    running --> queued : paused, claim refunded, no failure
    running --> queued : charged failure, retry in 10 min
    running --> queued : invalid output, retry at once
    running --> dead : third charged failure
    running --> queued : crashed at start-up (charged)
    dead --> queued : Retry (photo) or Ask Claude
    done --> [*]

The queue has one pause state (queue_state), shared by photo and enrich jobs. Pausing never touches a job’s failure count.

Reason

Cause

Queue resumes

credit

Out of credit or at a spend limit

On Resume

auth

Key refused, or cli backend without token or binary at start-up

On Resume

budget

SCRAP_BUDGET_DAY_USD spent (checked before each call)

At UTC midnight, or Resume

outage

API unreachable, 5xx or rate limited

After 60 s doubling up to an hour, or the API’s retry-after (at most an hour); any success or Resume

  • Charged classes (anything else): invalid, timeout, content, image (cannot decode or too large), internal, crashed. Retry after 10 minutes, invalid at once, dead on the third. dead shows as failed with Retry.

  • One photo, not the queue. A charged failure affects only its job; a dead photo counts as read for the page’s write-up and first related call, so it does not hold them back.

  • Announce once. The first pause since the last success or Resume, or one for a new reason, POSTs one fixed line to SCRAP_NOTIFY_URL, if set.

  • Clean shutdown gives the job in flight back for free. /healthz returns 503 if ingest is on and the worker thread has died.

Calling Claude#

claude.complete() makes one call with no tools and a JSON schema for the output. It never retries; the queue does, so one claim is one call.

        flowchart LR
    J["Claimed job"] --> B{"Budget left?"}
    B -- "no" --> P["Pause: budget"]
    B -- "yes" --> M["Build request<br/>prompt, schema"]
    M --> S{"SCRAP_LLM_BACKEND"}
    S -- "api" --> A(["Messages API<br/>ANTHROPIC_API_KEY"]):::claude
    S -- "cli" --> L(["claude -p<br/>Claude seat"]):::claude
    A --> V["Validate, sanitise"]
    L --> V
    V --> R[("Run row<br/>usage and cost")]:::store
    V --> D[("Derived version")]:::store
    classDef store fill:#e8f1ff,stroke:#4a7bd0;
    classDef claude fill:#fff3e0,stroke:#e08a00;

  • Budget is the sum of recorded cost today (UTC) across photo runs, enrich runs and topic runs.

  • Backends share everything after the call. The cli backend runs claude -p once per call on a seat token (Claude ingest section On a Claude seat (SCRAP_LLM_BACKEND=cli)).

  • Model output is data, not instructions. Elements, tags and links are sanitised and checked before storing (Sanitising).

  • Every call is recorded (model_run, enrich_run, topic_run) with the request, response, usage and cost.

Results are versions, not edits#

Derived rows sit beside the page and the owner’s text is not written by Claude; the page body stays the human’s.

  • derived (per capture, photo and kind): rows with version, producer (the model id, or human) and superseded_at. A drawing (excalidraw) keeps every version: a new current one supersedes the last by trigger. A reading (ocr, description) keeps only its current row: the next one deletes it first (ADR 39). Kinds are ocr, description, summary, fetched and excalidraw. A PDF’s or text file’s text is also an ocr version, read by scrap itself with no model run (35. Files are read locally, for search).

  • writeup (per page, for boards and topics): one row with prompt_version and the owner’s guidance it was written with. Regenerate replaces it; there is no history to restore. Nobody edits a write-up.

  • enrichment, model_tag, related: one summary per page, Claude’s tags marked apart from the owner’s (the owner can turn one off), and ranked related pages. These are replaced by the next reading and are not in the revisions list or revertible.

  • A human drawing wins. Reprocess keeps a human excalidraw version current unless the owner confirms Replace mine; the new drawing is then in the history only.

  • The page’s own fields are changed only where nothing of the owner’s exists: a photo page’s title (while still the original), tags (up to the limit, the owner’s first) and a note kind that is not fixed. From a list-of-actions photo, task boxes are appended as a Claude-authored revision, only if the text is unchanged (26. Claude appends task boxes to a list of actions).

Drawings as skeletons#

For a board or diagram the photo call also returns Excalidraw skeleton elements in the same call (there is no redraw job). They are sanitised, then stored as the excalidraw derived kind with format: skeleton and the sent image size, and the PWA converts them to a scene to view and edit (11. Whiteboards become Excalidraw drawings: model skeletons, human scenes, two sanitisers).

Invariants#

  • Queued Claude work runs on one worker thread, one job at a time; the scrap enrich passes call Claude from their own process, one page at a time, under the same pause and budget. claude.complete() never retries.

  • A pause never costs a job a failure; a charged failure dies on the third.

  • Photo, text and topic spend share one daily budget.

  • Claude’s output is stored apart from the owner’s text and can be regenerated.

  • A page in the trash is not claimed.

Settings and decisions#

Settings are in the configuration table: SCRAP_INGEST, SCRAP_ENRICH, SCRAP_FILE_ENRICH, SCRAP_ENRICH_DEBOUNCE, SCRAP_BUDGET_DAY_USD, SCRAP_VISION_MODEL, SCRAP_TEXT_MODEL, SCRAP_TEXT_EFFORT, SCRAP_LLM_BACKEND, SCRAP_NOTIFY_URL. Queue state is GET /api/ingest and POST /api/ingest/resume.

ADR

Decision

0009

Photo ingest through the Messages API

0010

The queue pauses instead of failing

0011

Drawings as Excalidraw skeletons

0020

Text enrichment and related pages

0021

Ingest on a Claude seat through claude -p

0026

Claude appends task boxes to a list of actions

0027

Board write-ups apart from the text

0028

Claude-maintained topic pages

0038

File text is sent to Claude on capture, bounded

More detail: Enrich: text pages and links (text enrichment), Related pages, Write-ups of boards, Topic pages, Spend and outage control and Troubleshooting in Claude ingest; what is asked per page in What Claude is asked, per page. Neighbouring pages: Data model and storage, Capture, files and backup and the architecture overview.