20. Text enrichment: Claude tags, summarises and relates text pages, apart from the text#

Date: 2026-10-22 (#85, migration 9)

Status#

Accepted. Extends 9. Claude ingest: Messages API, no tools, one structured call per photo, which sent only photos to Claude, and shares the queue pause and daily budget of 10. Queue error classes: outages pause, charged failures count, spend is capped. Amended by 26. Claude appends task boxes to a list of actions: enrich may now append task boxes to a list of actions, as a revision of Claude’s own, so “never edits the text” no longer holds. Extended by 27. Claude’s write-ups of boards, apart from the text: enrich jobs of reason writeup (migration 12), and a board page’s first related call follows its write-up.

Context#

Only photo pages were read by Claude, so a note, a todo or a saved link had whatever tags its owner typed, and no summary. Finding things (see 8. Search is SQLite FTS5 with a shared query syntax; embeddings are deferred) and the tag vocabulary suffered for it. Embeddings were still deferred, so any answer to “what else is like this” had to come from tags and words. A saved link is the awkward case: its page holds only a web address, and fetching an address a user saved is a server-side request to something untrusted.

Decision#

  • A second job kind, enrich (enrich/), for notes, todos, reads and saved links. It runs on the ingest worker’s thread, whenever there is no photo job to run, so there is still no second process. It is on when ingest is on (SCRAP_INGEST) unless SCRAP_ENRICH=off.

  • One tool-less, schema-checked Messages API call per page, on a small text model (SCRAP_TEXT_MODEL, default claude-haiku-5-5; SCRAP_TEXT_EFFORT, default low). It returns tags, a summary and, for a saved link, a title. The prompt lists the tags already in use, most used first, and asks for those; the page and anything fetched are data, and the prompt says to follow nothing written in them.

  • Related pages are a call of their own. relate gives Claude a numbered index of at most 150 candidate pages, chosen by shared tags and then full-text search (enrich.CANDIDATES, swappable; not embeddings), and it answers with index numbers, so it cannot name a page that does not exist. It runs on Ask Claude (POST /api/pages/{id}/enrich) and in a nightly pass, scrap enrich --related, for each page changed since its last picks (--all: every page). Amended (#255): also once when a page is first read (its first enrich, or photo ingest of all of its first photo capture), as an enrich job of reason related (migration 11), so a new page has related pages before the night. Saves do not queue one. To roll back to a scrap before migration 11 (0.5.1 refuses schema 11), run PRAGMA user_version = 10; and DELETE FROM enrich_job WHERE reason = 'related' AND state = 'queued'; (older code would run a queued related job as a full enrich, adding tags and redoing the summary). Upgrading again is safe: migration 11 skips the rebuild when the table already takes 'related'. Related shows both ways, so an older page learns of a newer one that picked it without a call of its own.

    Superseded in part by ADR 37: this rollback no longer works. Migrations 0001 to 0018 are folded into the baseline and 1.0 refuses a store below schema 18, so user_version cannot be set back to 10.

  • Triggers. A note or link capture queues a job (a link at once, a note after SCRAP_ENRICH_DEBOUNCE seconds, default 60). A save of a page’s title or text moves its one queued job that far ahead again, so a burst of edits makes one call. A save only refreshes the summary: it adds no tags. Tag edits alone queue nothing. scrap enrich --backfill queues every untagged page Claude has not read, by hand only.

  • Links are fetched safely and only for this (fetch.py): http and https on ports 80 and 443; every address the host resolves to must be public, checked again at each redirect, and the connection goes to the address that was checked; a size limit, a 20 s deadline for the whole fetch, no cookies, credentials or proxy; only HTML and plain text. A failed fetch is not a failed job: the page is still tagged from what is known.

  • The user wins, and the text is never edited. What Claude adds is stored apart from the page (migration 9): enrichment (summary, fetched address and title), model_tag (the tags Claude added, on, and off once the user removes one, after which it never comes back), related (picks in order), with every call kept in enrich_run as model_run is for photos (the fetched text is not kept, only its length and hash). At most five tags are added, at most two of them not already in use, within a page’s 64. A link’s title replaces the provisional one only while the title is still the address as captured; the kind becomes consume only while the page is an unfixed note.

  • Failures and money follow ADR 0010. Enrich jobs share the queue pause and the daily budget (SCRAP_BUDGET_DAY_USD): spent_today sums model_run and enrich_run. Outage, credit, auth and budget pause the queue and refund the job; any other failure is charged and retried in 10 minutes (invalid output at once), dead on the third. scrap enrich --related checks the pause and the budget before each call and stops, exiting non-zero, at either.

  • On a single server deploy/scrap-enrich-related.timer runs the pass nightly in the service’s container (podman exec scrap scrap enrich --related), after the backup. install.sh enables it if the image has the command. With ingest or enrich off, or no API key, the command says so and exits 0. The Helm chart has the equivalent enrichRelated CronJob (only while ingest is on), which runs the command in its own pod beside the scrap pod with the Deployment’s environment.

  • On a Claude seat with no API key, enrichment runs through claude -p instead: see 21. Ingest on a Claude seat: claude -p with a long-lived token.

Consequences#

  • Text now leaves the machine for the Claude API (the page’s text, up to 12,000 characters, the tag vocabulary, and a fetched page’s text), where 9. Claude ingest: Messages API, no tools, one structured call per photo sent photos only. Turning it off is SCRAP_ENRICH=off, with photo ingest still on.

  • The daily budget is shared with photos: when text calls (a backfill, say) use it up, the queue pauses until UTC midnight for photos too.

  • Tags can now be set by Claude on text pages, so a page may have tags its owner never typed. They are marked in model_tag, are not revisions (19. Kinds and tags editable over MCP, made safe by metadata revisions), and MCP read returns them separately as by_model.

  • Related pages come from tags and words, not meaning; if they prove too weak, embeddings (#53) would replace enrich.CANDIDATES.

  • The nightly pass runs in a process of its own (not the worker’s thread), so it shares the worker’s queue pause and spend only through the database.

  • Migration 9 adds tables only (IF NOT EXISTS), so an older scrap still runs on a database with user_version set back to 8.

  • The design and prompts are in Claude ingest, under “Enrich”.