bastion-agent/docs/catalog.md

# bastion agent `catalog` stage: the curated dataset

The catalog is the small, structured thing a host reasons over and the thing
`seal` ships: one card per file and one per directory, built only from the
sanitized text. It is `bastion_agent/catalog.py`, standard library only.

`python3 -m bastion_agent catalog <manifest> [--out DIR] [--require-signed]`
needs the walk, extract and sanitize outputs, and reads `chunks.jsonl` if it
is there. It writes:

- **`catalog.jsonl`**: one card per file the walk found, sorted by
  `(root, path)`.
- **`catalog.dirs.jsonl`**: one card per directory that holds files (every
  ancestor of every file; the root is `""`), sorted the same way.
- **`catalog.summary.json`**: counts, the summarizer that ran (or `null`),
  `stopped`, and `bytes_catalog` (the length of all the cards, in
  characters).

## The file card

| Field | From | Notes |
|---|---|---|
| `root`, `path`, `name`, `kind`, `size`, `mtime`, `mime`, `sha256` | walk | `path` and `name` after the name mask (below). |
| `text` | extract | `ok`, `empty`, `unsupported`, `error`, `skipped` or `not_extracted`. |
| `extractor` | extract | Which reader, e.g. `pdftotext`, `pdf-builtin`, `docx`, `text:utf-8`. |
| `sanitized` | sanitize | `ok`, `skipped`, `error`, `not_run`, or `null` when the sanitize stage has no record for it. |
| `title` | **sanitized text** | The first line of at least three characters, list and heading markers stripped, at most 120 characters. |
| `head` | **sanitized text** | The first 300 characters, whitespace collapsed. |
| `chars` | sanitized text | Length. |
| `keywords` | **sanitized text** | The top 8 by TF-IDF across the catalog: words of three or more letters, stopwords and placeholders removed, ties broken alphabetically (so the result is deterministic). |
| `entities` | sanitize | Token counts per type minted for this file, e.g. `{"EMAIL": 2, "PERSON": 1}`. Counts, never values. |
| `chunks` | chunk | How many windows the document produced (0 without the chunk stage). |
| `summary` | summarizer | Present only when the manifest asks for file summaries; `null` when there was no text. |
| `v` | | The card format version. |

Only files with sanitized text get the text fields. A file without text still
gets a card: the host can see that it exists, its type and size, and why it
has no text.

## The directory card

`files`, `with_text`, `bytes`, `latest` (newest mtime), `types` (the top 6
MIME types), `keywords` (the top 12 over the merged term counts), `entities`
(summed over the files), and `summary` when asked for.

## What never enters the catalog

- **Values.** Every field derived from text comes from the sanitized store. A
  test searches the whole catalog for the fixture's personal data and fails on
  any hit.
- **Placeholders as keywords.** `[PII_*_n]` tokens are removed before terms
  are counted.
- **Anchors in names.** File and directory names are plaintext by nature, so
  every segment of every path (parent directories included) goes through the
  rules pass; an e-mail address, phone number, card or key in a name is masked
  to `[PII_TYPE]`. The masking is conservative: the e-mail rule spans dots and
  hyphens, so `invoice-m@example.com.pdf` becomes `[PII_EMAIL]`; `mime` and
  `size` still say what it was. The summary counts masked segments as
  `names_masked`. **A name like `Marisol_medical.pdf` is not caught**: that is
  contextual personal data, the same limit as the rules-only `flag` mode. The
  mask is the catalog's: the bastion's own listing, for a grant with `read`,
  returns names as they are.
- **Not hidden: content hashes.** Each file card's `sha256`, also the `doc` id in
  search hits and excerpts, is the hash of the original file, so whoever holds
  a copy of a file can confirm it is in the folder.

## The summarizer (a sensor)

A manifest section `summarize: {model, per: [file, directory]}` turns it on
(it needs `summarize` in the mask). The model is reached through
`SUMMARIZE_URL` (default `http://127.0.0.1:11434`, a local Ollama),
`SUMMARIZE_API` (`ollama` or `openai`; unset means try Ollama's `/api/chat`,
then `/v1/chat/completions`) and `SUMMARIZE_TIMEOUT` (300 s): the
environment, never the manifest.

It sees only sanitized text (the first 12,000 characters), is told that
placeholders are redactions to keep as they are, and runs at temperature 0.
Its reply is cleaned (`<think>` blocks and surrounding quotes removed,
whitespace collapsed, at most 600 characters) into the `summary` string, the
only field it can touch. A directory's summary is built from its direct
children's summaries or heads (at most 60, 200 characters each). Summaries are
cached in `catalog.cache.json` by document, model and version, so a re-run
costs nothing for unchanged documents.

The first failure stops summarizing: every card is still written, one card
carries `summary_error`, the summary says `stopped`, and the exit code is 1.

## Tests

`python3 -m unittest bastion_agent.test_catalog` covers title, head and
keywords and their determinism; name masking (e-mail, phone, plain names,
parent directories); summary cleaning; the whole stage with and without a
summarizer (card fields, entities, keywords free of placeholders, a card for
a file with no text, directory totals including the root, masked names
counted); the catalog containing none of the fixture's personal data; file and
directory summaries with the cache reused on the second run and the model
seeing only sanitized text; OpenAI-style detection; a summarizer failure
stopping after one call with every card still written; an unreachable
summarizer; and the guard that the earlier stages ran first.

The stopword list is for prose: on source code, identifiers such as `self`
and `def` count as keywords.

## Not in this stage

Sealing (the next stage: the catalog encrypted to the owner and the
manifest's `seal_to` recipients) and any rendering of the cards on the host.