bastion-agent/docs/extract.md

# bastion agent `extract` stage: text out of the walked files

The extract stage turns every walked file it can read into plain text, with
built-in readers or with local tools when they are installed. It is
`bastion_agent/extract.py` and `bastion_agent/pdftext.py`, standard library
only.

## What it produces

`python3 -m bastion_agent extract <manifest.yaml> [--out DIR] [--no-external] [--require-signed]`
reads `walk.jsonl` and `walk.summary.json` from `DIR` (the walk must have run;
the stage refuses otherwise) and writes:

- **`extract/<aa>/<sha256>.txt`**: the text store, one file per distinct
  content hash, UTF-8, normalised (CRLF and lone CR become LF, NUL removed,
  NFC). Content addressing is the cache: two copies of a document are
  extracted once, and an unchanged file costs nothing on the next sweep.
- **`extract.jsonl`**: one record per walked file, sorted by `(root, path)`:
  `sha256`, `mime`, `status`, `extractor`, `chars`, `error`, `v` (the
  extractor version), `reused` when a prior verdict was kept, and on
  `unsupported` records the `tools` that were present, so the verdict is
  retried when a tool appears.
- **`extract.summary.json`**: counts, total characters, the tools found, the
  extractor version, the manifest digest, and the manifest digest the walk
  recorded, which ties the walk map and the text store to one manifest.

`DIR` defaults as for the walk (`<root>/.agent/` for one root). This text
store is the plaintext of every file in scope, and it never leaves the machine;
see [runtime.md](runtime.md#where-an-agent-lives-rootagent).

| `status` | Meaning |
|---|---|
| `ok` | Text written to the store. |
| `empty` | The reader ran and found no text (an empty file, a document with only pictures). |
| `unsupported` | No reader for this type on this machine; `error` names what would handle it. |
| `error` | The reader failed (a corrupt zip, bad XML, a tool failure); the stage continues. |
| `skipped` | The walk has no hash for it (too large, a read error, or a `--no-hash` walk). |

A record whose earlier verdict was kept carries `reused: true` and is counted
under `reused` in the summary.

Exit code: `0` clean; `1` the manifest is invalid, `extract` is not in its
stages, or there is no walk map; `2` completed with `error` records.

## Readers

Built in, no libraries:

| Type | How |
|---|---|
| `text/*`, JSON, YAML, TOML, XML, SQL, SVG, PEM, code | Decoded, and the encoding recorded (`text:utf-8`, `text:cp1252`, …): a byte-order mark first, then BOM-less UTF-16 by the share of NUL bytes, then strict UTF-8, then cp1252. |
| `text/html` | `html.parser`: scripts, styles, `noscript` and `template` dropped; block tags become line breaks; entities resolved. |
| CSV / TSV | `csv` module; each row written as tab-separated values. |
| `message/rfc822` | `email` (modern policy): From/To/Cc/Date/Subject lines, then the plain body, or the HTML body through the HTML reader. |
| RTF | A small reader of its own: control words, `\'hh` (cp1252), `\uN` (one fallback character skipped), and the groups that hold no body text (`fonttbl`, `colortbl`, `stylesheet`, `info`, `pict`, `object`, `header`, `footer`, and others) discarded. |
| DOCX | Paragraphs of `word/document.xml` (runs, tabs, breaks, text boxes after the paragraph that holds them), plus headers and footers. |
| XLSX | Shared strings, inline strings and values, one tab-separated line per row, a `# sheetN` line per sheet; empty rows dropped. |
| PPTX | Every paragraph on every slide, slides in numeric order, a `# slideN` line per slide. |
| ODT / ODS / ODP | `content.xml`: paragraphs and headings; table rows tab-separated. |
| PDF | **`pdftotext -layout -enc UTF-8` when poppler is on PATH.** Otherwise the built-in reader (below). |
| Images (PNG, JPEG, TIFF, BMP, GIF, WebP) | **`tesseract` when it is on PATH**; otherwise `unsupported`. |

Anything else (archives, executables, audio, video, unknown binaries) is
`unsupported`, with the MIME type named.

### The built-in PDF reader

Without poppler (on Windows, usually) PDFs are read by `pdftext.py`, which
walks the document far enough for text: the object table (a later definition
wins, so incremental updates are respected) and compressed object streams;
FlateDecode streams (a stream with any other filter is skipped); the page tree
in order with inherited resources; fonts through their `/ToUnicode` maps
(Type0 / Identity-H fonts, as browsers print them), or WinAnsi / MacRoman for
simple fonts without one; glyph widths, so a word placed by position still
gets its space; the text operators, and form XObjects. Pages are separated by
a form feed.

Encrypted files are refused. Output below 90% printable characters is refused
too, so a font without a usable map comes back as `unsupported` (`extractor:
pdf-builtin`, `needs pdftotext (poppler) on this machine`), never as byte salad
in the catalog.

It reads arbitrary files, so it reads on a budget per document: 60 s of wall
time, 256 MiB of decoded streams (64 MiB per stream), 5 million content
operators, 2,000 form invocations at most 6 deep, 200,000 CMap entries,
20 million tokens and 1 GiB lexed. Reaching a limit stops the read with the
pages read so far. Nothing in it raises out to the stage.

## Rules the stage enforces

- **External tools are sensors, not dependencies.** They are looked up on
  PATH at run time, named in every record they produced (`extractor:
  pdftotext`), and their absence is a recorded gap, never a failure.
  `--no-external` (in a sweep, `FREEAGENT_NO_EXTERNAL=1`) uses the built-in
  readers only. A manifest cannot name a tool or an address.
- **Nothing is written outside the output directory**, and every file is
  written atomically.
- **Bounded reads.** One zip member is read to at most 64 MiB decompressed
  (`ZIP_MEMBER_CAP`), so a hostile office file cannot balloon memory. Office
  documents are parsed as a stream, each finished paragraph or row dropped
  once read, and capped per document: 256 MiB of XML, 16 Mi characters of
  text, 60 s, and 200,000 parts in one paragraph or row. Reaching a cap keeps
  the text read so far and ends it with a line naming the cap. XML with a DTD
  or an entity declaration is refused (Office files never carry one, and it is
  how entity expansion gets in), as is XML that is neither UTF-8 nor UTF-16.
  Tool calls time out at 300 s.
- **Errors are entries.** A corrupt document is one `error` record; the stage
  finishes and reports the count.
- **Retry only when something changed.** An `unsupported` verdict is kept
  while the machine's tool set is the same; install `pdftotext` and the next
  sweep tries exactly those files again. `error` records are retried on every
  sweep. Every record carries the extractor version (`v`, `EXTRACT_VERSION`,
  now 2); raising it after a reader changes redoes every cached verdict and
  text on the next sweep.

## Tests

`python3 -m unittest bastion_agent.test_extract bastion_agent.test_pdftext`.
`test_extract` runs every reader against a generated fixture (docx, xlsx,
pptx and odt built with `zipfile`, plain and Flate PDFs, RTF with hex and
Unicode escapes, plain and HTML e-mail, encodings), the Office streaming caps,
DTD refusal and UTF-16 input, the zip member cap, an oversized CSV field, the
whole stage over a mixed tree (statuses, counts, duplicates), reuse across
sweeps and its invalidation when content or the extractor version changes,
tool discovery and failure (mocked), and the walk-first guard. `test_pdftext`
builds browser-shaped PDFs with a small writer (Type0 fonts with ToUnicode
maps, per-run positioning, ligatures, inherited resources, form XObjects,
object streams) and hostile ones (reference cycles, deep nesting, malformed
tables and maps), and checks each budget.

The external-tool paths are tested with mocks; a real scanned PDF or image
through `pdftotext` or `tesseract` is not part of the suite.

## Not in this stage

Sanitizing (the next stage: the text store goes through the PII sterilizer),
chunking, and change detection beyond the content hash. Audio and video are not
extracted.