# bastion agent `extract` stage: text out of the walked files The extract stage turns every walked file it can read into plain text, with built-in readers or with local tools when they are installed. It is `bastion_agent/extract.py` and `bastion_agent/pdftext.py`, standard library only. ## What it produces `python3 -m bastion_agent extract [--out DIR] [--no-external] [--require-signed]` reads `walk.jsonl` and `walk.summary.json` from `DIR` (the walk must have run; the stage refuses otherwise) and writes: - **`extract//.txt`**: the text store, one file per distinct content hash, UTF-8, normalised (CRLF and lone CR become LF, NUL removed, NFC). Content addressing is the cache: two copies of a document are extracted once, and an unchanged file costs nothing on the next sweep. - **`extract.jsonl`**: one record per walked file, sorted by `(root, path)`: `sha256`, `mime`, `status`, `extractor`, `chars`, `error`, `v` (the extractor version), `reused` when a prior verdict was kept, and on `unsupported` records the `tools` that were present, so the verdict is retried when a tool appears. - **`extract.summary.json`**: counts, total characters, the tools found, the extractor version, the manifest digest, and the manifest digest the walk recorded, which ties the walk map and the text store to one manifest. `DIR` defaults as for the walk (`/.agent/` for one root). This text store is the plaintext of every file in scope, and it never leaves the machine; see [runtime.md](runtime.md#where-an-agent-lives-rootagent). | `status` | Meaning | |---|---| | `ok` | Text written to the store. | | `empty` | The reader ran and found no text (an empty file, a document with only pictures). | | `unsupported` | No reader for this type on this machine; `error` names what would handle it. | | `error` | The reader failed (a corrupt zip, bad XML, a tool failure); the stage continues. | | `skipped` | The walk has no hash for it (too large, a read error, or a `--no-hash` walk). | A record whose earlier verdict was kept carries `reused: true` and is counted under `reused` in the summary. Exit code: `0` clean; `1` the manifest is invalid, `extract` is not in its stages, or there is no walk map; `2` completed with `error` records. ## Readers Built in, no libraries: | Type | How | |---|---| | `text/*`, JSON, YAML, TOML, XML, SQL, SVG, PEM, code | Decoded, and the encoding recorded (`text:utf-8`, `text:cp1252`, …): a byte-order mark first, then BOM-less UTF-16 by the share of NUL bytes, then strict UTF-8, then cp1252. | | `text/html` | `html.parser`: scripts, styles, `noscript` and `template` dropped; block tags become line breaks; entities resolved. | | CSV / TSV | `csv` module; each row written as tab-separated values. | | `message/rfc822` | `email` (modern policy): From/To/Cc/Date/Subject lines, then the plain body, or the HTML body through the HTML reader. | | RTF | A small reader of its own: control words, `\'hh` (cp1252), `\uN` (one fallback character skipped), and the groups that hold no body text (`fonttbl`, `colortbl`, `stylesheet`, `info`, `pict`, `object`, `header`, `footer`, and others) discarded. | | DOCX | Paragraphs of `word/document.xml` (runs, tabs, breaks, text boxes after the paragraph that holds them), plus headers and footers. | | XLSX | Shared strings, inline strings and values, one tab-separated line per row, a `# sheetN` line per sheet; empty rows dropped. | | PPTX | Every paragraph on every slide, slides in numeric order, a `# slideN` line per slide. | | ODT / ODS / ODP | `content.xml`: paragraphs and headings; table rows tab-separated. | | PDF | **`pdftotext -layout -enc UTF-8` when poppler is on PATH.** Otherwise the built-in reader (below). | | Images (PNG, JPEG, TIFF, BMP, GIF, WebP) | **`tesseract` when it is on PATH**; otherwise `unsupported`. | Anything else (archives, executables, audio, video, unknown binaries) is `unsupported`, with the MIME type named. ### The built-in PDF reader Without poppler (on Windows, usually) PDFs are read by `pdftext.py`, which walks the document far enough for text: the object table (a later definition wins, so incremental updates are respected) and compressed object streams; FlateDecode streams (a stream with any other filter is skipped); the page tree in order with inherited resources; fonts through their `/ToUnicode` maps (Type0 / Identity-H fonts, as browsers print them), or WinAnsi / MacRoman for simple fonts without one; glyph widths, so a word placed by position still gets its space; the text operators, and form XObjects. Pages are separated by a form feed. Encrypted files are refused. Output below 90% printable characters is refused too, so a font without a usable map comes back as `unsupported` (`extractor: pdf-builtin`, `needs pdftotext (poppler) on this machine`), never as byte salad in the catalog. It reads arbitrary files, so it reads on a budget per document: 60 s of wall time, 256 MiB of decoded streams (64 MiB per stream), 5 million content operators, 2,000 form invocations at most 6 deep, 200,000 CMap entries, 20 million tokens and 1 GiB lexed. Reaching a limit stops the read with the pages read so far. Nothing in it raises out to the stage. ## Rules the stage enforces - **External tools are sensors, not dependencies.** They are looked up on PATH at run time, named in every record they produced (`extractor: pdftotext`), and their absence is a recorded gap, never a failure. `--no-external` (in a sweep, `FREEAGENT_NO_EXTERNAL=1`) uses the built-in readers only. A manifest cannot name a tool or an address. - **Nothing is written outside the output directory**, and every file is written atomically. - **Bounded reads.** One zip member is read to at most 64 MiB decompressed (`ZIP_MEMBER_CAP`), so a hostile office file cannot balloon memory. Office documents are parsed as a stream, each finished paragraph or row dropped once read, and capped per document: 256 MiB of XML, 16 Mi characters of text, 60 s, and 200,000 parts in one paragraph or row. Reaching a cap keeps the text read so far and ends it with a line naming the cap. XML with a DTD or an entity declaration is refused (Office files never carry one, and it is how entity expansion gets in), as is XML that is neither UTF-8 nor UTF-16. Tool calls time out at 300 s. - **Errors are entries.** A corrupt document is one `error` record; the stage finishes and reports the count. - **Retry only when something changed.** An `unsupported` verdict is kept while the machine's tool set is the same; install `pdftotext` and the next sweep tries exactly those files again. `error` records are retried on every sweep. Every record carries the extractor version (`v`, `EXTRACT_VERSION`, now 2); raising it after a reader changes redoes every cached verdict and text on the next sweep. ## Tests `python3 -m unittest bastion_agent.test_extract bastion_agent.test_pdftext`. `test_extract` runs every reader against a generated fixture (docx, xlsx, pptx and odt built with `zipfile`, plain and Flate PDFs, RTF with hex and Unicode escapes, plain and HTML e-mail, encodings), the Office streaming caps, DTD refusal and UTF-16 input, the zip member cap, an oversized CSV field, the whole stage over a mixed tree (statuses, counts, duplicates), reuse across sweeps and its invalidation when content or the extractor version changes, tool discovery and failure (mocked), and the walk-first guard. `test_pdftext` builds browser-shaped PDFs with a small writer (Type0 fonts with ToUnicode maps, per-run positioning, ligatures, inherited resources, form XObjects, object streams) and hostile ones (reference cycles, deep nesting, malformed tables and maps), and checks each budget. The external-tool paths are tested with mocks; a real scanned PDF or image through `pdftotext` or `tesseract` is not part of the suite. ## Not in this stage Sanitizing (the next stage: the text store goes through the PII sterilizer), chunking, and change detection beyond the content hash. Audio and video are not extracted.