bastion-agent/docs/extract.md

bastion agent extract stage: text out of the walked files

The extract stage turns every walked file it can read into plain text, with built-in readers or with local tools when they are installed. It is bastion_agent/extract.py and bastion_agent/pdftext.py, standard library only.

What it produces

python3 -m bastion_agent extract <manifest.yaml> [--out DIR] [--no-external] [--require-signed] reads walk.jsonl and walk.summary.json from DIR (the walk must have run; the stage refuses otherwise) and writes:

• extract/<aa>/<sha256>.txt: the text store, one file per distinct content hash, UTF-8, normalised (CRLF and lone CR become LF, NUL removed, NFC). Content addressing is the cache: two copies of a document are extracted once, and an unchanged file costs nothing on the next sweep.
• extract.jsonl: one record per walked file, sorted by (root, path): sha256, mime, status, extractor, chars, error, v (the extractor version), reused when a prior verdict was kept, and on unsupported records the tools that were present, so the verdict is retried when a tool appears.
• extract.summary.json: counts, total characters, the tools found, the extractor version, the manifest digest, and the manifest digest the walk recorded, which ties the walk map and the text store to one manifest.

DIR defaults as for the walk (<root>/.agent/ for one root). This text store is the plaintext of every file in scope. The agent never serves it, but by default it sits in the folder, so if the folder is synced, the sync provider gets a copy. A bastion from 0.58.1 on never serves .agent/ to a host. On an older bastion, a host whose folder grant has read can read the store like any other file unless the grant excludes .agent; see runtime.md.

status	Meaning
ok	Text written to the store.
empty	The reader ran and found no text (an empty file, a document with only pictures).
unsupported	No reader for this type on this machine; error names what would handle it.
error	The reader failed (a corrupt zip, bad XML, a tool failure); the stage continues.
skipped	The walk has no hash for it (too large, a read error, or a --no-hash walk).

A record whose earlier verdict was kept carries reused: true and is counted under reused in the summary.

Exit code: 0 clean; 1 the manifest is invalid, extract is not in its stages, or there is no walk map; 2 completed with error records.

Readers

Built in, no libraries:

Type	How
text/*, JSON, YAML, TOML, XML, SQL, SVG, PEM, code	Decoded, and the encoding recorded (text:utf-8, text:cp1252, …): a byte-order mark first, then BOM-less UTF-16 by the share of NUL bytes, then strict UTF-8, then cp1252.
text/html	html.parser: scripts, styles, noscript and template dropped; block tags become line breaks; entities resolved.
CSV / TSV	csv module; each row written as tab-separated values.
message/rfc822	email (modern policy): From/To/Cc/Date/Subject lines, then the plain body, or the HTML body through the HTML reader.
RTF	A small reader of its own: control words, \'hh (cp1252), \uN (one fallback character skipped), and the groups that hold no body text (fonttbl, colortbl, stylesheet, info, pict, object, header, footer, and others) discarded.
DOCX	Paragraphs of word/document.xml (runs, tabs, breaks, text boxes after the paragraph that holds them), plus headers and footers.
XLSX	Shared strings, inline strings and values, one tab-separated line per row, a # sheetN line per sheet; empty rows dropped.
PPTX	Every paragraph on every slide, slides in numeric order, a # slideN line per slide.
ODT / ODS / ODP	content.xml: paragraphs and headings; table rows tab-separated.
PDF	pdftotext -layout -enc UTF-8 when poppler is on PATH. Otherwise the built-in reader (below).
Images (PNG, JPEG, TIFF, BMP, GIF, WebP)	tesseract when it is on PATH; otherwise unsupported.

Anything else (archives, executables, audio, video, unknown binaries) is unsupported, with the MIME type named.

The built-in PDF reader

Without poppler (on Windows, usually) PDFs are read by pdftext.py, which walks the document far enough for text: the object table (a later definition wins, so incremental updates are respected) and compressed object streams; FlateDecode streams (a stream with any other filter is skipped); the page tree in order with inherited resources; fonts through their /ToUnicode maps (Type0 / Identity-H fonts, as browsers print them), or WinAnsi / MacRoman for simple fonts without one; glyph widths, so a word placed by position still gets its space; the text operators, and form XObjects. Pages are separated by a form feed.

Encrypted files are refused. Output below 90% printable characters is refused too, so a font without a usable map comes back as unsupported (extractor: pdf-builtin, needs pdftotext (poppler) on this machine), never as byte salad in the catalog.

It reads arbitrary files, so it reads on a budget per document: 60 s of wall time, 256 MiB of decoded streams (64 MiB per stream), 5 million content operators, 2,000 form invocations at most 6 deep, 200,000 CMap entries, 20 million tokens and 1 GiB lexed. Reaching a limit stops the read with the pages read so far. Nothing in it raises out to the stage.

Rules the stage enforces

• External tools are sensors, not dependencies. They are looked up on PATH at run time, named in every record they produced (extractor: pdftotext), and their absence is a recorded gap, never a failure. --no-external (in a sweep, FREEAGENT_NO_EXTERNAL=1) uses the built-in readers only. A manifest cannot name a tool or an address.
• Nothing is written outside the output directory, and every file is written atomically.
• Bounded reads. One zip member is read to at most 64 MiB decompressed (ZIP_MEMBER_CAP), so a hostile office file cannot balloon memory. Office documents are parsed as a stream, each finished paragraph or row dropped once read, and capped per document: 256 MiB of XML, 16 Mi characters of text, 60 s, and 200,000 parts in one paragraph or row. Reaching a cap keeps the text read so far and ends it with a line naming the cap. XML with a DTD or an entity declaration is refused (Office files never carry one, and it is how entity expansion gets in), as is XML that is neither UTF-8 nor UTF-16. Tool calls time out at 300 s.
• Errors are entries. A corrupt document is one error record; the stage finishes and reports the count.
• Retry only when something changed. An unsupported verdict is kept while the machine's tool set is the same; install pdftotext and the next sweep tries exactly those files again. error records are retried on every sweep. Every record carries the extractor version (v, EXTRACT_VERSION, now 2); raising it after a reader changes redoes every cached verdict and text on the next sweep.

Tests

python3 -m unittest bastion_agent.test_extract bastion_agent.test_pdftext. test_extract runs every reader against a generated fixture (docx, xlsx, pptx and odt built with zipfile, plain and Flate PDFs, RTF with hex and Unicode escapes, plain and HTML e-mail, encodings), the Office streaming caps, DTD refusal and UTF-16 input, the zip member cap, an oversized CSV field, the whole stage over a mixed tree (statuses, counts, duplicates), reuse across sweeps and its invalidation when content or the extractor version changes, tool discovery and failure (mocked), and the walk-first guard. test_pdftext builds browser-shaped PDFs with a small writer (Type0 fonts with ToUnicode maps, per-run positioning, ligatures, inherited resources, form XObjects, object streams) and hostile ones (reference cycles, deep nesting, malformed tables and maps), and checks each budget.

The external-tool paths are tested with mocks; a real scanned PDF or image through pdftotext or tesseract is not part of the suite.

Not in this stage

Sanitizing (the next stage: the text store goes through the PII sterilizer), chunking, and change detection beyond the content hash. Audio and video are not extracted.