bastion agent 0.1.14
This commit is contained in:
+138
@@ -0,0 +1,138 @@
|
||||
# bastion agent `extract` stage: text out of the walked files
|
||||
|
||||
The extract stage turns every walked file it can read into plain text, with
|
||||
built-in readers or with local tools when they are installed. It is
|
||||
`bastion_agent/extract.py` and `bastion_agent/pdftext.py`, standard library
|
||||
only.
|
||||
|
||||
## What it produces
|
||||
|
||||
`python3 -m bastion_agent extract <manifest.yaml> [--out DIR] [--no-external] [--require-signed]`
|
||||
reads `walk.jsonl` and `walk.summary.json` from `DIR` (the walk must have run;
|
||||
the stage refuses otherwise) and writes:
|
||||
|
||||
- **`extract/<aa>/<sha256>.txt`**: the text store, one file per distinct
|
||||
content hash, UTF-8, normalised (CRLF and lone CR become LF, NUL removed,
|
||||
NFC). Content addressing is the cache: two copies of a document are
|
||||
extracted once, and an unchanged file costs nothing on the next sweep.
|
||||
- **`extract.jsonl`**: one record per walked file, sorted by `(root, path)`:
|
||||
`sha256`, `mime`, `status`, `extractor`, `chars`, `error`, `v` (the
|
||||
extractor version), `reused` when a prior verdict was kept, and on
|
||||
`unsupported` records the `tools` that were present, so the verdict is
|
||||
retried when a tool appears.
|
||||
- **`extract.summary.json`**: counts, total characters, the tools found, the
|
||||
extractor version, the manifest digest, and the manifest digest the walk
|
||||
recorded, which ties the walk map and the text store to one manifest.
|
||||
|
||||
`DIR` defaults as for the walk (`<root>/.agent/` for one root). This text
|
||||
store is the plaintext of every file in scope, and it never leaves the machine;
|
||||
see [runtime.md](runtime.md#where-an-agent-lives-rootagent).
|
||||
|
||||
| `status` | Meaning |
|
||||
|---|---|
|
||||
| `ok` | Text written to the store. |
|
||||
| `empty` | The reader ran and found no text (an empty file, a document with only pictures). |
|
||||
| `unsupported` | No reader for this type on this machine; `error` names what would handle it. |
|
||||
| `error` | The reader failed (a corrupt zip, bad XML, a tool failure); the stage continues. |
|
||||
| `skipped` | The walk has no hash for it (too large, a read error, or a `--no-hash` walk). |
|
||||
|
||||
A record whose earlier verdict was kept carries `reused: true` and is counted
|
||||
under `reused` in the summary.
|
||||
|
||||
Exit code: `0` clean; `1` the manifest is invalid, `extract` is not in its
|
||||
stages, or there is no walk map; `2` completed with `error` records.
|
||||
|
||||
## Readers
|
||||
|
||||
Built in, no libraries:
|
||||
|
||||
| Type | How |
|
||||
|---|---|
|
||||
| `text/*`, JSON, YAML, TOML, XML, SQL, SVG, PEM, code | Decoded, and the encoding recorded (`text:utf-8`, `text:cp1252`, …): a byte-order mark first, then BOM-less UTF-16 by the share of NUL bytes, then strict UTF-8, then cp1252. |
|
||||
| `text/html` | `html.parser`: scripts, styles, `noscript` and `template` dropped; block tags become line breaks; entities resolved. |
|
||||
| CSV / TSV | `csv` module; each row written as tab-separated values. |
|
||||
| `message/rfc822` | `email` (modern policy): From/To/Cc/Date/Subject lines, then the plain body, or the HTML body through the HTML reader. |
|
||||
| RTF | A small reader of its own: control words, `\'hh` (cp1252), `\uN` (one fallback character skipped), and the groups that hold no body text (`fonttbl`, `colortbl`, `stylesheet`, `info`, `pict`, `object`, `header`, `footer`, and others) discarded. |
|
||||
| DOCX | Paragraphs of `word/document.xml` (runs, tabs, breaks, text boxes after the paragraph that holds them), plus headers and footers. |
|
||||
| XLSX | Shared strings, inline strings and values, one tab-separated line per row, a `# sheetN` line per sheet; empty rows dropped. |
|
||||
| PPTX | Every paragraph on every slide, slides in numeric order, a `# slideN` line per slide. |
|
||||
| ODT / ODS / ODP | `content.xml`: paragraphs and headings; table rows tab-separated. |
|
||||
| PDF | **`pdftotext -layout -enc UTF-8` when poppler is on PATH.** Otherwise the built-in reader (below). |
|
||||
| Images (PNG, JPEG, TIFF, BMP, GIF, WebP) | **`tesseract` when it is on PATH**; otherwise `unsupported`. |
|
||||
|
||||
Anything else (archives, executables, audio, video, unknown binaries) is
|
||||
`unsupported`, with the MIME type named.
|
||||
|
||||
### The built-in PDF reader
|
||||
|
||||
Without poppler (on Windows, usually) PDFs are read by `pdftext.py`, which
|
||||
walks the document far enough for text: the object table (a later definition
|
||||
wins, so incremental updates are respected) and compressed object streams;
|
||||
FlateDecode streams (a stream with any other filter is skipped); the page tree
|
||||
in order with inherited resources; fonts through their `/ToUnicode` maps
|
||||
(Type0 / Identity-H fonts, as browsers print them), or WinAnsi / MacRoman for
|
||||
simple fonts without one; glyph widths, so a word placed by position still
|
||||
gets its space; the text operators, and form XObjects. Pages are separated by
|
||||
a form feed.
|
||||
|
||||
Encrypted files are refused. Output below 90% printable characters is refused
|
||||
too, so a font without a usable map comes back as `unsupported` (`extractor:
|
||||
pdf-builtin`, `needs pdftotext (poppler) on this machine`), never as byte salad
|
||||
in the catalog.
|
||||
|
||||
It reads arbitrary files, so it reads on a budget per document: 60 s of wall
|
||||
time, 256 MiB of decoded streams (64 MiB per stream), 5 million content
|
||||
operators, 2,000 form invocations at most 6 deep, 200,000 CMap entries,
|
||||
20 million tokens and 1 GiB lexed. Reaching a limit stops the read with the
|
||||
pages read so far. Nothing in it raises out to the stage.
|
||||
|
||||
## Rules the stage enforces
|
||||
|
||||
- **External tools are sensors, not dependencies.** They are looked up on
|
||||
PATH at run time, named in every record they produced (`extractor:
|
||||
pdftotext`), and their absence is a recorded gap, never a failure.
|
||||
`--no-external` (in a sweep, `FREEAGENT_NO_EXTERNAL=1`) uses the built-in
|
||||
readers only. A manifest cannot name a tool or an address.
|
||||
- **Nothing is written outside the output directory**, and every file is
|
||||
written atomically.
|
||||
- **Bounded reads.** One zip member is read to at most 64 MiB decompressed
|
||||
(`ZIP_MEMBER_CAP`), so a hostile office file cannot balloon memory. Office
|
||||
documents are parsed as a stream, each finished paragraph or row dropped
|
||||
once read, and capped per document: 256 MiB of XML, 16 Mi characters of
|
||||
text, 60 s, and 200,000 parts in one paragraph or row. Reaching a cap keeps
|
||||
the text read so far and ends it with a line naming the cap. XML with a DTD
|
||||
or an entity declaration is refused (Office files never carry one, and it is
|
||||
how entity expansion gets in), as is XML that is neither UTF-8 nor UTF-16.
|
||||
Tool calls time out at 300 s.
|
||||
- **Errors are entries.** A corrupt document is one `error` record; the stage
|
||||
finishes and reports the count.
|
||||
- **Retry only when something changed.** An `unsupported` verdict is kept
|
||||
while the machine's tool set is the same; install `pdftotext` and the next
|
||||
sweep tries exactly those files again. `error` records are retried on every
|
||||
sweep. Every record carries the extractor version (`v`, `EXTRACT_VERSION`,
|
||||
now 2); raising it after a reader changes redoes every cached verdict and
|
||||
text on the next sweep.
|
||||
|
||||
## Tests
|
||||
|
||||
`python3 -m unittest bastion_agent.test_extract bastion_agent.test_pdftext`.
|
||||
`test_extract` runs every reader against a generated fixture (docx, xlsx,
|
||||
pptx and odt built with `zipfile`, plain and Flate PDFs, RTF with hex and
|
||||
Unicode escapes, plain and HTML e-mail, encodings), the Office streaming caps,
|
||||
DTD refusal and UTF-16 input, the zip member cap, an oversized CSV field, the
|
||||
whole stage over a mixed tree (statuses, counts, duplicates), reuse across
|
||||
sweeps and its invalidation when content or the extractor version changes,
|
||||
tool discovery and failure (mocked), and the walk-first guard. `test_pdftext`
|
||||
builds browser-shaped PDFs with a small writer (Type0 fonts with ToUnicode
|
||||
maps, per-run positioning, ligatures, inherited resources, form XObjects,
|
||||
object streams) and hostile ones (reference cycles, deep nesting, malformed
|
||||
tables and maps), and checks each budget.
|
||||
|
||||
The external-tool paths are tested with mocks; a real scanned PDF or image
|
||||
through `pdftotext` or `tesseract` is not part of the suite.
|
||||
|
||||
## Not in this stage
|
||||
|
||||
Sanitizing (the next stage: the text store goes through the PII sterilizer),
|
||||
chunking, and change detection beyond the content hash. Audio and video are not
|
||||
extracted.
|
||||
Reference in New Issue
Block a user