bastion-agent/docs/sanitize.md

bastion agent sanitize stage: personal data out of the text

The sanitize stage runs the extracted text through the PII sterilizer (pii/, described in pii-sterilization.md) and keeps the result in a store of its own. Every later stage reads that store, never the extracted text. It is bastion_agent/sanitize.py.

What it produces

python3 -m bastion_agent sanitize <manifest.yaml> [--out DIR] [--require-signed] reads the extract index and store and writes:

• sanitize/<aa>/<sha256>.txt: the sanitized text store. Personal data is replaced by stable [PII_TYPE_n] tokens. From this stage on, later stages read only this store. The extract store keeps the plaintext and never leaves the machine.
• vault.db: the token-to-value map (in-process mode only), created 0600 under umask 077. It is the only way back from a token to a value, and it stays on the machine. It is <out>/sanitize/vault.db, or $FREEAGENT_VAULT_DIR/vault.db when that is set: the output folder may be synced to a cloud drive, the vault must not be, so the bastion sets FREEAGENT_VAULT_DIR to its own state directory.
• sanitize.jsonl: one record per extracted file: status, mode, stats (counts from the pipeline, summed over the file's chunks: rule hits, recall hits, dropped proposals, residuals and verify hits replaced, and literal_tokens_escaped, so a literal that appears in two chunks counts twice), types (token counts per type), chars, reused, error. Never a value. A test searches the index and the summary for the fixture's personal data and fails if any appears.
• sanitize.summary.json: counts, token totals per type, the sterilizer that ran (flag:rules-only, sterilize:llm=…, sterilize:gliner=…@0.50, service:http://…), and stopped when the stage failed closed.

status	Meaning
ok	Sanitized text written (or kept from an earlier run: then reused is true).
skipped	The extract stage had no text for it (unsupported, error, empty, too large).
error	A failure that is not the recall pass (the store unreadable, an HTTP error from the service other than 502). The stage continues.
not_run	The stage stopped (fail closed) at or before this file.

Exit code: 0 clean; 1 the manifest is invalid, sanitize is not in its stages, there is no extract index, or the stage stopped; 2 completed with error records.

Modes (manifest sanitize.mode)

• sterilize: the full pipeline: the rules pass, the recall pass, grounding, replacement by the program, the residual loop, and the optional adversarial verify pass. Contextual personal data (names, schools, employers, medical facts) is found by the recall pass.
• flag: the rules pass only: deterministic, no model, milliseconds. It replaces the anchors a pattern can find: e-mail addresses, phone numbers, SSNs, cards, IBANs, routing numbers, crypto addresses, IP and MAC addresses, API keys, private-key blocks, JWTs. It can also replace an ordinary number that looks like a phone or ID number. It does not find names, street addresses, dates of birth, or anything else that takes reading for meaning (an employer, a school, a medical fact). It is the mode the bastion writes into every manifest it deploys. A flag-mode catalog is not free of personal data, and the manifest doc says so too.

Where the sterilizer runs

It is chosen from the environment, never the manifest (a manifest carries no addresses).

Condition	Where	Vault
mode: flag	In this process, rules only. PII_ADDR is ignored.	vault.db as above
mode: sterilize, no PII_ADDR	In this process: pii inside the agent, the recall pass from PII_RECALL (llm → the server at LLM_URL; gliner → an in-process CPU model). The laptop case.	vault.db as above
mode: sterilize, PII_ADDR set	A running PII service over HTTP (POST /sterilize, Authorization: Bearer $PII_API_KEY).	The service's own

PII_ADDR without PII_API_KEY is refused: the service fails closed, and so does its caller.

Rules the stage enforces

• Fail closed. The first recall failure (the model unreachable, the service answering 502 or not at all) stops the stage. So does a service that does not confirm it escaped every placeholder-shaped string in the document (stats.escape_held_literals missing or false: a service older than the agent); upgrade the service. The file being sanitized and every file not yet reached are recorded not_run; nothing half-sanitized is written; the exit code is 1 and summary.stopped says why. A recall backend that cannot start (for example PII_RECALL=gliner without the gliner package) stops the stage before the first file, with exit code 1 and no index written.
• Tokens are stable within a scope. sanitize.scope (default: the manifest id) is the vault scope: the same e-mail address in two documents of one catalog gets one token, so the catalog stays coherent, and a different catalog gets different tokens, so two catalogs cannot be joined through shared placeholders.
• Long texts are split, not truncated. The sterilizer is called per chunk of at most 20,000 characters (SANITIZE_CHUNK_CHARS), split at paragraph and then line boundaries (a single longer line is cut). The chunks join back to the input exactly, and because tokens are stable a value that appears in two chunks gets one token. A cut never falls inside a placeholder: it moves back to before the placeholder, or just after it when the placeholder starts the chunk (only possible when the placeholder is longer than the limit), so that chunk runs over the limit by the rest of the placeholder. Any text in a document that looks like a placeholder is escaped, whether or not the vault holds that name, so a document quoting [PII_EMAIL_1] comes back as written and never as another document's value.
• What makes a file be done again: a change in its content, the mode, the scope, verify, where the sterilizer runs (in process or service), or SANITIZE_VERSION. Results are redone when the pipeline changes: SANITIZE_VERSION is raised whenever the same text would be sterilized differently, so an upgrade that raises it re-sanitizes every file once. Changing only the recall backend or its model does not redo files already sanitized; raise SANITIZE_VERSION, or start from an empty output folder, when that matters.
• Duplicate content is sanitized once, and every path that has it is indexed.

Tests

python3 -m unittest bastion_agent.test_sanitize covers flag mode with the real pipeline and vault (anchors replaced, a name left alone: the limit is asserted, not hidden), the index and summary carrying no values, the vault's 0600 mode and FREEAGENT_VAULT_DIR, reuse and its invalidation (content, scope), token stability across files, sterilize mode with a fake recall backend (grounding drops an invented span; stats reported), the fail-closed stop after the first recall failure (not_run, no store file written, the backend called once), a backend that cannot start, the chunking round trip (one token across every chunk) and hard cuts, the service path over a stub speaking the real wire format (auth header, scope, verify, stats.recall), a service 502 failing closed, a service without a key refused, flag mode never touching the service, and the extract-first guard.

A sterilize run against a real recall model is not part of the suite; the PII service's verify-live.sh does that against a running service.

Not in this stage

Chunking for embedding (the next stage, which reads the sanitized store), putting values back (the vault is local, and /rehydrate belongs to the PII service), and encryption of the vault at rest (not done: the controls are the file's 0600 mode and where it is kept).