privatae/bastion-agent
2
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

bastion agent 0.1.14

Browse Source
This commit is contained in:
Privatae
2026-09-26 18:31:40 +00:00
commit 3f1d48bf9b
65 changed files with 15548 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/catalog.md
+102
View File
@@ -0,0 +1,102 @@
# bastion agent `catalog` stage: the curated dataset
The catalog is the small, structured thing a host reasons over and the thing
`seal` ships: one card per file and one per directory, built only from the
sanitized text. It is `bastion_agent/catalog.py`, standard library only.
`python3 -m bastion_agent catalog <manifest> [--out DIR] [--require-signed]`
needs the walk, extract and sanitize outputs, and reads `chunks.jsonl` if it
is there. It writes:
- **`catalog.jsonl`**: one card per file the walk found, sorted by
`(root, path)`.
- **`catalog.dirs.jsonl`**: one card per directory that holds files (every
ancestor of every file; the root is `""`), sorted the same way.
- **`catalog.summary.json`**: counts, the summarizer that ran (or `null`),
`stopped`, and `bytes_catalog` (the length of all the cards, in
characters).
## The file card
| Field | From | Notes |
|---|---|---|
| `root`, `path`, `name`, `kind`, `size`, `mtime`, `mime`, `sha256` | walk | `path` and `name` after the name mask (below). |
| `text` | extract | `ok`, `empty`, `unsupported`, `error`, `skipped` or `not_extracted`. |
| `extractor` | extract | Which reader, e.g. `pdftotext`, `pdf-builtin`, `docx`, `text:utf-8`. |
| `sanitized` | sanitize | `ok`, `skipped`, `error`, `not_run`, or `null` when the sanitize stage has no record for it. |
| `title` | **sanitized text** | The first line of at least three characters, list and heading markers stripped, at most 120 characters. |
| `head` | **sanitized text** | The first 300 characters, whitespace collapsed. |
| `chars` | sanitized text | Length. |
| `keywords` | **sanitized text** | The top 8 by TF-IDF across the catalog: words of three or more letters, stopwords and placeholders removed, ties broken alphabetically (so the result is deterministic). |
| `entities` | sanitize | Token counts per type minted for this file, e.g. `{"EMAIL": 2, "PERSON": 1}`. Counts, never values. |
| `chunks` | chunk | How many windows the document produced (0 without the chunk stage). |
| `summary` | summarizer | Present only when the manifest asks for file summaries; `null` when there was no text. |
| `v` | | The card format version. |
Only files with sanitized text get the text fields. A file without text still
gets a card: the host can see that it exists, its type and size, and why it
has no text.
## The directory card
`files`, `with_text`, `bytes`, `latest` (newest mtime), `types` (the top 6
MIME types), `keywords` (the top 12 over the merged term counts), `entities`
(summed over the files), and `summary` when asked for.
## What never enters the catalog
- **Values.** Every field derived from text comes from the sanitized store. A
test searches the whole catalog for the fixture's personal data and fails on
any hit.
- **Placeholders as keywords.** `[PII_*_n]` tokens are removed before terms
are counted.
- **Anchors in names.** File and directory names are plaintext by nature, so
every segment of every path (parent directories included) goes through the
rules pass; an e-mail address, phone number, card or key in a name is masked
to `[PII_TYPE]`. The masking is conservative: the e-mail rule spans dots and
hyphens, so `invoice-m@example.com.pdf` becomes `[PII_EMAIL]`; `mime` and
`size` still say what it was. The summary counts masked segments as
`names_masked`. **A name like `Marisol_medical.pdf` is not caught**: that is
contextual personal data, the same limit as the rules-only `flag` mode.
## The summarizer (a sensor)
A manifest section `summarize: {model, per: [file, directory]}` turns it on
(it needs `summarize` in the mask). The model is reached through
`SUMMARIZE_URL` (default `http://127.0.0.1:11434`, a local Ollama),
`SUMMARIZE_API` (`ollama` or `openai`; unset means try Ollama's `/api/chat`,
then `/v1/chat/completions`) and `SUMMARIZE_TIMEOUT` (300 s): the
environment, never the manifest.
It sees only sanitized text (the first 12,000 characters), is told that
placeholders are redactions to keep as they are, and runs at temperature 0.
Its reply is cleaned (`<think>` blocks and surrounding quotes removed,
whitespace collapsed, at most 600 characters) into the `summary` string, the
only field it can touch. A directory's summary is built from its direct
children's summaries or heads (at most 60, 200 characters each). Summaries are
cached in `catalog.cache.json` by document, model and version, so a re-run
costs nothing for unchanged documents.
The first failure stops summarizing: every card is still written, one card
carries `summary_error`, the summary says `stopped`, and the exit code is 1.
## Tests
`python3 -m unittest bastion_agent.test_catalog` covers title, head and
keywords and their determinism; name masking (e-mail, phone, plain names,
parent directories); summary cleaning; the whole stage with and without a
summarizer (card fields, entities, keywords free of placeholders, a card for
a file with no text, directory totals including the root, masked names
counted); the catalog containing none of the fixture's personal data; file and
directory summaries with the cache reused on the second run and the model
seeing only sanitized text; OpenAI-style detection; a summarizer failure
stopping after one call with every card still written; an unreachable
summarizer; and the guard that the earlier stages ran first.
The stopword list is for prose: on source code, identifiers such as `self`
and `def` count as keywords.
## Not in this stage
Sealing (the next stage: the catalog encrypted to the owner and the
manifest's `seal_to` recipients) and any rendering of the cards on the host.
docs/embed.md
+85
View File
@@ -0,0 +1,85 @@
# bastion agent `chunk` and `embed` stages, and `search`
`chunk` splits the sanitized text into overlapping windows, `embed` turns each
window into a vector with a model on this machine, and `search` answers a
query against those vectors. They are `bastion_agent/embed.py`, standard
library only.
## chunk
`python3 -m bastion_agent chunk <manifest> [--out DIR] [--require-signed]`
reads the **sanitized** store (never the extract store) and writes
`chunks.jsonl`: one row per window (`chunk_id`, `doc` (the sanitized text's
sha256), `idx`, `start` and `end` character offsets, `chars`, `tokens`, `v`),
sorted by `(doc, idx)`, plus `chunk.summary.json`.
- Windows are `embed.chunk.size_tokens` long and advance by
`size - overlap` tokens; the last window may be shorter. With any overlap,
every character of the text is inside at least one window (with
`overlap_tokens: 0` the whitespace between two windows may fall outside
both).
- A "token" is a fixed regular-expression unit (a word or a punctuation mark),
not a model's tokenizer. The standard library has none, and the same text
must chunk the same way on every machine; the count is close enough to model
tokens to size windows by.
- `chunk_id = sha256(document sha256, NUL, window text)`. An unchanged
document keeps its chunk ids, and so its vectors, from one sweep to the
next. A changed document has a new sha256, so all of its windows get new ids
and are embedded again. Identical windows inside one document (a repeated
paragraph) share an id and are embedded once.
- Duplicate documents (the same sanitized content at several paths) are
chunked once.
## embed
`python3 -m bastion_agent embed <manifest> [--out DIR] [--require-signed]`
writes `vectors.db` (SQLite) and `embed.summary.json`.
**The endpoint is a sensor.** It comes from the environment, never the
manifest: `EMBED_URL` (default `http://127.0.0.1:11434`, a local Ollama),
`EMBED_API` (`ollama` or `openai`; unset means detect it with the probe),
`EMBED_BATCH` (32), `EMBED_TIMEOUT` (300 s). The manifest pins the **model
name and dimension**. Two wire formats: Ollama's `POST /api/embed` and the
OpenAI-style `POST /v1/embeddings` (llama.cpp server, TEI).
**The probe is the gate.** Before the first real call the stage embeds a fixed
probe string. If the served dimension is not the manifest's, the stage stops
before opening the store, so a wrong or mislabelled model never writes a
vector. An unreachable endpoint stops it the same way.
**Storage** (`vectors.db`):
| Table | Role |
|---|---|
| `chunks(chunk_id PK, doc, idx, start, end, text)` | This sweep's windows, rebuilt on every run. Holds the sanitized window text, so a search can return it. |
| `vectors(chunk_id, model, dim, vec BLOB float32, created)`, PK `(chunk_id, model)` | Kept across runs; rows are only ever added. |
Switching models **adds** rows for the new model; the old model's rows stay.
Vectors whose chunk no longer exists in this sweep also stay, and are counted
as `stale_vectors` in the summary: nothing is thrown away, so going back to an
earlier model or document costs nothing. A re-run embeds only chunk ids that
have no vector for the manifest's model (the rest are `reused`).
## search
`python3 -m bastion_agent search <manifest> "<query>" [-k N] [--out DIR]`
embeds the query through the same endpoint, scores cosine similarity against
every vector of the manifest's model, and prints the top k (default 5)
sanitized windows with their score, document id and index. It is a linear scan
in pure Python, which is fine to tens of thousands of chunks; an index
(HNSW or sqlite-vec) is the next step for a folder that outgrows it. The
query endpoint (`serve`, [serve.md](serve.md)) uses the same search.
## Tests
`python3 -m unittest bastion_agent.test_embed` runs against a loopback stub
that speaks both wire formats with a deterministic bag-of-words embedding:
window coverage, overlap and step; empty and short inputs; content-addressed
ids; the float32 packing and cosine edge cases; the chunk stage (duplicate
documents, sort order, determinism, the sanitize-first guard); the probe
refusing a wrong dimension before the store exists; an unreachable endpoint
failing closed; batching; reuse on a re-run; a second model adding rows while
the first model's rows stay; stale vectors kept and counted after a document
changes; OpenAI-style detection and a forced wrong API refused; and search
ranking the right document, with an empty result for a model that has no
vectors.
docs/extract.md
+138
View File
@@ -0,0 +1,138 @@
# bastion agent `extract` stage: text out of the walked files
The extract stage turns every walked file it can read into plain text, with
built-in readers or with local tools when they are installed. It is
`bastion_agent/extract.py` and `bastion_agent/pdftext.py`, standard library
only.
## What it produces
`python3 -m bastion_agent extract <manifest.yaml> [--out DIR] [--no-external] [--require-signed]`
reads `walk.jsonl` and `walk.summary.json` from `DIR` (the walk must have run;
the stage refuses otherwise) and writes:
- **`extract/<aa>/<sha256>.txt`**: the text store, one file per distinct
content hash, UTF-8, normalised (CRLF and lone CR become LF, NUL removed,
NFC). Content addressing is the cache: two copies of a document are
extracted once, and an unchanged file costs nothing on the next sweep.
- **`extract.jsonl`**: one record per walked file, sorted by `(root, path)`:
`sha256`, `mime`, `status`, `extractor`, `chars`, `error`, `v` (the
extractor version), `reused` when a prior verdict was kept, and on
`unsupported` records the `tools` that were present, so the verdict is
retried when a tool appears.
- **`extract.summary.json`**: counts, total characters, the tools found, the
extractor version, the manifest digest, and the manifest digest the walk
recorded, which ties the walk map and the text store to one manifest.
`DIR` defaults as for the walk (`<root>/.agent/` for one root). This text
store is the plaintext of every file in scope, and it never leaves the machine;
see [runtime.md](runtime.md#where-an-agent-lives-rootagent).
| `status` | Meaning |
|---|---|
| `ok` | Text written to the store. |
| `empty` | The reader ran and found no text (an empty file, a document with only pictures). |
| `unsupported` | No reader for this type on this machine; `error` names what would handle it. |
| `error` | The reader failed (a corrupt zip, bad XML, a tool failure); the stage continues. |
| `skipped` | The walk has no hash for it (too large, a read error, or a `--no-hash` walk). |
A record whose earlier verdict was kept carries `reused: true` and is counted
under `reused` in the summary.
Exit code: `0` clean; `1` the manifest is invalid, `extract` is not in its
stages, or there is no walk map; `2` completed with `error` records.
## Readers
Built in, no libraries:
| Type | How |
|---|---|
| `text/*`, JSON, YAML, TOML, XML, SQL, SVG, PEM, code | Decoded, and the encoding recorded (`text:utf-8`, `text:cp1252`, …): a byte-order mark first, then BOM-less UTF-16 by the share of NUL bytes, then strict UTF-8, then cp1252. |
| `text/html` | `html.parser`: scripts, styles, `noscript` and `template` dropped; block tags become line breaks; entities resolved. |
| CSV / TSV | `csv` module; each row written as tab-separated values. |
| `message/rfc822` | `email` (modern policy): From/To/Cc/Date/Subject lines, then the plain body, or the HTML body through the HTML reader. |
| RTF | A small reader of its own: control words, `\'hh` (cp1252), `\uN` (one fallback character skipped), and the groups that hold no body text (`fonttbl`, `colortbl`, `stylesheet`, `info`, `pict`, `object`, `header`, `footer`, and others) discarded. |
| DOCX | Paragraphs of `word/document.xml` (runs, tabs, breaks, text boxes after the paragraph that holds them), plus headers and footers. |
| XLSX | Shared strings, inline strings and values, one tab-separated line per row, a `# sheetN` line per sheet; empty rows dropped. |
| PPTX | Every paragraph on every slide, slides in numeric order, a `# slideN` line per slide. |
| ODT / ODS / ODP | `content.xml`: paragraphs and headings; table rows tab-separated. |
| PDF | **`pdftotext -layout -enc UTF-8` when poppler is on PATH.** Otherwise the built-in reader (below). |
| Images (PNG, JPEG, TIFF, BMP, GIF, WebP) | **`tesseract` when it is on PATH**; otherwise `unsupported`. |
Anything else (archives, executables, audio, video, unknown binaries) is
`unsupported`, with the MIME type named.
### The built-in PDF reader
Without poppler (on Windows, usually) PDFs are read by `pdftext.py`, which
walks the document far enough for text: the object table (a later definition
wins, so incremental updates are respected) and compressed object streams;
FlateDecode streams (a stream with any other filter is skipped); the page tree
in order with inherited resources; fonts through their `/ToUnicode` maps
(Type0 / Identity-H fonts, as browsers print them), or WinAnsi / MacRoman for
simple fonts without one; glyph widths, so a word placed by position still
gets its space; the text operators, and form XObjects. Pages are separated by
a form feed.
Encrypted files are refused. Output below 90% printable characters is refused
too, so a font without a usable map comes back as `unsupported` (`extractor:
pdf-builtin`, `needs pdftotext (poppler) on this machine`), never as byte salad
in the catalog.
It reads arbitrary files, so it reads on a budget per document: 60 s of wall
time, 256 MiB of decoded streams (64 MiB per stream), 5 million content
operators, 2,000 form invocations at most 6 deep, 200,000 CMap entries,
20 million tokens and 1 GiB lexed. Reaching a limit stops the read with the
pages read so far. Nothing in it raises out to the stage.
## Rules the stage enforces
- **External tools are sensors, not dependencies.** They are looked up on
PATH at run time, named in every record they produced (`extractor:
pdftotext`), and their absence is a recorded gap, never a failure.
`--no-external` (in a sweep, `FREEAGENT_NO_EXTERNAL=1`) uses the built-in
readers only. A manifest cannot name a tool or an address.
- **Nothing is written outside the output directory**, and every file is
written atomically.
- **Bounded reads.** One zip member is read to at most 64 MiB decompressed
(`ZIP_MEMBER_CAP`), so a hostile office file cannot balloon memory. Office
documents are parsed as a stream, each finished paragraph or row dropped
once read, and capped per document: 256 MiB of XML, 16 Mi characters of
text, 60 s, and 200,000 parts in one paragraph or row. Reaching a cap keeps
the text read so far and ends it with a line naming the cap. XML with a DTD
or an entity declaration is refused (Office files never carry one, and it is
how entity expansion gets in), as is XML that is neither UTF-8 nor UTF-16.
Tool calls time out at 300 s.
- **Errors are entries.** A corrupt document is one `error` record; the stage
finishes and reports the count.
- **Retry only when something changed.** An `unsupported` verdict is kept
while the machine's tool set is the same; install `pdftotext` and the next
sweep tries exactly those files again. `error` records are retried on every
sweep. Every record carries the extractor version (`v`, `EXTRACT_VERSION`,
now 2); raising it after a reader changes redoes every cached verdict and
text on the next sweep.
## Tests
`python3 -m unittest bastion_agent.test_extract bastion_agent.test_pdftext`.
`test_extract` runs every reader against a generated fixture (docx, xlsx,
pptx and odt built with `zipfile`, plain and Flate PDFs, RTF with hex and
Unicode escapes, plain and HTML e-mail, encodings), the Office streaming caps,
DTD refusal and UTF-16 input, the zip member cap, an oversized CSV field, the
whole stage over a mixed tree (statuses, counts, duplicates), reuse across
sweeps and its invalidation when content or the extractor version changes,
tool discovery and failure (mocked), and the walk-first guard. `test_pdftext`
builds browser-shaped PDFs with a small writer (Type0 fonts with ToUnicode
maps, per-run positioning, ligatures, inherited resources, form XObjects,
object streams) and hostile ones (reference cycles, deep nesting, malformed
tables and maps), and checks each budget.
The external-tool paths are tested with mocks; a real scanned PDF or image
through `pdftotext` or `tesseract` is not part of the suite.
## Not in this stage
Sanitizing (the next stage: the text store goes through the PII sterilizer),
chunking, and change detection beyond the content hash. Audio and video are not
extracted.
docs/manifest.md
+291
View File
@@ -0,0 +1,291 @@
# bastion agent job manifest: schema
The manifest is the contract between the agent and whoever directs it: what to
read, what to do with it, and who may query the result. Every part of the
agent validates against it. The loader is `bastion_agent/manifest.py`; the
signature is checked by the runtime ([runtime.md](runtime.md)) with
`bastion_agent/x509gate.py`.
## What a bastion agent is
A bastion agent is a program that runs on the owner's own machine and does the
data work that must stay there: walk a folder, extract text, sterilize personal
data, chunk, embed with a local model, catalog, and seal. A model on the
machine may serve as a sensor (for recall of personal data, embeddings and
summaries), and that is all: the same rule as the PII sterilizer
([pii-sterilization.md](pii-sterilization.md)), a deterministic program owns
the control flow and every change, and a model proposes, never acts.
The owner's host writes a **job manifest**, and it is signed (by the bastion,
in the managed install). The agent executes exactly that manifest and nothing
else. The output is a **catalog** (small, structured cards per file and per
directory) plus local vectors. The files themselves never move; a host
reasons over them by querying the agent. Sharing is a grant that lets another
host query this catalog: access moves, data does not.
## Rules the schema enforces
1. **Plaintext never leaves.** `sink.plaintext` is the constant `never`. Any
other value is a validation error, not a setting.
2. **Nothing runs that was not granted.** `mask` is the ceiling; every stage
must be covered by an op in the mask, or the manifest is refused before it
runs.
3. **Nothing leaves unsealed.** A catalog sent to the host, `seal_to`
recipients, or any grant requires the `seal` stage.
4. **No addresses.** The sanitizer and the embedder are found from the
machine's environment when the agent runs. The manifest never carries a
host:port.
5. **Signed by a CA-issued identity.** The runtime refuses a manifest whose
signing certificate does not chain to the root in `FREEAGENT_CA_ROOT`, and
refuses to judge at all without that root, because "signed by the key in
the certificate next to the manifest" proves nothing. The signing
certificate must carry exactly one CN. The loader checks the shape and
computes the signing payload; the runtime checks the signature and the
chain in the standard library (ECDSA P-256 / SHA-256), so the agent needs
neither openssl nor a third-party package on the machine.
## File layout
```yaml
freeagent: 1
id: <slug>
owner:
host: <slug> # the authoring host's cert CN
scope:
roots: [<abs-or-~path>, ...]
exclude: [<glob>, ...] # optional
follow_symlinks: false # optional
max_file_bytes: 268435456 # optional
mask: [read, search, index, summarize] # subset; 'read' is mandatory
stages: [walk, extract, sanitize, chunk, embed, catalog, seal] # ordered subset
sanitize: # required iff 'sanitize' in stages
mode: sterilize | flag
scope: <token scope> # optional, default = id
verify: false # optional
embed: # required iff 'embed' in stages
model: <name>
dim: <int>
chunk:
size_tokens: <int>
overlap_tokens: <int>
summarize: # optional; needs 'summarize' in mask
model: <name>
per: [file, directory]
sweep: 15m | hourly | daily | manual
sink:
catalog: local | host
vectors: local
plaintext: never
seal_to: [<key-id>, ...] # optional
grants: # optional, default []
- host: <slug>
ops: [search, read_range]
until: <YYYY-MM-DD>
signature: # optional in the file; runtime requires it
alg: ecdsa-p256-sha256
cert: <path>
value: <base64>
```
The first key, `freeagent: 1`, keeps the program's first name on purpose: the
bastion writes it into every manifest it generates, and renaming it would make
the agent refuse them.
## Fields
### Top level
| Field | Required | Rules |
|---|---|---|
| `freeagent` | yes | Schema version. Integer `1`. |
| `id` | yes | Lowercase slug `[a-z0-9-]+`, 1–63 characters. Names the catalog and is the default token scope for personal data. |
| `owner.host` | yes | Slug, 1–63 characters. The authoring host's certificate CN; the runtime checks the signing certificate's CN against it. |
Unknown keys anywhere are a validation error, so a misspelled `excludes:`
fails loudly instead of silently indexing everything.
### `scope`
| Field | Required | Rules |
|---|---|---|
| `roots` | yes | Non-empty list of paths. Each must be absolute (POSIX `/srv/a`, a Windows drive `C:\Users\me` or `C:/Users/me`, UNC `\\server\share`) or start with `~/` (or be exactly `~`), which expands to the running user's home on every OS. Relative paths are refused, so a manifest means the same thing wherever it lands. No duplicates. |
| `exclude` | no | List of glob strings, matched against the `/`-separated path relative to each root (`**` spans directories; a pattern with no `/` also matches any single path segment, so `node_modules` prunes it at any depth). Case-insensitive on Windows only. Default `[]`. See [walk.md](walk.md). |
| `follow_symlinks` | no | Boolean, default `false`. |
| `max_file_bytes` | no | Positive integer, default `268435456` (256 MiB). Larger files are catalogued by metadata only, never read. |
### `mask`: the ceiling
A non-empty list from `read`, `search`, `index`, `summarize`, with no
duplicates.
- `read`: **mandatory**; every stage reads.
- `search`: the agent may answer `search` queries over the vectors.
- `index`: the agent may write vectors and catalog entries. Required by
`embed` and `catalog`.
- `summarize`: a local model may write summaries. Required by the `summarize`
section.
There is no `write` and no `delete`: an agent never changes the files it
reads. Any other value is an error.
### `stages`: what runs, in order
A non-empty list, a subset of the canonical order, in canonical order:
```
walk → extract → sanitize → chunk → embed → catalog → seal
```
| Stage | Needs | Does |
|---|---|---|
| `walk` | (always first, always present) | The file system and metadata map: path, size, mtime, type, hash. The cheapest thing a host can plan from. |
| `extract` | `walk` | Text out of documents, code and (with a local OCR tool) images. |
| `sanitize` | `extract` | Personal data replaced by tokens in the text every later stage reads; never in the source files. |
| `chunk` | `sanitize` | Splits the sanitized text for embedding. |
| `embed` | `chunk`; mask `index` | Vectors from a local model, stored locally. |
| `catalog` | `sanitize`; mask `index` | Per-file and per-directory cards: what it is, when, entity counts, summary. **This is the curated dataset.** |
| `seal` | `catalog` | Encrypts the catalog to the owner's key and `sink.seal_to`. |
Out-of-order, duplicate or unknown stage names are errors, as is a stage whose
prerequisite stage is absent.
### `sanitize`
Present **if and only if** `sanitize` is in `stages`.
| Field | Required | Rules |
|---|---|---|
| `mode` | yes | `sterilize`: the full pipeline (rules, local-model recall, vault). `flag`: the rules pass only; deterministic, no model, and **not** "no personal data" (see [sanitize.md](sanitize.md)). |
| `scope` | no | Token scope, 1–63 characters. Default: the manifest `id`. The same (scope, type, value) gets the same token; a different scope gets a different token, so two catalogs cannot be linked through placeholders. |
| `verify` | no | Boolean, default `false`. A second, adversarial model pass (`sterilize` only; an error with `flag`). |
Where the sterilizer runs is never in the manifest; the agent reads it from
its environment and fails closed if it is absent.
### `embed`
Present **if and only if** `embed` is in `stages`.
| Field | Required | Rules |
|---|---|---|
| `model` | yes | Model name, 1–128 characters. |
| `dim` | yes | Integer 1–8192. The stage refuses to run if the local endpoint's vectors have another dimension. Vectors are stored per model, so a later model change never discards existing vectors. |
| `chunk.size_tokens` | yes | Integer 1–8192. |
| `chunk.overlap_tokens` | yes | Integer, 0 ≤ overlap < size. |
There is no endpoint field. Embedding is local, always.
### `summarize`
Optional. Requires `summarize` in `mask` and `catalog` in `stages`.
| Field | Required | Rules |
|---|---|---|
| `model` | yes | Local model name, 1–128 characters. |
| `per` | yes | Non-empty subset of `file`, `directory`. |
### `sweep`
One of `15m`, `hourly`, `daily`, `manual`.
### `sink`
| Field | Required | Rules |
|---|---|---|
| `catalog` | yes | `local`: the sealed catalog stays here, and the host queries it. `host`: the sealed catalog is sent to the owner's host, so it can plan while this machine is asleep. Requires the `seal` stage. |
| `vectors` | yes | `local`, the only value. |
| `plaintext` | yes | `never`, the only value, ever. It is there so the rule is written down in every manifest and can be searched for, not remembered. |
| `seal_to` | no | List of recipient keys, 1–128 characters each, no duplicates. Requires the `seal` stage. The owner's key is always added by the agent and need not be listed. |
### `grants`
An optional list, default `[]`. Each entry lets another host query this
catalog. Requires the `seal` stage.
| Field | Required | Rules |
|---|---|---|
| `host` | yes | Slug, 1–63 characters: the grantee host's certificate CN. Must not equal `owner.host`. Unique across grants. |
| `ops` | yes | Non-empty subset of `search`, `read_range`. `search` requires `search` in the mask; `read_range` requires `read` (always present). |
| `until` | yes | `YYYY-MM-DD` expiry, a real calendar date. |
Revoking is removing the entry and re-signing; `serve` stops serving that host
on its next request. The grantee's client certificate must also chain to
`FREEAGENT_CA_ROOT` ([serve.md](serve.md)).
### `signature`
Optional in the file, so a manifest can be written, validated and reviewed
before it is signed. **The runtime refuses an unsigned manifest.**
`python3 -m bastion_agent validate --require-signed` enforces that here.
| Field | Required | Rules |
|---|---|---|
| `alg` | yes | `ecdsa-p256-sha256`. |
| `cert` | yes | Path to the signing host's PEM certificate, 1–256 characters, read relative to the manifest. |
| `value` | yes | Base64 signature over the signing payload. |
**Signing payload** = the manifest with `signature` removed, serialized as
canonical JSON (sorted keys, no whitespace, UTF-8, dates as ISO strings).
`bastion_agent.signing_payload(doc)` produces it and `signing_digest(doc)` its
SHA-256 hex, so the signer and the runtime agree byte for byte. Any ECDSA P-256
/ SHA-256 signer will do; the managed install signs in the bastion and records
the result with `sign --value`. By hand, with openssl:
```
python3 -m bastion_agent payload manifests/home-documents.yaml > payload.json
openssl dgst -sha256 -sign <host key> payload.json | base64 -w0
python3 -m bastion_agent sign manifests/home-documents.yaml --value <that base64> --cert <cert path>
```
(`sign --key <host key> --cert <cert path>` does the same in one step.)
The gate, in the standard library: exactly one CN, equal to `owner.host`; the
certificate inside its validity window and either identical to a certificate
in `FREEAGENT_CA_ROOT` (the managed install: its certificate is its own root,
and its self-signature must verify) or signed directly by one that is a CA
(basicConstraints, and keyCertSign when the root has a keyUsage extension);
and the signature over this payload. It is a one-level PKI only: longer chains
are not built.
## Loader
`bastion_agent/manifest.py` (Python 3, PyYAML). Library API:
```python
from bastion_agent import load_manifest, signing_payload, signing_digest
m = load_manifest("manifests/home-documents.yaml") # raises ManifestError listing every problem
m.id, m.owner_host, m.scope.roots, m.mask, m.stages # typed, frozen
m.sanitize, m.embed, m.summarize # None when the section is absent
m.sink.catalog, m.grants, m.signed # grants: tuple of Grant
signing_payload(m.raw) # bytes the signature covers
```
CLI (from the repository root):
```
python3 -m bastion_agent validate [--require-signed] <manifest.yaml> [...]
python3 -m bastion_agent show <manifest.yaml> # stages, mask, grants, signed?
python3 -m bastion_agent payload <manifest.yaml> # canonical signing payload to stdout
```
Validation collects **all** errors before failing, so one pass shows
everything wrong with the file. `python3 -m bastion_agent validate manifests/*.yaml`
validates every example. `python3 -m unittest bastion_agent.test_manifest`
runs the loader's tests, which also load `manifests/home-documents.yaml`, so
the example can never drift from the schema.
## Deliberately absent
- A `write` op. An agent reads; adding a way to change files is a decision of
its own, not a field.
- An endpoint or address field anywhere, for the reason in rule 4.
docs/packaging.md
+183
View File
@@ -0,0 +1,183 @@
# bastion agent packaging: one file for three operating systems
## The artifact
`python3 -m bastion_agent.build --out dist` → `dist/bastion-agent-<version>.pyz`
and `.sha256`.
A **zipapp**, the standard library's own single-file format: one file of about
150 KB that runs on any Python 3.8 or later on Mac, Windows or Linux:
```
python3 bastion-agent-<version>.pyz doctor
python3 bastion-agent-<version>.pyz sweep manifest.yaml --once
python3 bastion-agent-<version>.pyz serve manifest.yaml
```
Inside: the `bastion_agent` package (every stage, the scheduler, the
endpoint; no tests, and neither `build.py` nor `verify_release.py`, which are
tools for making and checking a release), the `pii` package (the sterilizer; no
tests, no Dockerfile), a vendored **pure-Python PyYAML** (MIT, its notice
included; the C accelerator is left out so one file runs everywhere), which is
the one import the manifest loader needs beyond the standard library, and the
repository's `LICENSE`, so the license travels with the program. The version
comes from `bastion_agent/VERSION`, read with `pkgutil` so it works from inside
the zip.
`bastion_agent/VERSION` on `main` is the next release, not necessarily the one
bastions run. A merged change waits there until it is published, and the
published version is whatever <https://privatae.ai/bastion-agent/version.json>
names at the moment: a bastion installs that version and no other. Between a
merge and a publish, `main` is ahead of the manifest; when a second release is
merged before the first is published, only the later one is published.
**Reproducible.** Fixed member timestamps, sorted members, no bytecode, and
the vendored PyYAML pinned to one release (`PYYAML_VERSION`; the build refuses
any other): two builds of the same tree are byte-identical on any machine, so
the `.sha256` is a real statement about the file, and anyone can rebuild a
release and compare (`python3 -m bastion_agent.verify_release`, in the README).
`test_build` asserts identical bytes across two builds.
`age` stays what it is, a single external binary, because the standard library
cannot do X25519 or ChaCha20. The manifest gate does **not** need `openssl`: it
reads the certificate and verifies the ECDSA P-256 signature in the standard
library (`x509gate.py`), so the agent runs on a machine without it (Windows
ships none). `openssl` is needed only to sign with `sign --key` and for
`SEAL_SIGN_KEY` seal signatures. `doctor` reports both.
## doctor
`bastion-agent doctor [--json]` is the preflight; it changes nothing:
| Check | Used by | Required? |
|---|---|---|
| Python ≥ 3.8, PyYAML | everything | yes |
| `age` | seal | yes |
| `openssl` | `sign --key`, seal signatures | no: the gate verifies in the standard library |
| `pdftotext`, `tesseract` | extract | no: without them the gaps are recorded |
| `gliner` package | sanitize | no: the laptop recall backend |
| `EMBED_URL` (default `http://127.0.0.1:11434`), and `PII_ADDR` / `SUMMARIZE_URL` when set | embed / sanitize / catalog | no: reachable? Any HTTP answer counts, so a proxy that allows only the embedding path (and answers 403 on `/`) is reachable |
| `FREEAGENT_OWNER_RECIPIENT` set; `FREEAGENT_CA_ROOT`, `FREEAGENT_TLS_CERT`, `FREEAGENT_TLS_KEY` set and the files present | seal / sweep / serve | no, for the preflight: the stages that need them refuse without them |
The exit code is 1 only when a required item is missing; optional gaps are
printed as `opt`.
## Installers (a hand install)
The bastion installs the agent itself from a grant; these are for running it
by hand.
- `install.sh <pyz>`: Linux and macOS, POSIX sh, no root, no downloads. It
checks for `python3` 3.8 or later, verifies the `.sha256` when it sits next
to the `.pyz` (`sha256sum` or `shasum`), copies to
`~/.local/bin/bastion-agent.pyz`, writes a `bastion-agent` wrapper (and
repoints an existing `freeagent` wrapper, so an earlier hand install keeps
working), runs `doctor`, and prints the next step. `BASTION_AGENT_BIN_DIR`,
or else `FREEAGENT_BIN_DIR`, overrides the directory.
- `install.ps1 <pyz>`: Windows PowerShell, no administrator. The same checks
with `Get-FileHash`; installs to `%LOCALAPPDATA%\bastion-agent`, writes
`bastion-agent.cmd`, adds the directory to the user's PATH, and runs
`--version` and `doctor`.
Neither installer fetches anything. Get the `.pyz` and its `.sha256` from the
download address, then run one command (Linux or macOS):
```
V=<version from https://privatae.ai/bastion-agent/version.json>
curl -fsSLO https://privatae.ai/bastion-agent/dl/bastion-agent-$V.pyz && \
curl -fsSLO https://privatae.ai/bastion-agent/dl/bastion-agent-$V.pyz.sha256 && \
curl -fsSL https://privatae.ai/bastion-agent/install.sh | sh -s -- bastion-agent-$V.pyz
```
## The download address
`bastion_agent/publish.sh` publishes to every edge server named in its
environment, each serving from its own disk, at `/bastion-agent/*`, with
backups and dotfiles refused. The legacy `/freeagent/*` path is retired (see
"The legacy name" below):
```
https://privatae.ai/bastion-agent/version.json version, built_from, python_floor, files (url, sha256, bytes), installers, notes
https://privatae.ai/bastion-agent/version.json.<h>.sig ed25519 signature; <h> = first 16 hex of version.json's sha256
https://privatae.ai/bastion-agent/dl/bastion-agent-<v>.pyz the artifact
https://privatae.ai/bastion-agent/dl/bastion-agent-<v>.pyz.sha256
https://privatae.ai/bastion-agent/install.sh and install.ps1
```
The manifest is signed with the bastion's release key, which stays on the key
holder's own computer. A publish therefore runs twice: the first run builds
and prints the `release-sign sign bastion-agent <version> <sha256>` command to
run there, and the second, with `SIGNATURE=<the line it printed>` and
`BASTION_TREE` pointing at a privatae-bastion checkout, checks the signature
against the keys that checkout pins and the keys the published bastion pins,
and only then publishes. What is signed is a short statement naming the kind
of release, its version and the manifest's sha256. A bastion from 0.41.0 on
fetches `version.json.<h>.sig`, refuses a manifest that does not verify, and
never replaces an installed agent with an older version.
The script builds reproducibly, runs the built `.pyz` once (`--version`),
refuses to publish a version that is not newer than the one already served
(`ALLOW_REPUBLISH=1` overrides it deliberately), uploads to every edge, then
checks each edge's bytes against the manifest it serves. Where it publishes
is not written into the script: the edges, the ssh key, the web root and the
backup directory come from its environment, and it stops if one is missing.
The manifest's `notes` link to the tag `v<version>` on the public repository,
which marks the source of that release. `publish.sh` does not make the tag; it
is made when that release's source is published to the public repository.
### The legacy name
The agent was first published as freeagent, at `privatae.ai/freeagent/`, and
a bastion older than 0.20.0 knows only that address and the
`freeagent-<v>.pyz` file name. Bastions from 0.20.0 read the new address first
and fall back to the legacy one; from 0.42.0 they read only the new one.
The legacy address is retired: no edge has the `/freeagent/*` route, and
`https://privatae.ai/freeagent/version.json` answers 404. A bastion that
cannot fetch the manifest keeps the agent it has installed, so a bastion older
than 0.20.0 keeps the agent it has and gets no newer one. What would strand a
machine is the other case, a legacy manifest that answers but does not list
the legacy file name, and a retired address cannot produce it.
`publish.sh` still checks on every run, on every edge (the route in the Caddy
configuration each edge has loaded) and at the public address. With no route
anywhere and a 404 in public it prints, per edge, "no /freeagent route" and
publishes the new file name only. It publishes both names again, the same
bytes under each, if any edge still has the route, if any edge cannot be
asked (ssh fails or times out, or the configuration Caddy has loaded cannot
be read), or if the public address answers anything but 404 (no answer
included), so that address could not serve a manifest without the name an
old bastion asks for.
The names inside the agent that keep the word freeagent (the `FREEAGENT_*`
environment variables and the manifest's `freeagent: 1` field) are stable
names, not part of this retirement: the bastion sets and writes them, so
renaming one would have to change the bastion and the agent in step, for no
change in behaviour.
## What this is not
A native binary. PyInstaller or Nuitka make one per OS, but each has to be
built on that OS, and none is produced; it would be built from the same
`members()` list. Until then the `.pyz` and a system Python are the artifact.
Python 3.8 or later is on every Linux and current Mac (with the Xcode command
line tools or Homebrew) and one Store install away on Windows, and the floor
test keeps the code there.
## Tests
`python3 -m unittest bastion_agent.test_build` covers byte-identical builds;
the contents (the entry point, both packages, VERSION, the vendored YAML and
its notice; no tests, no `.so` or `.pyc`, no `build.py`, `verify_release.py`
or Dockerfile; every member at the fixed time; under 1 MB); the `.pyz`
answering `--version` and `doctor --json` and validating the example manifest;
**a full sweep from the `.pyz` in a scratch directory with `PYTHONPATH=""`**
(nothing from the repository on the path) producing a catalog with sanitized
heads; the `.agent` default output directory; and the doctor's checks against
unreachable and missing inputs.
This repository has no CI. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the
checks to run before proposing a change, and `publish.sh` builds the release
and runs it before anything is uploaded. The installers are not exercised by
the test suite.
docs/pii-sterilization.md
+226
View File
@@ -0,0 +1,226 @@
# PII sterilization
The `pii` package finds personal data in text and replaces it with
placeholder tokens, keeping the real values in a local vault. The agent's
`sanitize` stage runs it in process ([sanitize.md](sanitize.md)); it also
runs on its own as a small HTTP service. It is a deterministic program with
a model as one of its sensors: the program owns the control flow and every
change to the text, and the model only proposes.
## Purpose
Text that is going to leave the machine (a catalog, a chunk returned to a
host, context sent to a cloud model) first passes through `/sterilize`:
personal data is replaced with stable placeholder tokens, the real values stay
in a local vault, and `/rehydrate` puts them back, locally, on demand. A cloud
model can then work over the text without the values in it. Patterns alone
cannot do this: "my daughter goes to Lincoln Elementary" contains no pattern,
only meaning.
## Pipeline (`pii/pipeline.py`)
1. **Rules pass** (`detect.py`): deterministic detectors (below) for the
anchors a pattern can find. They never depend on a model. Overlaps are
resolved by type priority, then by length (an IP address is not a dotted
phone number).
2. **Recall pass** (`recall.py`): a model proposes verbatim spans with a type
from a closed vocabulary: names, addresses, employers, schools, medical
facts, credentials. Which model is chosen when the service starts (below).
3. **Grounding**: every proposed span must occur exactly in the input, be at
least two characters, have a known type and not contain a token or a
token's bare name (`PII_PERSON_1`), or it is dropped and counted. An invented span can never mint a token or touch the
text.
4. **Replacement**, by the program only: longest value first, every
occurrence. The model never rewrites the text, so a model that misbehaves
can cost recall but cannot corrupt the rest of the text. Replacement is by
exact substring, so a value is also replaced where it appears inside a
longer word ("Garcia" in "Garcias", "Marisol" in `Marisol_resume.pdf`): a
missed inflection would be a leak, and an extra match only costs
readability. No match may overlap a placeholder already in the text.
Matches that overlap each other, including a value that overlaps itself
("aba" in "ababa"), become one placeholder over their union, typed by the
longer value, so no fragment of either is left in the clear.
Placeholder-shaped text in the input is first replaced by a placeholder of
its own, so rehydrating gives back the literal text. Documents the agent
reads are raw files, so every such literal in them is escaped. Text sent to
the service may be the service's own earlier output (a stored conversation
sent back on every save), so there a literal the vault holds for the scope
is taken to be that token and passes through unchanged, and only a literal
the vault does not hold (`[PII_FOO_99]`) is escaped. A caller can ask for
every literal to be escaped with `escape_held_literals: true`. The cost of
replacement grows with the number of occurrences times the number of
placeholders in the text.
5. **Residual loop**: the rules pass runs again on the output until it is
clean, at most three rounds; residuals are replaced, not only reported.
6. **Optional verify** (`verify: true`): a second, adversarial model pass over
the sterilized text; its grounded findings go through replacement too.
If the recall pass fails, `/sterilize` answers 502: fail closed. Text without
the recall pass never leaves as "sterilized".
## Tokens, scopes, vault
- Tokens look like `[PII_<TYPE>_<n>]`: plain ASCII, they survive a round trip
through a cloud model, and the model can reason about `[PII_PERSON_1]`
across a conversation.
- **Stable within a scope**: the same (scope, type, value) always gets the
same token. **Isolated across scopes**: the same value in another scope gets
a different token, so two conversations or catalogs cannot be linked through
shared placeholders. Stability is per value as replaced, not per detection:
where a model's span overlaps a rule's hit (a phone number inside an address
the model proposed), the two become one placeholder for the combined text,
typed by the longer value. That placeholder differs from the one the phone
number gets on its own elsewhere. If a later run proposes the longer span
with a different type, the same bytes get a different placeholder. The
caller chooses the scope (default `default`; the agent uses the manifest's
`sanitize.scope`).
- The vault is SQLite, created 0600 under umask 077. Rows are never deleted:
the next number in a scope is the count of its rows plus one, so after a
delete that number either names a token that still exists (and the insert
fails) or, if the newest row was deleted, gives its token to a different
value. A delete feature would first need a separate sequence. In the service, every sterilize and rehydrate also
writes an audit row; `GET /scope/<id>` lists tokens, types and creation
times, never values.
## The service (`pii/service.py`)
Standard library only: the process holds the most sensitive data on the
machine, and every dependency is attack surface.
| Route | Auth | Does |
|---|---|---|
| `GET /health` | none | Liveness. |
| `GET /scope/<id>` | key | The token inventory of a scope (no values). |
| `POST /sterilize {text, scope?, verify?, escape_held_literals?}` | key | The pipeline; returns `{text, findings, stats}`, with `stats.recall` naming the recall backend. `escape_held_literals: true` escapes every placeholder-shaped string in `text`, even one the vault holds; the agent sends it for documents. |
| `POST /rehydrate {text, scope?}` | key | Values back in place of known tokens; returns `{text, restored, unknown_tokens}`. |
| `POST /flag {text}` | key | The rules pass alone: `{flagged, counts}`. No model, no vault rows, no audit. |
Request bodies are limited to 2 MiB.
`stats` in a `/sterilize` answer counts what this call did: `rules` (rule
findings), `llm` (grounded model proposals), `llm_ungrounded` (proposals
dropped by grounding), `residual_tokenized` and `verify_tokenized` (values
those passes actually replaced), and `literal_tokens_escaped`
(placeholder-shaped strings in the input that were escaped, each distinct
string counted once per call). Escaped literals appear in neither `findings`
nor the per-type counts. `stats.recall` names the recall backend, and
`stats.escape_held_literals` echoes which literal rule was applied, so a
caller that asked for it can check.
**A clean `/flag` does not mean "no personal data".** It sees only what the
rules find, never contextual data such as a name; anything that is kept or
sent on goes through `/sterilize`.
Configuration, from the environment:
| Variable | Default | Meaning |
|---|---|---|
| `PII_API_KEY` | (none) | Required. Without it the service refuses to start: `/rehydrate` returns real values, so there is no unauthenticated mode. (`PII_ALLOW_NOAUTH=1` exists for the unit tests only.) |
| `PII_BIND` | `0.0.0.0:8091` | The address the service listens on; this is its exposure control. |
| `PII_DB` | `/data/vault.db` | The vault. |
| `PII_RECALL` | `llm` | The recall backend (below). |
| `LLM_URL` | `http://llm:8080` | The model server for `llm`. |
| `PII_LLM_TIMEOUT` | `300` | Seconds per model call. |
Two ways to run it, the same code:
- **Container**: `pii/Dockerfile` (a pinned `python:*-slim` image, no pip
install; the vault on a mounted `/data`).
- **systemd**: `pii/pii.service`, with install steps in its header. It runs
with `DynamicUser`, a private 0700 state directory for the vault
(`/var/lib/privatae-pii/vault.db`), loopback bind (`127.0.0.1:8091`), and
the key in a 0600 environment file.
The bastion can make a service on the machine reachable to the owner's host:
with `PRIVATAE_PII_ADDR` set to the service's address, it forwards to it
through its tunnel and reports in its heartbeat that the service is there and
whether it answers. The bastion passes requests through without reading them
and never holds the API key; the caller sends it.
## Recall backends (`pii/recall.py`)
The recall pass is a pluggable sensor, chosen by whoever runs the service
with `PII_RECALL` when it starts, never per request, so a request cannot
downgrade it. Whichever sensor proposes, grounding drops any span that is not
exactly in the input and the program makes every replacement, so swapping the
sensor changes recall, never the integrity rules.
| `PII_RECALL` | What | Where it fits |
|---|---|---|
| `llm` (default) | A llama.cpp-compatible server's `/v1/chat/completions` at `LLM_URL`, with the output constrained by a JSON schema (`response_format: json_schema`), temperature 0, thinking turned off per request (`llmclient.py`). | A machine that can reach a model server. Best recall. |
| `gliner` | GLiNER (Apache-2.0), zero-shot named-entity recognition in this process, on CPU. Its labels are plain-English phrases mapped onto the closed type vocabulary (`recall.GLINER_LABELS`); `OTHER` is deliberately not a label. `GLINER_MODEL` (default `urchade/gliner_multi_pii-v1`), `GLINER_THRESHOLD` (0.5), `GLINER_VERIFY_THRESHOLD` (0.3, the more aggressive second pass). | A laptop with no GPU and no model server. |
Fail closed when it starts: `PII_RECALL=gliner` without the `gliner` package,
a model that will not load, or an unknown backend name stops the service before
it serves a request. (The `llm` server is not probed at start; an unreachable
one fails each `/sterilize` with 502.)
Why GLiNER and not Presidio: Presidio (MIT) is the usual framework, but it
brings spaCy and optionally torch into the process that holds the data. GLiNER
is one optional import, loaded only when chosen. What was taken from Presidio
instead is its list of recognizers, as deterministic rules. The cost is real:
the `gliner` package pulls in torch (a CPU build), hundreds of megabytes on
the machine.
## The rules pass (`pii/detect.py`)
Every detector is anchored by a checksum, a prefix or a fixed shape; none
depends on a model.
- **E-mail, phone, SSN, IPv4/IPv6, MAC.**
- **Cards** → `CREDIT_CARD`, checked with Luhn.
- **IBAN** → `BANK_ACCOUNT`: ISO 13616 mod-97 plus a per-country length
table. It runs before the card detector, so an IBAN's digits cannot be
claimed as a card number.
- **US ABA routing number** → `BANK_ACCOUNT`: the weighted checksum and a
`routing`, `ABA`, `RTN` or `transit` cue within 40 characters before it
(one random 9-digit string in ten passes the checksum alone).
- **Bitcoin legacy and P2SH addresses** → `BANK_ACCOUNT`: base58check
(double SHA-256). Bech32 (`bc1…`) addresses are not detected.
- **Secrets** → `CREDENTIAL`: AWS access key ids, GitHub classic and
fine-grained tokens, Slack tokens, Google API keys, Stripe keys, GitLab
tokens, OpenAI-style `sk-` keys, JWTs, and whole PEM private-key blocks (one
finding per block, at the highest priority, so the digits inside are never
claimed piece by piece).
## Portability
The package parses at the Python 3.8 floor shared with the agent
(`test_syntax_floor.py`), with `typing` generics and no newer standard-library
calls, so the systemd form runs on older interpreters as it is.
## Tests
- `python3 -m unittest pii.test_pii`: detectors, vault stability and
isolation, grounding, longest match first, the HTTP wire format against a
stub llama.cpp server, auth, and fail-closed configuration. No model is
needed.
- `PII_API_KEY=<key> pii/verify-live.sh [host:port]` runs eight checks
against a running service and its real model: rules-class sterilization,
contextual recall (school, address, doctor, medical), a round trip back to
the original, token stability, scope isolation, untouched non-personal text,
auth, and health.
## Limits
- **Sterilization narrows what is missed; it cannot guarantee nothing is.** The
recall pass is probabilistic. The layers (rules, model, residual loop,
optional verify) reduce misses but cannot make them impossible. Anything
that tells a person their data is protected has to say so.
- **The vault is not encrypted at rest.** The controls are the file's 0600 mode,
its directory, and the bind address.
- **Rows are never deleted** (see above), so there is no erasure yet.
- **Placeholders are plain tokens.** Surrogate values (invented names in place
of real ones, for fluency) are not done.
- **Through the service, a quoted placeholder that names a live token is read
as that token.** If text sent to `/sterilize` quotes `[PII_PERSON_1]` and the
vault holds `[PII_PERSON_1]` for the same scope, rehydrating gives the
vault's value in its place. The placeholder cannot be told apart from one
the service minted earlier, and treating it as a token is what lets
already-sterilized text be sterilized again without change. A quoted
placeholder the vault does not hold is escaped and comes back as written.
The agent's document stage is not affected: it asks for every literal to be
escaped.
- **One model call per `/sterilize`**, two with `verify`; the agent makes one
per 20,000-character chunk. That suits writes to context and memory; an
interactive path would want smaller calls or a smaller model.
docs/runtime.md
+131
View File
@@ -0,0 +1,131 @@
# bastion agent runtime: the `sweep` scheduler and the manifest gate
`sweep` is the agent proper: the loop that checks the manifest and runs its
stages on the manifest's cadence, on the owner's machine. It is
`bastion_agent/run.py`, with the signature check in `bastion_agent/x509gate.py`,
standard library only apart from the manifest loader's PyYAML.
`python3 -m bastion_agent sweep <manifest> [--out DIR] [--once] [--allow-unsigned]`
Each cycle:
1. **Read the manifest again.** A re-signed manifest takes effect on the next
sweep: a revoked grant, a new root, a changed cadence.
2. **The gate.** The manifest must carry a signature that verifies: ECDSA
P-256 over the canonical payload (`signing_payload`, the same bytes the
`payload` command prints), from a certificate with exactly one CN, equal to
`owner.host`, that chains to `FREEAGENT_CA_ROOT`. The root is required:
without one the gate refuses rather than trust the certificate beside the
manifest. Anything else (unsigned, the wrong key, edited after signing,
someone else's certificate, a certificate outside the CA, a certificate with
two CNs) and **nothing runs**. The refusal is appended to `run.jsonl` like
any other run; a manifest too broken to have an id is logged under
`build/freeagent/refused/` unless `--out` is given. `--allow-unsigned` is
for development only: the CLI help says so in capitals, and it is written
into every run record it touches.
3. **The stages, in the manifest's order.** The first failure stops the run,
so a later stage never consumes a half-finished earlier one. A stage that
completes but reports `stopped` (sanitize or catalog failing closed) counts
as failed.
4. **Record and wait.** One line is appended to `run.jsonl`, `run.state.json`
is rewritten, and the loop sleeps until the next run is due, or exits for
`manual` (and after `--once`).
Signing, on the host's side: `python3 -m bastion_agent sign <manifest> --cert <cert> --key <key>`
signs with openssl and writes (or replaces) the signature block;
`--value <base64>` records a signature made elsewhere instead. The bastion
signs in its own process with the key it minted for the agent and records it
with `--value`. The certificate path is recorded as given and read relative to
the manifest. [manifest.md](manifest.md) has the details.
## Where an agent lives: `<root>/.agent/`
One agent per folder, and the agent lives with its folder. For a manifest with
one root, the commands that run a single stage, `sweep` and `serve` default
their output directory to `<root>/.agent/`: the signed manifest and every prepared output
(walk map, extract index, sanitized store, chunks, vectors, catalog, run log,
sealed bundles). Copy the folder and the agent comes with it; fork it and you
have a new one. The walker always excludes `.agent/`, so an agent never
indexes its own output.
What is in `.agent/`: the **extract store is the plaintext of every file in
scope** (a PDF or DOCX becomes a flat `.txt`), and the sanitized store is that
text with personal data tokenized (only the rules-pass anchors, unless the
manifest uses `sterilize`). That is the same trust boundary as the folder
itself, since the source files are already there, but a sync provider now
receives a second, flattened, indexable copy. If that matters for a folder,
exclude `.agent` from sync or keep the output elsewhere (`--out`).
The identity key is never in `.agent/`: it is wherever `FREEAGENT_TLS_KEY`
points. The PII vault (plaintext values) is in `.agent/sanitize/vault.db` for
an agent run by hand. A Documents folder is often cloud-synced, so set
`FREEAGENT_VAULT_DIR` to keep the vault out of it; the bastion always sets it
to its own state directory. The certificate is public and may sit anywhere.
**One machine per folder.** The manifest is signed by the machine's own key,
and the gate proves "signed by this key", not "the newest". If a synced folder
is served from two machines, or a sync conflict restores an earlier signed
manifest, the endpoint serves the older terms until the manifest is rewritten.
Run one agent per folder, on one machine.
## Cadence and backoff
| `sweep` | Period | On failure |
|---|---|---|
| `15m` / `hourly` / `daily` | 900 / 3600 / 86400 s from the run's start | Retry after 60 s, 120 s, 240 s … capped at the period; `consecutive_failures` counts up and resets on success. |
| `manual` | | One run, then exit. |
Waits are taken in naps of at most 30 s, so SIGTERM or SIGINT lands quickly; a
stop request lets the current run finish and never interrupts a stage.
## The lock
`<out>/sweep.lock`, created atomically (`O_EXCL`) with the pid and time
inside. A second sweep on the same output directory is refused, with the holder
named. A lock is stale only when its holder is provably gone: on POSIX the pid
decides, and a live pid keeps its lock however long its sweep runs; only where
the pid cannot be checked (Windows) or the file cannot be read does age decide,
by the file's mtime, after 12 hours. A reclaimed lock is noted in the run
record.
## Files
| File | Content |
|---|---|
| `run.jsonl` | Append-only, one record per run: `v`, `started`, `finished`, `seconds`, `manifest`, `out`, `manifest_id`, `manifest_digest`, `gate` (`cn`, `cert`, `chained`, or `bypassed` with the reason), `stages[]` (`stage`, `status`, `seconds`, a few counts, `error`), `ok`, `error` (on failure), `allow_unsigned`, `lock_reclaimed` (when it happened). |
| `run.state.json` | The latest state: `last_started`, `last_finished`, `last_ok`, `last_error`, `consecutive_failures`, `sweep`, `next_due`, `pid`, `runs_this_process`. What a health check reads. |
| `sweep.lock` | Present only while a sweep runs. |
The stages read their settings from the environment, never the manifest:
`FREEAGENT_OWNER_RECIPIENT`, `FREEAGENT_VAULT_DIR`, `FREEAGENT_CA_ROOT`,
`FREEAGENT_NO_EXTERNAL=1` (built-in readers only), `EMBED_URL` / `EMBED_API`,
`PII_ADDR` / `PII_API_KEY` / `PII_RECALL` / `LLM_URL`, `SUMMARIZE_URL`, and
`SEAL_SIGN_KEY`.
## Running it as a service by hand
The bastion runs the agent itself. To run one without a bastion on Linux,
`bastion_agent/bastion-agent.service` is a systemd **user** unit: the sweep
runs under the owner's account with their permissions, reads its environment
from `~/.config/bastion-agent/env` (create it 0600, as the unit's header
shows), writes to `~/.local/share/bastion-agent/out`, and runs with
`UMask=0077`, `NoNewPrivileges` and `PrivateTmp`, restarting 60 s after a
failure. There is no launchd or Scheduled Task file yet; on a Mac or Windows
machine run the same command from either.
## Tests
`python3 -m unittest bastion_agent.test_run bastion_agent.test_x509gate`.
`test_run` covers the gate (sign then verify; re-signing replaces the block; an
edit after signing refused; a CN mismatch refused; a certificate with two CNs
refused; the CA chain accepted against the right root and refused against
another; unsigned refused unless `--allow-unsigned`, and then recorded), runs
(stages in order with counts logged; a failing stage stops the run and later
stages never execute; an invalid manifest recorded as an error; the lock
refusing a second holder and reclaiming a dead one; `next_due` arithmetic
including the backoff cap), and the loop with a fake clock (`manual` runs
once; `15m` runs three times, sleeping exactly two periods in naps of at most
30 s, with `next_due` right; hourly failures retry after 60 s then 120 s with
the counter at 3). `test_x509gate` runs the gate with openssl removed from
PATH, through the CLI with no openssl, and against bit flips, expiry and a
foreign CA with the same name.
docs/sanitize.md
+134
View File
@@ -0,0 +1,134 @@
# bastion agent `sanitize` stage: personal data out of the text
The sanitize stage runs the extracted text through the PII sterilizer
(`pii/`, described in [pii-sterilization.md](pii-sterilization.md)) and keeps
the result in a store of its own. Every later stage reads that store, never the
extracted text. It is `bastion_agent/sanitize.py`.
## What it produces
`python3 -m bastion_agent sanitize <manifest.yaml> [--out DIR] [--require-signed]`
reads the extract index and store and writes:
- **`sanitize/<aa>/<sha256>.txt`**: the sanitized text store. Personal data is
replaced by stable `[PII_TYPE_n]` tokens. **From this stage on, later
stages read only this store.** The extract store keeps the plaintext and
never leaves the machine.
- **`vault.db`**: the token-to-value map (in-process mode only), created 0600
under umask 077. It is the only way back from a token to a value, and it
stays on the machine. It is `<out>/sanitize/vault.db`, or
`$FREEAGENT_VAULT_DIR/vault.db` when that is set: the output folder may be
synced to a cloud drive, the vault must not be, so the bastion sets
`FREEAGENT_VAULT_DIR` to its own state directory.
- **`sanitize.jsonl`**: one record per extracted file: `status`, `mode`,
`stats` (counts from the pipeline, summed over the file's chunks: rule hits,
recall hits, dropped proposals, residuals and verify hits replaced, and
`literal_tokens_escaped`, so a literal that appears in two chunks counts
twice), `types` (token counts per type), `chars`, `reused`,
`error`. **Never a value.** A test searches the index and the summary for
the fixture's personal data and fails if any appears.
- **`sanitize.summary.json`**: counts, token totals per type, the sterilizer
that ran (`flag:rules-only`, `sterilize:llm=…`, `sterilize:gliner=…@0.50`,
`service:http://…`), and `stopped` when the stage failed closed.
| `status` | Meaning |
|---|---|
| `ok` | Sanitized text written (or kept from an earlier run: then `reused` is true). |
| `skipped` | The extract stage had no text for it (unsupported, error, empty, too large). |
| `error` | A failure that is not the recall pass (the store unreadable, an HTTP error from the service other than 502). The stage continues. |
| `not_run` | The stage stopped (fail closed) at or before this file. |
Exit code: `0` clean; `1` the manifest is invalid, `sanitize` is not in its
stages, there is no extract index, or the stage **stopped**; `2` completed with
`error` records.
## Modes (manifest `sanitize.mode`)
- **`sterilize`**: the full pipeline: the rules pass, the recall pass,
grounding, replacement by the program, the residual loop, and the optional
adversarial `verify` pass. Contextual personal data (names, schools,
employers, medical facts) is found by the recall pass.
- **`flag`**: the rules pass **only**: deterministic, no model, milliseconds.
It replaces the anchors a pattern can find: e-mail addresses, phone
numbers, SSNs, cards, IBANs, routing numbers, crypto addresses, IP and MAC
addresses, API keys, private-key blocks, JWTs. **It does not find a name.**
A flag-mode catalog is not free of personal data, and the manifest doc says
so too.
## Where the sterilizer runs
It is chosen from the environment, never the manifest (a manifest carries no
addresses).
| Condition | Where | Vault |
|---|---|---|
| `mode: flag` | In this process, rules only. `PII_ADDR` is ignored. | `vault.db` as above |
| `mode: sterilize`, no `PII_ADDR` | In this process: `pii` inside the agent, the recall pass from `PII_RECALL` (`llm` → the server at `LLM_URL`; `gliner` → an in-process CPU model). The laptop case. | `vault.db` as above |
| `mode: sterilize`, `PII_ADDR` set | A running PII service over HTTP (`POST /sterilize`, `Authorization: Bearer $PII_API_KEY`). | The service's own |
`PII_ADDR` without `PII_API_KEY` is refused: the service fails closed, and so
does its caller.
## Rules the stage enforces
- **Fail closed.** The first recall failure (the model unreachable, the service
answering 502 or not at all) stops the stage. So does a service that does
not confirm it escaped every placeholder-shaped string in the document
(`stats.escape_held_literals` missing or false: a service older than the
agent); upgrade the service. The file being sanitized and
every file not yet reached are recorded `not_run`; nothing half-sanitized is
written; the exit code is 1 and `summary.stopped` says why. A recall backend
that cannot start (for example `PII_RECALL=gliner` without the `gliner`
package) stops the stage before the first file, with exit code 1 and no
index written.
- **Tokens are stable within a scope.** `sanitize.scope` (default: the
manifest id) is the vault scope: the same e-mail address in two documents of
one catalog gets one token, so the catalog stays coherent, and a different
catalog gets different tokens, so two catalogs cannot be joined through
shared placeholders.
- **Long texts are split, not truncated.** The sterilizer is called per chunk
of at most 20,000 characters (`SANITIZE_CHUNK_CHARS`), split at paragraph
and then line boundaries (a single longer line is cut). The chunks join back
to the input exactly, and because tokens are stable a value that appears in
two chunks gets one token. A cut never falls inside a placeholder: it moves
back to before the placeholder, or just after it when the placeholder starts
the chunk (only possible when the placeholder is longer than the limit), so
that chunk runs over the limit by the rest of the placeholder.
Any text in a document that looks like a placeholder is escaped, whether or
not the vault holds that name, so a document quoting `[PII_EMAIL_1]` comes
back as written and never as another document's value.
- **What makes a file be done again**: a change in its content, the mode, the
scope, `verify`, where the sterilizer runs (in process or service), or
`SANITIZE_VERSION`. **Results are redone when the pipeline changes:**
`SANITIZE_VERSION` is raised whenever the same text would be sterilized
differently, so an upgrade that raises it re-sanitizes every file once.
Changing only the recall backend or its model does not redo files already
sanitized; raise `SANITIZE_VERSION`, or start from an empty output folder,
when that matters.
- **Duplicate content is sanitized once**, and every path that has it is
indexed.
## Tests
`python3 -m unittest bastion_agent.test_sanitize` covers flag mode with the
real pipeline and vault (anchors replaced, a name left alone: the limit is
asserted, not hidden), the index and summary carrying no values, the vault's
0600 mode and `FREEAGENT_VAULT_DIR`, reuse and its invalidation (content,
scope), token stability across files, sterilize mode with a fake recall
backend (grounding drops an invented span; stats reported), the fail-closed
stop after the first recall failure (`not_run`, no store file written, the
backend called once), a backend that cannot start, the chunking round trip
(one token across every chunk) and hard cuts, the service path over a stub
speaking the real wire format (auth header, scope, verify, `stats.recall`), a
service 502 failing closed, a service without a key refused, flag mode never
touching the service, and the extract-first guard.
A `sterilize` run against a real recall model is not part of the suite; the
PII service's `verify-live.sh` does that against a running service.
## Not in this stage
Chunking for embedding (the next stage, which reads the sanitized store),
putting values back (the vault is local, and `/rehydrate` belongs to the PII
service), and encryption of the vault at rest (not done: the controls are the
file's 0600 mode and where it is kept).
docs/seal.md
+89
View File
@@ -0,0 +1,89 @@
# bastion agent `seal` stage: the catalog leaves, nothing else does
`seal` is the last stage of a run and the one way a dataset leaves the machine:
an allowlisted bundle of the catalog, encrypted with age to the owner and the
manifest's named recipients. It is `bastion_agent/seal.py`.
## What it does
`python3 -m bastion_agent seal <manifest> [--out DIR] [--require-signed]`
builds the bundle and encrypts it to the owner and the manifest's
`sink.seal_to` recipients:
```
<out>/sealed/<id>-<stamp>.tar.age the envelope (0600)
<out>/sealed/<id>-<stamp>.tar.age.sig optional detached signature
<out>/sealed/latest.json pointer to the newest envelope
<out>/seal.summary.json sizes, digests, recipients, members
```
`python3 -m bastion_agent unseal <file.tar.age> --identity <age key> --dest DIR`
is the owner's side: it decrypts, refuses unsafe paths, and checks every
member's SHA-256 against the inventory the sealer wrote inside the bundle.
## What is in the bundle, and what cannot be
| Member | Content |
|---|---|
| `manifest.json` | The manifest as executed (signature block removed). |
| `catalog.jsonl`, `catalog.dirs.jsonl` | The cards: sanitized text only. |
| `<stage>.summary.json` | The earlier stages' summaries: counts, digests, tool and model names. |
| `seal.manifest.json` | The inventory: sha256 and size per member, the manifest digest, the time of sealing. |
The bundle is built from an **allowlist**: the two catalog files
(`BUNDLE_ALLOW`), the stage summaries (names matching `<stage>.summary.json`),
and the two files the sealer writes itself. There is no code path that adds
the extract store, the sanitized store, the vault, `vectors.db`, or any other
`*.jsonl` index. A test opens the bundle and checks exactly that, and searches
it for the fixture's personal data. The tar is built with fixed member times
and a fixed order, so the same catalog sealed at the same moment is the same
plaintext.
## Recipients
- **Owner**: `FREEAGENT_OWNER_RECIPIENT`, an `age1…` public key, from the
environment. Mandatory: no owner recipient, no seal. It is never in the
manifest, because the manifest is written by the host and the owner's key is
the machine's business.
- **`sink.seal_to`**: each entry is an `age1…` recipient or an `ssh-ed25519`
or `ssh-rsa` public-key line (age accepts both). Any other string is refused
before anything is written.
- Duplicates collapse; the owner is always first.
- **Grantees are not recipients.** A `grants` entry lets another host *query*
this agent; it never receives the sealed catalog. Sharing is access, not a
copy.
## Why age
age (X25519 and ChaCha20-Poly1305) is a small, widely reviewed format with a
single static binary for Mac, Windows and Linux. The envelope cannot be made
with the standard library (it has no X25519 and no ChaCha20-Poly1305), so,
unlike the extract stage's optional tools, **age is required here**: a catalog
that cannot be sealed does not leave, and the stage says so and exits 1.
## Signature (optional)
`SEAL_SIGN_KEY=<path to a PEM private key>` adds a detached
`openssl dgst -sha256` signature over the envelope, so a receiving host can
check who sealed it. The signature covers that one envelope (age encrypts
differently every time), not the plaintext. The stage only produces the
signature; checking it against a certificate is the receiver's job. A test
proves it verifies against the public half.
## Tests
`python3 -m unittest bastion_agent.test_seal` covers: no owner recipient,
refused with nothing written; malformed recipients refused and named; no `age`
on PATH, refused; `seal` not in the stages, refused; the bundle allowlist
(present members, forbidden members absent, no values, sanitized placeholders
present); reproducible plaintext; a round trip with two real recipients and a
stranger refused; owner and recipient de-duplication and an ECDSA P-256
signature that verifies with openssl; and a tampered member refused on unseal.
The round-trip tests are skipped where `age` or `age-keygen` is not installed.
## Where it sits
walk → extract → sanitize → chunk → embed → catalog → seal, each usable on its
own and each with its own document here. The runtime around them (the `sweep`
scheduler and the manifest gate) is [runtime.md](runtime.md); the query
endpoint is [serve.md](serve.md).
docs/serve.md
+97
View File
@@ -0,0 +1,97 @@
# bastion agent query endpoint: `serve`
`serve` is how a host reads what the agent prepared without the data moving:
a host (the owner's, or a grantee's) sends a query and gets **sanitized**
chunks and catalog cards back. Nothing else has a route: not the extract
store, not the vault, not the vectors table. It is `bastion_agent/serve.py`,
mutual TLS with the standard library's `ssl`.
`python3 -m bastion_agent serve <manifest> [--out DIR] [--bind HOST:PORT] [--port-file FILE] [--allow-unsigned]`
## Identity: the client certificate
TLS runs with `ssl.CERT_REQUIRED` against `FREEAGENT_CA_ROOT`, TLS 1.2 at
least. Each connection's handshake runs in that connection's own thread with a
10 s timeout, never on the accept loop, so a peer that connects and says
nothing costs its own thread and nothing else; an idle connection is closed
after 60 s. A client with no certificate, or one from another CA, never
reaches a handler: the handshake fails, one line goes to stderr, and there is
no audit row (it never became a request). For a client that did chain, the
certificate's CN is who they are:
| CN | Gets |
|---|---|
| equal to the manifest's `owner.host` | `search` (if `search` is in the mask), `read_range`, `catalog`, `status`, `health` |
| a host in `manifest.grants`, on or before its `until` date | exactly that entry's `ops` (`search` and/or `read_range`), plus `health` |
| anyone else who chained | `health` only; everything else is 403, audited with the CN |
There is no other credential: no token, no password, no header.
## Routes
| Route | Body | Returns |
|---|---|---|
| `POST /search` | `{query, k?}` (k defaults to 5 and is held to 1–50) | `{hits, k}`: the top k sanitized windows with score, document id, chunk id, index and offsets. The same search as the CLI, through the same `EMBED_URL` model ([embed.md](embed.md)); 502 if the model cannot be reached. |
| `POST /read_range` | `{doc, start?, end}` | Up to 65,536 characters of the **sanitized** text of that document, with `total`. |
| `GET /catalog`, `GET /catalog/dirs` | query `prefix`, `offset`, `limit` (default 500, at most 2,000) | The file cards or the directory cards (owner only), a page at a time: `{cards, kind, total, offset, limit, next_offset, prefix}`, `next_offset` being `null` on the last page. 404 until the first sweep has written a catalog. |
| `GET /status` | | `{state}`: `run.state.json` (owner only). |
| `GET /health` | | `{ok, id, serving}` for any client that chained. |
`doc` is the document's content hash (64 lowercase hex), the same id the
extract and sanitized stores share. The route reads only the `sanitize/`
directory, so the plaintext for that id, which exists on disk, is never what
comes back; a test checks both halves of that sentence.
## The manifest is the policy, live
On each request the endpoint checks whether the manifest file
changed, and if so re-reads it and runs the same gate as the scheduler: the
signature over the canonical payload, the signing certificate's CN equal to
`owner.host`, and the chain to `FREEAGENT_CA_ROOT` (required). So:
- **Revoke a grant**: delete the entry and re-sign. The next request from that
CN is 403.
- **Edit without re-signing**: the endpoint serves nothing but `/health`
(which reports `serving: false`), and answers everything else with 503 and
the gate's reason until the manifest verifies again.
- `--allow-unsigned` is for development only, as for `sweep`.
## Audit
Every request that reached a handler is one line in `<out>/serve.audit.jsonl`:
`ts`, `cn`, `op`, `path`, `status`, `bytes`; for `search` the first 16 hex
characters of the query's sha256 (never the query) and the number of hits; for
`read_range` the start of the document id and the characters served; for the
catalog the page size and the total. A test checks that the query text is not
in the file.
## Bind and certificates
- `--bind` or `FREEAGENT_BIND`: `host:port` or `[v6]:port`, default
`[::]:8470`. `--bind` wins. An IPv6 address gets an `AF_INET6` socket.
- `--port-file FILE` writes the port actually bound. The bastion runs `serve`
on `127.0.0.1:0` and reads the port from this file, so the endpoint is not
reachable from the network; the bastion is its client.
Run by hand for another machine to reach, bind the interface it reaches
instead.
- `FREEAGENT_TLS_CERT` and `FREEAGENT_TLS_KEY`: the machine's certificate and
key. `FREEAGENT_CA_ROOT`: the CA that client certificates must chain to. All
three are required; the server refuses to start without them.
## Tests
`python3 -m unittest bastion_agent.test_serve` builds a small PKI per test
class (a CA root, a server certificate, owner, grantee, expired-grant and
unknown-CN client certificates, and a client certificate with the owner's CN
from a different CA) and checks: the owner gets every route and sees sanitized
heads in the catalog, paged; the grantee gets `search` and is 403 on
`read_range`, `catalog` and `status`; an expired grant and an unknown CN are
403; no certificate, or a foreign CA, fails the handshake and leaves no audit
row; a connection that never speaks does not stall the server; `read_range` is
capped, returns sanitized text for the shared id while the plaintext sits on
disk with no route, and gives 404 for an unknown id and 400 for malformed ids
and queries; the audit hashes the query; a grant revoked by re-signing is
refused on the next request; an edit without re-signing turns the endpoint
into 503 with `serving: false`, and re-signing restores it; an unreadable
manifest fails closed; and `--port-file` reports the bound port. Plus bind
parsing, CN extraction, and the required TLS inputs.
docs/walk.md
+116
View File
@@ -0,0 +1,116 @@
# bastion agent `walk` stage: the filesystem and metadata map
The walk is the first stage of every run and the cheapest thing a host can
plan from: every entry under the manifest's roots, with its size, time, type
and hash. It is `bastion_agent/walk.py`, standard library only.
## What it produces
`python3 -m bastion_agent walk <manifest.yaml> [--out DIR] [--no-hash] [--require-signed]`
reads the manifest's `scope`, walks every root, and writes two files into
`DIR`. Without `--out`, `DIR` is `<root>/.agent/` for a manifest with one
root, and `build/freeagent/<id>/` (under the current directory) for one with
several. The second keeps the program's first name on purpose: it is output a
person may already have, and renaming it would orphan it. Both files are
written atomically (a temporary file, then a rename), so a reader never sees a
half-written map.
**`walk.jsonl`**: one JSON object per entry, sorted by `(root, path)`, keys
sorted, UTF-8. Identical trees produce byte-identical files.
| Key | Present on | Meaning |
|---|---|---|
| `root` | all | Index into the manifest's `scope.roots`. |
| `path` | all | Path relative to that root, `/`-separated on every OS. |
| `kind` | all | `file`, `dir`, `symlink` (Windows junctions too), `other` (socket, fifo, device). |
| `size` | file | Bytes. |
| `mtime`, `mtime_ns` | file, dir | Modification time as UTC ISO-8601 and as integer nanoseconds (the cache key). |
| `mime` | file | Sniffed type: magic bytes first, then a fixed extension table, then a text/binary probe. `inode/x-empty` for empty files. The `mimetypes` module is not used, because it reads per-machine registries and would make the map depend on the machine. |
| `sha256` | file | Hex digest, or `null` when the file was too large, could not be read, or `--no-hash` was given. |
| `skip` | file | `too_large` when `size > scope.max_file_bytes` (default 256 MiB). The file is still sniffed and listed, never read in full. |
| `target` | symlink | The link text, `/`-separated. |
| `error` | any | `ExceptionName: message`. The entry is recorded and the walk continues. |
**`walk.summary.json`**: `manifest_id`, `manifest_digest` (the signing
digest, which ties the map to the exact manifest that produced it),
`walked_at`, `platform`, `roots` (each with its `resolved` absolute path, and
an `error` when it is absent on this machine), `entries`, `bytes`, `counts`
(`files dirs symlinks other excluded errors too_large hashed hash_reused`) and
`hashing`.
Exit code: `0` clean; `1` the manifest failed validation (or was unsigned
under `--require-signed`); `2` the walk completed with errors recorded
(a missing root, an unreadable directory). The map is written in every case
but the first.
## Rules the walker enforces
- **Excludes prune.** An excluded directory is not descended, so its subtree
costs nothing. Patterns (`scope.exclude`) are `/`-separated and relative to
the root. `**` spans directories; `*` and `?` stay within one segment;
`[..]` and `[!..]` classes work. A pattern with no `/` also matches any
single path segment, so `node_modules` or `*.key` prune at any depth. A
pattern ending in `/**` prunes the directory itself as well as its contents.
Backslashes in patterns are read as `/`. Matching is case-insensitive on
Windows only, as the file system is.
- **The agent's own folder is always excluded.** `.agent` is pruned at any
depth and in any letter case on every OS, whatever the manifest says, so an
agent never indexes its own output (on a case-insensitive Mac volume,
`.Agent` is the same folder).
- **Links are recorded, never followed**, unless `scope.follow_symlinks` is
true. This covers symlinks on every OS and junctions on Windows
(`is_junction()` on Python 3.12 and later, the reparse tag before that).
Following is opt-in because a link out of the root is the usual way a
scoped walk ends up reading the whole disk.
- **Large files are metadata only.** Over `max_file_bytes` the walker reads
8 KiB to sniff the type and stops.
- **Errors are entries, not aborts.** Permission denied on a directory gives a
`dir` entry with `error`, and the walk moves on; a root that does not exist
on this machine is reported in the summary and skipped. The host can see
exactly what it could not see.
- **Hashes are reused.** A previous `walk.jsonl` in the output directory is a
cache keyed on `(root, path)` → `(size, mtime_ns)`. An unchanged file keeps
its digest without being read again, so a daily sweep of a large folder
costs one `stat` per file, not one read. `--no-hash` skips hashing (and the
cache) for a fast first survey.
## Portability
The stage runs on the owner's machine, whatever it is.
- **Standard library only**: nothing to install on the machine that holds the
files.
- **Python 3.8 syntax floor**, enforced by `test_syntax_floor.py` at the
repository root, which parses every module in `bastion_agent/` and `pii/` at
3.8 and denies the newer standard-library calls that parse on 3.8 and then
fail at run time.
- **Roots** may be POSIX absolute paths, `~` or `~/...` (expanded with
`os.path.expanduser`: `HOME` on Unix, `USERPROFILE` on Windows), Windows
drive paths (`C:\...` or `C:/...`) and UNC paths (`\\server\share`).
- **Paths in the map are always `/`-separated**, so a map made on Windows
reads the same as one made on a Mac, and exclude patterns mean the same thing
everywhere.
- **Unicode file names** are written as they are (`ensure_ascii=False`);
undecodable bytes in link targets go through `os.fsdecode`.
- **Timestamps** are nanosecond integers from `st_mtime_ns`, which exists on
all three platforms; the ISO form is derived from it.
## What the stage does not do
- It does not read file contents beyond the 8 KiB sniff and the hash stream.
Extraction is the next stage.
- It does not detect changes by itself. Two maps diff trivially because they
are sorted, and later stages find their work from the hashes.
- It does not treat mounts specially. A network share under a root is walked
like any directory; exclude it if that is not wanted.
## Tests
`python3 -m unittest bastion_agent.test_walk` covers the map's shape and sort
order, MIME sniffing (magic, extension, empty, binary), excludes (segment,
anchored, `/**` pruning, backslashes, Windows case folding, `.agent` in any
case), the too-large skip, `--no-hash`, hash reuse and its invalidation when a
file changes, byte-identical re-runs, a missing root, an unreadable directory,
Unicode names, `~` expansion, and symlinks recorded but not followed, or
followed when the manifest allows it. The symlink tests are skipped on Windows,
where creating a symlink needs a privilege.