bastion agent 0.1.17

docs/catalog.md

@@ -57,7 +57,12 @@ MIME types), `keywords` (the top 12 over the merged term counts), `entities`
   hyphens, so `invoice-m@example.com.pdf` becomes `[PII_EMAIL]`; `mime` and
   `size` still say what it was. The summary counts masked segments as
   `names_masked`. **A name like `Marisol_medical.pdf` is not caught**: that is
-  contextual personal data, the same limit as the rules-only `flag` mode.
+  contextual personal data, the same limit as the rules-only `flag` mode. The
+  mask is the catalog's: the bastion's own listing, for a grant with `read`,
+  returns names as they are.
+- **Not hidden: content hashes.** Each file card's `sha256`, also the `doc` id in
+  search hits and excerpts, is the hash of the original file, so whoever holds
+  a copy of a file can confirm it is in the folder.
 
 ## The summarizer (a sensor)
 

docs/embed.md

@@ -1,7 +1,8 @@
 # bastion agent `chunk` and `embed` stages, and `search`
 
 `chunk` splits the sanitized text into overlapping windows, `embed` turns each
-window into a vector with a model on this machine, and `search` answers a
+window into a vector with the model server `EMBED_URL` names (by default the
+Ollama port on the same machine), and `search` answers a
 query against those vectors. They are `bastion_agent/embed.py`, standard
 library only.
 
@@ -9,9 +10,10 @@ library only.
 
 `python3 -m bastion_agent chunk <manifest> [--out DIR] [--require-signed]`
 reads the **sanitized** store (never the extract store) and writes
-`chunks.jsonl`: one row per window (`chunk_id`, `doc` (the sanitized text's
-sha256), `idx`, `start` and `end` character offsets, `chars`, `tokens`, `v`),
-sorted by `(doc, idx)`, plus `chunk.summary.json`.
+`chunks.jsonl`: one row per window (`chunk_id`, `doc` (the original file's
+sha256, the id the extract and sanitized stores share), `idx`, `start` and
+`end` character offsets, `chars`, `tokens`, `v`), sorted by `(doc, idx)`, plus
+`chunk.summary.json`.
 
 - Windows are `embed.chunk.size_tokens` long and advance by
   `size - overlap` tokens; the last window may be shorter. With any overlap,
@@ -27,8 +29,8 @@ sorted by `(doc, idx)`, plus `chunk.summary.json`.
   next. A changed document has a new sha256, so all of its windows get new ids
   and are embedded again. Identical windows inside one document (a repeated
   paragraph) share an id and are embedded once.
-- Duplicate documents (the same sanitized content at several paths) are
-  chunked once.
+- Duplicate documents (the same file content at several paths) are chunked
+  once.
 
 ## embed
 

docs/extract.md

@@ -25,8 +25,12 @@ the stage refuses otherwise) and writes:
   recorded, which ties the walk map and the text store to one manifest.
 
 `DIR` defaults as for the walk (`<root>/.agent/` for one root). This text
-store is the plaintext of every file in scope, and it never leaves the machine;
-see [runtime.md](runtime.md#where-an-agent-lives-rootagent).
+store is the plaintext of every file in scope. The agent never serves it, but
+by default it sits in the folder, so if the folder is synced, the sync provider
+gets a copy. A bastion from 0.58.1 on never serves `.agent/` to a host. On an
+older bastion, a host whose folder grant has `read` can read the store like
+any other file unless the grant excludes `.agent`; see
+[runtime.md](runtime.md#where-an-agent-lives-rootagent).
 
 | `status` | Meaning |
 |---|---|

docs/manifest.md

@@ -19,19 +19,21 @@ the control flow and every change, and a model proposes, never acts.
 The owner's host writes a **job manifest**, and it is signed (by the bastion,
 in the managed install). The agent executes exactly that manifest and nothing
 else. The output is a **catalog** (small, structured cards per file and per
-directory) plus local vectors. The files themselves never move; a host
-reasons over them by querying the agent. Sharing is a grant that lets another
-host query this catalog: access moves, data does not.
+directory) plus local vectors. The agent never sends the files themselves; a
+host reasons over them by querying the agent. Sharing is a grant that lets
+another host search the sanitized text or read ranges of it, until a set date
+([`grants`](#grants)).
 
 ## Rules the schema enforces
 
-1. **Plaintext never leaves.** `sink.plaintext` is the constant `never`. Any
+1. **No plaintext sink.** `sink.plaintext` is the constant `never`. Any
    other value is a validation error, not a setting.
 2. **Nothing runs that was not granted.** `mask` is the ceiling; every stage
    must be covered by an op in the mask, or the manifest is refused before it
    runs.
-3. **Nothing leaves unsealed.** A catalog sent to the host, `seal_to`
-   recipients, or any grant requires the `seal` stage.
+3. **Sharing needs `seal`.** `sink.catalog: host`, `seal_to` recipients, or
+   any grant requires the `seal` stage. What `serve` returns, to the owner or
+   to a grantee, is never sealed.
 4. **No addresses.** The sanitizer and the embedder are found from the
    machine's environment when the agent runs. The manifest never carries a
    host:port.
@@ -179,11 +181,14 @@ Present **if and only if** `embed` is in `stages`.
 | Field | Required | Rules |
 |---|---|---|
 | `model` | yes | Model name, 1–128 characters. |
-| `dim` | yes | Integer 1–8192. The stage refuses to run if the local endpoint's vectors have another dimension. Vectors are stored per model, so a later model change never discards existing vectors. |
+| `dim` | yes | Integer 1–8192. The stage refuses to write if the embedding server (`EMBED_URL`) returns vectors of another dimension. Vectors are stored per model, so a later model change never discards existing vectors. |
 | `chunk.size_tokens` | yes | Integer 1–8192. |
 | `chunk.overlap_tokens` | yes | Integer, 0 ≤ overlap < size. |
 
-There is no endpoint field. Embedding is local, always.
+There is no endpoint field: the embedder is the server `EMBED_URL` names in
+the agent's environment, by default the Ollama port on the same machine. The
+bastion sets it to `127.0.0.1:11434`; an agent run by hand can be pointed
+anywhere.
 
 ### `summarize`
 
@@ -191,7 +196,7 @@ Optional. Requires `summarize` in `mask` and `catalog` in `stages`.
 
 | Field | Required | Rules |
 |---|---|---|
-| `model` | yes | Local model name, 1–128 characters. |
+| `model` | yes | Model name, 1–128 characters, at the server `SUMMARIZE_URL` names. |
 | `per` | yes | Non-empty subset of `file`, `directory`. |
 
 ### `sweep`
@@ -202,7 +207,7 @@ One of `15m`, `hourly`, `daily`, `manual`.
 
 | Field | Required | Rules |
 |---|---|---|
-| `catalog` | yes | `local`: the sealed catalog stays here, and the host queries it. `host`: the sealed catalog is sent to the owner's host, so it can plan while this machine is asleep. Requires the `seal` stage. |
+| `catalog` | yes | `local`: the catalog stays here, and the host queries it through `serve`. `host`: the sealed catalog is meant for the owner's host. The agent writes it to `<out>/sealed/` and neither the agent nor the bastion sends it. In the default `<out>`, `<root>/.agent/`, a bastion from 0.58.1 on never serves it to a host; on an older bastion, a host whose folder grant has `read` can fetch it like any other file unless the grant excludes `.agent`. When `<out>` is inside the folder under any other name, any bastion serves it to such a host like any other file. Requires the `seal` stage. |
 | `vectors` | yes | `local`, the only value. |
 | `plaintext` | yes | `never`, the only value, ever. It is there so the rule is written down in every manifest and can be searched for, not remembered. |
 | `seal_to` | no | List of recipient keys, 1–128 characters each, no duplicates. Requires the `seal` stage. The owner's key is always added by the agent and need not be listed. |

docs/packaging.md

@@ -111,7 +111,7 @@ run there, and the second, with `SIGNATURE=<the line it printed>` and
 `BASTION_TREE` pointing at a privatae-bastion checkout, checks the signature
 against the keys that checkout pins and the keys the published bastion pins,
 and only then publishes. What is signed is a short statement naming the kind
-of release, its version and the manifest's sha256. A bastion from 0.41.0 on
+of release, its version and the manifest's sha256. A bastion from 0.42.0 on
 fetches `version.json.<h>.sig`, refuses a manifest that does not verify, and
 never replaces an installed agent with an older version.
 
@@ -180,7 +180,10 @@ answering `--version` and `doctor --json` and validating the example manifest;
 heads; the `.agent` default output directory; and the doctor's checks against
 unreachable and missing inputs.
 
-This repository has no CI. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the
+Every push to `main` and every pull request into it runs CI
+(`.gitea/workflows/ci.yml`, `.github/workflows/ci.yml`): the 3.8 floor,
+pyflakes, every test on Python 3.8 and a current Python, a personal-data scan
+and a reproducibility check. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the
 checks to run before proposing a change, and `publish.sh` builds the release
 and runs it before anything is uploaded. The installers are not exercised by
 the test suite.

docs/runtime.md

@@ -54,7 +54,25 @@ text with personal data tokenized (only the rules-pass anchors, unless the
 manifest uses `sterilize`). That is the same trust boundary as the folder
 itself, since the source files are already there, but a sync provider now
 receives a second, flattened, indexable copy. If that matters for a folder,
-exclude `.agent` from sync or keep the output elsewhere (`--out`).
+exclude `.agent` from sync or, for an agent run by hand, keep the output
+outside the folder (`--out`).
+
+Whether a host can reach `.agent/` depends on the bastion's version. From
+bastion 0.58.1 on, the bastion never serves it: a host's listings leave out
+every folder named `.agent`, at any depth and in any letter case, and the
+bastion refuses to stat, read or move anything in one, however the path is
+written and whatever link it goes through, and says why. It knows the folder
+by that name only, so an output folder inside the shared folder under any
+other name (`--out`) is served like any other folder. A bastion older than
+0.58.1 treats `.agent/` like any other folder: a host whose folder grant has
+`read` can list it and read the extract store as it is. On those bastions,
+add `.agent` to the paths the grant excludes; the bastion then leaves it out
+of a host's listings and refuses reads of it.
+
+The bastion keeps its agent in `<root>/.agent/` too, except on
+Windows when the bastion runs as the system service, or when it cannot create
+`.agent` in the folder (a read-only folder, for one); it then keeps it in its
+own state directory.
 
 The identity key is never in `.agent/`: it is wherever `FREEAGENT_TLS_KEY`
 points. The PII vault (plaintext values) is in `.agent/sanitize/vault.db` for

docs/sanitize.md

@@ -12,8 +12,11 @@ reads the extract index and store and writes:
 
 - **`sanitize/<aa>/<sha256>.txt`**: the sanitized text store. Personal data is
   replaced by stable `[PII_TYPE_n]` tokens. **From this stage on, later
-  stages read only this store.** The extract store keeps the plaintext and
-  never leaves the machine.
+  stages read only this store.** The extract store keeps the plaintext. The
+  agent never serves it, and while it is in `.agent/` (the default) a
+  bastion from 0.58.1 on does not either. An older bastion lets a folder
+  grant with `read` reach it there unless the grant excludes `.agent`
+  ([runtime.md](runtime.md#where-an-agent-lives-rootagent)).
 - **`vault.db`**: the token-to-value map (in-process mode only), created 0600
   under umask 077. It is the only way back from a token to a value, and it
   stays on the machine. It is `<out>/sanitize/vault.db`, or
@@ -50,10 +53,14 @@ stages, there is no extract index, or the stage **stopped**; `2` completed with
   employers, medical facts) is found by the recall pass.
 - **`flag`**: the rules pass **only**: deterministic, no model, milliseconds.
   It replaces the anchors a pattern can find: e-mail addresses, phone
-  numbers, SSNs, cards, IBANs, routing numbers, crypto addresses, IP and MAC
-  addresses, API keys, private-key blocks, JWTs. It can also replace an
-  ordinary number that looks like a phone or ID number. **It does not find names,
-  street addresses, dates of birth**, or anything else that takes reading for
+  numbers, SSNs, cards, IBANs, US routing numbers after a cue word, legacy and
+  P2SH Bitcoin addresses, IP and MAC addresses, API keys, private-key blocks,
+  JWTs ([the list](pii-sterilization.md#the-rules-pass-piidetectpy)). Other
+  bank account numbers, other crypto addresses (`bc1…`, Ethereum) and ID
+  numbers such as passport numbers are not found. It can also replace an
+  ordinary number that looks like a phone, card or social security number.
+  **It does not find names, street addresses, dates of birth**, or anything
+  else that takes reading for
   meaning (an employer, a school, a medical fact). It is the mode the bastion
   writes into every manifest it deploys. A flag-mode catalog is not free of
   personal data, and the manifest doc says so too.

docs/seal.md

@@ -1,8 +1,10 @@
-# bastion agent `seal` stage: the catalog leaves, nothing else does
+# bastion agent `seal` stage: the catalog in one encrypted bundle
 
-`seal` is the last stage of a run and the one way a dataset leaves the machine:
-an allowlisted bundle of the catalog, encrypted with age to the owner and the
-manifest's named recipients. It is `bastion_agent/seal.py`.
+`seal` is the last stage of a run: it packs an allowlisted bundle of the
+catalog into one file, encrypted with age to the owner and the manifest's named
+recipients. It writes the bundle into the output folder and sends it nowhere.
+`serve` answers queries with cards and excerpts, unsealed
+([serve.md](serve.md)). It is `bastion_agent/seal.py`.
 
 ## What it does
 
@@ -58,8 +60,8 @@ plaintext.
 age (X25519 and ChaCha20-Poly1305) is a small, widely reviewed format with a
 single static binary for Mac, Windows and Linux. The envelope cannot be made
 with the standard library (it has no X25519 and no ChaCha20-Poly1305), so,
-unlike the extract stage's optional tools, **age is required here**: a catalog
-that cannot be sealed does not leave, and the stage says so and exits 1.
+unlike the extract stage's optional tools, **age is required here**: without
+it no bundle is written at all, and the stage says so and exits 1.
 
 ## Signature (optional)
 

docs/serve.md

@@ -1,7 +1,7 @@
 # bastion agent query endpoint: `serve`
 
-`serve` is how a host reads what the agent prepared without the data moving:
-a host (the owner's, or a grantee's) sends a query and gets **sanitized**
+`serve` is how a host reads what the agent prepared without being sent the
+files themselves: a host (the owner's, or a grantee's) sends a query and gets **sanitized**
 chunks and catalog cards back (sanitized by the manifest's mode: in `flag`
 mode, which the bastion deploys, names, street addresses and dates of birth
 remain; [sanitize.md](sanitize.md#modes-manifest-sanitizemode)). Nothing else has a route: not the extract