bastion agent 0.1.17

CHANGELOG.md

@@ -14,6 +14,49 @@ Changes to the release tooling (`publish.sh`, `build.py`, `verify_release.py`)
 and to the documentation are listed under the version whose source first
 carried them, although they do not change what the `.pyz` does.
 
+## 0.1.17
+
+- **Messages no longer claim what the code does not do.** A manifest with
+  `sink.catalog: host` and no `seal` stage is refused with "the copy meant
+  for the host is the sealed bundle", and one with a grant and no `seal`
+  stage with "any grant requires the 'seal' stage"; neither says that
+  nothing leaves or is served unsealed, since `serve`'s replies are never
+  sealed. A malformed `doc` in `read_range` is refused as "the original
+  file's sha256", which is what the id is. What the manifest accepts and
+  refuses is unchanged.
+- **The documentation says what reaches a Host.** What the agent
+  returns (catalog cards, search hits, excerpts by document id) comes from
+  the sanitized text, and the agent sends no file or extracted text to a
+  Host. A folder grant with `read`, the default, also lets the Host ask the
+  bastion itself for the folder's file names, sizes and times and for raw
+  ranges of its files, up to 512 KiB per read. From bastion 0.58.1 on, none
+  of that reaches the agent's own `.agent/` folder. An older bastion serves
+  `.agent/` and the plaintext extract store in it wherever it keeps the agent
+  inside the folder, unless the grant's excluded paths list `.agent`.
+  `seal` writes its bundle and sends it nowhere. A file card's `sha256` is
+  the original file's hash.
+- **`flag` mode's limits are listed.** It does not look for bank account
+  numbers other than IBANs and US routing numbers after a word such as
+  "routing", crypto addresses other than Bitcoin addresses that start with
+  1 or 3, or ID numbers such as passport numbers.
+- **Every model server is the one the environment names.** `EMBED_URL` and
+  `SUMMARIZE_URL` default to the Ollama port on the same machine, and the
+  bastion sets `EMBED_URL` to `127.0.0.1:11434`. `sterilize` mode sends the
+  unsanitized text to the PII service at `PII_ADDR` when that is set;
+  otherwise to the server at `LLM_URL`, unless `PII_RECALL=gliner` runs the
+  model inside the agent.
+- **Smaller corrections.** The `doc` id is the original file's sha256
+  (embed.md); CI runs on every push and pull request (packaging.md); signed
+  manifests are checked from bastion 0.42.0, since 0.41.0 was never
+  published; the bastion installs Privatae's signed build, which anyone can
+  rebuild from this source.
+
+- **Tests look for the whole test card number.** Four tests checked that
+  the four digits `4111` never appear in a sealed bundle, an index or a
+  catalog. Those also hold sha256 hex and times, where the same four digits
+  turn up by chance: CI failed on a seal manifest hash ending `...81934111`.
+  They now look for the full card value, spaced and unspaced.
+
 ## 0.1.16
 
 - **Every `search` and `read_range` reply names its sanitize mode**

CONTRIBUTING.md

@@ -43,11 +43,19 @@ refuses adds a test that it refuses.
 
 ## What a change has to respect
 
-- **Bytes never leave the machine.** The files, their extracted text and the
+- **The agent sends no file and no extracted text to a Host.** It keeps the
-  vault stay local. What leaves is the sealed catalog, to the recipients the
+  files, their extracted text and the vault on the machine. The one exception
-  manifest names, and what `serve` returns to a certificate the manifest
+  is `sterilize` mode with a server: the PII service at `PII_ADDR`, or the
-  allows. There is no plaintext sink: `sink.plaintext` is `never` and has no
+  recall model at `LLM_URL`, is sent the extracted text, and the PII service
-  other value.
+  keeps its own vault ([docs/sanitize.md](docs/sanitize.md#where-the-sterilizer-runs)).
+  What the agent makes for others is the sealed catalog, for the recipients
+  the manifest names, and what `serve` returns to a certificate the manifest
+  allows. There is no plaintext sink:
+  `sink.plaintext` is `never` and has no other value. (The bastion's own
+  `read` op is separate: it returns ranges of any file in the granted folder
+  that the grant does not exclude, as they are on disk. From bastion 0.58.1
+  on it returns nothing in `.agent/`; an older bastion does unless the grant
+  excludes `.agent`.)
 - **Sanitize before anything is written for later stages.** Chunks, vectors,
   keywords, titles, summaries, the catalog and everything `serve` returns come
   from the sanitized store, never the extract store. A test that searches an

README.md

@@ -9,10 +9,21 @@ It is deployed **from a grant**, and you do not install it yourself: the bastion
 (`version.json.<first 16 hex of its sha256>.sig`) against the keys compiled into the bastion, downloads the `.pyz`
 and checks it against the sha256 in that signed manifest,
 mints a P-256 identity per store, writes and signs a manifest from the grant you
-approved, and supervises `sweep` and `serve`. Your files stay on the machine;
+approved, and supervises `sweep` and `serve`. Nothing is uploaded or synced;
-what crosses the tunnel is what the grant's mask allows: catalog cards, and
+your Host asks, and gets what the grant's mask allows. What the agent returns
-the sanitized excerpts that `search` and `read_range` return, each reply naming
+is sanitized: catalog cards, search hits, and the excerpts a `read_range` by
-the sanitize mode that made it (`sanitize_mode`, [serve.md](docs/serve.md)).
+document id returns, each reply naming the sanitize mode that made it
+(`sanitize_mode`, [serve.md](docs/serve.md)). A grant with `read`, which a
+connected folder has by default, also lets the Host ask the bastion itself for
+the folder's file names, sizes and modification times, and for one range of
+one file per read, up to 512 KiB, as it is on disk and not sanitized. From
+bastion 0.58.1 on, none of this reaches the agent's own `.agent/` folder: the
+bastion leaves every folder named `.agent` out of a listing and refuses any
+request for a path in one. An older bastion serves `.agent/` like any other
+folder, including the unsanitized text the agent extracted from every file,
+unless `.agent` is among the grant's excluded paths
+([runtime.md](docs/runtime.md#where-an-agent-lives-rootagent) says when the
+agent's folder is inside the shared folder).
 
 The published version is whatever `privatae.ai/bastion-agent/version.json`
 names. `bastion_agent/VERSION` on `main` is the next release, which may not be
@@ -20,7 +31,7 @@ published yet. Release manifests are published signed from 0.1.13 on, and the
 `.pyz` can be rebuilt from this source and compared
 ([Verify a release](#verify-a-release)).
 
-This repository is free software, licensed under the GNU General Public License version 3 or (at your option) any later version (see `LICENSE`). Copyright Privatae LLC. You may run it, read it, change it and redistribute it under that license. The Host the bastion connects to is separate software and is not covered by this license: a C-Series Host is [Privatae/cseries](https://github.com/Privatae/cseries), under AGPL-3.0-only, and a Host on Privatae's own platform is proprietary software. A member runs their own build on their own machine — nothing here depends on a Privatae-held credential or a Privatae-side build.
+This repository is free software, licensed under the GNU General Public License version 3 or (at your option) any later version (see `LICENSE`). Copyright Privatae LLC. You may run it, read it, change it and redistribute it under that license. The Host the bastion connects to is separate software and is not covered by this license: a C-Series Host is [Privatae/cseries](https://github.com/Privatae/cseries), under AGPL-3.0-only, and a Host on Privatae's own platform is proprietary software. Anyone can build and run the agent from this source with no Privatae account or key; the bastion installs Privatae's signed build, which this source reproduces.
 
 ## What it does
 
@@ -32,10 +43,10 @@ failure stops the run:
 |---|---|---|
 | walk | the map of the folder: files, sizes, types, what the grant excludes | [walk.md](docs/walk.md) |
 | extract | text out of each file, with built-in extractors or local tools | [extract.md](docs/extract.md) |
-| sanitize | replaces the personal data its mode finds with tokens; the originals stay in a vault on this machine. In the mode the bastion deploys it does not find names, street addresses or dates of birth ([below](#how-the-bastion-runs-it)) | [sanitize.md](docs/sanitize.md), [pii-sterilization.md](docs/pii-sterilization.md) |
+| sanitize | replaces the personal data its mode finds with tokens; the originals stay in a vault on this machine, or in a PII service's own vault when `PII_ADDR` names one. In the mode the bastion deploys it does not find names, street addresses or dates of birth ([below](#how-the-bastion-runs-it)) | [sanitize.md](docs/sanitize.md), [pii-sterilization.md](docs/pii-sterilization.md) |
-| chunk, embed | splits the text and embeds it with a model on this machine | [embed.md](docs/embed.md) |
+| chunk, embed | splits the text and embeds it with the model server `EMBED_URL` names: by default, and always under the bastion, the Ollama port on the same machine | [embed.md](docs/embed.md) |
 | catalog | the curated dataset built from the stages above | [catalog.md](docs/catalog.md) |
-| seal | packs an allowlisted bundle of the catalog and encrypts it with age to the owner and the manifest's named recipients; the one way a dataset leaves the machine | [seal.md](docs/seal.md) |
+| seal | packs an allowlisted bundle of the catalog into one file and encrypts it with age to the owner and the manifest's named recipients; it writes the file into the output folder and sends it nowhere | [seal.md](docs/seal.md) |
 
 `serve` answers queries over the index with mutual TLS, returning chunks and
 catalog cards from the sanitized text ([serve.md](docs/serve.md)). The gate refuses an
@@ -56,13 +67,22 @@ For each shared folder the bastion:
 - mints the agent its own identity and writes and signs its manifest from the
   grant, so the agent works only on what you granted;
 - sets the sanitize stage to `flag` mode: rules only, no model. It replaces
-  e-mail addresses, phone numbers, card, bank and social security numbers, IP
+  e-mail addresses, phone numbers, US social security numbers, card numbers,
-  and MAC addresses and keys. **It does not find names, street addresses or
+  IBANs, US routing numbers that follow a word such as "routing", Bitcoin
-  dates of birth**, so text a Host receives from the agent still contains them.
+  addresses that start with 1 or 3, IP and MAC addresses, and keys and tokens
-  `sterilize` mode adds a model on this machine that looks for those, and the
+  of known formats ([the full list](docs/pii-sterilization.md#the-rules-pass-piidetectpy)).
-  bastion does not set it ([sanitize.md](docs/sanitize.md#modes-manifest-sanitizemode));
+  It does not look for other bank account numbers, other crypto addresses or
-- points the embed stage at the model server on this machine (a local ollama)
+  ID numbers such as passport numbers, though digits in one can still be
-  and at nothing else;
+  replaced when they look like a phone, card or social security number.
+  **It does not find names, street addresses or dates of birth**, so text a
+  Host receives from the agent still contains them. `sterilize` mode adds a
+  model that looks for those. When `PII_ADDR` is set, the unsanitized text
+  goes to the PII service it names. Otherwise the model is the server
+  `LLM_URL` names, which is sent the unsanitized text, or, with
+  `PII_RECALL=gliner`, one that runs inside the agent. The bastion does not
+  set this mode ([sanitize.md](docs/sanitize.md#where-the-sterilizer-runs));
+- points the embed stage at `127.0.0.1:11434`, the Ollama port on the same
+  machine, and at nothing else;
 - runs `serve` on the loopback address only, with mutual TLS, so the bastion is
   the only client;
 - keeps the personal-data vault in the bastion's own state directory rather

bastion_agent/VERSION

@@ -1 +1 @@
-0.1.16
+0.1.17

bastion_agent/catalog.py

@@ -1,6 +1,6 @@
 """bastion agent `catalog` stage — the curated dataset.
 
-The catalog is what a host reasons over and what `seal` ships. One card
+The catalog is what a host reasons over and what `seal` packs. One card
 per file, one per directory, small and structured, built from:
 
     walk.jsonl          metadata (path, size, mtime, mime, hash)
@@ -22,9 +22,10 @@ masked as [PII_TYPE] in the card. Contextual PII in a filename
 ("Marisol_medical.pdf") is not caught by rules — the honest limit of the
 sanitize stage applies here too, and the doc says so.
 
-The optional summarizer (manifest `summarize`) is a local chat model — a
+The optional summarizer (manifest `summarize`) is a chat model at the
-sensor: it fills the `summary` string and nothing else, is reached via
+server SUMMARIZE_URL names (never the manifest; by default the Ollama port
-SUMMARIZE_URL (never the manifest), sees only sanitized text, and is
+on this machine, 127.0.0.1:11434) — a sensor: it fills the `summary`
+string and nothing else, is sent only sanitized text, and is
 cached per (document, model, version). The first failure stops
 summarizing (cards still get written, `summary_error` set, exit 1).
 """
@@ -151,7 +152,7 @@ def mask_name(name, detect):
 def mask_path(path, detect):
     """Every component of a relative path through mask_name — a parent
     directory can carry an address as easily as a filename, and the file
-    card's path is what seal ships. Returns (masked
+    card's path is what serve returns and seal packs. Returns (masked
     path, number of components changed)."""
     if not path:
         return path, 0

bastion_agent/manifest.py

@@ -539,9 +539,10 @@ def validate_document(doc, require_signed=False):
             errors.append("summarize: requires the 'catalog' stage (summaries live in the catalog)")
     if stages is not None:
         if sink is not None and sink.catalog == "host" and "seal" not in stages:
-            errors.append("sink: catalog 'host' requires the 'seal' stage — nothing leaves unsealed")
+            errors.append("sink: catalog 'host' requires the 'seal' stage — the copy meant for the host "
+                          "is the sealed bundle")
         if grants and "seal" not in stages:
-            errors.append("grants: any grant requires the 'seal' stage — nothing is served unsealed")
+            errors.append("grants: any grant requires the 'seal' stage")
         if sink is not None and sink.seal_to and "seal" not in stages:
             errors.append("sink: seal_to given but 'seal' is not in stages")
 

bastion_agent/sanitize.py

@@ -12,8 +12,15 @@ SECOND store:
     <out>/sanitize.summary.json
 
 From here on, later stages read ONLY the sanitized store. The extract
-store keeps the plaintext and stays on this machine; the vault is the
+store keeps the plaintext. This agent never serves it, but it sits in
-only way back (`/rehydrate` semantics, local).
+<out>, by default <root>/.agent/ inside the folder, where a sync provider
+can reach it. A bastion from 0.58.1 on never serves a folder named .agent
+to a host; an older bastion does to a host whose folder grant has `read`
+unless the grant excludes .agent, and any bastion serves an <out> inside
+the folder under another name.
+The vault is the only way back from a token to a value (`/rehydrate`
+semantics): a file on this machine in-process, the service's own with
+PII_ADDR.
 
 Two ways to reach the sterilizer, chosen from the environment, never the
 manifest (no addresses in manifests — docs/pii-sterilization.md):

bastion_agent/serve.py

@@ -1,9 +1,10 @@
 """bastion agent query endpoint — the container on the mesh.
 
-The other half of "access moves, data does not". Serves the owner's host
+Serves the owner's host and the manifest's grantees over MUTUAL TLS;
-and the manifest's grantees over MUTUAL TLS; nothing here ever reads the
+nothing here ever reads the extract store, the vault, or the vectors' raw
-extract store, the vault, or the vectors' raw table for a client — the
+table for a client. What it returns is sanitized chunks, catalog cards, the
-only bytes that leave are sanitized chunks and catalog cards.
+run state and health. (The bastion's own `read` op is not this server: it
+reads files in the granted folder as they are on disk.)
 
 Identity is the presented client certificate:
   - the TLS layer requires a client cert that chains to FREEAGENT_CA_ROOT
@@ -16,7 +17,7 @@ Ops (JSON over HTTPS):
   POST /search      {query, k?}           search  (owner, grantee w/ op)
   POST /read_range  {doc, start, end}     read_range (owner, grantee w/ op)
                                           from the SANITIZED store only,
-                                          <= READ_RANGE_MAX bytes
+                                          <= READ_RANGE_MAX characters
   Both answer with `sanitize_mode`, the mode the sanitize stage recorded for
   the text in the reply ("flag" or "sterilize"; the weakest across a
   search's hits; "unknown" when the record does not vouch for that text; a
@@ -414,7 +415,8 @@ class Handler(BaseHTTPRequestHandler):
                 start = int(req.get("start", 0))
                 end = int(req["end"])
                 if not isinstance(doc, str) or len(doc) != 64 or not all(c in "0123456789abcdef" for c in doc):
-                    raise ValueError("doc must be a sanitized-document sha256")
+                    raise ValueError("doc must be a document id: the original file's sha256, "
+                                     "64 lowercase hex")
                 if start < 0 or end <= start:
                     raise ValueError("start/end out of order")
                 end = min(end, start + READ_RANGE_MAX)

bastion_agent/test_catalog.py

@@ -189,7 +189,8 @@ class StageTests(Case):
     def test_catalog_carries_no_values(self):
         _, files, dirs = self.run_stage(manifest(self.root))
         blob = json.dumps(files) + json.dumps(dirs)
-        for leak in ("m@example.com", "4111"):
+        # the whole card: four digits alone also turn up inside hashes and times
+        for leak in ("m@example.com", "4111 1111 1111 1111", "4111111111111111"):
             self.assertNotIn(leak, blob)
 
     def test_summaries_file_and_directory_with_cache(self):
@@ -205,7 +206,8 @@ class StageTests(Case):
         # the model only ever saw sanitized text
         for _, _, prompt in _StubChat.calls:
             self.assertNotIn("m@example.com", prompt)
-            self.assertNotIn("4111", prompt)
+            for card in ("4111 1111 1111 1111", "4111111111111111"):
+                self.assertNotIn(card, prompt)
         # second run: all summaries from cache, no calls
         _StubChat.calls = []
         s2, _, _ = self.run_stage(m)

bastion_agent/test_manifest.py

@@ -123,12 +123,18 @@ class DoctrineTests(unittest.TestCase):
     def test_host_catalog_needs_seal(self):
         doc = with_(FULL, stages=["walk", "extract", "sanitize", "chunk", "embed", "catalog"],
                     grants=None, sink={"catalog": "host", "vectors": "local", "plaintext": "never"})
-        self.assertTrue(any("unsealed" in e for e in errors_of(doc)))
+        errs = errors_of(doc)
+        self.assertTrue(any("sink: catalog 'host' requires the 'seal' stage" in e for e in errs))
+        # serve's replies are never sealed, so the message must not claim that
+        # nothing leaves, or is served, unsealed
+        self.assertFalse(any("unsealed" in e for e in errs), errs)
 
     def test_grant_needs_seal(self):
         doc = with_(FULL, stages=["walk", "extract", "sanitize", "chunk", "embed", "catalog"],
                     sink={"catalog": "local", "vectors": "local", "plaintext": "never"})
-        self.assertTrue(any("grants: any grant requires the 'seal'" in e for e in errors_of(doc)))
+        errs = errors_of(doc)
+        self.assertTrue(any("grants: any grant requires the 'seal'" in e for e in errs))
+        self.assertFalse(any("unsealed" in e for e in errs), errs)
 
     def test_grant_search_needs_mask_search(self):
         doc = with_(FULL, mask=["read", "index", "summarize"])

bastion_agent/test_sanitize.py

@@ -104,7 +104,8 @@ class FlagModeTests(Case):
         s, idx = self.run_stage(manifest(self.root, "flag"))
         out = self.text_of(idx["note.txt"])
         self.assertNotIn("m@example.com", out)
-        self.assertNotIn("4111", out)
+        for card in ("4111 1111 1111 1111", "4111111111111111"):
+            self.assertNotIn(card, out)
         self.assertIn("Marisol", out)  # contextual PII is NOT caught by flag mode
         self.assertIn("[PII_EMAIL_1]", out)
         self.assertIn("[PII_CREDIT_CARD_1]", out)
@@ -121,7 +122,9 @@ class FlagModeTests(Case):
         s, idx = self.run_stage(manifest(self.root, "flag"))
         blob = json.dumps(idx) + json.dumps(s)
         self.assertNotIn("m@example.com", blob)
-        self.assertNotIn("4111", blob)
+        # the whole card: four digits alone also turn up inside hashes and times
+        for card in ("4111 1111 1111 1111", "4111111111111111"):
+            self.assertNotIn(card, blob)
 
     @unittest.skipIf(IS_WINDOWS, "POSIX permission bits")
     def test_vault_is_private(self):

bastion_agent/test_seal.py

@@ -112,9 +112,12 @@ class FailClosedTests(Case):
         for forbidden in ("walk.jsonl", "extract.jsonl", "sanitize.jsonl", "vectors.db", "vault.db"):
             self.assertFalse(any(n.endswith(forbidden) for n in names), forbidden)
         self.assertFalse(any(n.startswith(("extract/", "sanitize/")) for n in names))
-        # nothing in the bundle is a value
+        # nothing in the bundle is a value. The card is looked for whole, spaced
+        # or not: four of its digits also turn up by chance inside the bundle's
+        # sha256 hex (CI run 2483 failed on "...81934111").
         self.assertNotIn(b"m@example.com", plain)
-        self.assertNotIn(b"4111", plain)
+        self.assertNotIn(b"4111 1111 1111 1111", plain)
+        self.assertNotIn(b"4111111111111111", plain)
         self.assertIn(b"[PII_EMAIL_1]", plain)  # sanitized head is there, as designed
         self.assertEqual(set(inv["members"]), set(names))
 

bastion_agent/test_serve.py

@@ -233,7 +233,10 @@ class ServeTests(unittest.TestCase):
             self.assertIn("m@example.com", fh.read())  # the plaintext exists on disk; it has no route
         st, _ = self.call("owner", "POST", "/read_range", {"doc": "0" * 64, "start": 0, "end": 10})
         self.assertEqual(st, 404)
-        self.assertEqual(self.call("owner", "POST", "/read_range", {"doc": "zz", "start": 0, "end": 1})[0], 400)
+        st, body = self.call("owner", "POST", "/read_range", {"doc": "zz", "start": 0, "end": 1})
+        self.assertEqual(st, 400)
+        # the id is the original file's hash (the extract and sanitized stores share it)
+        self.assertIn("original file's sha256", body["error"])
         self.assertEqual(self.call("owner", "POST", "/search", {"query": ""})[0], 400)
         self.assertEqual(self.call("owner", "POST", "/nope", {})[0], 404)
 

docs/catalog.md

@@ -57,7 +57,12 @@ MIME types), `keywords` (the top 12 over the merged term counts), `entities`
   hyphens, so `invoice-m@example.com.pdf` becomes `[PII_EMAIL]`; `mime` and
   `size` still say what it was. The summary counts masked segments as
   `names_masked`. **A name like `Marisol_medical.pdf` is not caught**: that is
-  contextual personal data, the same limit as the rules-only `flag` mode.
+  contextual personal data, the same limit as the rules-only `flag` mode. The
+  mask is the catalog's: the bastion's own listing, for a grant with `read`,
+  returns names as they are.
+- **Not hidden: content hashes.** Each file card's `sha256`, also the `doc` id in
+  search hits and excerpts, is the hash of the original file, so whoever holds
+  a copy of a file can confirm it is in the folder.
 
 ## The summarizer (a sensor)
 

docs/embed.md

@@ -1,7 +1,8 @@
 # bastion agent `chunk` and `embed` stages, and `search`
 
 `chunk` splits the sanitized text into overlapping windows, `embed` turns each
-window into a vector with a model on this machine, and `search` answers a
+window into a vector with the model server `EMBED_URL` names (by default the
+Ollama port on the same machine), and `search` answers a
 query against those vectors. They are `bastion_agent/embed.py`, standard
 library only.
 
@@ -9,9 +10,10 @@ library only.
 
 `python3 -m bastion_agent chunk <manifest> [--out DIR] [--require-signed]`
 reads the **sanitized** store (never the extract store) and writes
-`chunks.jsonl`: one row per window (`chunk_id`, `doc` (the sanitized text's
+`chunks.jsonl`: one row per window (`chunk_id`, `doc` (the original file's
-sha256), `idx`, `start` and `end` character offsets, `chars`, `tokens`, `v`),
+sha256, the id the extract and sanitized stores share), `idx`, `start` and
-sorted by `(doc, idx)`, plus `chunk.summary.json`.
+`end` character offsets, `chars`, `tokens`, `v`), sorted by `(doc, idx)`, plus
+`chunk.summary.json`.
 
 - Windows are `embed.chunk.size_tokens` long and advance by
   `size - overlap` tokens; the last window may be shorter. With any overlap,
@@ -27,8 +29,8 @@ sorted by `(doc, idx)`, plus `chunk.summary.json`.
   next. A changed document has a new sha256, so all of its windows get new ids
   and are embedded again. Identical windows inside one document (a repeated
   paragraph) share an id and are embedded once.
-- Duplicate documents (the same sanitized content at several paths) are
+- Duplicate documents (the same file content at several paths) are chunked
-  chunked once.
+  once.
 
 ## embed
 

docs/extract.md

@@ -25,8 +25,12 @@ the stage refuses otherwise) and writes:
   recorded, which ties the walk map and the text store to one manifest.
 
 `DIR` defaults as for the walk (`<root>/.agent/` for one root). This text
-store is the plaintext of every file in scope, and it never leaves the machine;
+store is the plaintext of every file in scope. The agent never serves it, but
-see [runtime.md](runtime.md#where-an-agent-lives-rootagent).
+by default it sits in the folder, so if the folder is synced, the sync provider
+gets a copy. A bastion from 0.58.1 on never serves `.agent/` to a host. On an
+older bastion, a host whose folder grant has `read` can read the store like
+any other file unless the grant excludes `.agent`; see
+[runtime.md](runtime.md#where-an-agent-lives-rootagent).
 
 | `status` | Meaning |
 |---|---|

docs/manifest.md

@@ -19,19 +19,21 @@ the control flow and every change, and a model proposes, never acts.
 The owner's host writes a **job manifest**, and it is signed (by the bastion,
 in the managed install). The agent executes exactly that manifest and nothing
 else. The output is a **catalog** (small, structured cards per file and per
-directory) plus local vectors. The files themselves never move; a host
+directory) plus local vectors. The agent never sends the files themselves; a
-reasons over them by querying the agent. Sharing is a grant that lets another
+host reasons over them by querying the agent. Sharing is a grant that lets
-host query this catalog: access moves, data does not.
+another host search the sanitized text or read ranges of it, until a set date
+([`grants`](#grants)).
 
 ## Rules the schema enforces
 
-1. **Plaintext never leaves.** `sink.plaintext` is the constant `never`. Any
+1. **No plaintext sink.** `sink.plaintext` is the constant `never`. Any
    other value is a validation error, not a setting.
 2. **Nothing runs that was not granted.** `mask` is the ceiling; every stage
    must be covered by an op in the mask, or the manifest is refused before it
    runs.
-3. **Nothing leaves unsealed.** A catalog sent to the host, `seal_to`
+3. **Sharing needs `seal`.** `sink.catalog: host`, `seal_to` recipients, or
-   recipients, or any grant requires the `seal` stage.
+   any grant requires the `seal` stage. What `serve` returns, to the owner or
+   to a grantee, is never sealed.
 4. **No addresses.** The sanitizer and the embedder are found from the
    machine's environment when the agent runs. The manifest never carries a
    host:port.
@@ -179,11 +181,14 @@ Present **if and only if** `embed` is in `stages`.
 | Field | Required | Rules |
 |---|---|---|
 | `model` | yes | Model name, 1–128 characters. |
-| `dim` | yes | Integer 1–8192. The stage refuses to run if the local endpoint's vectors have another dimension. Vectors are stored per model, so a later model change never discards existing vectors. |
+| `dim` | yes | Integer 1–8192. The stage refuses to write if the embedding server (`EMBED_URL`) returns vectors of another dimension. Vectors are stored per model, so a later model change never discards existing vectors. |
 | `chunk.size_tokens` | yes | Integer 1–8192. |
 | `chunk.overlap_tokens` | yes | Integer, 0 ≤ overlap < size. |
 
-There is no endpoint field. Embedding is local, always.
+There is no endpoint field: the embedder is the server `EMBED_URL` names in
+the agent's environment, by default the Ollama port on the same machine. The
+bastion sets it to `127.0.0.1:11434`; an agent run by hand can be pointed
+anywhere.
 
 ### `summarize`
 
@@ -191,7 +196,7 @@ Optional. Requires `summarize` in `mask` and `catalog` in `stages`.
 
 | Field | Required | Rules |
 |---|---|---|
-| `model` | yes | Local model name, 1–128 characters. |
+| `model` | yes | Model name, 1–128 characters, at the server `SUMMARIZE_URL` names. |
 | `per` | yes | Non-empty subset of `file`, `directory`. |
 
 ### `sweep`
@@ -202,7 +207,7 @@ One of `15m`, `hourly`, `daily`, `manual`.
 
 | Field | Required | Rules |
 |---|---|---|
-| `catalog` | yes | `local`: the sealed catalog stays here, and the host queries it. `host`: the sealed catalog is sent to the owner's host, so it can plan while this machine is asleep. Requires the `seal` stage. |
+| `catalog` | yes | `local`: the catalog stays here, and the host queries it through `serve`. `host`: the sealed catalog is meant for the owner's host. The agent writes it to `<out>/sealed/` and neither the agent nor the bastion sends it. In the default `<out>`, `<root>/.agent/`, a bastion from 0.58.1 on never serves it to a host; on an older bastion, a host whose folder grant has `read` can fetch it like any other file unless the grant excludes `.agent`. When `<out>` is inside the folder under any other name, any bastion serves it to such a host like any other file. Requires the `seal` stage. |
 | `vectors` | yes | `local`, the only value. |
 | `plaintext` | yes | `never`, the only value, ever. It is there so the rule is written down in every manifest and can be searched for, not remembered. |
 | `seal_to` | no | List of recipient keys, 1–128 characters each, no duplicates. Requires the `seal` stage. The owner's key is always added by the agent and need not be listed. |

docs/packaging.md

@@ -111,7 +111,7 @@ run there, and the second, with `SIGNATURE=<the line it printed>` and
 `BASTION_TREE` pointing at a privatae-bastion checkout, checks the signature
 against the keys that checkout pins and the keys the published bastion pins,
 and only then publishes. What is signed is a short statement naming the kind
-of release, its version and the manifest's sha256. A bastion from 0.41.0 on
+of release, its version and the manifest's sha256. A bastion from 0.42.0 on
 fetches `version.json.<h>.sig`, refuses a manifest that does not verify, and
 never replaces an installed agent with an older version.
 
@@ -180,7 +180,10 @@ answering `--version` and `doctor --json` and validating the example manifest;
 heads; the `.agent` default output directory; and the doctor's checks against
 unreachable and missing inputs.
 
-This repository has no CI. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the
+Every push to `main` and every pull request into it runs CI
+(`.gitea/workflows/ci.yml`, `.github/workflows/ci.yml`): the 3.8 floor,
+pyflakes, every test on Python 3.8 and a current Python, a personal-data scan
+and a reproducibility check. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the
 checks to run before proposing a change, and `publish.sh` builds the release
 and runs it before anything is uploaded. The installers are not exercised by
 the test suite.

docs/runtime.md

@@ -54,7 +54,25 @@ text with personal data tokenized (only the rules-pass anchors, unless the
 manifest uses `sterilize`). That is the same trust boundary as the folder
 itself, since the source files are already there, but a sync provider now
 receives a second, flattened, indexable copy. If that matters for a folder,
-exclude `.agent` from sync or keep the output elsewhere (`--out`).
+exclude `.agent` from sync or, for an agent run by hand, keep the output
+outside the folder (`--out`).
+
+Whether a host can reach `.agent/` depends on the bastion's version. From
+bastion 0.58.1 on, the bastion never serves it: a host's listings leave out
+every folder named `.agent`, at any depth and in any letter case, and the
+bastion refuses to stat, read or move anything in one, however the path is
+written and whatever link it goes through, and says why. It knows the folder
+by that name only, so an output folder inside the shared folder under any
+other name (`--out`) is served like any other folder. A bastion older than
+0.58.1 treats `.agent/` like any other folder: a host whose folder grant has
+`read` can list it and read the extract store as it is. On those bastions,
+add `.agent` to the paths the grant excludes; the bastion then leaves it out
+of a host's listings and refuses reads of it.
+
+The bastion keeps its agent in `<root>/.agent/` too, except on
+Windows when the bastion runs as the system service, or when it cannot create
+`.agent` in the folder (a read-only folder, for one); it then keeps it in its
+own state directory.
 
 The identity key is never in `.agent/`: it is wherever `FREEAGENT_TLS_KEY`
 points. The PII vault (plaintext values) is in `.agent/sanitize/vault.db` for

docs/sanitize.md

@@ -12,8 +12,11 @@ reads the extract index and store and writes:
 
 - **`sanitize/<aa>/<sha256>.txt`**: the sanitized text store. Personal data is
   replaced by stable `[PII_TYPE_n]` tokens. **From this stage on, later
-  stages read only this store.** The extract store keeps the plaintext and
+  stages read only this store.** The extract store keeps the plaintext. The
-  never leaves the machine.
+  agent never serves it, and while it is in `.agent/` (the default) a
+  bastion from 0.58.1 on does not either. An older bastion lets a folder
+  grant with `read` reach it there unless the grant excludes `.agent`
+  ([runtime.md](runtime.md#where-an-agent-lives-rootagent)).
 - **`vault.db`**: the token-to-value map (in-process mode only), created 0600
   under umask 077. It is the only way back from a token to a value, and it
   stays on the machine. It is `<out>/sanitize/vault.db`, or
@@ -50,10 +53,14 @@ stages, there is no extract index, or the stage **stopped**; `2` completed with
   employers, medical facts) is found by the recall pass.
 - **`flag`**: the rules pass **only**: deterministic, no model, milliseconds.
   It replaces the anchors a pattern can find: e-mail addresses, phone
-  numbers, SSNs, cards, IBANs, routing numbers, crypto addresses, IP and MAC
+  numbers, SSNs, cards, IBANs, US routing numbers after a cue word, legacy and
-  addresses, API keys, private-key blocks, JWTs. It can also replace an
+  P2SH Bitcoin addresses, IP and MAC addresses, API keys, private-key blocks,
-  ordinary number that looks like a phone or ID number. **It does not find names,
+  JWTs ([the list](pii-sterilization.md#the-rules-pass-piidetectpy)). Other
-  street addresses, dates of birth**, or anything else that takes reading for
+  bank account numbers, other crypto addresses (`bc1…`, Ethereum) and ID
+  numbers such as passport numbers are not found. It can also replace an
+  ordinary number that looks like a phone, card or social security number.
+  **It does not find names, street addresses, dates of birth**, or anything
+  else that takes reading for
   meaning (an employer, a school, a medical fact). It is the mode the bastion
   writes into every manifest it deploys. A flag-mode catalog is not free of
   personal data, and the manifest doc says so too.

docs/seal.md

@@ -1,8 +1,10 @@
-# bastion agent `seal` stage: the catalog leaves, nothing else does
+# bastion agent `seal` stage: the catalog in one encrypted bundle
 
-`seal` is the last stage of a run and the one way a dataset leaves the machine:
+`seal` is the last stage of a run: it packs an allowlisted bundle of the
-an allowlisted bundle of the catalog, encrypted with age to the owner and the
+catalog into one file, encrypted with age to the owner and the manifest's named
-manifest's named recipients. It is `bastion_agent/seal.py`.
+recipients. It writes the bundle into the output folder and sends it nowhere.
+`serve` answers queries with cards and excerpts, unsealed
+([serve.md](serve.md)). It is `bastion_agent/seal.py`.
 
 ## What it does
 
@@ -58,8 +60,8 @@ plaintext.
 age (X25519 and ChaCha20-Poly1305) is a small, widely reviewed format with a
 single static binary for Mac, Windows and Linux. The envelope cannot be made
 with the standard library (it has no X25519 and no ChaCha20-Poly1305), so,
-unlike the extract stage's optional tools, **age is required here**: a catalog
+unlike the extract stage's optional tools, **age is required here**: without
-that cannot be sealed does not leave, and the stage says so and exits 1.
+it no bundle is written at all, and the stage says so and exits 1.
 
 ## Signature (optional)
 

docs/serve.md

@@ -1,7 +1,7 @@
 # bastion agent query endpoint: `serve`
 
-`serve` is how a host reads what the agent prepared without the data moving:
+`serve` is how a host reads what the agent prepared without being sent the
-a host (the owner's, or a grantee's) sends a query and gets **sanitized**
+files themselves: a host (the owner's, or a grantee's) sends a query and gets **sanitized**
 chunks and catalog cards back (sanitized by the manifest's mode: in `flag`
 mode, which the bastion deploys, names, street addresses and dates of birth
 remain; [sanitize.md](sanitize.md#modes-manifest-sanitizemode)). Nothing else has a route: not the extract