privatae/bastion-agent
2
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

bastion agent 0.1.17

Browse Source
This commit is contained in:
Privatae
2026-10-29 21:42:06 +00:00
parent e3074f50b0
commit 2a6e63ef7b
22 changed files with 226 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+43
View File
@@ -14,6 +14,49 @@ Changes to the release tooling (`publish.sh`, `build.py`, `verify_release.py`)
and to the documentation are listed under the version whose source first
carried them, although they do not change what the `.pyz` does.
## 0.1.17
- **Messages no longer claim what the code does not do.** A manifest with
`sink.catalog: host` and no `seal` stage is refused with "the copy meant
for the host is the sealed bundle", and one with a grant and no `seal`
stage with "any grant requires the 'seal' stage"; neither says that
nothing leaves or is served unsealed, since `serve`'s replies are never
sealed. A malformed `doc` in `read_range` is refused as "the original
file's sha256", which is what the id is. What the manifest accepts and
refuses is unchanged.
- **The documentation says what reaches a Host.** What the agent
returns (catalog cards, search hits, excerpts by document id) comes from
the sanitized text, and the agent sends no file or extracted text to a
Host. A folder grant with `read`, the default, also lets the Host ask the
bastion itself for the folder's file names, sizes and times and for raw
ranges of its files, up to 512 KiB per read. From bastion 0.58.1 on, none
of that reaches the agent's own `.agent/` folder. An older bastion serves
`.agent/` and the plaintext extract store in it wherever it keeps the agent
inside the folder, unless the grant's excluded paths list `.agent`.
`seal` writes its bundle and sends it nowhere. A file card's `sha256` is
the original file's hash.
- **`flag` mode's limits are listed.** It does not look for bank account
numbers other than IBANs and US routing numbers after a word such as
"routing", crypto addresses other than Bitcoin addresses that start with
1 or 3, or ID numbers such as passport numbers.
- **Every model server is the one the environment names.** `EMBED_URL` and
`SUMMARIZE_URL` default to the Ollama port on the same machine, and the
bastion sets `EMBED_URL` to `127.0.0.1:11434`. `sterilize` mode sends the
unsanitized text to the PII service at `PII_ADDR` when that is set;
otherwise to the server at `LLM_URL`, unless `PII_RECALL=gliner` runs the
model inside the agent.
- **Smaller corrections.** The `doc` id is the original file's sha256
(embed.md); CI runs on every push and pull request (packaging.md); signed
manifests are checked from bastion 0.42.0, since 0.41.0 was never
published; the bastion installs Privatae's signed build, which anyone can
rebuild from this source.
- **Tests look for the whole test card number.** Four tests checked that
the four digits `4111` never appear in a sealed bundle, an index or a
catalog. Those also hold sha256 hex and times, where the same four digits
turn up by chance: CI failed on a seal manifest hash ending `...81934111`.
They now look for the full card value, spaced and unspaced.
## 0.1.16
- **Every `search` and `read_range` reply names its sanitize mode**
CONTRIBUTING.md
+13 -5
View File
@@ -43,11 +43,19 @@ refuses adds a test that it refuses.
## What a change has to respect
- **Bytes never leave the machine.** The files, their extracted text and the
vault stay local. What leaves is the sealed catalog, to the recipients the
manifest names, and what `serve` returns to a certificate the manifest
allows. There is no plaintext sink: `sink.plaintext` is `never` and has no
other value.
- **The agent sends no file and no extracted text to a Host.** It keeps the
files, their extracted text and the vault on the machine. The one exception
is `sterilize` mode with a server: the PII service at `PII_ADDR`, or the
recall model at `LLM_URL`, is sent the extracted text, and the PII service
keeps its own vault ([docs/sanitize.md](docs/sanitize.md#where-the-sterilizer-runs)).
What the agent makes for others is the sealed catalog, for the recipients
the manifest names, and what `serve` returns to a certificate the manifest
allows. There is no plaintext sink:
`sink.plaintext` is `never` and has no other value. (The bastion's own
`read` op is separate: it returns ranges of any file in the granted folder
that the grant does not exclude, as they are on disk. From bastion 0.58.1
on it returns nothing in `.agent/`; an older bastion does unless the grant
excludes `.agent`.)
- **Sanitize before anything is written for later stages.** Chunks, vectors,
keywords, titles, summaries, the catalog and everything `serve` returns come
from the sanitized store, never the extract store. A test that searches an
README.md
+35 -15
View File
@@ -9,10 +9,21 @@ It is deployed **from a grant**, and you do not install it yourself: the bastion
(`version.json.<first 16 hex of its sha256>.sig`) against the keys compiled into the bastion, downloads the `.pyz`
and checks it against the sha256 in that signed manifest,
mints a P-256 identity per store, writes and signs a manifest from the grant you
approved, and supervises `sweep` and `serve`. Your files stay on the machine;
what crosses the tunnel is what the grant's mask allows: catalog cards, and
the sanitized excerpts that `search` and `read_range` return, each reply naming
the sanitize mode that made it (`sanitize_mode`, [serve.md](docs/serve.md)).
approved, and supervises `sweep` and `serve`. Nothing is uploaded or synced;
your Host asks, and gets what the grant's mask allows. What the agent returns
is sanitized: catalog cards, search hits, and the excerpts a `read_range` by
document id returns, each reply naming the sanitize mode that made it
(`sanitize_mode`, [serve.md](docs/serve.md)). A grant with `read`, which a
connected folder has by default, also lets the Host ask the bastion itself for
the folder's file names, sizes and modification times, and for one range of
one file per read, up to 512 KiB, as it is on disk and not sanitized. From
bastion 0.58.1 on, none of this reaches the agent's own `.agent/` folder: the
bastion leaves every folder named `.agent` out of a listing and refuses any
request for a path in one. An older bastion serves `.agent/` like any other
folder, including the unsanitized text the agent extracted from every file,
unless `.agent` is among the grant's excluded paths
([runtime.md](docs/runtime.md#where-an-agent-lives-rootagent) says when the
agent's folder is inside the shared folder).
The published version is whatever `privatae.ai/bastion-agent/version.json`
names. `bastion_agent/VERSION` on `main` is the next release, which may not be
@@ -20,7 +31,7 @@ published yet. Release manifests are published signed from 0.1.13 on, and the
`.pyz` can be rebuilt from this source and compared
([Verify a release](#verify-a-release)).
This repository is free software, licensed under the GNU General Public License version 3 or (at your option) any later version (see `LICENSE`). Copyright Privatae LLC. You may run it, read it, change it and redistribute it under that license. The Host the bastion connects to is separate software and is not covered by this license: a C-Series Host is [Privatae/cseries](https://github.com/Privatae/cseries), under AGPL-3.0-only, and a Host on Privatae's own platform is proprietary software. A member runs their own build on their own machine — nothing here depends on a Privatae-held credential or a Privatae-side build.
This repository is free software, licensed under the GNU General Public License version 3 or (at your option) any later version (see `LICENSE`). Copyright Privatae LLC. You may run it, read it, change it and redistribute it under that license. The Host the bastion connects to is separate software and is not covered by this license: a C-Series Host is [Privatae/cseries](https://github.com/Privatae/cseries), under AGPL-3.0-only, and a Host on Privatae's own platform is proprietary software. Anyone can build and run the agent from this source with no Privatae account or key; the bastion installs Privatae's signed build, which this source reproduces.
## What it does
@@ -32,10 +43,10 @@ failure stops the run:
|---|---|---|
| walk | the map of the folder: files, sizes, types, what the grant excludes | [walk.md](docs/walk.md) |
| extract | text out of each file, with built-in extractors or local tools | [extract.md](docs/extract.md) |
| sanitize | replaces the personal data its mode finds with tokens; the originals stay in a vault on this machine. In the mode the bastion deploys it does not find names, street addresses or dates of birth ([below](#how-the-bastion-runs-it)) | [sanitize.md](docs/sanitize.md), [pii-sterilization.md](docs/pii-sterilization.md) |
| chunk, embed | splits the text and embeds it with a model on this machine | [embed.md](docs/embed.md) |
| sanitize | replaces the personal data its mode finds with tokens; the originals stay in a vault on this machine, or in a PII service's own vault when `PII_ADDR` names one. In the mode the bastion deploys it does not find names, street addresses or dates of birth ([below](#how-the-bastion-runs-it)) | [sanitize.md](docs/sanitize.md), [pii-sterilization.md](docs/pii-sterilization.md) |
| chunk, embed | splits the text and embeds it with the model server `EMBED_URL` names: by default, and always under the bastion, the Ollama port on the same machine | [embed.md](docs/embed.md) |
| catalog | the curated dataset built from the stages above | [catalog.md](docs/catalog.md) |
| seal | packs an allowlisted bundle of the catalog and encrypts it with age to the owner and the manifest's named recipients; the one way a dataset leaves the machine | [seal.md](docs/seal.md) |
| seal | packs an allowlisted bundle of the catalog into one file and encrypts it with age to the owner and the manifest's named recipients; it writes the file into the output folder and sends it nowhere | [seal.md](docs/seal.md) |
`serve` answers queries over the index with mutual TLS, returning chunks and
catalog cards from the sanitized text ([serve.md](docs/serve.md)). The gate refuses an
@@ -56,13 +67,22 @@ For each shared folder the bastion:
- mints the agent its own identity and writes and signs its manifest from the
grant, so the agent works only on what you granted;
- sets the sanitize stage to `flag` mode: rules only, no model. It replaces
e-mail addresses, phone numbers, card, bank and social security numbers, IP
and MAC addresses and keys. **It does not find names, street addresses or
dates of birth**, so text a Host receives from the agent still contains them.
`sterilize` mode adds a model on this machine that looks for those, and the
bastion does not set it ([sanitize.md](docs/sanitize.md#modes-manifest-sanitizemode));
- points the embed stage at the model server on this machine (a local ollama)
and at nothing else;
e-mail addresses, phone numbers, US social security numbers, card numbers,
IBANs, US routing numbers that follow a word such as "routing", Bitcoin
addresses that start with 1 or 3, IP and MAC addresses, and keys and tokens
of known formats ([the full list](docs/pii-sterilization.md#the-rules-pass-piidetectpy)).
It does not look for other bank account numbers, other crypto addresses or
ID numbers such as passport numbers, though digits in one can still be
replaced when they look like a phone, card or social security number.
**It does not find names, street addresses or dates of birth**, so text a
Host receives from the agent still contains them. `sterilize` mode adds a
model that looks for those. When `PII_ADDR` is set, the unsanitized text
goes to the PII service it names. Otherwise the model is the server
`LLM_URL` names, which is sent the unsanitized text, or, with
`PII_RECALL=gliner`, one that runs inside the agent. The bastion does not
set this mode ([sanitize.md](docs/sanitize.md#where-the-sterilizer-runs));
- points the embed stage at `127.0.0.1:11434`, the Ollama port on the same
machine, and at nothing else;
- runs `serve` on the loopback address only, with mutual TLS, so the bastion is
the only client;
- keeps the personal-data vault in the bastion's own state directory rather
bastion_agent/VERSION
+1 -1
View File
@@ -1 +1 @@
0.1.16
0.1.17
bastion_agent/catalog.py
+6 -5
View File
@@ -1,6 +1,6 @@
"""bastion agent `catalog` stage — the curated dataset.
The catalog is what a host reasons over and what `seal` ships. One card
The catalog is what a host reasons over and what `seal` packs. One card
per file, one per directory, small and structured, built from:
walk.jsonl metadata (path, size, mtime, mime, hash)
@@ -22,9 +22,10 @@ masked as [PII_TYPE] in the card. Contextual PII in a filename
("Marisol_medical.pdf") is not caught by rules — the honest limit of the
sanitize stage applies here too, and the doc says so.
The optional summarizer (manifest `summarize`) is a local chat model — a
sensor: it fills the `summary` string and nothing else, is reached via
SUMMARIZE_URL (never the manifest), sees only sanitized text, and is
The optional summarizer (manifest `summarize`) is a chat model at the
server SUMMARIZE_URL names (never the manifest; by default the Ollama port
on this machine, 127.0.0.1:11434) — a sensor: it fills the `summary`
string and nothing else, is sent only sanitized text, and is
cached per (document, model, version). The first failure stops
summarizing (cards still get written, `summary_error` set, exit 1).
"""
@@ -151,7 +152,7 @@ def mask_name(name, detect):
def mask_path(path, detect):
"""Every component of a relative path through mask_name — a parent
directory can carry an address as easily as a filename, and the file
card's path is what seal ships. Returns (masked
card's path is what serve returns and seal packs. Returns (masked
path, number of components changed)."""
if not path:
return path, 0
bastion_agent/manifest.py
+3 -2
View File
@@ -539,9 +539,10 @@ def validate_document(doc, require_signed=False):
errors.append("summarize: requires the 'catalog' stage (summaries live in the catalog)")
if stages is not None:
if sink is not None and sink.catalog == "host" and "seal" not in stages:
errors.append("sink: catalog 'host' requires the 'seal' stage — nothing leaves unsealed")
errors.append("sink: catalog 'host' requires the 'seal' stage — the copy meant for the host "
"is the sealed bundle")
if grants and "seal" not in stages:
errors.append("grants: any grant requires the 'seal' stage — nothing is served unsealed")
errors.append("grants: any grant requires the 'seal' stage")
if sink is not None and sink.seal_to and "seal" not in stages:
errors.append("sink: seal_to given but 'seal' is not in stages")
bastion_agent/sanitize.py
+9 -2
View File
@@ -12,8 +12,15 @@ SECOND store:
<out>/sanitize.summary.json
From here on, later stages read ONLY the sanitized store. The extract
store keeps the plaintext and stays on this machine; the vault is the
only way back (`/rehydrate` semantics, local).
store keeps the plaintext. This agent never serves it, but it sits in
<out>, by default <root>/.agent/ inside the folder, where a sync provider
can reach it. A bastion from 0.58.1 on never serves a folder named .agent
to a host; an older bastion does to a host whose folder grant has `read`
unless the grant excludes .agent, and any bastion serves an <out> inside
the folder under another name.
The vault is the only way back from a token to a value (`/rehydrate`
semantics): a file on this machine in-process, the service's own with
PII_ADDR.
Two ways to reach the sterilizer, chosen from the environment, never the
manifest (no addresses in manifests — docs/pii-sterilization.md):
bastion_agent/serve.py
+8 -6
View File
@@ -1,9 +1,10 @@
"""bastion agent query endpoint — the container on the mesh.
The other half of "access moves, data does not". Serves the owner's host
and the manifest's grantees over MUTUAL TLS; nothing here ever reads the
extract store, the vault, or the vectors' raw table for a client — the
only bytes that leave are sanitized chunks and catalog cards.
Serves the owner's host and the manifest's grantees over MUTUAL TLS;
nothing here ever reads the extract store, the vault, or the vectors' raw
table for a client. What it returns is sanitized chunks, catalog cards, the
run state and health. (The bastion's own `read` op is not this server: it
reads files in the granted folder as they are on disk.)
Identity is the presented client certificate:
- the TLS layer requires a client cert that chains to FREEAGENT_CA_ROOT
@@ -16,7 +17,7 @@ Ops (JSON over HTTPS):
POST /search {query, k?} search (owner, grantee w/ op)
POST /read_range {doc, start, end} read_range (owner, grantee w/ op)
from the SANITIZED store only,
<= READ_RANGE_MAX bytes
<= READ_RANGE_MAX characters
Both answer with `sanitize_mode`, the mode the sanitize stage recorded for
the text in the reply ("flag" or "sterilize"; the weakest across a
search's hits; "unknown" when the record does not vouch for that text; a
@@ -414,7 +415,8 @@ class Handler(BaseHTTPRequestHandler):
start = int(req.get("start", 0))
end = int(req["end"])
if not isinstance(doc, str) or len(doc) != 64 or not all(c in "0123456789abcdef" for c in doc):
raise ValueError("doc must be a sanitized-document sha256")
raise ValueError("doc must be a document id: the original file's sha256, "
"64 lowercase hex")
if start < 0 or end <= start:
raise ValueError("start/end out of order")
end = min(end, start + READ_RANGE_MAX)
bastion_agent/test_catalog.py
+4 -2
View File
@@ -189,7 +189,8 @@ class StageTests(Case):
def test_catalog_carries_no_values(self):
_, files, dirs = self.run_stage(manifest(self.root))
blob = json.dumps(files) + json.dumps(dirs)
for leak in ("m@example.com", "4111"):
# the whole card: four digits alone also turn up inside hashes and times
for leak in ("m@example.com", "4111 1111 1111 1111", "4111111111111111"):
self.assertNotIn(leak, blob)
def test_summaries_file_and_directory_with_cache(self):
@@ -205,7 +206,8 @@ class StageTests(Case):
# the model only ever saw sanitized text
for _, _, prompt in _StubChat.calls:
self.assertNotIn("m@example.com", prompt)
self.assertNotIn("4111", prompt)
for card in ("4111 1111 1111 1111", "4111111111111111"):
self.assertNotIn(card, prompt)
# second run: all summaries from cache, no calls
_StubChat.calls = []
s2, _, _ = self.run_stage(m)
bastion_agent/test_manifest.py
+8 -2
View File
@@ -123,12 +123,18 @@ class DoctrineTests(unittest.TestCase):
def test_host_catalog_needs_seal(self):
doc = with_(FULL, stages=["walk", "extract", "sanitize", "chunk", "embed", "catalog"],
grants=None, sink={"catalog": "host", "vectors": "local", "plaintext": "never"})
self.assertTrue(any("unsealed" in e for e in errors_of(doc)))
errs = errors_of(doc)
self.assertTrue(any("sink: catalog 'host' requires the 'seal' stage" in e for e in errs))
# serve's replies are never sealed, so the message must not claim that
# nothing leaves, or is served, unsealed
self.assertFalse(any("unsealed" in e for e in errs), errs)
def test_grant_needs_seal(self):
doc = with_(FULL, stages=["walk", "extract", "sanitize", "chunk", "embed", "catalog"],
sink={"catalog": "local", "vectors": "local", "plaintext": "never"})
self.assertTrue(any("grants: any grant requires the 'seal'" in e for e in errors_of(doc)))
errs = errors_of(doc)
self.assertTrue(any("grants: any grant requires the 'seal'" in e for e in errs))
self.assertFalse(any("unsealed" in e for e in errs), errs)
def test_grant_search_needs_mask_search(self):
doc = with_(FULL, mask=["read", "index", "summarize"])
bastion_agent/test_sanitize.py
+5 -2
View File
@@ -104,7 +104,8 @@ class FlagModeTests(Case):
s, idx = self.run_stage(manifest(self.root, "flag"))
out = self.text_of(idx["note.txt"])
self.assertNotIn("m@example.com", out)
self.assertNotIn("4111", out)
for card in ("4111 1111 1111 1111", "4111111111111111"):
self.assertNotIn(card, out)
self.assertIn("Marisol", out) # contextual PII is NOT caught by flag mode
self.assertIn("[PII_EMAIL_1]", out)
self.assertIn("[PII_CREDIT_CARD_1]", out)
@@ -121,7 +122,9 @@ class FlagModeTests(Case):
s, idx = self.run_stage(manifest(self.root, "flag"))
blob = json.dumps(idx) + json.dumps(s)
self.assertNotIn("m@example.com", blob)
self.assertNotIn("4111", blob)
# the whole card: four digits alone also turn up inside hashes and times
for card in ("4111 1111 1111 1111", "4111111111111111"):
self.assertNotIn(card, blob)
@unittest.skipIf(IS_WINDOWS, "POSIX permission bits")
def test_vault_is_private(self):
bastion_agent/test_seal.py
+5 -2
View File
@@ -112,9 +112,12 @@ class FailClosedTests(Case):
for forbidden in ("walk.jsonl", "extract.jsonl", "sanitize.jsonl", "vectors.db", "vault.db"):
self.assertFalse(any(n.endswith(forbidden) for n in names), forbidden)
self.assertFalse(any(n.startswith(("extract/", "sanitize/")) for n in names))
# nothing in the bundle is a value
# nothing in the bundle is a value. The card is looked for whole, spaced
# or not: four of its digits also turn up by chance inside the bundle's
# sha256 hex (CI run 2483 failed on "...81934111").
self.assertNotIn(b"m@example.com", plain)
self.assertNotIn(b"4111", plain)
self.assertNotIn(b"4111 1111 1111 1111", plain)
self.assertNotIn(b"4111111111111111", plain)
self.assertIn(b"[PII_EMAIL_1]", plain) # sanitized head is there, as designed
self.assertEqual(set(inv["members"]), set(names))
bastion_agent/test_serve.py
+4 -1
View File
@@ -233,7 +233,10 @@ class ServeTests(unittest.TestCase):
self.assertIn("m@example.com", fh.read()) # the plaintext exists on disk; it has no route
st, _ = self.call("owner", "POST", "/read_range", {"doc": "0" * 64, "start": 0, "end": 10})
self.assertEqual(st, 404)
self.assertEqual(self.call("owner", "POST", "/read_range", {"doc": "zz", "start": 0, "end": 1})[0], 400)
st, body = self.call("owner", "POST", "/read_range", {"doc": "zz", "start": 0, "end": 1})
self.assertEqual(st, 400)
# the id is the original file's hash (the extract and sanitized stores share it)
self.assertIn("original file's sha256", body["error"])
self.assertEqual(self.call("owner", "POST", "/search", {"query": ""})[0], 400)
self.assertEqual(self.call("owner", "POST", "/nope", {})[0], 404)
docs/catalog.md
+6 -1
View File
@@ -57,7 +57,12 @@ MIME types), `keywords` (the top 12 over the merged term counts), `entities`
hyphens, so `invoice-m@example.com.pdf` becomes `[PII_EMAIL]`; `mime` and
`size` still say what it was. The summary counts masked segments as
`names_masked`. **A name like `Marisol_medical.pdf` is not caught**: that is
contextual personal data, the same limit as the rules-only `flag` mode.
contextual personal data, the same limit as the rules-only `flag` mode. The
mask is the catalog's: the bastion's own listing, for a grant with `read`,
returns names as they are.
- **Not hidden: content hashes.** Each file card's `sha256`, also the `doc` id in
search hits and excerpts, is the hash of the original file, so whoever holds
a copy of a file can confirm it is in the folder.
## The summarizer (a sensor)
docs/embed.md
+8 -6
View File
@@ -1,7 +1,8 @@
# bastion agent `chunk` and `embed` stages, and `search`
`chunk` splits the sanitized text into overlapping windows, `embed` turns each
window into a vector with a model on this machine, and `search` answers a
window into a vector with the model server `EMBED_URL` names (by default the
Ollama port on the same machine), and `search` answers a
query against those vectors. They are `bastion_agent/embed.py`, standard
library only.
@@ -9,9 +10,10 @@ library only.
`python3 -m bastion_agent chunk <manifest> [--out DIR] [--require-signed]`
reads the **sanitized** store (never the extract store) and writes
`chunks.jsonl`: one row per window (`chunk_id`, `doc` (the sanitized text's
sha256), `idx`, `start` and `end` character offsets, `chars`, `tokens`, `v`),
sorted by `(doc, idx)`, plus `chunk.summary.json`.
`chunks.jsonl`: one row per window (`chunk_id`, `doc` (the original file's
sha256, the id the extract and sanitized stores share), `idx`, `start` and
`end` character offsets, `chars`, `tokens`, `v`), sorted by `(doc, idx)`, plus
`chunk.summary.json`.
- Windows are `embed.chunk.size_tokens` long and advance by
`size - overlap` tokens; the last window may be shorter. With any overlap,
@@ -27,8 +29,8 @@ sorted by `(doc, idx)`, plus `chunk.summary.json`.
next. A changed document has a new sha256, so all of its windows get new ids
and are embedded again. Identical windows inside one document (a repeated
paragraph) share an id and are embedded once.
- Duplicate documents (the same sanitized content at several paths) are
chunked once.
- Duplicate documents (the same file content at several paths) are chunked
once.
## embed
docs/extract.md
+6 -2
View File
@@ -25,8 +25,12 @@ the stage refuses otherwise) and writes:
recorded, which ties the walk map and the text store to one manifest.
`DIR` defaults as for the walk (`<root>/.agent/` for one root). This text
store is the plaintext of every file in scope, and it never leaves the machine;
see [runtime.md](runtime.md#where-an-agent-lives-rootagent).
store is the plaintext of every file in scope. The agent never serves it, but
by default it sits in the folder, so if the folder is synced, the sync provider
gets a copy. A bastion from 0.58.1 on never serves `.agent/` to a host. On an
older bastion, a host whose folder grant has `read` can read the store like
any other file unless the grant excludes `.agent`; see
[runtime.md](runtime.md#where-an-agent-lives-rootagent).
| `status` | Meaning |
|---|---|
docs/manifest.md
+15 -10
View File
@@ -19,19 +19,21 @@ the control flow and every change, and a model proposes, never acts.
The owner's host writes a **job manifest**, and it is signed (by the bastion,
in the managed install). The agent executes exactly that manifest and nothing
else. The output is a **catalog** (small, structured cards per file and per
directory) plus local vectors. The files themselves never move; a host
reasons over them by querying the agent. Sharing is a grant that lets another
host query this catalog: access moves, data does not.
directory) plus local vectors. The agent never sends the files themselves; a
host reasons over them by querying the agent. Sharing is a grant that lets
another host search the sanitized text or read ranges of it, until a set date
([`grants`](#grants)).
## Rules the schema enforces
1. **Plaintext never leaves.** `sink.plaintext` is the constant `never`. Any
1. **No plaintext sink.** `sink.plaintext` is the constant `never`. Any
other value is a validation error, not a setting.
2. **Nothing runs that was not granted.** `mask` is the ceiling; every stage
must be covered by an op in the mask, or the manifest is refused before it
runs.
3. **Nothing leaves unsealed.** A catalog sent to the host, `seal_to`
recipients, or any grant requires the `seal` stage.
3. **Sharing needs `seal`.** `sink.catalog: host`, `seal_to` recipients, or
any grant requires the `seal` stage. What `serve` returns, to the owner or
to a grantee, is never sealed.
4. **No addresses.** The sanitizer and the embedder are found from the
machine's environment when the agent runs. The manifest never carries a
host:port.
@@ -179,11 +181,14 @@ Present **if and only if** `embed` is in `stages`.
| Field | Required | Rules |
|---|---|---|
| `model` | yes | Model name, 1–128 characters. |
| `dim` | yes | Integer 1–8192. The stage refuses to run if the local endpoint's vectors have another dimension. Vectors are stored per model, so a later model change never discards existing vectors. |
| `dim` | yes | Integer 1–8192. The stage refuses to write if the embedding server (`EMBED_URL`) returns vectors of another dimension. Vectors are stored per model, so a later model change never discards existing vectors. |
| `chunk.size_tokens` | yes | Integer 1–8192. |
| `chunk.overlap_tokens` | yes | Integer, 0 ≤ overlap < size. |
There is no endpoint field. Embedding is local, always.
There is no endpoint field: the embedder is the server `EMBED_URL` names in
the agent's environment, by default the Ollama port on the same machine. The
bastion sets it to `127.0.0.1:11434`; an agent run by hand can be pointed
anywhere.
### `summarize`
@@ -191,7 +196,7 @@ Optional. Requires `summarize` in `mask` and `catalog` in `stages`.
| Field | Required | Rules |
|---|---|---|
| `model` | yes | Local model name, 1–128 characters. |
| `model` | yes | Model name, 1–128 characters, at the server `SUMMARIZE_URL` names. |
| `per` | yes | Non-empty subset of `file`, `directory`. |
### `sweep`
@@ -202,7 +207,7 @@ One of `15m`, `hourly`, `daily`, `manual`.
| Field | Required | Rules |
|---|---|---|
| `catalog` | yes | `local`: the sealed catalog stays here, and the host queries it. `host`: the sealed catalog is sent to the owner's host, so it can plan while this machine is asleep. Requires the `seal` stage. |
| `catalog` | yes | `local`: the catalog stays here, and the host queries it through `serve`. `host`: the sealed catalog is meant for the owner's host. The agent writes it to `<out>/sealed/` and neither the agent nor the bastion sends it. In the default `<out>`, `<root>/.agent/`, a bastion from 0.58.1 on never serves it to a host; on an older bastion, a host whose folder grant has `read` can fetch it like any other file unless the grant excludes `.agent`. When `<out>` is inside the folder under any other name, any bastion serves it to such a host like any other file. Requires the `seal` stage. |
| `vectors` | yes | `local`, the only value. |
| `plaintext` | yes | `never`, the only value, ever. It is there so the rule is written down in every manifest and can be searched for, not remembered. |
| `seal_to` | no | List of recipient keys, 1–128 characters each, no duplicates. Requires the `seal` stage. The owner's key is always added by the agent and need not be listed. |
docs/packaging.md
+5 -2
View File
@@ -111,7 +111,7 @@ run there, and the second, with `SIGNATURE=<the line it printed>` and
`BASTION_TREE` pointing at a privatae-bastion checkout, checks the signature
against the keys that checkout pins and the keys the published bastion pins,
and only then publishes. What is signed is a short statement naming the kind
of release, its version and the manifest's sha256. A bastion from 0.41.0 on
of release, its version and the manifest's sha256. A bastion from 0.42.0 on
fetches `version.json.<h>.sig`, refuses a manifest that does not verify, and
never replaces an installed agent with an older version.
@@ -180,7 +180,10 @@ answering `--version` and `doctor --json` and validating the example manifest;
heads; the `.agent` default output directory; and the doctor's checks against
unreachable and missing inputs.
This repository has no CI. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the
Every push to `main` and every pull request into it runs CI
(`.gitea/workflows/ci.yml`, `.github/workflows/ci.yml`): the 3.8 floor,
pyflakes, every test on Python 3.8 and a current Python, a personal-data scan
and a reproducibility check. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the
checks to run before proposing a change, and `publish.sh` builds the release
and runs it before anything is uploaded. The installers are not exercised by
the test suite.
docs/runtime.md
+19 -1
View File
@@ -54,7 +54,25 @@ text with personal data tokenized (only the rules-pass anchors, unless the
manifest uses `sterilize`). That is the same trust boundary as the folder
itself, since the source files are already there, but a sync provider now
receives a second, flattened, indexable copy. If that matters for a folder,
exclude `.agent` from sync or keep the output elsewhere (`--out`).
exclude `.agent` from sync or, for an agent run by hand, keep the output
outside the folder (`--out`).
Whether a host can reach `.agent/` depends on the bastion's version. From
bastion 0.58.1 on, the bastion never serves it: a host's listings leave out
every folder named `.agent`, at any depth and in any letter case, and the
bastion refuses to stat, read or move anything in one, however the path is
written and whatever link it goes through, and says why. It knows the folder
by that name only, so an output folder inside the shared folder under any
other name (`--out`) is served like any other folder. A bastion older than
0.58.1 treats `.agent/` like any other folder: a host whose folder grant has
`read` can list it and read the extract store as it is. On those bastions,
add `.agent` to the paths the grant excludes; the bastion then leaves it out
of a host's listings and refuses reads of it.
The bastion keeps its agent in `<root>/.agent/` too, except on
Windows when the bastion runs as the system service, or when it cannot create
`.agent` in the folder (a read-only folder, for one); it then keeps it in its
own state directory.
The identity key is never in `.agent/`: it is wherever `FREEAGENT_TLS_KEY`
points. The PII vault (plaintext values) is in `.agent/sanitize/vault.db` for
docs/sanitize.md
+13 -6
View File
@@ -12,8 +12,11 @@ reads the extract index and store and writes:
- **`sanitize/<aa>/<sha256>.txt`**: the sanitized text store. Personal data is
replaced by stable `[PII_TYPE_n]` tokens. **From this stage on, later
stages read only this store.** The extract store keeps the plaintext and
never leaves the machine.
stages read only this store.** The extract store keeps the plaintext. The
agent never serves it, and while it is in `.agent/` (the default) a
bastion from 0.58.1 on does not either. An older bastion lets a folder
grant with `read` reach it there unless the grant excludes `.agent`
([runtime.md](runtime.md#where-an-agent-lives-rootagent)).
- **`vault.db`**: the token-to-value map (in-process mode only), created 0600
under umask 077. It is the only way back from a token to a value, and it
stays on the machine. It is `<out>/sanitize/vault.db`, or
@@ -50,10 +53,14 @@ stages, there is no extract index, or the stage **stopped**; `2` completed with
employers, medical facts) is found by the recall pass.
- **`flag`**: the rules pass **only**: deterministic, no model, milliseconds.
It replaces the anchors a pattern can find: e-mail addresses, phone
numbers, SSNs, cards, IBANs, routing numbers, crypto addresses, IP and MAC
addresses, API keys, private-key blocks, JWTs. It can also replace an
ordinary number that looks like a phone or ID number. **It does not find names,
street addresses, dates of birth**, or anything else that takes reading for
numbers, SSNs, cards, IBANs, US routing numbers after a cue word, legacy and
P2SH Bitcoin addresses, IP and MAC addresses, API keys, private-key blocks,
JWTs ([the list](pii-sterilization.md#the-rules-pass-piidetectpy)). Other
bank account numbers, other crypto addresses (`bc1…`, Ethereum) and ID
numbers such as passport numbers are not found. It can also replace an
ordinary number that looks like a phone, card or social security number.
**It does not find names, street addresses, dates of birth**, or anything
else that takes reading for
meaning (an employer, a school, a medical fact). It is the mode the bastion
writes into every manifest it deploys. A flag-mode catalog is not free of
personal data, and the manifest doc says so too.
docs/seal.md
+8 -6
View File
@@ -1,8 +1,10 @@
# bastion agent `seal` stage: the catalog leaves, nothing else does
# bastion agent `seal` stage: the catalog in one encrypted bundle
`seal` is the last stage of a run and the one way a dataset leaves the machine:
an allowlisted bundle of the catalog, encrypted with age to the owner and the
manifest's named recipients. It is `bastion_agent/seal.py`.
`seal` is the last stage of a run: it packs an allowlisted bundle of the
catalog into one file, encrypted with age to the owner and the manifest's named
recipients. It writes the bundle into the output folder and sends it nowhere.
`serve` answers queries with cards and excerpts, unsealed
([serve.md](serve.md)). It is `bastion_agent/seal.py`.
## What it does
@@ -58,8 +60,8 @@ plaintext.
age (X25519 and ChaCha20-Poly1305) is a small, widely reviewed format with a
single static binary for Mac, Windows and Linux. The envelope cannot be made
with the standard library (it has no X25519 and no ChaCha20-Poly1305), so,
unlike the extract stage's optional tools, **age is required here**: a catalog
that cannot be sealed does not leave, and the stage says so and exits 1.
unlike the extract stage's optional tools, **age is required here**: without
it no bundle is written at all, and the stage says so and exits 1.
## Signature (optional)
docs/serve.md
+2 -2
View File
@@ -1,7 +1,7 @@
# bastion agent query endpoint: `serve`
`serve` is how a host reads what the agent prepared without the data moving:
a host (the owner's, or a grantee's) sends a query and gets **sanitized**
`serve` is how a host reads what the agent prepared without being sent the
files themselves: a host (the owner's, or a grantee's) sends a query and gets **sanitized**
chunks and catalog cards back (sanitized by the manifest's mode: in `flag`
mode, which the bastion deploys, names, street addresses and dates of birth
remain; [sanitize.md](sanitize.md#modes-manifest-sanitizemode)). Nothing else has a route: not the extract