bastion agent 0.1.14

docs/runtime.md

@@ -0,0 +1,131 @@
+# bastion agent runtime: the `sweep` scheduler and the manifest gate
+
+`sweep` is the agent proper: the loop that checks the manifest and runs its
+stages on the manifest's cadence, on the owner's machine. It is
+`bastion_agent/run.py`, with the signature check in `bastion_agent/x509gate.py`,
+standard library only apart from the manifest loader's PyYAML.
+
+`python3 -m bastion_agent sweep <manifest> [--out DIR] [--once] [--allow-unsigned]`
+
+Each cycle:
+
+1. **Read the manifest again.** A re-signed manifest takes effect on the next
+   sweep: a revoked grant, a new root, a changed cadence.
+2. **The gate.** The manifest must carry a signature that verifies: ECDSA
+   P-256 over the canonical payload (`signing_payload`, the same bytes the
+   `payload` command prints), from a certificate with exactly one CN, equal to
+   `owner.host`, that chains to `FREEAGENT_CA_ROOT`. The root is required:
+   without one the gate refuses rather than trust the certificate beside the
+   manifest. Anything else (unsigned, the wrong key, edited after signing,
+   someone else's certificate, a certificate outside the CA, a certificate with
+   two CNs) and **nothing runs**. The refusal is appended to `run.jsonl` like
+   any other run; a manifest too broken to have an id is logged under
+   `build/freeagent/refused/` unless `--out` is given. `--allow-unsigned` is
+   for development only: the CLI help says so in capitals, and it is written
+   into every run record it touches.
+3. **The stages, in the manifest's order.** The first failure stops the run,
+   so a later stage never consumes a half-finished earlier one. A stage that
+   completes but reports `stopped` (sanitize or catalog failing closed) counts
+   as failed.
+4. **Record and wait.** One line is appended to `run.jsonl`, `run.state.json`
+   is rewritten, and the loop sleeps until the next run is due, or exits for
+   `manual` (and after `--once`).
+
+Signing, on the host's side: `python3 -m bastion_agent sign <manifest> --cert <cert> --key <key>`
+signs with openssl and writes (or replaces) the signature block;
+`--value <base64>` records a signature made elsewhere instead. The bastion
+signs in its own process with the key it minted for the agent and records it
+with `--value`. The certificate path is recorded as given and read relative to
+the manifest. [manifest.md](manifest.md) has the details.
+
+## Where an agent lives: `<root>/.agent/`
+
+One agent per folder, and the agent lives with its folder. For a manifest with
+one root, the commands that run a single stage, `sweep` and `serve` default
+their output directory to `<root>/.agent/`: the signed manifest and every prepared output
+(walk map, extract index, sanitized store, chunks, vectors, catalog, run log,
+sealed bundles). Copy the folder and the agent comes with it; fork it and you
+have a new one. The walker always excludes `.agent/`, so an agent never
+indexes its own output.
+
+What is in `.agent/`: the **extract store is the plaintext of every file in
+scope** (a PDF or DOCX becomes a flat `.txt`), and the sanitized store is that
+text with personal data tokenized (only the rules-pass anchors, unless the
+manifest uses `sterilize`). That is the same trust boundary as the folder
+itself, since the source files are already there, but a sync provider now
+receives a second, flattened, indexable copy. If that matters for a folder,
+exclude `.agent` from sync or keep the output elsewhere (`--out`).
+
+The identity key is never in `.agent/`: it is wherever `FREEAGENT_TLS_KEY`
+points. The PII vault (plaintext values) is in `.agent/sanitize/vault.db` for
+an agent run by hand. A Documents folder is often cloud-synced, so set
+`FREEAGENT_VAULT_DIR` to keep the vault out of it; the bastion always sets it
+to its own state directory. The certificate is public and may sit anywhere.
+
+**One machine per folder.** The manifest is signed by the machine's own key,
+and the gate proves "signed by this key", not "the newest". If a synced folder
+is served from two machines, or a sync conflict restores an earlier signed
+manifest, the endpoint serves the older terms until the manifest is rewritten.
+Run one agent per folder, on one machine.
+
+## Cadence and backoff
+
+| `sweep` | Period | On failure |
+|---|---|---|
+| `15m` / `hourly` / `daily` | 900 / 3600 / 86400 s from the run's start | Retry after 60 s, 120 s, 240 s … capped at the period; `consecutive_failures` counts up and resets on success. |
+| `manual` | | One run, then exit. |
+
+Waits are taken in naps of at most 30 s, so SIGTERM or SIGINT lands quickly; a
+stop request lets the current run finish and never interrupts a stage.
+
+## The lock
+
+`<out>/sweep.lock`, created atomically (`O_EXCL`) with the pid and time
+inside. A second sweep on the same output directory is refused, with the holder
+named. A lock is stale only when its holder is provably gone: on POSIX the pid
+decides, and a live pid keeps its lock however long its sweep runs; only where
+the pid cannot be checked (Windows) or the file cannot be read does age decide,
+by the file's mtime, after 12 hours. A reclaimed lock is noted in the run
+record.
+
+## Files
+
+| File | Content |
+|---|---|
+| `run.jsonl` | Append-only, one record per run: `v`, `started`, `finished`, `seconds`, `manifest`, `out`, `manifest_id`, `manifest_digest`, `gate` (`cn`, `cert`, `chained`, or `bypassed` with the reason), `stages[]` (`stage`, `status`, `seconds`, a few counts, `error`), `ok`, `error` (on failure), `allow_unsigned`, `lock_reclaimed` (when it happened). |
+| `run.state.json` | The latest state: `last_started`, `last_finished`, `last_ok`, `last_error`, `consecutive_failures`, `sweep`, `next_due`, `pid`, `runs_this_process`. What a health check reads. |
+| `sweep.lock` | Present only while a sweep runs. |
+
+The stages read their settings from the environment, never the manifest:
+`FREEAGENT_OWNER_RECIPIENT`, `FREEAGENT_VAULT_DIR`, `FREEAGENT_CA_ROOT`,
+`FREEAGENT_NO_EXTERNAL=1` (built-in readers only), `EMBED_URL` / `EMBED_API`,
+`PII_ADDR` / `PII_API_KEY` / `PII_RECALL` / `LLM_URL`, `SUMMARIZE_URL`, and
+`SEAL_SIGN_KEY`.
+
+## Running it as a service by hand
+
+The bastion runs the agent itself. To run one without a bastion on Linux,
+`bastion_agent/bastion-agent.service` is a systemd **user** unit: the sweep
+runs under the owner's account with their permissions, reads its environment
+from `~/.config/bastion-agent/env` (create it 0600, as the unit's header
+shows), writes to `~/.local/share/bastion-agent/out`, and runs with
+`UMask=0077`, `NoNewPrivileges` and `PrivateTmp`, restarting 60 s after a
+failure. There is no launchd or Scheduled Task file yet; on a Mac or Windows
+machine run the same command from either.
+
+## Tests
+
+`python3 -m unittest bastion_agent.test_run bastion_agent.test_x509gate`.
+`test_run` covers the gate (sign then verify; re-signing replaces the block; an
+edit after signing refused; a CN mismatch refused; a certificate with two CNs
+refused; the CA chain accepted against the right root and refused against
+another; unsigned refused unless `--allow-unsigned`, and then recorded), runs
+(stages in order with counts logged; a failing stage stops the run and later
+stages never execute; an invalid manifest recorded as an error; the lock
+refusing a second holder and reclaiming a dead one; `next_due` arithmetic
+including the backoff cap), and the loop with a fake clock (`manual` runs
+once; `15m` runs three times, sleeping exactly two periods in naps of at most
+30 s, with `next_due` right; hourly failures retry after 60 s then 120 s with
+the counter at 3). `test_x509gate` runs the gate with openssl removed from
+PATH, through the CLI with no openssl, and against bit flips, expiry and a
+foreign CA with the same name.