bastion-agent/docs/runtime.md

bastion agent runtime: the sweep scheduler and the manifest gate

sweep is the agent proper: the loop that checks the manifest and runs its stages on the manifest's cadence, on the owner's machine. It is bastion_agent/run.py, with the signature check in bastion_agent/x509gate.py, standard library only apart from the manifest loader's PyYAML.

python3 -m bastion_agent sweep <manifest> [--out DIR] [--once] [--allow-unsigned]

Each cycle:

1. Read the manifest again. A re-signed manifest takes effect on the next sweep: a revoked grant, a new root, a changed cadence.
2. The gate. The manifest must carry a signature that verifies: ECDSA P-256 over the canonical payload (signing_payload, the same bytes the payload command prints), from a certificate with exactly one CN, equal to owner.host, that chains to FREEAGENT_CA_ROOT. The root is required: without one the gate refuses rather than trust the certificate beside the manifest. Anything else (unsigned, the wrong key, edited after signing, someone else's certificate, a certificate outside the CA, a certificate with two CNs) and nothing runs. The refusal is appended to run.jsonl like any other run; a manifest too broken to have an id is logged under build/freeagent/refused/ unless --out is given. --allow-unsigned is for development only: the CLI help says so in capitals, and it is written into every run record it touches.
3. The stages, in the manifest's order. The first failure stops the run, so a later stage never consumes a half-finished earlier one. A stage that completes but reports stopped (sanitize or catalog failing closed) counts as failed.
4. Record and wait. One line is appended to run.jsonl, run.state.json is rewritten, and the loop sleeps until the next run is due, or exits for manual (and after --once).

Signing, on the host's side: python3 -m bastion_agent sign <manifest> --cert <cert> --key <key> signs with openssl and writes (or replaces) the signature block; --value <base64> records a signature made elsewhere instead. The bastion signs in its own process with the key it minted for the agent and records it with --value. The certificate path is recorded as given and read relative to the manifest. manifest.md has the details.

Where an agent lives: <root>/.agent/

One agent per folder, and the agent lives with its folder. For a manifest with one root, the commands that run a single stage, sweep and serve default their output directory to <root>/.agent/: the signed manifest and every prepared output (walk map, extract index, sanitized store, chunks, vectors, catalog, run log, sealed bundles). Copy the folder and the agent comes with it; fork it and you have a new one. The walker always excludes .agent/, so an agent never indexes its own output.

What is in .agent/: the extract store is the plaintext of every file in scope (a PDF or DOCX becomes a flat .txt), and the sanitized store is that text with personal data tokenized (only the rules-pass anchors, unless the manifest uses sterilize). That is the same trust boundary as the folder itself, since the source files are already there, but a sync provider now receives a second, flattened, indexable copy. If that matters for a folder, exclude .agent from sync or, for an agent run by hand, keep the output outside the folder (--out).

Whether a host can reach .agent/ depends on the bastion's version. From bastion 0.58.1 on, the bastion never serves it: a host's listings leave out every folder named .agent, at any depth and in any letter case, and the bastion refuses to stat, read or move anything in one, however the path is written and whatever link it goes through, and says why. It knows the folder by that name only, so an output folder inside the shared folder under any other name (--out) is served like any other folder. A bastion older than 0.58.1 treats .agent/ like any other folder: a host whose folder grant has read can list it and read the extract store as it is. On those bastions, add .agent to the paths the grant excludes; the bastion then leaves it out of a host's listings and refuses reads of it.

The bastion keeps its agent in <root>/.agent/ too, except on Windows when the bastion runs as the system service, or when it cannot create .agent in the folder (a read-only folder, for one); it then keeps it in its own state directory.

The identity key is never in .agent/: it is wherever FREEAGENT_TLS_KEY points. The PII vault (plaintext values) is in .agent/sanitize/vault.db for an agent run by hand. A Documents folder is often cloud-synced, so set FREEAGENT_VAULT_DIR to keep the vault out of it; the bastion always sets it to its own state directory. The certificate is public and may sit anywhere.

One machine per folder. The manifest is signed by the machine's own key, and the gate proves "signed by this key", not "the newest". If a synced folder is served from two machines, or a sync conflict restores an earlier signed manifest, the endpoint serves the older terms until the manifest is rewritten. Run one agent per folder, on one machine.

Cadence and backoff

sweep	Period	On failure
15m / hourly / daily	900 / 3600 / 86400 s from the run's start	Retry after 60 s, 120 s, 240 s … capped at the period; consecutive_failures counts up and resets on success.
manual		One run, then exit.

Waits are taken in naps of at most 30 s, so SIGTERM or SIGINT lands quickly; a stop request lets the current run finish and never interrupts a stage.

The lock

<out>/sweep.lock, created atomically (O_EXCL) with the pid and time inside. A second sweep on the same output directory is refused, with the holder named. A lock is stale only when its holder is provably gone: on POSIX the pid decides, and a live pid keeps its lock however long its sweep runs; only where the pid cannot be checked (Windows) or the file cannot be read does age decide, by the file's mtime, after 12 hours. A reclaimed lock is noted in the run record.

Files

File	Content
run.jsonl	Append-only, one record per run: v, started, finished, seconds, manifest, out, manifest_id, manifest_digest, gate (cn, cert, chained, or bypassed with the reason), stages[] (stage, status, seconds, a few counts, error), ok, error (on failure), allow_unsigned, lock_reclaimed (when it happened).
run.state.json	The latest state: last_started, last_finished, last_ok, last_error, consecutive_failures, sweep, next_due, pid, runs_this_process. What a health check reads.
sweep.lock	Present only while a sweep runs.

The stages read their settings from the environment, never the manifest: FREEAGENT_OWNER_RECIPIENT, FREEAGENT_VAULT_DIR, FREEAGENT_CA_ROOT, FREEAGENT_NO_EXTERNAL=1 (built-in readers only), EMBED_URL / EMBED_API, PII_ADDR / PII_API_KEY / PII_RECALL / LLM_URL, SUMMARIZE_URL, and SEAL_SIGN_KEY.

Running it as a service by hand

The bastion runs the agent itself. To run one without a bastion on Linux, bastion_agent/bastion-agent.service is a systemd user unit: the sweep runs under the owner's account with their permissions, reads its environment from ~/.config/bastion-agent/env (create it 0600, as the unit's header shows), writes to ~/.local/share/bastion-agent/out, and runs with UMask=0077, NoNewPrivileges and PrivateTmp, restarting 60 s after a failure. There is no launchd or Scheduled Task file yet; on a Mac or Windows machine run the same command from either.

Tests

python3 -m unittest bastion_agent.test_run bastion_agent.test_x509gate. test_run covers the gate (sign then verify; re-signing replaces the block; an edit after signing refused; a CN mismatch refused; a certificate with two CNs refused; the CA chain accepted against the right root and refused against another; unsigned refused unless --allow-unsigned, and then recorded), runs (stages in order with counts logged; a failing stage stops the run and later stages never execute; an invalid manifest recorded as an error; the lock refusing a second holder and reclaiming a dead one; next_due arithmetic including the backoff cap), and the loop with a fake clock (manual runs once; 15m runs three times, sleeping exactly two periods in naps of at most 30 s, with next_due right; hourly failures retry after 60 s then 120 s with the counter at 3). test_x509gate runs the gate with openssl removed from PATH, through the CLI with no openssl, and against bit flips, expiry and a foreign CA with the same name.