privatae/bastion-agent
2
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

bastion agent 0.1.14

Browse Source
This commit is contained in:
Privatae
2026-09-26 18:31:40 +00:00
commit 3f1d48bf9b
65 changed files with 15548 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/walk.md
+116
View File
@@ -0,0 +1,116 @@
# bastion agent `walk` stage: the filesystem and metadata map
The walk is the first stage of every run and the cheapest thing a host can
plan from: every entry under the manifest's roots, with its size, time, type
and hash. It is `bastion_agent/walk.py`, standard library only.
## What it produces
`python3 -m bastion_agent walk <manifest.yaml> [--out DIR] [--no-hash] [--require-signed]`
reads the manifest's `scope`, walks every root, and writes two files into
`DIR`. Without `--out`, `DIR` is `<root>/.agent/` for a manifest with one
root, and `build/freeagent/<id>/` (under the current directory) for one with
several. The second keeps the program's first name on purpose: it is output a
person may already have, and renaming it would orphan it. Both files are
written atomically (a temporary file, then a rename), so a reader never sees a
half-written map.
**`walk.jsonl`**: one JSON object per entry, sorted by `(root, path)`, keys
sorted, UTF-8. Identical trees produce byte-identical files.
| Key | Present on | Meaning |
|---|---|---|
| `root` | all | Index into the manifest's `scope.roots`. |
| `path` | all | Path relative to that root, `/`-separated on every OS. |
| `kind` | all | `file`, `dir`, `symlink` (Windows junctions too), `other` (socket, fifo, device). |
| `size` | file | Bytes. |
| `mtime`, `mtime_ns` | file, dir | Modification time as UTC ISO-8601 and as integer nanoseconds (the cache key). |
| `mime` | file | Sniffed type: magic bytes first, then a fixed extension table, then a text/binary probe. `inode/x-empty` for empty files. The `mimetypes` module is not used, because it reads per-machine registries and would make the map depend on the machine. |
| `sha256` | file | Hex digest, or `null` when the file was too large, could not be read, or `--no-hash` was given. |
| `skip` | file | `too_large` when `size > scope.max_file_bytes` (default 256 MiB). The file is still sniffed and listed, never read in full. |
| `target` | symlink | The link text, `/`-separated. |
| `error` | any | `ExceptionName: message`. The entry is recorded and the walk continues. |
**`walk.summary.json`**: `manifest_id`, `manifest_digest` (the signing
digest, which ties the map to the exact manifest that produced it),
`walked_at`, `platform`, `roots` (each with its `resolved` absolute path, and
an `error` when it is absent on this machine), `entries`, `bytes`, `counts`
(`files dirs symlinks other excluded errors too_large hashed hash_reused`) and
`hashing`.
Exit code: `0` clean; `1` the manifest failed validation (or was unsigned
under `--require-signed`); `2` the walk completed with errors recorded
(a missing root, an unreadable directory). The map is written in every case
but the first.
## Rules the walker enforces
- **Excludes prune.** An excluded directory is not descended, so its subtree
costs nothing. Patterns (`scope.exclude`) are `/`-separated and relative to
the root. `**` spans directories; `*` and `?` stay within one segment;
`[..]` and `[!..]` classes work. A pattern with no `/` also matches any
single path segment, so `node_modules` or `*.key` prune at any depth. A
pattern ending in `/**` prunes the directory itself as well as its contents.
Backslashes in patterns are read as `/`. Matching is case-insensitive on
Windows only, as the file system is.
- **The agent's own folder is always excluded.** `.agent` is pruned at any
depth and in any letter case on every OS, whatever the manifest says, so an
agent never indexes its own output (on a case-insensitive Mac volume,
`.Agent` is the same folder).
- **Links are recorded, never followed**, unless `scope.follow_symlinks` is
true. This covers symlinks on every OS and junctions on Windows
(`is_junction()` on Python 3.12 and later, the reparse tag before that).
Following is opt-in because a link out of the root is the usual way a
scoped walk ends up reading the whole disk.
- **Large files are metadata only.** Over `max_file_bytes` the walker reads
8 KiB to sniff the type and stops.
- **Errors are entries, not aborts.** Permission denied on a directory gives a
`dir` entry with `error`, and the walk moves on; a root that does not exist
on this machine is reported in the summary and skipped. The host can see
exactly what it could not see.
- **Hashes are reused.** A previous `walk.jsonl` in the output directory is a
cache keyed on `(root, path)` → `(size, mtime_ns)`. An unchanged file keeps
its digest without being read again, so a daily sweep of a large folder
costs one `stat` per file, not one read. `--no-hash` skips hashing (and the
cache) for a fast first survey.
## Portability
The stage runs on the owner's machine, whatever it is.
- **Standard library only**: nothing to install on the machine that holds the
files.
- **Python 3.8 syntax floor**, enforced by `test_syntax_floor.py` at the
repository root, which parses every module in `bastion_agent/` and `pii/` at
3.8 and denies the newer standard-library calls that parse on 3.8 and then
fail at run time.
- **Roots** may be POSIX absolute paths, `~` or `~/...` (expanded with
`os.path.expanduser`: `HOME` on Unix, `USERPROFILE` on Windows), Windows
drive paths (`C:\...` or `C:/...`) and UNC paths (`\\server\share`).
- **Paths in the map are always `/`-separated**, so a map made on Windows
reads the same as one made on a Mac, and exclude patterns mean the same thing
everywhere.
- **Unicode file names** are written as they are (`ensure_ascii=False`);
undecodable bytes in link targets go through `os.fsdecode`.
- **Timestamps** are nanosecond integers from `st_mtime_ns`, which exists on
all three platforms; the ISO form is derived from it.
## What the stage does not do
- It does not read file contents beyond the 8 KiB sniff and the hash stream.
Extraction is the next stage.
- It does not detect changes by itself. Two maps diff trivially because they
are sorted, and later stages find their work from the hashes.
- It does not treat mounts specially. A network share under a root is walked
like any directory; exclude it if that is not wanted.
## Tests
`python3 -m unittest bastion_agent.test_walk` covers the map's shape and sort
order, MIME sniffing (magic, extension, empty, binary), excludes (segment,
anchored, `/**` pruning, backslashes, Windows case folding, `.agent` in any
case), the too-large skip, `--no-hash`, hash reuse and its invalidation when a
file changes, byte-identical re-runs, a missing root, an unreadable directory,
Unicode names, `~` expansion, and symlinks recorded but not followed, or
followed when the manifest allows it. The symlink tests are skipped on Windows,
where creating a symlink needs a privilege.