117 lines
6.5 KiB
Markdown
117 lines
6.5 KiB
Markdown
# bastion agent `walk` stage: the filesystem and metadata map
|
|
|
|
The walk is the first stage of every run and the cheapest thing a host can
|
|
plan from: every entry under the manifest's roots, with its size, time, type
|
|
and hash. It is `bastion_agent/walk.py`, standard library only.
|
|
|
|
## What it produces
|
|
|
|
`python3 -m bastion_agent walk <manifest.yaml> [--out DIR] [--no-hash] [--require-signed]`
|
|
reads the manifest's `scope`, walks every root, and writes two files into
|
|
`DIR`. Without `--out`, `DIR` is `<root>/.agent/` for a manifest with one
|
|
root, and `build/freeagent/<id>/` (under the current directory) for one with
|
|
several. The second keeps the program's first name on purpose: it is output a
|
|
person may already have, and renaming it would orphan it. Both files are
|
|
written atomically (a temporary file, then a rename), so a reader never sees a
|
|
half-written map.
|
|
|
|
**`walk.jsonl`**: one JSON object per entry, sorted by `(root, path)`, keys
|
|
sorted, UTF-8. Identical trees produce byte-identical files.
|
|
|
|
| Key | Present on | Meaning |
|
|
|---|---|---|
|
|
| `root` | all | Index into the manifest's `scope.roots`. |
|
|
| `path` | all | Path relative to that root, `/`-separated on every OS. |
|
|
| `kind` | all | `file`, `dir`, `symlink` (Windows junctions too), `other` (socket, fifo, device). |
|
|
| `size` | file | Bytes. |
|
|
| `mtime`, `mtime_ns` | file, dir | Modification time as UTC ISO-8601 and as integer nanoseconds (the cache key). |
|
|
| `mime` | file | Sniffed type: magic bytes first, then a fixed extension table, then a text/binary probe. `inode/x-empty` for empty files. The `mimetypes` module is not used, because it reads per-machine registries and would make the map depend on the machine. |
|
|
| `sha256` | file | Hex digest, or `null` when the file was too large, could not be read, or `--no-hash` was given. |
|
|
| `skip` | file | `too_large` when `size > scope.max_file_bytes` (default 256 MiB). The file is still sniffed and listed, never read in full. |
|
|
| `target` | symlink | The link text, `/`-separated. |
|
|
| `error` | any | `ExceptionName: message`. The entry is recorded and the walk continues. |
|
|
|
|
**`walk.summary.json`**: `manifest_id`, `manifest_digest` (the signing
|
|
digest, which ties the map to the exact manifest that produced it),
|
|
`walked_at`, `platform`, `roots` (each with its `resolved` absolute path, and
|
|
an `error` when it is absent on this machine), `entries`, `bytes`, `counts`
|
|
(`files dirs symlinks other excluded errors too_large hashed hash_reused`) and
|
|
`hashing`.
|
|
|
|
Exit code: `0` clean; `1` the manifest failed validation (or was unsigned
|
|
under `--require-signed`); `2` the walk completed with errors recorded
|
|
(a missing root, an unreadable directory). The map is written in every case
|
|
but the first.
|
|
|
|
## Rules the walker enforces
|
|
|
|
- **Excludes prune.** An excluded directory is not descended, so its subtree
|
|
costs nothing. Patterns (`scope.exclude`) are `/`-separated and relative to
|
|
the root. `**` spans directories; `*` and `?` stay within one segment;
|
|
`[..]` and `[!..]` classes work. A pattern with no `/` also matches any
|
|
single path segment, so `node_modules` or `*.key` prune at any depth. A
|
|
pattern ending in `/**` prunes the directory itself as well as its contents.
|
|
Backslashes in patterns are read as `/`. Matching is case-insensitive on
|
|
Windows only, as the file system is.
|
|
- **The agent's own folder is always excluded.** `.agent` is pruned at any
|
|
depth and in any letter case on every OS, whatever the manifest says, so an
|
|
agent never indexes its own output (on a case-insensitive Mac volume,
|
|
`.Agent` is the same folder).
|
|
- **Links are recorded, never followed**, unless `scope.follow_symlinks` is
|
|
true. This covers symlinks on every OS and junctions on Windows
|
|
(`is_junction()` on Python 3.12 and later, the reparse tag before that).
|
|
Following is opt-in because a link out of the root is the usual way a
|
|
scoped walk ends up reading the whole disk.
|
|
- **Large files are metadata only.** Over `max_file_bytes` the walker reads
|
|
8 KiB to sniff the type and stops.
|
|
- **Errors are entries, not aborts.** Permission denied on a directory gives a
|
|
`dir` entry with `error`, and the walk moves on; a root that does not exist
|
|
on this machine is reported in the summary and skipped. The host can see
|
|
exactly what it could not see.
|
|
- **Hashes are reused.** A previous `walk.jsonl` in the output directory is a
|
|
cache keyed on `(root, path)` → `(size, mtime_ns)`. An unchanged file keeps
|
|
its digest without being read again, so a daily sweep of a large folder
|
|
costs one `stat` per file, not one read. `--no-hash` skips hashing (and the
|
|
cache) for a fast first survey.
|
|
|
|
## Portability
|
|
|
|
The stage runs on the owner's machine, whatever it is.
|
|
|
|
- **Standard library only**: nothing to install on the machine that holds the
|
|
files.
|
|
- **Python 3.8 syntax floor**, enforced by `test_syntax_floor.py` at the
|
|
repository root, which parses every module in `bastion_agent/` and `pii/` at
|
|
3.8 and denies the newer standard-library calls that parse on 3.8 and then
|
|
fail at run time.
|
|
- **Roots** may be POSIX absolute paths, `~` or `~/...` (expanded with
|
|
`os.path.expanduser`: `HOME` on Unix, `USERPROFILE` on Windows), Windows
|
|
drive paths (`C:\...` or `C:/...`) and UNC paths (`\\server\share`).
|
|
- **Paths in the map are always `/`-separated**, so a map made on Windows
|
|
reads the same as one made on a Mac, and exclude patterns mean the same thing
|
|
everywhere.
|
|
- **Unicode file names** are written as they are (`ensure_ascii=False`);
|
|
undecodable bytes in link targets go through `os.fsdecode`.
|
|
- **Timestamps** are nanosecond integers from `st_mtime_ns`, which exists on
|
|
all three platforms; the ISO form is derived from it.
|
|
|
|
## What the stage does not do
|
|
|
|
- It does not read file contents beyond the 8 KiB sniff and the hash stream.
|
|
Extraction is the next stage.
|
|
- It does not detect changes by itself. Two maps diff trivially because they
|
|
are sorted, and later stages find their work from the hashes.
|
|
- It does not treat mounts specially. A network share under a root is walked
|
|
like any directory; exclude it if that is not wanted.
|
|
|
|
## Tests
|
|
|
|
`python3 -m unittest bastion_agent.test_walk` covers the map's shape and sort
|
|
order, MIME sniffing (magic, extension, empty, binary), excludes (segment,
|
|
anchored, `/**` pruning, backslashes, Windows case folding, `.agent` in any
|
|
case), the too-large skip, `--no-hash`, hash reuse and its invalidation when a
|
|
file changes, byte-identical re-runs, a missing root, an unreadable directory,
|
|
Unicode names, `~` expansion, and symlinks recorded but not followed, or
|
|
followed when the manifest allows it. The symlink tests are skipped on Windows,
|
|
where creating a symlink needs a privilege.
|