2 Commits

Author SHA1 Message Date
Privatae 081763a675 bastion 0.57.0
ci / personal-data (push) Successful in 4s
ci / test (push) Successful in 4m30s
ci / fuzz (push) Successful in 3m25s
2026-10-28 20:35:45 +00:00
Privatae 4012da0b75 bastion 0.56.0
ci / personal-data (push) Successful in 10s
ci / test (push) Successful in 5m30s
ci / fuzz (push) Successful in 3m16s
2026-10-28 02:24:07 +00:00
75 changed files with 8547 additions and 306 deletions
+6
View File
@@ -131,8 +131,14 @@ jobs:
- name: every fuzz target, 5 seconds each - name: every fuzz target, 5 seconds each
# A short pass on every commit. Run deploy/fuzz.sh with its default of # A short pass on every commit. Run deploy/fuzz.sh with its default of
# 20 seconds before changing a wall a host's input goes through. # 20 seconds before changing a wall a host's input goes through.
# fuzz.sh does not fail on Go's own -fuzztime report, on the Go
# releases whose fuzzer reports its deadline as a failure, and fails on
# anything else; test_fuzz.sh checks both, with planted failing
# inputs, before the targets run, and fails once the image's Go no
# longer needs the workaround.
run: | run: |
docker run --rm --cpus 6 -e GOMAXPROCS=6 -v "$PWD":/src -w /src \ docker run --rm --cpus 6 -e GOMAXPROCS=6 -v "$PWD":/src -w /src \
-e GOTOOLCHAIN=local -e CGO_ENABLED=0 "$GO_IMAGE" sh -euc ' -e GOTOOLCHAIN=local -e CGO_ENABLED=0 "$GO_IMAGE" sh -euc '
apk add -q --no-cache bash apk add -q --no-cache bash
bash deploy/test_fuzz.sh
deploy/fuzz.sh 5' deploy/fuzz.sh 5'
+7
View File
@@ -166,6 +166,13 @@ jobs:
git fetch -q --depth=1 "$REPO_URL" "$WANT" git fetch -q --depth=1 "$REPO_URL" "$WANT"
git checkout -q --detach FETCH_HEAD git checkout -q --detach FETCH_HEAD
test "$(git rev-parse HEAD)" = "$WANT" test "$(git rev-parse HEAD)" = "$WANT"
- name: the fuzz runner fails on a failing input and on nothing else
# fuzz.sh does not fail on Go's own -fuzztime report, on the Go
# releases whose fuzzer reports its deadline as a failure, and fails on
# anything else; this checks both, with planted failing inputs, before
# the targets run, and fails once the image's Go no longer needs the
# workaround.
run: bash deploy/test_fuzz.sh
- name: every fuzz target, 5 seconds each - name: every fuzz target, 5 seconds each
# A short pass on every commit. Run deploy/fuzz.sh with its default of # A short pass on every commit. Run deploy/fuzz.sh with its default of
# 20 seconds before changing a wall a host's input goes through. # 20 seconds before changing a wall a host's input goes through.
+121
View File
@@ -23,6 +23,127 @@ From 0.39.1 on, a heading marks a version *not published* or *held* once
that is settled. The newest entries may not be published yet; the manifest that is settled. The newest entries may not be published yet; the manifest
says which version is. Older entries are not marked. says which version is. Older entries are not marked.
## 0.57.0
- **Pairing with a Cloud Host is one pasted line.** A Host's "Set up a
bastion" card now gives a pairing code, `skp_…`, that carries the token,
the Host's enroll address and the SHA-256 of the Host's CA. `pair <code>`
(or the code on standard input after `pair -`) takes the CA from the enroll
port's own TLS chain, keeps it only under that fingerprint and only when the
port's certificate verifies under it for `host.cseries.internal`, then pairs
exactly as `--redeem-url` with `--host-ca` does. There is no certificate
file to save, so the step people skipped ("open host-ca.pem: The system
cannot find the file specified") is gone. `--host-ca` still works as before.
- **The installers pair and start in one go.** `install.sh` with
`PRIVATAE_PAIR_CODE='<code>'` in its environment (the card's line is
`curl -fsSL https://privatae.ai/bastion/install.sh | PRIVATAE_PAIR_CODE='skp_…' sh`)
and `install.ps1 -Pair '<code>'` install or update the bastion as before,
hand the code to `pair -` on standard input, then start the bastion, or restart it when it
was running, and end with one plain sentence and `status`. On Windows that
is the logon task (every running copy is stopped first, so the new one gets
its control port), or the service on a networked machine; on a machine whose
bastion is already the Windows service, `-Pair` pairs that service from an
Administrator PowerShell instead of installing a second bastion. On Linux as
yourself it is a systemd user unit, with lingering asked for and the outcome
said; on macOS a LaunchAgent; as root, the system service. Without the code
both installers behave as they did.
- **The code is in no process's arguments.** On macOS and Linux it comes
only from the environment: an argument would sit in the installer shell's
command line for the whole install, where any user of the machine can read
it with `ps`, and one at the same address could redeem it first. A
process's environment is readable only by its owner and root. `install.sh`
reads the variable before it starts any program and unsets it, so nothing
it starts inherits it. An empty or partial code is refused before the
download. `install.sh` now names no argument it refuses, so a code given in
the wrong place is never printed back. On Windows the code stays a
parameter of the script, inside the person's own PowerShell.
- **On Linux, as yourself, the copy that serves is the one the installer
started.** Before the user unit starts, every copy of your bastion running
outside it (left in a terminal, or from another path) is stopped, as
install.ps1 does on Windows. Otherwise the unit's copy only waits behind the
old one's run lock, and the old copy never reads the new pairing. The
installer then checks for a few seconds that the unit is running and keeps
the same process. A unit that keeps restarting is reported as not running,
with its log command, and the install fails.
- **A local bastion can serve a second Host, when you say so at the machine.**
A local bastion used to serve one Host and refuse any other, so a person
with two Hosts of their own (a Cloud machine beside a platform Host) could
not pair the second at all. `pair` now asks on this machine's terminal,
`/dev/tty` or the Windows console, never standard input, so a code piped
in by an installer still needs a person to answer: "This computer already
serves your Host … Add … as well? [y/N]". Only y or yes adds it; the first
pairing is untouched; with no terminal it is refused. Other callers
(`claim-anchor`, the pairing op) keep the refusal. On a local machine that
serves two Hosts, each changes only the servers it built and stops only the
coder runs it started. The name in the question is the token's, so anything
in it but a plain host name is shown quoted and escaped.
- **`pair` exits 3 when the Host is already paired here.** Nothing changes, and
the installers report it as already connected. Over TCP, with a pairing
code or `--host-ca`, a Host of the same name under another CA (Cloud names
repeat) is a plain refusal, not exit 3, before the redeem and after it. A
pairing code that names a field twice is refused.
- **An enroll port that does not answer says why within 10 seconds.** A Cloud
Host opens its pairing port only to the address the browser used when the
card was opened, for 15 minutes; from anywhere else the port does not
answer, and `pair` used to wait for the system's connect timeout. It now
stops after 10 seconds with that reason and what to do.
- **The local-mode refusal names this system's way to a networked bastion**
(`install.ps1 -Networked` from an Administrator PowerShell on Windows,
`install.sh --networked` as root elsewhere) and says that the Host already
paired would have to be paired again with it. It used to name the Linux
script on Windows.
## 0.56.0
- **New: a Host with no platform can use the model servers on this
computer.** A Host running on this computer, or in a virtual machine on it,
can reach the model servers here (Ollama, a local language model server, a
personal-data filter) by name through the bastion, and is told which of
them it can use. It starts working once your Host supports it.
- **Changed: for a Host with no platform, model management reaches only this
computer.** Listing, downloading and removing models for such a Host now
reach the model server set here only when that server is on this computer.
One set to another computer is refused, and the bastion's log says which
setting is not used and what to set. Such a Host is also told only of
the model servers it can use, and no longer of the service slots, or of
coder runs when it is in a virtual machine, since neither runs for it.
- **Changed: on a local machine, the internet lane reaches only public
addresses.** A Host can no longer use a local machine's internet lane (the
egress lane) to reach devices on your home or office network at their
private addresses, such as 192.168.x.x or 10.x.x.x, or the private ranges
many VPNs use, as was already the case on an anchored machine. Your own
machines are reached through their approved lanes. A device that has a
public address, as many do over IPv6, is still reached at that address.
- **Changed: plain-HTTP requests through the bastion are checked one per
connection.** On the internet lane and a coder run's web proxy, a
plain-HTTP (not HTTPS) connection now carries one request, which is checked
before it is sent on; anything sent after it closes the connection. Before,
later requests on the same connection reached the destination unchecked. A
malformed or ambiguous request is refused, and every refusal now says why.
- **Safer: file access in a folder you shared stays inside it.** When a Host
lists, reads or moves files in a folder you shared with it, the bastion
stays inside that folder. If the folder, or any folder on its path, is later
swapped for a link to somewhere else, the bastion refuses until the folder
is shared again, and a file or folder inside it swapped for a link is never
followed out of it. Search, which goes through the folder's own search
agent, is not part of this change.
- **Safer: the bastion limits how much it holds of what a Host sends.** On
the internet lane, the terminal and root commands, coder runs, the memory
mirror (the copy of your Host's database kept here) and the bastion's
check-in with its Host, a message from the Host is read only up to a set
size and refused past it, or passed along or dropped as it arrives. A Host
can no longer make the bastion hold a message of any size in memory. A
coder run's web proxy now carries at most 1024 connections at once, and the
memory mirror at most 8.
- **Safer: asking to share a folder with a Host has limits.** Any user on the
computer can send that ask to the bastion, so each ask is now read only up
to a set size, and the number of asks open at once is capped, per user and
in total; past the cap, the ask is told the bastion is busy and to try
again.
- **For contributors: the fuzz check in CI no longer fails on a timing bug in
Go's own fuzzer.** A real failure still fails it. The program itself is
unchanged by this.
## 0.55.3 ## 0.55.3
- **A pairing change reaches every Host on the machine at once.** Each pairing - **A pairing change reaches every Host on the machine at once.** Each pairing
+14 -4
View File
@@ -49,6 +49,11 @@ privatae-bastion run
A Windows server installs from an Administrator PowerShell with `-Networked`. A Windows server installs from an Administrator PowerShell with `-Networked`.
See [Install](#install). See [Install](#install).
A Cloud Host's **Set up a bastion** card gives one line that installs the
bastion, pairs it and starts it, so it runs now and after a restart:
`curl -fsSL https://privatae.ai/bastion/install.sh | PRIVATAE_PAIR_CODE='skp_…' sh`, or
on Windows `& ([scriptblock]::Create((irm https://privatae.ai/bastion/install.ps1))) -Pair 'skp_…'`.
**AI readers and reviewers:** [docs/README.md](docs/README.md) lists every reference page, one line each, and [docs/claims.md](docs/claims.md) is the full table of security claims and the test behind each. **AI readers and reviewers:** [docs/README.md](docs/README.md) lists every reference page, one line each, and [docs/claims.md](docs/claims.md) is the full table of security claims and the test behind each.
**Contents:** [Why a bastion](#why-a-bastion) · [Install](#install) · **Contents:** [Why a bastion](#why-a-bastion) · [Install](#install) ·
@@ -106,7 +111,8 @@ privatae-bastion stepup-enroll
``` ```
**On your own computer** (run the installer as yourself). The bastion runs **On your own computer** (run the installer as yourself). The bastion runs
as your user in **local** mode and serves one host. The pairing's as your user in **local** mode and serves the host you pair it with, and
another only when you answer yes on this computer. The pairing's
capabilities are the authorization, and approvals work once you enrol a capabilities are the authorization, and approvals work once you enrol a
second factor. second factor.
@@ -166,7 +172,7 @@ share on the host with your code.
| Runs as | your user is fine | the system's service: root under systemd or launchd, LocalSystem on Windows | | Runs as | your user is fine | the system's service: root under systemd or launchd, LocalSystem on Windows |
| A terminal the host opens | opens on the pairing's capabilities | waits for your approval, for that one session | | A terminal the host opens | opens on the pairing's capabilities | waits for your approval, for that one session |
| A write (an op, a ship step) | runs on the pairing's capabilities | waits for your approval, bound to that op and its arguments | | A write (an op, a ship step) | runs on the pairing's capabilities | waits for your approval, bound to that op and its arguments |
| Second pairing | refused | waits for the **anchor** host's approval | | Second pairing | asked on this computer's terminal, added only on a yes | waits for the **anchor** host's approval |
| The anchor | none | the first host paired, or `claim-anchor` | | The anchor | none | the first host paired, or `claim-anchor` |
In networked mode one host is the **anchor**: the one host that approves every In networked mode one host is the **anchor**: the one host that approves every
@@ -248,6 +254,10 @@ Each point in full: [docs/security-model.md](docs/security-model.md); the link a
connection is accepted. ([tunnel](docs/lanes.md#tunnel)) connection is accepted. ([tunnel](docs/lanes.md#tunnel))
- **A Host on this machine gets the same walls**: mutual TLS against the CA - **A Host on this machine gets the same walls**: mutual TLS against the CA
pinned at pairing, and the bearer on every lane. ([page](docs/host-on-this-machine.md)) pinned at pairing, and the bearer on every lane. ([page](docs/host-on-this-machine.md))
- **A Host with no platform reaches your model servers by name, never by
address**: it names `ollama`, `llm` or `pii`, and the bastion reaches the
server set on this machine for it, through the forwards' filter, only when
that address is this machine. ([page](docs/host-on-this-machine.md#the-model-servers-on-this-machine))
- **Scoped tunnel.** The WireGuard peer's only allowed address is the paired - **Scoped tunnel.** The WireGuard peer's only allowed address is the paired
host's single tunnel address. ([tunnel](docs/lanes.md#tunnel)) host's single tunnel address. ([tunnel](docs/lanes.md#tunnel))
- **No MCP endpoint.** Nothing on this machine drives the host. - **No MCP endpoint.** Nothing on this machine drives the host.
@@ -269,8 +279,8 @@ Each point in full: [docs/security-model.md](docs/security-model.md); the link a
- **Egress for someone else reaches the public internet, never this - **Egress for someone else reaches the public internet, never this
machine**: a coder run's proxy and an anchored machine's egress lane refuse machine**: a coder run's proxy and an anchored machine's egress lane refuse
internal addresses and this machine's own (a coder run reaches a LAN host internal addresses and this machine's own (a coder run reaches a LAN host
you named). A local machine's egress lane refuses this machine and keeps you named). A local machine's egress lane refuses this machine and your
your LAN. An address that reaches this machine only through a 1:1 NAT is LAN too; your own machines are reached through their approved lanes. An address that reaches this machine only through a 1:1 NAT is
not covered. ([full text](docs/security-model.md)) not covered. ([full text](docs/security-model.md))
- **A terminal is checked here and logged.** On a local machine it opens when - **A terminal is checked here and logged.** On a local machine it opens when
the host asks; on an anchored one each needs an approval for that session; the host asks; on an anchored one each needs an approval for that session;
+63 -5
View File
@@ -512,6 +512,30 @@ func resolveLLMIPs() map[string]string {
// to the anchor host, because approving is the anchor's job. "" = no pairing // to the anchor host, because approving is the anchor's job. "" = no pairing
// identity (tests, tools): mode, anchor and unapproved attempts still travel. // identity (tests, tools): mode, anchor and unapproved attempts still travel.
func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, tunnelIP, endpoint string, func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, tunnelIP, endpoint string,
savePending bool, mirror *mirrorReport, enforcing map[string]any, peers []peerAnnounce) string {
return buildHeartbeatBodyOn(beatLanes{}, self, hostname, platform, version, wgPub, fingerprint, tunnelIP, endpoint,
savePending, mirror, enforcing, peers)
}
// beatLanes is which of a pairing's lanes run, so its beat advertises those
// and no others. The zero value is a tunnel pairing: the forwards listen on
// the tunnel's stack and the service slots are reachable there.
type beatLanes struct {
// local: a Host with no platform (run.go runUnixPairing). The forwards and
// the service slots are absent; the model servers ride the serve lane
// (servelane.go), and the beat says which in lane_serves.
local bool
// noCoders: a tcp pairing, whose coder runs have no control plane to use
// (pair_tcp.go coderTransportRefusal), so it is never offered one.
noCoders bool
}
func beatLanesFor(cfg *Config) beatLanes {
return beatLanes{local: cfg.localTransport(), noCoders: cfg.tcpTransport()}
}
// buildHeartbeatBodyOn is buildHeartbeatBody for a pairing whose lanes are lanes.
func buildHeartbeatBodyOn(lanes beatLanes, self, hostname, platform, version, wgPub, fingerprint, tunnelIP, endpoint string,
savePending bool, mirror *mirrorReport, enforcing map[string]any, peers []peerAnnounce) string { savePending bool, mirror *mirrorReport, enforcing map[string]any, peers []peerAnnounce) string {
caps := []string{} caps := []string{}
// WHY SOME CAPABILITIES ARE WITHHELD HERE AND OTHERS ARE NOT — the asymmetry // WHY SOME CAPABILITIES ARE WITHHELD HERE AND OTHERS ARE NOT — the asymmetry
@@ -549,7 +573,17 @@ func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, t
// gets exactly what it got before: scope.filterModels is the identity then. // gets exactly what it got before: scope.filterModels is the identity then.
scope := pairingModelScope(self) scope := pairingModelScope(self)
reachable, models := false, []string(nil) reachable, models := false, []string(nil)
if inference { // A local pairing has no forward to advertise: its model servers ride the
// serve lane, and each lane below is advertised only when lane_serves
// carries its service (the target set and this machine's, the server
// answering, the family held). Its probes are of the address the lane
// dials, never the tunnel's guesses.
var served laneServeReport
if lanes.local {
served = laneServes(self)
reachable = containsStr(served.served, "ollama")
models = served.models
} else if inference {
reachable, models = ollamaModels(tunnelIP) reachable, models = ollamaModels(tunnelIP)
models = scope.filterModels(models) models = scope.filterModels(models)
// Reachable but nothing shared: advertising `embed` would put this box // Reachable but nothing shared: advertising `embed` would put this box
@@ -572,7 +606,15 @@ func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, t
// box running both. // box running both.
llmModels := []string{} llmModels := []string{}
llmPort := llmForwardPort llmPort := llmForwardPort
if !inference { if lanes.local {
// No port listens for it here: the Host reaches the server through the
// serve lane, so llm_port is the "no port" value.
llmPort = 0
if containsStr(served.served, "llm") {
caps = append(caps, "llm-openai")
llmModels = served.llmModels
}
} else if !inference {
// The host stores 0 as NULL; with llm-openai withheld and // The host stores 0 as NULL; with llm-openai withheld and
// llm_models empty nothing on the host resolves a port for this row. // llm_models empty nothing on the host resolves a port for this row.
llmPort = 0 llmPort = 0
@@ -598,7 +640,9 @@ func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, t
// probe: an up forwarder with a dead upstream is a distinct state the // probe: an up forwarder with a dead upstream is a distinct state the
// host wants before committing a fail-closed write path to it. // host wants before committing a fail-closed write path to it.
piiAdvertised, piiOK := false, false piiAdvertised, piiOK := false, false
if inference && scope.allowsPII() { if lanes.local {
piiAdvertised, piiOK = containsStr(served.served, "pii"), served.piiOK
} else if inference && scope.allowsPII() {
piiAdvertised, piiOK = piiProbe() piiAdvertised, piiOK = piiProbe()
} }
if piiAdvertised { if piiAdvertised {
@@ -638,7 +682,7 @@ func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, t
// "build a server on this machine" on it, and an offer the box would refuse // "build a server on this machine" on it, and an offer the box would refuse
// is a control that fails after the user chose it. The slot list rides with // is a control that fails after the user chose it. The slot list rides with
// it, withheld (absent, not empty) from a pairing outside the family. // it, withheld (absent, not empty) from a pairing outside the family.
servicesOn := servicesCapable(self) servicesOn := !lanes.local && servicesCapable(self)
if servicesOn { if servicesOn {
caps = append(caps, serviceCapability) caps = append(caps, serviceCapability)
} }
@@ -646,7 +690,10 @@ func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, t
// only when this machine would accept a run-up from this pairing — Linux, // only when this machine would accept a run-up from this pairing — Linux,
// root, coders on, a runtime, the family, and (anchored) a standing coder // root, coders on, a runtime, the family, and (anchored) a standing coder
// grant. Absent, never empty, otherwise. // grant. Absent, never empty, otherwise.
coders := coderHeartbeat(self) var coders map[string]any
if !lanes.noCoders {
coders = coderHeartbeat(self)
}
if coders != nil { if coders != nil {
caps = append(caps, coderCapability) caps = append(caps, coderCapability)
} }
@@ -774,6 +821,13 @@ func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, t
// absence as a client too old to say. // absence as a client too old to say.
"endpoint_save_pending": savePending, "endpoint_save_pending": savePending,
} }
// What this pairing serves over the serve lane (servelane.go), on a local
// pairing only: ["ollama","llm","pii"] or fewer, [] when nothing is. Absent
// on a tunnel pairing, which has the forwards instead, and from a bastion
// that predates the lane.
if lanes.local {
body["lane_serves"] = served.served
}
// The service slots this pairing may reach, with the tunnel port for each // The service slots this pairing may reach, with the tunnel port for each
// (service.go). Absent when the capability is withheld, never empty-for-no. // (service.go). Absent when the capability is withheld, never empty-for-no.
if servicesOn { if servicesOn {
@@ -1004,7 +1058,11 @@ func buildHeartbeatBody(self, hostname, platform, version, wgPub, fingerprint, t
} }
} }
if piiAdvertised { if piiAdvertised {
// pii_port is the tunnel's port; a local pairing has none and reaches
// the sterilizer through the serve lane.
if !lanes.local {
body["pii_port"] = piiForwardPort body["pii_port"] = piiForwardPort
}
body["pii_ok"] = piiOK body["pii_ok"] = piiOK
} }
if faAdvertised { if faAdvertised {
+5 -4
View File
@@ -574,9 +574,10 @@ var allCaps = []string{"dcn", "egress", "harness", "inference", "models", "obser
// writes a pairing PENDING when an anchor exists (pair.go), and state "" reads // writes a pairing PENDING when an anchor exists (pair.go), and state "" reads
// active (config.go), so B comes straight up with full lanes. // active (config.go), so B comes straight up with full lanes.
// //
// LOCAL is exempt and must stay so: a local bastion serves one host and cannot // LOCAL is exempt and must stay so: a local bastion serves the one host it was
// be anchored at all (cmdClaimAnchor refuses outside networked mode), so the // paired with, and a second only on a yes typed at the machine (pair_terminal.go),
// pairing IS the authorization there. Most bastions are local. // and it cannot be anchored at all (cmdClaimAnchor refuses outside networked
// mode), so the pairing IS the authorization there. Most bastions are local.
// //
// self-update is deliberately INSIDE the refusal. Directing a binary swap is the // self-update is deliberately INSIDE the refusal. Directing a binary swap is the
// most dangerous thing on the list, and an unanchored networked box cannot say // most dangerous thing on the list, and an unanchored networked box cannot say
@@ -1033,7 +1034,7 @@ func cmdClaimAnchor(token string, rotate bool, owner, host string) error {
return err return err
} }
if boxMode() != modeNetworked { if boxMode() != modeNetworked {
return fmt.Errorf("claim-anchor needs networked mode — a local bastion serves one host and needs no anchor (%s, then `privatae-bastion set-mode networked`)", adminHowTo()) return fmt.Errorf("claim-anchor needs networked mode — a local bastion serves only the Hosts its person said yes to and needs no anchor (%s, then `privatae-bastion set-mode networked`)", adminHowTo())
} }
cfgs, err := loadPairings() cfgs, err := loadPairings()
if err != nil { if err != nil {
+62 -31
View File
@@ -23,6 +23,7 @@ package main
import ( import (
"bufio" "bufio"
"context" "context"
"io"
"net" "net"
"os" "os"
"path/filepath" "path/filepath"
@@ -150,17 +151,32 @@ func coderVettedAddrs(ctx context.Context, hostport string, allowLAN []string) (
} }
func coderRefuse(c net.Conn, why string) { func coderRefuse(c net.Conn, why string) {
_, _ = c.Write([]byte("HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\nX-Privatae-Refused: " + writeEgressRefusal(c, "403 Forbidden", why, why)
strings.NewReplacer("\r", " ", "\n", " ").Replace(why) + "\r\n\r\n"))
} }
// handleCoderEgressConn serves one connection from a run: HTTP CONNECT or an // handleCoderEgressConn serves one connection from a run: HTTP CONNECT or an
// absolute-form plain HTTP request, the same parsing as handleEgressConn. // absolute-form plain HTTP request, the same parsing as handleEgressConn.
// Destinations are never logged; refusals are counted on the run. // Destinations are never logged; refusals are counted on the run.
// coderEgressMaxHeadBytes bounds a request's line and headers together.
const coderEgressMaxHeadBytes = 64 << 10
// coderEgressMaxConns bounds the connections one run's proxy carries at once.
// A whole Chromium profile opens at most 256 sockets at a time (its socket
// pool limit), and package managers far fewer, so four times the browser's
// figure never refuses real work; past it a connection is answered 503 and
// closed, and each one held costs this machine at most its head and two copy
// buffers.
const coderEgressMaxConns = 1024
func handleCoderEgressConn(client net.Conn, allowLAN []string, refused *int64) { func handleCoderEgressConn(client net.Conn, allowLAN []string, refused *int64) {
defer client.Close() defer client.Close()
_ = client.SetReadDeadline(time.Now().Add(30 * time.Second)) _ = client.SetReadDeadline(time.Now().Add(30 * time.Second))
br := bufio.NewReader(client) // The request line and headers are read under one bound, the head limit
// below has always named: the run's own processes send them, and a line
// that never ends must not grow in this machine's memory before that
// limit is looked at.
headLim := &io.LimitedReader{R: client, N: coderEgressMaxHeadBytes}
br := bufio.NewReader(headLim)
line, err := br.ReadString('\n') line, err := br.ReadString('\n')
if err != nil { if err != nil {
return return
@@ -172,7 +188,6 @@ func handleCoderEgressConn(client net.Conn, allowLAN []string, refused *int64) {
method, target := parts[0], parts[1] method, target := parts[0], parts[1]
var head strings.Builder var head strings.Builder
head.WriteString(line) head.WriteString(line)
hostHdr := ""
for { for {
h, err := br.ReadString('\n') h, err := br.ReadString('\n')
if err != nil { if err != nil {
@@ -182,38 +197,41 @@ func handleCoderEgressConn(client net.Conn, allowLAN []string, refused *int64) {
if h == "\r\n" || h == "\n" { if h == "\r\n" || h == "\n" {
break break
} }
if head.Len() > 64<<10 { if head.Len() > coderEgressMaxHeadBytes {
return return
} }
if len(h) >= 5 && strings.EqualFold(h[:5], "host:") {
hostHdr = strings.TrimSpace(h[5:])
}
} }
_ = client.SetReadDeadline(time.Time{}) _ = client.SetReadDeadline(time.Time{})
// The head is in; what follows is the stream itself. What the reader
// already buffered past the head stays first in line.
headLim.N = 1<<63 - 1
ctx, cancel := context.WithTimeout(context.Background(), egressDialTimeout) ctx, cancel := context.WithTimeout(context.Background(), egressDialTimeout)
defer cancel() defer cancel()
// Read the way the bastion's own egress lane reads it (egress_http.go): a
// CONNECT target through parseEgressAuthority, and plain HTTP as ONE
// request, rebuilt and sent with `Connection: close`, so the run's later
// requests on a keep-alive connection never reach the destination unvetted.
connect := strings.EqualFold(method, "CONNECT") connect := strings.EqualFold(method, "CONNECT")
hp := "" hp := ""
var req egressRequest
if connect { if connect {
hp = ensurePort(target, "443") host, port, ok := parseEgressAuthority(target, 443)
} else { if !ok {
host := hostHdr atomic.AddInt64(refused, 1)
if i := strings.Index(target, "://"); i >= 0 { writeEgressRefusal(client, "400 Bad Request", egressNoHost, egressNoHost)
rest := target[i+3:]
if s := strings.IndexAny(rest, "/?#"); s >= 0 {
host = rest[:s]
} else {
host = rest
}
if at := strings.LastIndexByte(host, '@'); at >= 0 {
host = host[at+1:]
}
}
if host == "" {
return return
} }
hp = ensurePort(host, "80") hp = egressHostPort(host, port)
} else {
r, bad := parseEgressRequest(head.String())
if bad != nil {
atomic.AddInt64(refused, 1)
writeEgressRefusal(client, bad.status, bad.body, bad.body)
return
}
req = r
hp = egressHostPort(req.host, req.port)
} }
dialAddrs, why := coderVettedAddrs(ctx, hp, allowLAN) dialAddrs, why := coderVettedAddrs(ctx, hp, allowLAN)
if why != "" { if why != "" {
@@ -223,18 +241,17 @@ func handleCoderEgressConn(client net.Conn, allowLAN []string, refused *int64) {
} }
upstream, err := dialEach(ctx, coderDialOut, dialAddrs) upstream, err := dialEach(ctx, coderDialOut, dialAddrs)
if err != nil { if err != nil {
_, _ = client.Write([]byte("HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")) writeEgressRefusal(client, "502 Bad Gateway", egressUnreachable, "")
return
}
if !connect {
oneEgressExchange(br, client, upstream, req)
return return
} }
if connect {
if _, err := client.Write([]byte("HTTP/1.1 200 Connection established\r\n\r\n")); err != nil { if _, err := client.Write([]byte("HTTP/1.1 200 Connection established\r\n\r\n")); err != nil {
_ = upstream.Close() _ = upstream.Close()
return return
} }
} else if _, err := upstream.Write([]byte(head.String())); err != nil {
_ = upstream.Close()
return
}
splice(br, client, upstream) splice(br, client, upstream)
} }
@@ -277,6 +294,7 @@ func coderListenEgress(runID, dir string, uid int) error {
} }
run := &coderEgressRun{ln: ln} run := &coderEgressRun{ln: ln}
coderEgress[runID] = run coderEgress[runID] = run
slots := make(chan struct{}, coderEgressMaxConns)
go func() { go func() {
for { for {
c, err := ln.Accept() c, err := ln.Accept()
@@ -285,7 +303,20 @@ func coderListenEgress(runID, dir string, uid int) error {
} }
// allow_lan is read per connection: the owner's edit applies to // allow_lan is read per connection: the owner's edit applies to
// running runs at once. // running runs at once.
go handleCoderEgressConn(c, loadCoderConfig().AllowLAN, &run.refused) select {
case slots <- struct{}{}:
go func() {
defer func() { <-slots }()
handleCoderEgressConn(c, loadCoderConfig().AllowLAN, &run.refused)
}()
default:
go func() {
defer c.Close()
_ = c.SetWriteDeadline(time.Now().Add(time.Second))
why := "this run already has " + strconv.Itoa(coderEgressMaxConns) + " connections open"
writeEgressRefusal(c, "503 Service Unavailable", why, why)
}()
}
} }
}() }()
return nil return nil
+7
View File
@@ -1229,10 +1229,17 @@ func coderContainerGone(rt, cname string) bool {
// anchor pairing on an anchored box (serviceOwnerRefusal's rule). // anchor pairing on an anchored box (serviceOwnerRefusal's rule).
// An unknown owner proves nothing about self, so it gets the anchor-only rule // An unknown owner proves nothing about self, so it gets the anchor-only rule
// too: every run this bastion starts records and labels its owner. // too: every run this bastion starts records and labels its owner.
//
// A local machine has no anchor, so anchorOnly lets every pairing through
// there; once it serves several Hosts (localServesSeveralHosts), each stops
// only the runs it started, as with the servers each built.
func coderOwnerRefusal(self, owner string) string { func coderOwnerRefusal(self, owner string) string {
if owner != "" && owner == self { if owner != "" && owner == self {
return "" return ""
} }
if localServesSeveralHosts() {
return "refused: that run was started by " + orUnknown(owner) + "; on a local machine that serves several Hosts each stops only the runs it started"
}
if why := anchorOnly(self, "stopping a run another host started"); why != "" { if why := anchorOnly(self, "stopping a run another host started"); why != "" {
return "refused: that run was started by " + orUnknown(owner) + "; " + why return "refused: that run was started by " + orUnknown(owner) + "; " + why
} }
+8 -2
View File
@@ -474,8 +474,14 @@ func TestCoderEgressProxyOverTheSocket(t *testing.T) {
} { } {
head, c := ask(req) head, c := ask(req)
c.Close() c.Close()
if !strings.HasPrefix(head, "HTTP/1.1 403") || !strings.Contains(head, "X-Privatae-Refused:") { want := "HTTP/1.1 403"
t.Fatalf("must be refused with 403 + X-Privatae-Refused: %q -> %q", req, head) if strings.Contains(req, "user@") {
// Userinfo is refused outright, before any lookup, as the Host's
// egress gate refuses it (parseEgressAuthority); still counted.
want = "HTTP/1.1 400"
}
if !strings.HasPrefix(head, want) || !strings.Contains(head, "X-Privatae-Refused:") {
t.Fatalf("must be refused with %s + X-Privatae-Refused: %q -> %q", want, req, head)
} }
} }
head, c := ask("CONNECT public.example:443 HTTP/1.1\r\n\r\n") head, c := ask("CONNECT public.example:443 HTTP/1.1\r\n\r\n")
+1 -1
View File
@@ -615,7 +615,7 @@ func removeStalePairingFile(hostName string) error {
func checkPairConflict(cfgs []*Config, handle string, replace bool) (*Config, error) { func checkPairConflict(cfgs []*Config, handle string, replace bool) (*Config, error) {
existing := pairingFor(cfgs, handle) existing := pairingFor(cfgs, handle)
if existing != nil && !replace { if existing != nil && !replace {
return nil, fmt.Errorf("already paired with %s — this daemon serves multiple hosts, so pairing never overwrites; to rotate that host's credentials run `privatae-bastion pair --replace <token>`", existing.HostName) return nil, &alreadyPairedError{host: existing.HostName, msg: fmt.Sprintf("already paired with %s — this daemon serves multiple hosts, so pairing never overwrites; to rotate that host's credentials run `privatae-bastion pair --replace <token>`", existing.HostName)}
} }
if existing == nil && replace { if existing == nil && replace {
return nil, fmt.Errorf("pair --replace: no existing pairing matches '%s' — pair it normally first", handle) return nil, fmt.Errorf("pair --replace: no existing pairing matches '%s' — pair it normally first", handle)
+77 -5
View File
@@ -43,9 +43,11 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"io"
"net" "net"
"os" "os"
"path/filepath" "path/filepath"
"sync"
"time" "time"
) )
@@ -126,16 +128,79 @@ func startContextSock(done <-chan struct{}) {
_ = ln.Close() _ = ln.Close()
_ = os.Remove(p) _ = os.Remove(p)
}() }()
go func() { go contextSockAccept(ln)
uilogf("context socket listening on %s\n", p)
}
// The socket is 0666 and the daemon runs as root (or SYSTEM), so every user on
// the machine can make it read. These bound what they can make it hold: an ask
// is a folder path and a display name, far under contextAskMaxBytes, and no
// more than contextSockMaxInFlight asks are read at once. Past that a
// connection is told the daemon is busy and closed; the asker tries again.
//
// One person may hold at most contextSockMaxPerUser of those slots, keyed on
// the identity the kernel gives (never on anything the asker says): without
// it, one user opening contextSockMaxInFlight idle connections and dialling
// again as each deadline ran out kept every other user's asks "busy" for as
// long as they liked. A person adds a folder at a time; four at once is far
// past that.
const (
contextAskMaxBytes = 64 << 10
contextSockMaxInFlight = 32
)
// contextSockMaxPerUser is a var only so the test of the overall cap can lift
// it (every test connection is the same user).
var contextSockMaxPerUser = 4
var contextSockUsers = struct {
sync.Mutex
held map[string]int
}{held: map[string]int{}}
// contextSockTakeUser takes one of user's slots, or reports there is none.
func contextSockTakeUser(user string) bool {
contextSockUsers.Lock()
defer contextSockUsers.Unlock()
if contextSockUsers.held[user] >= contextSockMaxPerUser {
return false
}
contextSockUsers.held[user]++
return true
}
func contextSockReleaseUser(user string) {
contextSockUsers.Lock()
defer contextSockUsers.Unlock()
if contextSockUsers.held[user]--; contextSockUsers.held[user] <= 0 {
delete(contextSockUsers.held, user)
}
}
// contextSockAccept serves ln until it is closed, at most
// contextSockMaxInFlight connections at a time.
func contextSockAccept(ln net.Listener) {
slots := make(chan struct{}, contextSockMaxInFlight)
for { for {
c, err := ln.Accept() c, err := ln.Accept()
if err != nil { if err != nil {
return return
} }
go contextSockServe(c) select {
} case slots <- struct{}{}:
go func() {
defer func() { <-slots }()
contextSockServe(c)
}() }()
uilogf("context socket listening on %s\n", p) default:
go func() {
defer c.Close()
_ = c.SetDeadline(time.Now().Add(time.Second))
_ = json.NewEncoder(c).Encode(contextAskReply{Why: "the bastion is busy " +
"answering other asks on this machine; try again in a moment"})
}()
}
}
} }
func contextSockServe(c net.Conn) { func contextSockServe(c net.Conn) {
@@ -154,8 +219,15 @@ func contextSockServe(c net.Conn) {
return return
} }
if !contextSockTakeUser(user) {
_ = json.NewEncoder(c).Encode(contextAskReply{User: user, Why: "the bastion is busy " +
"answering other asks of yours; try again in a moment"})
return
}
defer contextSockReleaseUser(user)
var req contextAskRequest var req contextAskRequest
if err := json.NewDecoder(c).Decode(&req); err != nil { if err := json.NewDecoder(io.LimitReader(c, contextAskMaxBytes)).Decode(&req); err != nil {
_ = json.NewEncoder(c).Encode(contextAskReply{User: user, Why: "the request could not be read"}) _ = json.NewEncoder(c).Encode(contextAskReply{User: user, Why: "the request could not be read"})
return return
} }
+130
View File
@@ -0,0 +1,130 @@
package main
// The context socket is 0666: every user on the machine may connect, and the
// daemon behind it runs as root (or SYSTEM). So what one connection can make
// the daemon hold must be bounded by the daemon, not by the asker.
import (
"net"
"path/filepath"
"runtime"
"strings"
"testing"
"time"
)
// An ask is a folder path and a display name. A client that streams a request
// with no end gets the daemon to read only a bounded amount of it, then a
// reply and a close; before, the decoder kept every byte until the 30 s
// deadline, and nothing capped how many such connections were open.
func TestAContextAskWithNoEndIsReadOnlyToABound(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("no unix-socket peer credentials on windows; contextsock_windows.go says why")
}
sock := filepath.Join(t.TempDir(), "s.sock")
ln, err := net.Listen("unix", sock)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
go func() {
c, err := ln.Accept()
if err != nil {
return
}
contextSockServe(c)
}()
c, err := net.Dial("unix", sock)
if err != nil {
t.Fatal(err)
}
defer c.Close()
const total = 16 << 20
chunk := []byte(strings.Repeat("a", 64<<10))
sent := 0
if _, err := c.Write([]byte(`{"root":"`)); err != nil {
t.Fatal(err)
}
for sent < total {
_ = c.SetWriteDeadline(time.Now().Add(2 * time.Second))
n, err := c.Write(chunk)
sent += n
if err != nil {
break
}
}
// The kernel's socket buffers hold a few hundred KiB the daemon never
// read; anything near the whole stream means the daemon read it all.
if sent >= 4<<20 {
t.Fatalf("the daemon accepted %d MiB of one ask with no end; it must stop at a bound", sent>>20)
}
}
// Asks in flight are capped: a connection past the cap is closed at once, so
// many idle or endless connections cannot multiply the bound above.
func TestContextAsksInFlightAreCapped(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("no unix-socket peer credentials on windows; contextsock_windows.go says why")
}
sock := filepath.Join(t.TempDir(), "s.sock")
ln, err := net.Listen("unix", sock)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
// Every connection here is the same user; lift the per-user share so the
// overall cap is what this test meets (TestOneUserCannotHoldEveryContextSockSlot
// holds the share).
oldShare := contextSockMaxPerUser
contextSockMaxPerUser = contextSockMaxInFlight
t.Cleanup(func() { contextSockMaxPerUser = oldShare })
go contextSockAccept(ln)
var held []net.Conn
defer func() {
for _, h := range held {
_ = h.Close()
}
}()
// Each of these sends nothing and holds its slot until its deadline.
for i := 0; i < contextSockMaxInFlight; i++ {
h, err := net.Dial("unix", sock)
if err != nil {
t.Fatal(err)
}
held = append(held, h)
}
// Give the daemon's accept loop time to take every held connection.
time.Sleep(200 * time.Millisecond)
extra, err := net.Dial("unix", sock)
if err != nil {
t.Fatal(err)
}
defer extra.Close()
_ = extra.SetReadDeadline(time.Now().Add(2 * time.Second))
buf := make([]byte, 512)
n, rerr := extra.Read(buf)
if ne, ok := rerr.(net.Error); ok && ne.Timeout() {
t.Fatalf("a connection past the cap of %d was left open; it must be refused", contextSockMaxInFlight)
}
if n > 0 && !strings.Contains(string(buf[:n]), "busy") {
t.Fatalf("a connection past the cap got %q; it must be told the daemon is busy", buf[:n])
}
// Once a held connection goes, a new ask gets a slot again.
_ = held[0].Close()
held = held[1:]
time.Sleep(200 * time.Millisecond)
again, err := net.Dial("unix", sock)
if err != nil {
t.Fatal(err)
}
defer again.Close()
_ = again.SetReadDeadline(time.Now().Add(500 * time.Millisecond))
n, rerr = again.Read(buf)
if ne, ok := rerr.(net.Error); !ok || !ne.Timeout() {
t.Fatalf("after a slot was freed, a new ask was not served (read %q, %v)", buf[:n], rerr)
}
}
+86
View File
@@ -0,0 +1,86 @@
package main
import (
"net"
"path/filepath"
"runtime"
"strings"
"testing"
"time"
)
// One person cannot take every slot: past their own share they are told the
// daemon is busy at once, while slots remain for everyone else. Before, one
// user's idle connections, re-dialled as each deadline ran out, held all of
// them and every other user's ask was refused.
func TestOneUserCannotHoldEveryContextSockSlot(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("no unix-socket peer credentials on windows; contextsock_windows.go says why")
}
const share = 4 // contextSockMaxPerUser
// A test before this one may still be releasing its slots.
if n := contextSockSlotsHeld(t, 0); n != 0 {
t.Fatalf("%d slots still held from an earlier test", n)
}
sock := filepath.Join(t.TempDir(), "s.sock")
ln, err := net.Listen("unix", sock)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
go contextSockAccept(ln)
var held []net.Conn
defer func() {
for _, h := range held {
_ = h.Close()
}
}()
for i := 0; i < share; i++ {
h, err := net.Dial("unix", sock)
if err != nil {
t.Fatal(err)
}
held = append(held, h)
}
// Wait until the daemon has given those four their slots (a fixed sleep
// lost this race on a loaded machine and handed the 5th one first).
if n := contextSockSlotsHeld(t, share); n != share {
t.Fatalf("the daemon gave %d slots to %d idle connections of one user", n, share)
}
extra, err := net.Dial("unix", sock)
if err != nil {
t.Fatal(err)
}
defer extra.Close()
_ = extra.SetReadDeadline(time.Now().Add(2 * time.Second))
buf := make([]byte, 512)
n, rerr := extra.Read(buf)
if ne, ok := rerr.(net.Error); ok && ne.Timeout() {
t.Fatalf("one user held %d idle connections and a %dth was still given a slot: "+
"one user can take all %d", share, share+1, contextSockMaxInFlight)
}
if !strings.Contains(string(buf[:n]), "busy") {
t.Fatalf("past one user's share the ask got %q; it must be told the daemon is busy", buf[:n])
}
}
// contextSockSlotsHeld waits up to 10 s for the per-user slots held across
// all users to reach want, and returns the count it last saw.
func contextSockSlotsHeld(t *testing.T, want int) int {
t.Helper()
n := -1
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(10 * time.Millisecond) {
contextSockUsers.Lock()
n = 0
for _, k := range contextSockUsers.held {
n += k
}
contextSockUsers.Unlock()
if n == want {
break
}
}
return n
}
+68 -7
View File
@@ -26,6 +26,7 @@ import (
"encoding/base64" "encoding/base64"
"encoding/json" "encoding/json"
"fmt" "fmt"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime" "runtime"
@@ -74,6 +75,10 @@ type dcnGrantRec struct {
// the signed bytes (a public key; the terms stay exactly what was // the signed bytes (a public key; the terms stay exactly what was
// signed). The managed local agent seals the store's catalog to it. // signed). The managed local agent seals the store's catalog to it.
SealRecipient string `json:"seal_recipient,omitempty"` SealRecipient string `json:"seal_recipient,omitempty"`
// RootReal is where the granted folder led when this box installed the
// grant: its real path, pinned here (dcn_store.go), outside the signed
// bytes. An op whose root leads anywhere else is refused.
RootReal string `json:"root_real,omitempty"`
} }
// dcnTerms is parsed FROM the verified canonical bytes — never from anything else. // dcnTerms is parsed FROM the verified canonical bytes — never from anything else.
@@ -524,6 +529,17 @@ func handleDcnGrantSync(ws *websocket.Conn, dir string, args []string) {
dcnRefuse(ws, dir, "grant_sync", payload.StoreID, "", "grant does not verify: "+err.Error()) dcnRefuse(ws, dir, "grant_sync", payload.StoreID, "", "grant does not verify: "+err.Error())
return return
} }
// The folder this grant names is pinned as it resolves now; a re-sync or a
// renewal of the same root keeps the pin it already has (dcn_store.go).
if prev, held := f.Stores[fmt.Sprint(payload.StoreID)]; held {
prevRoot := ""
if pt, perr := dcnVerifyRec(f.VerifyPubkey, prev); perr == nil {
prevRoot = pt.Root
}
rec.RootReal = dcnPinFor(&prev, prevRoot, terms.Root)
} else {
rec.RootReal = dcnPinFor(nil, "", terms.Root)
}
if hn, _ := os.Hostname(); hn != "" && terms.Server != "" && terms.Server != hn { if hn, _ := os.Hostname(); hn != "" && terms.Server != "" && terms.Server != hn {
dcnRefuse(ws, dir, "grant_sync", payload.StoreID, "", dcnRefuse(ws, dir, "grant_sync", payload.StoreID, "",
fmt.Sprintf("grant is for server '%s', this box is '%s'", terms.Server, hn)) fmt.Sprintf("grant is for server '%s', this box is '%s'", terms.Server, hn))
@@ -658,15 +674,26 @@ func handleDcnList(ws *websocket.Conn, dir string, args []string) {
if terms == nil { if terms == nil {
return return
} }
base := terms.Root base := "."
if a.Prefix != "" { if a.Prefix != "" {
p, ok := dcnResolvePath(terms.Root, a.Prefix) p, ok := dcnResolvePath(terms.Root, a.Prefix)
if !ok { if !ok {
dcnRefuse(ws, dir, "list", storeID, a.Prefix, "prefix escapes the store root") dcnRefuse(ws, dir, "list", storeID, a.Prefix, "prefix escapes the store root")
return return
} }
base = p rel, err := dcnRel(terms.Root, p)
if err != nil {
dcnRefuse(ws, dir, "list", storeID, a.Prefix, "prefix escapes the store root")
return
} }
base = filepath.ToSlash(rel)
}
r, err := dcnOpenStore(dir, storeID, terms)
if err != nil {
dcnRefuse(ws, dir, "list", storeID, a.Prefix, err.Error())
return
}
defer r.Close()
limit := a.Limit limit := a.Limit
if limit <= 0 || limit > dcnMaxListEntries { if limit <= 0 || limit > dcnMaxListEntries {
limit = dcnMaxListEntries limit = dcnMaxListEntries
@@ -678,12 +705,14 @@ func handleDcnList(ws *websocket.Conn, dir string, args []string) {
} }
var files []entry var files []entry
truncated := false truncated := false
_ = filepath.WalkDir(base, func(p string, d os.DirEntry, err error) error { // Walked through the store's handle: a link that leads out of it is not
// followed, whatever the names do during the walk.
_ = fs.WalkDir(r.FS(), base, func(p string, d fs.DirEntry, err error) error {
if err != nil { if err != nil {
return nil // unreadable subtree: skip, never abort the listing return nil // unreadable subtree: skip, never abort the listing
} }
rel, rerr := filepath.Rel(terms.Root, p) rel := filepath.FromSlash(p)
if rerr != nil || rel == "." { if rel == "." {
return nil return nil
} }
if dcnExcluded(terms.Excludes, rel) { if dcnExcluded(terms.Excludes, rel) {
@@ -727,8 +756,23 @@ func handleDcnStat(ws *websocket.Conn, dir string, args []string) {
dcnRefuse(ws, dir, "stat", storeID, a.Path, "path escapes the store root") dcnRefuse(ws, dir, "stat", storeID, a.Path, "path escapes the store root")
return return
} }
info, err := os.Stat(abs) rel, err := dcnRel(terms.Root, abs)
if err != nil { if err != nil {
dcnRefuse(ws, dir, "stat", storeID, a.Path, err.Error())
return
}
r, err := dcnOpenStore(dir, storeID, terms)
if err != nil {
dcnRefuse(ws, dir, "stat", storeID, a.Path, err.Error())
return
}
defer r.Close()
info, err := r.Stat(rel)
if err != nil {
if dcnEscaped(err) {
dcnRefuse(ws, dir, "stat", storeID, a.Path, "path escapes the store root")
return
}
if os.IsNotExist(err) { if os.IsNotExist(err) {
dcnResult(ws, dir, "stat", storeID, a.Path, map[string]any{ dcnResult(ws, dir, "stat", storeID, a.Path, map[string]any{
"state": "gone", "path": a.Path}) "state": "gone", "path": a.Path})
@@ -771,8 +815,25 @@ func handleDcnReadRange(ws *websocket.Conn, dir string, args []string) {
dcnRefuse(ws, dir, "read_range", storeID, a.Path, "path escapes the store root") dcnRefuse(ws, dir, "read_range", storeID, a.Path, "path escapes the store root")
return return
} }
fh, err := os.Open(abs) rel, err := dcnRel(terms.Root, abs)
if err != nil { if err != nil {
dcnRefuse(ws, dir, "read_range", storeID, a.Path, err.Error())
return
}
r, err := dcnOpenStore(dir, storeID, terms)
if err != nil {
dcnRefuse(ws, dir, "read_range", storeID, a.Path, err.Error())
return
}
defer r.Close()
// Opened through the store's handle: a name swapped for a link to a file
// outside the store after the check above is refused, never followed.
fh, err := r.Open(rel)
if err != nil {
if dcnEscaped(err) {
dcnRefuse(ws, dir, "read_range", storeID, a.Path, "path escapes the store root")
return
}
if os.IsNotExist(err) { if os.IsNotExist(err) {
dcnResult(ws, dir, "read_range", storeID, a.Path, map[string]any{ dcnResult(ws, dir, "read_range", storeID, a.Path, map[string]any{
"state": "gone", "path": a.Path}) "state": "gone", "path": a.Path})
+56 -20
View File
@@ -191,10 +191,28 @@ func dcnReportPath(root string, planID int64) string {
return p return p
} }
// dcnApplyPlan runs the plan: check, rename in order (each re-checked just // dcnApplyPlan is dcnApplyPlanIn on root opened where it leads now (the op
// before it runs; rolling back on the first failure), verify, report. Every // handler opens the PINNED folder instead, dcn_store.go).
// path it touches has passed dcnCheckPlan.
func dcnApplyPlan(root string, excludes []string, plan *dcnPlanTerms) map[string]any { func dcnApplyPlan(root string, excludes []string, plan *dcnPlanTerms) map[string]any {
real, err := dcnRealRoot(root)
if err != nil {
return map[string]any{"state": "refused", "mode": "apply", "plan_id": plan.PlanID,
"reason": "the store's folder cannot be opened: " + err.Error()}
}
r, err := dcnOpenRootAt(real)
if err != nil {
return map[string]any{"state": "refused", "mode": "apply", "plan_id": plan.PlanID, "reason": err.Error()}
}
defer r.Close()
return dcnApplyPlanIn(r, root, excludes, plan)
}
// dcnApplyPlanIn runs the plan: check, rename in order (each re-checked just
// before it runs; rolling back on the first failure), verify, report. Every
// path it touches has passed dcnCheckPlan, and every change is made through
// r, the store's handle, so a folder that becomes a link to somewhere else
// between the check and the rename is refused rather than followed.
func dcnApplyPlanIn(r *os.Root, root string, excludes []string, plan *dcnPlanTerms) map[string]any {
problems := dcnCheckPlan(root, excludes, plan.Moves) problems := dcnCheckPlan(root, excludes, plan.Moves)
if len(problems) > 0 { if len(problems) > 0 {
return map[string]any{"state": "refused", "mode": "apply", "plan_id": plan.PlanID, return map[string]any{"state": "refused", "mode": "apply", "plan_id": plan.PlanID,
@@ -207,11 +225,11 @@ func dcnApplyPlan(root string, excludes []string, plan *dcnPlanTerms) map[string
// and what is NOT (so the user knows exactly what is still displaced). // and what is NOT (so the user knows exactly what is still displaced).
rollback := func() (back, stuck []string) { rollback := func() (back, stuck []string) {
for i := len(applied) - 1; i >= 0; i-- { for i := len(applied) - 1; i >= 0; i-- {
if _, err := os.Lstat(applied[i].src); err == nil { if _, err := r.Lstat(applied[i].src); err == nil {
stuck = append(stuck, applied[i].from) // something now sits at the source; never replace it stuck = append(stuck, applied[i].from) // something now sits at the source; never replace it
continue continue
} }
if err := os.Rename(applied[i].dst, applied[i].src); err == nil { if err := r.Rename(applied[i].dst, applied[i].src); err == nil {
back = append(back, applied[i].from) back = append(back, applied[i].from)
} else { } else {
stuck = append(stuck, applied[i].from) stuck = append(stuck, applied[i].from)
@@ -230,24 +248,30 @@ func dcnApplyPlan(root string, excludes []string, plan *dcnPlanTerms) map[string
return rep return rep
} }
for i, m := range plan.Moves { for i, m := range plan.Moves {
src, _ := dcnResolvePath(root, m.From) srcAbs, _ := dcnResolvePath(root, m.From)
dst, _ := cleanRel(root, m.To) dstAbs, _ := cleanRel(root, m.To)
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { src, serr := dcnRel(root, srcAbs)
dst, derr := dcnRel(root, dstAbs)
if serr != nil || derr != nil {
return fail(i, m, "path escapes the store root")
}
if d := filepath.Dir(dst); d != "." {
if err := r.MkdirAll(d, 0o755); err != nil {
return fail(i, m, "could not create the target folder: "+err.Error()) return fail(i, m, "could not create the target folder: "+err.Error())
} }
}
// Re-check the instant before the rename: the world may have changed since // Re-check the instant before the rename: the world may have changed since
// the pre-check (a file appeared at the target; a parent became a symlink). // the pre-check (a file appeared at the target; a parent became a symlink).
// rename(2) would replace silently — this lane never lets it. // rename(2) would replace silently — this lane never lets it.
if _, ok := dcnResolvePath(root, m.To); !ok { if _, ok := dcnResolvePath(root, m.To); !ok {
return fail(i, m, "target's parent no longer resolves inside the store root") return fail(i, m, "target's parent no longer resolves inside the store root")
} }
if _, err := os.Lstat(dst); err == nil { if _, err := r.Lstat(dst); err == nil {
return fail(i, m, "target appeared since the check (a plan never overwrites)") return fail(i, m, "target appeared since the check (a plan never overwrites)")
} }
if err := os.Rename(src, dst); err != nil { if err := r.Rename(src, dst); err != nil {
why := err.Error() why := err.Error()
var le *os.LinkError if errors.Is(err, syscall.EXDEV) {
if errors.As(err, &le) && errors.Is(le.Err, syscall.EXDEV) {
why = "target is on a different filesystem than the source (a plan only renames; it never copies and removes)" why = "target is on a different filesystem than the source (a plan only renames; it never copies and removes)"
} }
return fail(i, m, why) return fail(i, m, why)
@@ -257,14 +281,16 @@ func dcnApplyPlan(root string, excludes []string, plan *dcnPlanTerms) map[string
// Verify pass: every target present, every source gone, declared sizes hold. // Verify pass: every target present, every source gone, declared sizes hold.
var mismatches []dcnPlanProblem var mismatches []dcnPlanProblem
for i, m := range plan.Moves { for i, m := range plan.Moves {
src, _ := dcnResolvePath(root, m.From) srcAbs, _ := dcnResolvePath(root, m.From)
dst, _ := cleanRel(root, m.To) dstAbs, _ := cleanRel(root, m.To)
info, err := os.Lstat(dst) src, _ := dcnRel(root, srcAbs)
dst, _ := dcnRel(root, dstAbs)
info, err := r.Lstat(dst)
if err != nil { if err != nil {
mismatches = append(mismatches, dcnPlanProblem{i, m.From, m.To, "target missing after move"}) mismatches = append(mismatches, dcnPlanProblem{i, m.From, m.To, "target missing after move"})
continue continue
} }
if _, err := os.Lstat(src); err == nil { if _, err := r.Lstat(src); err == nil {
mismatches = append(mismatches, dcnPlanProblem{i, m.From, m.To, "source still present after move"}) mismatches = append(mismatches, dcnPlanProblem{i, m.From, m.To, "source still present after move"})
} }
if m.SizeBytes != nil && !info.IsDir() && info.Size() != *m.SizeBytes { if m.SizeBytes != nil && !info.IsDir() && info.Size() != *m.SizeBytes {
@@ -372,9 +398,19 @@ func handleDcnPlanApply(ws *websocket.Conn, dir string, args []string) {
// a retry (a lost reply, a host re-run) gets the earlier report back BEFORE // a retry (a lost reply, a host re-run) gets the earlier report back BEFORE
// any gate can turn it into a refusal. (The signed expiry is the guard that // any gate can turn it into a refusal. (The signed expiry is the guard that
// does not depend on a file the user can remove.) // does not depend on a file the user can remove.)
store, err := dcnOpenStore(dir, storeID, terms)
if err != nil {
dcnRefuse(ws, dir, op, storeID, planKey, err.Error())
return
}
defer store.Close()
repPath := dcnReportPath(terms.Root, plan.PlanID) repPath := dcnReportPath(terms.Root, plan.PlanID)
repRel := ""
if repPath != "" { if repPath != "" {
if prev, err := os.ReadFile(repPath); err == nil { repRel, _ = dcnRel(terms.Root, repPath)
}
if repRel != "" {
if prev, err := store.ReadFile(repRel); err == nil {
var old map[string]any var old map[string]any
if json.Unmarshal(prev, &old) == nil && old["state"] == "ok" { if json.Unmarshal(prev, &old) == nil && old["state"] == "ok" {
old["note"] = "already applied on this machine; this is the earlier report" old["note"] = "already applied on this machine; this is the earlier report"
@@ -398,13 +434,13 @@ func handleDcnPlanApply(ws *websocket.Conn, dir string, args []string) {
return return
} }
} }
rep := dcnApplyPlan(terms.Root, terms.Excludes, plan) rep := dcnApplyPlanIn(store, terms.Root, terms.Excludes, plan)
if rep["state"] == "ok" && plan.Approval != "" { if rep["state"] == "ok" && plan.Approval != "" {
markApprovalUsed(dir, plan.Approval) // single use: the files have moved markApprovalUsed(dir, plan.Approval) // single use: the files have moved
} }
if repPath != "" { if repRel != "" {
if raw, err := json.MarshalIndent(rep, "", " "); err == nil { if raw, err := json.MarshalIndent(rep, "", " "); err == nil {
_ = atomicWrite(repPath, raw) // the machine's own record; best-effort _ = dcnRootAtomicWrite(store, repRel, raw) // the machine's own record; best-effort
} }
} }
dcnResult(ws, dir, op, storeID, planKey, rep) dcnResult(ws, dir, op, storeID, planKey, rep)
+322
View File
@@ -0,0 +1,322 @@
package main
// A machine store is the folder the person granted, for as long as the grant
// lives. The bastion often runs as root and the folder belongs to someone who
// is not, so whoever can write in it, or above it, must not be able to point
// a read or a move somewhere else: by turning the granted folder into a link
// after the grant (no race needed), or by swapping a file for a link between
// the check and the open.
import (
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strconv"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/gorilla/websocket"
)
// dcnRunOp runs one DCN handler on the far side of a real WebSocket and
// returns what it sent: the O payload, the R text and the exit code.
func dcnRunOp(t *testing.T, h func(ws *websocket.Conn)) (out, errText string, exit int) {
t.Helper()
up := websocket.Upgrader{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ws, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer ws.Close()
h(ws)
}))
defer srv.Close()
ws, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(srv.URL, "http"), nil)
if err != nil {
t.Fatal(err)
}
defer ws.Close()
_ = ws.SetReadDeadline(time.Now().Add(10 * time.Second))
exit = -1
for {
_, data, err := ws.ReadMessage()
if err != nil || len(data) == 0 {
return
}
switch data[0] {
case 'O':
out += string(data[1:])
case 'R':
errText += string(data[1:])
case 'X':
exit, _ = strconv.Atoi(string(data[1:]))
return
}
}
}
type dcnStoreFixture struct {
dir string // the pairing's dcn dir
base string
store string // the granted folder
other string // a folder the grant never named
priv ed25519.PrivateKey
pub ed25519.PublicKey
}
func newDcnStoreFixture(t *testing.T) *dcnStoreFixture {
t.Helper()
fx := &dcnStoreFixture{dir: t.TempDir(), base: t.TempDir()}
fx.store = filepath.Join(fx.base, "store")
fx.other = filepath.Join(fx.base, "other")
for _, d := range []string{fx.store, fx.other, filepath.Join(fx.store, "docs")} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(filepath.Join(fx.store, "a.txt"), []byte("MINE"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(fx.other, "a.txt"), []byte("OTHER"), 0o600); err != nil {
t.Fatal(err)
}
fx.pub, fx.priv, _ = ed25519.GenerateKey(nil)
return fx
}
// sync installs a read grant for the store through the real grant_sync op.
func (fx *dcnStoreFixture) sync(t *testing.T, mask ...string) {
t.Helper()
if len(mask) == 0 {
mask = []string{"read"}
}
rec := signedRec(t, fx.priv, map[string]any{"kind": "dcn_store_grant", "class": "machine",
"root": fx.store, "mask": mask,
"expires": time.Now().Add(24 * time.Hour).UTC().Format(time.RFC3339)})
payload, _ := json.Marshal(map[string]any{"store_id": 1, "terms_canonical": rec.TermsCanonical,
"signature": rec.Signature, "verify_pubkey": base64.StdEncoding.EncodeToString(fx.pub)})
_, why, exit := dcnRunOp(t, func(ws *websocket.Conn) {
handleDcnGrantSync(ws, fx.dir, []string{base64.StdEncoding.EncodeToString(payload)})
})
if exit != 0 {
t.Fatalf("grant_sync: exit %d: %s", exit, why)
}
}
func (fx *dcnStoreFixture) op(t *testing.T, h func(*websocket.Conn, string, []string), args map[string]any) (map[string]any, string, int) {
t.Helper()
raw, _ := json.Marshal(args)
out, why, exit := dcnRunOp(t, func(ws *websocket.Conn) {
h(ws, fx.dir, []string{"1", string(raw)})
})
var obj map[string]any
_ = json.Unmarshal([]byte(out), &obj)
return obj, why, exit
}
func (fx *dcnStoreFixture) read(t *testing.T, path string) (string, string, int) {
t.Helper()
obj, why, exit := fx.op(t, handleDcnReadRange, map[string]any{"path": path})
b, _ := base64.StdEncoding.DecodeString(func() string { s, _ := obj["content_b64"].(string); return s }())
return string(b), why, exit
}
// repoint turns the granted folder into a link to the other one.
func (fx *dcnStoreFixture) repoint(t *testing.T) {
t.Helper()
if err := os.Rename(fx.store, fx.store+".was"); err != nil {
t.Fatal(err)
}
if err := os.Symlink(fx.other, fx.store); err != nil {
t.Skipf("symlinks unavailable: %v", err)
}
}
func TestDcnStoreRepointedAfterTheGrantIsRefused(t *testing.T) {
fx := newDcnStoreFixture(t)
fx.sync(t)
if got, why, exit := fx.read(t, "a.txt"); exit != 0 || got != "MINE" {
t.Fatalf("the granted file must read: exit %d %q %s", exit, got, why)
}
fx.repoint(t)
if got, _, exit := fx.read(t, "a.txt"); exit == 0 || got == "OTHER" {
t.Fatalf("after the granted folder became a link elsewhere, read_range returned %q (exit %d); it must refuse", got, exit)
}
if obj, _, exit := fx.op(t, handleDcnStat, map[string]any{"path": "a.txt"}); exit == 0 {
t.Fatalf("stat through the repointed folder must refuse, got %v", obj)
}
if obj, _, exit := fx.op(t, handleDcnList, map[string]any{}); exit == 0 {
t.Fatalf("list through the repointed folder must refuse, got %v", obj)
}
}
// A folder inside the store that becomes a link to somewhere else after the
// grant is the same escape one level down.
func TestDcnSubfolderRepointedIsRefused(t *testing.T) {
fx := newDcnStoreFixture(t)
fx.sync(t)
if err := os.RemoveAll(filepath.Join(fx.store, "docs")); err != nil {
t.Fatal(err)
}
if err := os.Symlink(fx.other, filepath.Join(fx.store, "docs")); err != nil {
t.Skipf("symlinks unavailable: %v", err)
}
if got, _, exit := fx.read(t, "docs/a.txt"); exit == 0 || got == "OTHER" {
t.Fatalf("a link inside the store to a folder outside it read %q (exit %d)", got, exit)
}
}
// The race: a file swapped for a link to a file outside the store between the
// check and the open. Run for long enough that the old check-then-open loses.
func TestDcnReadRangeSwapRaceNeverLeavesTheStore(t *testing.T) {
fx := newDcnStoreFixture(t)
fx.sync(t)
target := filepath.Join(fx.store, "a.txt")
realTmp := filepath.Join(fx.store, ".real")
linkTmp := filepath.Join(fx.store, ".link")
var stop atomic.Bool
done := make(chan struct{})
go func() {
defer close(done)
for !stop.Load() {
_ = os.Symlink(filepath.Join(fx.other, "a.txt"), linkTmp)
_ = os.Rename(linkTmp, target)
_ = os.WriteFile(realTmp, []byte("MINE"), 0o644)
_ = os.Rename(realTmp, target)
}
}()
deadline := time.Now().Add(3 * time.Second)
n := 0
for time.Now().Before(deadline) {
n++
if got, _, _ := fx.read(t, "a.txt"); got == "OTHER" {
stop.Store(true)
<-done
t.Fatalf("read %d returned the file outside the store through a swapped link", n)
}
}
stop.Store(true)
<-done
}
// A move plan applies through the store too: a target folder that leads
// outside the store for an instant between the re-check and the rename must
// not receive the file.
func TestDcnPlanMoveRaceNeverLeavesTheStore(t *testing.T) {
fx := newDcnStoreFixture(t)
if err := os.Remove(filepath.Join(fx.other, "a.txt")); err != nil {
t.Fatal(err)
}
docs := filepath.Join(fx.store, "docs")
var stop atomic.Bool
done := make(chan struct{})
go func() {
defer close(done)
for !stop.Load() {
_ = os.Rename(docs, docs+".real")
_ = os.Symlink(fx.other, docs+".link")
_ = os.Rename(docs+".link", docs)
_ = os.Rename(docs, docs+".link")
_ = os.Remove(docs + ".link")
_ = os.Rename(docs+".real", docs)
}
}()
defer func() { stop.Store(true); <-done }()
deadline := time.Now().Add(3 * time.Second)
for i := 0; time.Now().Before(deadline); i++ {
_ = os.WriteFile(filepath.Join(fx.store, "a.txt"), []byte("MINE"), 0o644)
plan := &dcnPlanTerms{PlanID: int64(i), StoreID: 1, Root: fx.store,
Moves: []dcnMove{{From: "a.txt", To: "docs/a.txt"}}}
_ = dcnApplyPlan(fx.store, nil, plan)
if _, err := os.Lstat(filepath.Join(fx.other, "a.txt")); err == nil {
t.Fatalf("move %d landed outside the store, through a folder that was a link for an instant", i)
}
_ = os.Remove(filepath.Join(fx.store, "docs.real", "a.txt"))
_ = os.Remove(filepath.Join(docs, "a.txt"))
}
}
// The plan report is written beside its name and renamed into place. Whoever
// can write in the store could plant a link at that side name; the write must
// never follow it, or the root bastion truncates whatever file the link names
// (an excluded .ssh/authorized_keys included) and fills it with the report.
func TestDcnReportWriteNeverFollowsAPlantedSideName(t *testing.T) {
store := t.TempDir()
victim := filepath.Join(store, ".ssh", "authorized_keys")
if err := os.MkdirAll(filepath.Dir(victim), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(victim, []byte("ssh-ed25519 AAAAexample user@example\n"), 0o600); err != nil {
t.Fatal(err)
}
plans := filepath.Join(store, ".agent", "plans")
if err := os.MkdirAll(plans, 0o700); err != nil {
t.Fatal(err)
}
rel := filepath.Join(".agent", "plans", "7.json")
// Every side name a writer could guess: this process's pid, and a spread
// of others, all pointing at the victim.
for _, pid := range []int{os.Getpid(), 1, 2, 100, 1000} {
side := filepath.Join(store, rel+".tmp-"+strconv.Itoa(pid))
if err := os.Symlink(filepath.Join("..", "..", ".ssh", "authorized_keys"), side); err != nil {
t.Fatal(err)
}
}
r, err := os.OpenRoot(store)
if err != nil {
t.Fatal(err)
}
defer r.Close()
if err := dcnRootAtomicWrite(r, rel, []byte(`{"state":"ok"}`)); err != nil {
t.Fatalf("report write: %v", err)
}
if got, _ := os.ReadFile(victim); string(got) != "ssh-ed25519 AAAAexample user@example\n" {
t.Fatalf("the report write followed a planted link: the victim now holds %q", got)
}
if got, err := os.ReadFile(filepath.Join(store, rel)); err != nil || string(got) != `{"state":"ok"}` {
t.Fatalf("report not in place: %q %v", got, err)
}
if fi, err := os.Lstat(filepath.Join(store, rel)); err != nil || fi.Mode()&os.ModeSymlink != 0 || fi.Mode().Perm() != 0o600 {
t.Fatalf("report is not a 0600 regular file: %v %v", fi, err)
}
}
// A grant installed before pins existed has no root_real. It still works,
// pins where its folder leads at first use, keeps that pin, and a repoint
// after that is refused like any other.
func TestDcnGrantFromBeforePinsPinsAtFirstUse(t *testing.T) {
fx := newDcnStoreFixture(t)
fx.sync(t)
f, err := dcnLoadFile(fx.dir)
if err != nil {
t.Fatal(err)
}
rec := f.Stores["1"]
rec.RootReal = ""
f.Stores["1"] = rec
if err := dcnSaveFile(fx.dir, f); err != nil {
t.Fatal(err)
}
if got, why, exit := fx.read(t, "a.txt"); exit != 0 || got != "MINE" {
t.Fatalf("a grant from before pins no longer reads: exit %d %q %s", exit, got, why)
}
f, err = dcnLoadFile(fx.dir)
if err != nil {
t.Fatal(err)
}
want, _ := filepath.EvalSymlinks(fx.store)
if got := f.Stores["1"].RootReal; got != want {
t.Fatalf("first use pinned %q, want %q", got, want)
}
fx.repoint(t)
if got, _, exit := fx.read(t, "a.txt"); exit == 0 || got == "OTHER" {
t.Fatalf("after the first-use pin a repointed folder was read: exit %d %q", exit, got)
}
}
+199
View File
@@ -0,0 +1,199 @@
package main
// dcn_store.go — every byte a machine-store op touches is reached through ONE
// handle on the folder the person granted.
//
// WHY. The bastion is often root (a networked install, the macOS and Windows
// system service), and the granted folder is often someone else's. Checking a
// path and then opening it by name leaves two ways out of the store for
// whoever can write in it or above it:
//
// - the granted folder itself turned into a link to another folder after the
// grant. The old check resolved the root again on every op and judged the
// path against wherever it led NOW, so no race was needed;
// - a file or folder inside the store swapped for a link between the check
// and the open (or the rename, for a move plan).
//
// So: the folder's real path is PINNED at grant sync (rec.RootReal, this box's
// own field, outside the signed bytes), an op whose root now leads elsewhere
// is refused, and the op opens the pinned folder as an os.Root, which never
// follows a link out of it, whatever happens to the names afterwards. That
// handle is then checked to be the folder at the pinned path reached with no
// link at all (dcnNoLinkStat: an open that refuses a link in every component
// on Linux and macOS), so a swap of a folder above the store between the two
// opens is refused too.
import (
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strconv"
)
// errDcnRepointed is the refusal for a granted folder that leads elsewhere now.
var errDcnRepointed = errors.New("the granted folder now leads somewhere other than the folder that was granted; " +
"a new grant for it is the repair")
// dcnRealRoot is where the grant's root leads now: its links resolved.
func dcnRealRoot(root string) (string, error) {
real, err := filepath.EvalSymlinks(filepath.Clean(root))
if err != nil {
return "", err
}
if !filepath.IsAbs(real) {
return "", errors.New("the granted folder is not an absolute path")
}
return filepath.Clean(real), nil
}
// dcnPinFor is the real root a sync records: the one already pinned for the
// same root while the store is held (a re-sync, a renewed grant), else where
// the root leads now ("" when it cannot be resolved yet; pinned at first use).
func dcnPinFor(prev *dcnGrantRec, prevRoot, root string) string {
if prev != nil && prev.RootReal != "" && filepath.Clean(prevRoot) == filepath.Clean(root) {
return prev.RootReal
}
real, err := dcnRealRoot(root)
if err != nil {
return ""
}
return real
}
// dcnStorePin is the pinned real root of a held store, pinning it now (and
// saving it) for a grant synced before pins existed.
func dcnStorePin(dir string, storeID int64, root string) (string, error) {
grantsMu.Lock()
defer grantsMu.Unlock()
f, err := dcnLoadFile(dir)
if err != nil {
return "", err
}
key := strconv.FormatInt(storeID, 10)
rec, ok := f.Stores[key]
if !ok {
return "", errors.New("no grant for this store on this box")
}
if rec.RootReal != "" {
return rec.RootReal, nil
}
real, err := dcnRealRoot(root)
if err != nil {
return "", err
}
rec.RootReal = real
f.Stores[key] = rec
if err := dcnSaveFile(dir, f); err != nil {
return "", err
}
return real, nil
}
// dcnOpenStore opens the store a verified grant names, as the folder that was
// granted, or refuses.
func dcnOpenStore(dir string, storeID int64, terms *dcnTerms) (*os.Root, error) {
pin, err := dcnStorePin(dir, storeID, terms.Root)
if err != nil {
return nil, fmt.Errorf("the granted folder cannot be opened: %w", err)
}
real, err := dcnRealRoot(terms.Root)
if err != nil {
return nil, fmt.Errorf("the granted folder cannot be opened: %w", err)
}
if real != pin {
return nil, errDcnRepointed
}
return dcnOpenRootAt(pin)
}
// dcnOpenRootAt opens real (a resolved, absolute path) as an os.Root and
// checks that the handle is the folder reached at real with no link in any
// component, so nothing above the folder was swapped while it was opened.
func dcnOpenRootAt(real string) (*os.Root, error) {
r, err := os.OpenRoot(real)
if err != nil {
return nil, fmt.Errorf("the granted folder cannot be opened: %w", err)
}
rfi, err := r.Stat(".")
if err != nil {
_ = r.Close()
return nil, fmt.Errorf("the granted folder cannot be opened: %w", err)
}
nfi, err := dcnNoLinkStat(real)
if err != nil || !nfi.IsDir() || !os.SameFile(rfi, nfi) {
_ = r.Close()
return nil, errDcnRepointed
}
return r, nil
}
// dcnRel is a path the sandbox accepted (cleanRel/dcnResolvePath's joined
// form) as a name inside the root handle.
func dcnRel(root, joined string) (string, error) {
rel, err := filepath.Rel(filepath.Clean(root), joined)
if err != nil || rel == ".." || filepath.IsAbs(rel) || len(rel) >= 3 && rel[:3] == ".."+string(os.PathSeparator) {
return "", errors.New("path escapes the store root")
}
return rel, nil
}
// dcnEscaped: the root handle refused a name because it leads out of the store.
func dcnEscaped(err error) bool {
var pe *os.PathError
return errors.As(err, &pe) && pe.Err != nil && pe.Err.Error() == "path escapes from parent"
}
// dcnRootAtomicWrite is atomicWrite inside the store's handle: folders 0700,
// the file 0600, written beside its name and renamed into place.
func dcnRootAtomicWrite(r *os.Root, rel string, data []byte) error {
if d := filepath.Dir(rel); d != "." {
if err := r.MkdirAll(d, 0o700); err != nil {
return err
}
}
// The side name is unguessable and opened O_EXCL, as atomicWrite's
// CreateTemp is: whoever can write in the store could otherwise plant a
// link at a guessable name, and an open that follows it would truncate
// and fill whatever file in the store the link names. O_EXCL refuses any
// existing name, a link included, without following it.
var f *os.File
var tmp string
for try := 0; ; try++ {
var nb [12]byte
if _, err := rand.Read(nb[:]); err != nil {
return err
}
tmp = filepath.Join(filepath.Dir(rel), ".dcn-"+hex.EncodeToString(nb[:])+".tmp")
var err error
f, err = r.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
if err == nil {
break
}
if !errors.Is(err, fs.ErrExist) || try >= 8 {
return err
}
}
if _, err := f.Write(data); err != nil {
_ = f.Close()
_ = r.Remove(tmp)
return err
}
if err := f.Sync(); err != nil {
_ = f.Close()
_ = r.Remove(tmp)
return err
}
if err := f.Close(); err != nil {
_ = r.Remove(tmp)
return err
}
if err := r.Rename(tmp, rel); err != nil {
_ = r.Remove(tmp)
return err
}
return nil
}
+21
View File
@@ -0,0 +1,21 @@
//go:build darwin
package main
import (
"os"
"golang.org/x/sys/unix"
)
// dcnNoLinkStat describes the directory at p opened with O_NOFOLLOW_ANY, which
// fails on a link in any component (macOS 11 and later).
func dcnNoLinkStat(p string) (os.FileInfo, error) {
fd, err := unix.Open(p, unix.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW_ANY|unix.O_CLOEXEC, 0)
if err != nil {
return nil, err
}
f := os.NewFile(uintptr(fd), p)
defer f.Close()
return f.Stat()
}
+48
View File
@@ -0,0 +1,48 @@
//go:build linux
package main
import (
"errors"
"os"
"strings"
"golang.org/x/sys/unix"
)
// dcnNoLinkStat describes the directory at p (absolute, clean) reached one
// component at a time with O_NOFOLLOW, so a link anywhere in the path fails
// instead of being followed. O_PATH: a component the bastion may traverse but
// not list (a home at 0711) still opens.
func dcnNoLinkStat(p string) (os.FileInfo, error) {
if !strings.HasPrefix(p, "/") {
return nil, errors.New("not an absolute path")
}
fd, err := unix.Open("/", unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
if err != nil {
return nil, err
}
for _, c := range strings.Split(p, "/") {
if c == "" {
continue
}
if c == "." || c == ".." {
_ = unix.Close(fd)
return nil, errors.New("not a clean path")
}
next, err := unix.Openat(fd, c, unix.O_PATH|unix.O_NOFOLLOW|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
_ = unix.Close(fd)
if err != nil {
return nil, err
}
var st unix.Stat_t
if err := unix.Fstat(next, &st); err != nil || st.Mode&unix.S_IFMT != unix.S_IFDIR {
_ = unix.Close(next)
return nil, errors.New("a component is not a directory")
}
fd = next
}
f := os.NewFile(uintptr(fd), p)
defer f.Close()
return f.Stat()
}
+14
View File
@@ -0,0 +1,14 @@
//go:build !linux && !darwin
package main
import "os"
// dcnNoLinkStat on a platform with no open that refuses a link in every
// component (Windows): the final component is judged without following it.
// The store is still reached only through its os.Root and only while its real
// path matches the pin; what this leaves is a swap of a folder above the
// store, by someone who can rename it, timed between the two opens.
func dcnNoLinkStat(p string) (os.FileInfo, error) {
return os.Lstat(p)
}
+79 -46
View File
@@ -6,6 +6,7 @@ import (
"errors" "errors"
"io" "io"
"net" "net"
"net/http"
"strings" "strings"
"time" "time"
) )
@@ -24,94 +25,110 @@ type egressDialer func(ctx context.Context, network, addr string) (net.Conn, err
// egress is refused (403, before any dial) unless its pairing grant carries // egress is refused (403, before any dial) unless its pairing grant carries
// the "egress" cap: this lane makes requests leave THIS machine's connection // the "egress" cap: this lane makes requests leave THIS machine's connection
// under its address, and nothing in the mgmt op table gates it. // under its address, and nothing in the mgmt op table gates it.
func handleEgressConn(client net.Conn, dialOut egressDialer, self string) { //
// It reports whether it took the job: true once the request passed every check
// and was dialled out, whatever the destination then answered; false when the
// worker closed before a request, or the request was malformed or refused. The
// worker pool (runEgressWorkers) parks again at once after a job it took, and
// waits after one it did not.
func handleEgressConn(client net.Conn, dialOut egressDialer, self string) bool {
defer client.Close() defer client.Close()
br := bufio.NewReader(client) // The request line and headers are read under a bound, the one Go's own HTTP
// server uses (http.DefaultMaxHeaderBytes, 1 MiB): the host sends them, and a
// head that never ends must not grow in this machine's memory.
headLim := &io.LimitedReader{R: client, N: egressMaxHeadBytes}
br := bufio.NewReader(headLim)
line, err := br.ReadString('\n') line, err := br.ReadString('\n')
if err != nil { if err != nil {
return return false
} }
parts := strings.Fields(strings.TrimSpace(line)) parts := strings.Fields(strings.TrimSpace(line))
if len(parts) < 2 { if len(parts) < 2 {
return return false
} }
method, target := parts[0], parts[1] method, target := parts[0], parts[1]
// Buffer the head so a plain-HTTP request can be replayed verbatim; capture Host. // Buffer the head: a plain-HTTP request is parsed from it and rebuilt
// (parseEgressRequest), never replayed verbatim.
var head strings.Builder var head strings.Builder
head.WriteString(line) head.WriteString(line)
hostHdr := ""
for { for {
h, err := br.ReadString('\n') h, err := br.ReadString('\n')
if err != nil { if err != nil {
return return false
} }
head.WriteString(h) head.WriteString(h)
if h == "\r\n" || h == "\n" { if h == "\r\n" || h == "\n" {
break break
} }
if len(h) >= 5 && strings.EqualFold(h[:5], "host:") {
hostHdr = strings.TrimSpace(h[5:])
}
} }
// The head is in; what follows is the stream itself, unbounded. What the
// reader already buffered past the head stays first in line.
headLim.N = 1<<63 - 1
if why := capsAllow(self, "egress"); why != "" { if why := capsAllow(self, "egress"); why != "" {
_, _ = client.Write([]byte("HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nX-Privatae-Refused: " + why + "\r\n\r\n")) writeEgressRefusal(client, "403 Forbidden", why, why)
return return false
} }
ctx, cancel := context.WithTimeout(context.Background(), egressDialTimeout) ctx, cancel := context.WithTimeout(context.Background(), egressDialTimeout)
defer cancel() defer cancel()
if strings.EqualFold(method, "CONNECT") { if strings.EqualFold(method, "CONNECT") {
dialAddrs, why := egressVettedAddrs(ctx, ensurePort(target, "443")) // The target is read the way the Host's egress gate reads it
// (parseEgressAuthority): userinfo, an unbracketed IPv6 literal or a
// malformed port is refused, never guessed at.
host, port, ok := parseEgressAuthority(target, 443)
if !ok {
writeEgressRefusal(client, "400 Bad Request", egressNoHost, egressNoHost)
return false
}
dialAddrs, why := egressVettedAddrs(ctx, egressHostPort(host, port))
if why != "" { if why != "" {
_, _ = client.Write([]byte("HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nX-Privatae-Refused: " + why + "\r\n\r\n")) writeEgressRefusal(client, "403 Forbidden", why, why)
return return false
} }
upstream, err := dialEach(ctx, dialOut, dialAddrs) upstream, err := dialEach(ctx, dialOut, dialAddrs)
if err != nil { if err != nil {
_, _ = client.Write([]byte("HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n")) writeEgressRefusal(client, "502 Bad Gateway", egressUnreachable, "")
return return true
} }
if _, err := client.Write([]byte("HTTP/1.1 200 Connection established\r\n\r\n")); err != nil { if _, err := client.Write([]byte("HTTP/1.1 200 Connection established\r\n\r\n")); err != nil {
_ = upstream.Close() _ = upstream.Close()
return return true
} }
splice(br, client, upstream) splice(br, client, upstream)
return return true
} }
// Plain HTTP, absolute-form: "GET http://host/path HTTP/1.1". // Plain HTTP: ONE request per connection, parsed and rebuilt so the
host := hostHdr // destination reads the host, port and body the lane vetted, sent with
if i := strings.Index(target, "://"); i >= 0 { // `Connection: close` (egress_http.go). Anything the Host sends after that
rest := target[i+3:] // request's body closes the connection unforwarded: a keep-alive connection
if s := strings.IndexByte(rest, '/'); s >= 0 { // can no longer carry later requests past the vetting.
host = rest[:s] req, refused := parseEgressRequest(head.String())
} else { if refused != nil {
host = rest writeEgressRefusal(client, refused.status, refused.body, refused.body)
return false
} }
} dialAddrs, why := egressVettedAddrs(ctx, egressHostPort(req.host, req.port))
if host == "" {
return
}
dialAddrs, why := egressVettedAddrs(ctx, ensurePort(host, "80"))
if why != "" { if why != "" {
_, _ = client.Write([]byte("HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nX-Privatae-Refused: " + why + "\r\n\r\n")) writeEgressRefusal(client, "403 Forbidden", why, why)
return return false
} }
upstream, err := dialEach(ctx, dialOut, dialAddrs) upstream, err := dialEach(ctx, dialOut, dialAddrs)
if err != nil { if err != nil {
_, _ = client.Write([]byte("HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n")) writeEgressRefusal(client, "502 Bad Gateway", egressUnreachable, "")
return return true
} }
if _, err := upstream.Write([]byte(head.String())); err != nil { oneEgressExchange(br, client, upstream, req)
_ = upstream.Close() return true
return
}
splice(br, client, upstream)
} }
// egressMaxHeadBytes bounds an egress request's line and headers together.
const egressMaxHeadBytes = http.DefaultMaxHeaderBytes
// ensurePort returns host:port, adding defPort when target has no explicit port. // ensurePort returns host:port, adding defPort when target has no explicit port.
func ensurePort(target, defPort string) string { func ensurePort(target, defPort string) string {
target = strings.TrimSpace(target) target = strings.TrimSpace(target)
@@ -157,11 +174,14 @@ func splice(clientReader io.Reader, client, upstream net.Conn) {
// included), and any address the machine takes as its own (ownaddr.go), such // included), and any address the machine takes as its own (ownaddr.go), such
// as a public one on its NIC or a range a `local` route delivers here. // as a public one on its NIC or a range a `local` route delivers here.
// //
// What differs is the network around it. On an ANCHORED box the lane reaches // The network around it is refused too, on every box: the lane reaches the
// the public internet and nothing else: private, CGNAT and reserved addresses // public internet and nothing else, so private, CGNAT and reserved addresses
// (the LAN, the mesh) are refused too. On a LOCAL box the LAN stays reachable: // (the LAN, the mesh) are refused, by address and by any address a name
// the host there is the one the person paired, and their own network is // resolves to. A LOCAL box used to keep its LAN reachable here; it no longer
// theirs to reach. // does (2026-10-26): the Host's egress gate holds every dial it makes to the
// same rule, and a person's own machines
// are reached through their approved lanes, never through egress. Only the
// wording differs between the two kinds of box.
// //
// Resolution failures refuse too: a name this box cannot resolve is not one // Resolution failures refuse too: a name this box cannot resolve is not one
// it can vouch for. // it can vouch for.
@@ -245,6 +265,19 @@ func egressVettedAddrs(ctx context.Context, hostport string) ([]string, string)
} }
return nil, "refused: egress to " + host + " — that is this machine; this lane never reaches this machine" return nil, "refused: egress to " + host + " — that is this machine; this lane never reaches this machine"
} }
// The network around this machine, on a local box as on an anchored one
// (above): any one address of the name that is not the public internet
// refuses it. Checked after this machine's own addresses, so those keep
// saying "that is this machine".
if !anchored {
for _, ip := range addrs {
if internalIP(ip) {
return nil, "refused: egress to " + host + " (" + ip.String() + ") — this lane reaches the " +
"public internet only, never this machine's network or the mesh; reach your own " +
"machines through their approved lanes"
}
}
}
out := make([]string, 0, len(addrs)) out := make([]string, 0, len(addrs))
for _, ip := range addrs { for _, ip := range addrs {
out = append(out, net.JoinHostPort(ip.String(), port)) out = append(out, net.JoinHostPort(ip.String(), port))
+420
View File
@@ -0,0 +1,420 @@
package main
import (
"bufio"
"errors"
"io"
"net"
"net/http/httputil"
"strconv"
"strings"
)
// ONE plain-HTTP request per egress connection, read the way the Host's egress
// gate reads it (its strict request parse and its authority parse: the same
// refusals, the same normalised host, the same rebuilt head).
//
// Everything the lane writes to a destination is what it checked, and nothing
// more:
// - the head is parsed strictly and REBUILT, so the destination reads the same
// host, port and body length the lane vetted: never a second reading of the
// Host's bytes (a Host header naming another site, a query or userinfo glued
// to the host, a padded Transfer-Encoding);
// - the request goes out with `Connection: close`, its body is framed by
// Content-Length or chunked encoding, and a byte the Host sends after the
// body closes the connection with nothing of it forwarded. Before this the
// lane wrote the first head and then spliced the whole connection, so every
// later request on a keep-alive connection reached the destination unread.
//
// An honest client sends its next request only after the response, which the
// destination closes behind `Connection: close`, so it opens a new connection
// and meets the vetting again.
// egressHostChars are the characters a host name may have here (lower-cased).
const egressHostChars = "abcdefghijklmnopqrstuvwxyz0123456789-._"
// egressTokenChars are RFC 9110 token characters: methods and header names.
const egressTokenChars = "!#$%&'*+-.^_`|~0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
// egressHopByHop are headers that belong to one hop. The lane speaks for itself
// on the next one, and a proxy credential never goes on to the destination.
var egressHopByHop = map[string]bool{
"connection": true, "proxy-connection": true, "keep-alive": true, "upgrade": true,
"te": true, "trailer": true, "proxy-authorization": true, "proxy-authenticate": true,
}
// egressTrailerLimit bounds the trailer section of a chunked body, which is
// read to its end and dropped (the gate's limit).
const egressTrailerLimit = 8192
// egressRefusal is a request the lane refuses before it dials anything.
type egressRefusal struct{ status, body string }
func egressBad(body string) *egressRefusal {
return &egressRefusal{"400 Bad Request", body}
}
const egressNoHost = "The bastion could not read where this request is going (no host in the " +
"request line or in a Host header), so it sent nothing."
// egressUnreachable is the body of a 502: the destination passed the vetting,
// and no address of it answered from this machine.
const egressUnreachable = "The bastion could not reach that destination from this machine's own " +
"connection, so nothing was sent."
// egressHostPort is host and port as the vetting takes them.
func egressHostPort(host string, port int) string {
return net.JoinHostPort(host, strconv.Itoa(port))
}
// writeEgressRefusal answers the Host with a status and a short plain-text
// body saying why, and closes the exchange. why, when set, also goes in
// X-Privatae-Refused, the header the Host already reads.
func writeEgressRefusal(w io.Writer, status, body, why string) {
body += "\n"
var b strings.Builder
b.WriteString("HTTP/1.1 " + status + "\r\n")
b.WriteString("Content-Type: text/plain; charset=utf-8\r\n")
b.WriteString("Content-Length: " + strconv.Itoa(len(body)) + "\r\n")
b.WriteString("Connection: close\r\n")
if why != "" {
b.WriteString("X-Privatae-Refused: " + strings.NewReplacer("\r", " ", "\n", " ").Replace(why) + "\r\n")
}
b.WriteString("\r\n")
b.WriteString(body)
_, _ = io.WriteString(w, b.String())
}
// parseEgressAuthority: (host, port, true) from "host", "host:port", "[v6]" or
// "[v6]:port", or false when the lane can't read it exactly as the destination
// would. Refused rather than guessed: userinfo ("a@b"), an unbracketed IPv6
// literal, characters no host name has, a non-numeric or out-of-range port.
// The host is lower-cased and loses one trailing dot.
func parseEgressAuthority(s string, defaultPort int) (string, int, bool) {
s = strings.TrimSpace(s)
if s == "" || strings.Contains(s, "@") {
return "", 0, false
}
var host string
port := defaultPort
digits := func(p string) bool {
if len(p) == 0 || len(p) > 5 {
return false
}
for _, c := range p {
if c < '0' || c > '9' {
return false
}
}
return true
}
if strings.HasPrefix(s, "[") {
end := strings.IndexByte(s, ']')
if end < 0 {
return "", 0, false
}
// An IPv6 literal, IPv4-mapped forms included (the vetting judges the
// address they map to); a bare IPv4 address in brackets or a zone is
// not one.
inner := s[1:end]
ip := net.ParseIP(inner)
if ip == nil || !strings.Contains(inner, ":") || strings.Contains(inner, "%") {
return "", 0, false
}
host = ip.String()
rest := s[end+1:]
switch {
case rest == "":
case strings.HasPrefix(rest, ":") && digits(rest[1:]):
port, _ = strconv.Atoi(rest[1:])
default:
return "", 0, false
}
} else {
if strings.Count(s, ":") > 1 {
return "", 0, false
}
h, p, hasPort := strings.Cut(s, ":")
if hasPort {
if !digits(p) {
return "", 0, false
}
port, _ = strconv.Atoi(p)
}
h = strings.ToLower(h)
h = strings.TrimSuffix(h, ".")
if h == "" || strings.HasPrefix(h, ".") || strings.Contains(h, "..") {
return "", 0, false
}
for _, c := range h {
if !strings.ContainsRune(egressHostChars, c) {
return "", 0, false
}
}
host = h
}
if port <= 0 || port >= 65536 {
return "", 0, false
}
return host, port, true
}
// fmtEgressAuthority writes host[:port] back, the port left out when it is the
// scheme's default.
func fmtEgressAuthority(host string, port, defaultPort int) string {
h := host
if strings.Contains(host, ":") {
h = "[" + host + "]"
}
if port == defaultPort {
return h
}
return h + ":" + strconv.Itoa(port)
}
// egressRequest is the one plain-HTTP request the lane forwards.
type egressRequest struct {
host string
port int
head []byte // rebuilt, CRLF, ends at its blank line
chunked bool
length int64
}
func egressTokenOK(s string) bool {
if s == "" {
return false
}
for _, c := range s {
if !strings.ContainsRune(egressTokenChars, c) {
return false
}
}
return true
}
// parseEgressRequest reads the one request in head (the request line and
// headers through their blank line, as handleEgressConn read them) and
// rebuilds it, or refuses it. CONNECT never comes here.
func parseEgressRequest(head string) (egressRequest, *egressRefusal) {
lines := strings.Split(head, "\n")
// The read stops at the first blank line, so the head ends "\n", blank, "\n":
// drop the empty tail after the last "\n" and the blank line itself.
if len(lines) < 3 {
return egressRequest{}, egressBad("The request ended before its headers did, so the bastion sent nothing.")
}
lines = lines[:len(lines)-2]
for i, ln := range lines {
lines[i] = strings.TrimSuffix(ln, "\r")
if strings.ContainsAny(lines[i], "\r\x00") {
return egressRequest{}, egressBad("The request's headers carry a stray carriage return or NUL, so the bastion sent nothing.")
}
}
parts := strings.Split(lines[0], " ")
if len(parts) != 3 || parts[0] == "" || parts[1] == "" || parts[2] == "" || !egressTokenOK(parts[0]) {
return egressRequest{}, egressBad("The request line is not 'METHOD target HTTP/1.x', so the bastion sent nothing.")
}
method, rawTarget, version := strings.ToUpper(parts[0]), parts[1], parts[2]
if version != "HTTP/1.1" && version != "HTTP/1.0" {
return egressRequest{}, egressBad("The bastion forwards HTTP/1.0 and HTTP/1.1 only, so it sent nothing.")
}
type header struct{ name, value string }
var headers []header
for _, ln := range lines[1:] {
if strings.HasPrefix(ln, " ") || strings.HasPrefix(ln, "\t") {
return egressRequest{}, egressBad("A header is folded onto a second line (obsolete line folding), so the bastion sent nothing.")
}
name, value, ok := strings.Cut(ln, ":")
if !ok || !egressTokenOK(name) {
return egressRequest{}, egressBad("A header line is not 'Name: value' (no space may come before the colon), so the bastion sent nothing.")
}
headers = append(headers, header{name, strings.Trim(value, " \t")})
}
values := func(n string) []string {
var out []string
for _, h := range headers {
if strings.EqualFold(h.name, n) {
out = append(out, h.value)
}
}
return out
}
hosts := values("host")
if len(hosts) > 1 {
return egressRequest{}, egressBad("The request carries more than one Host header, so the bastion sent nothing.")
}
var host, target, authority string
var port int
if scheme, after, abs := strings.Cut(rawTarget, "://"); abs {
if strings.ToLower(scheme) != "http" {
// https:// in absolute form asks the lane to make the TLS connection,
// which it does not do: it used to send the request in plain text to
// port 80. Every client sends https as a CONNECT.
return egressRequest{}, egressBad("The bastion forwards plain http:// requests only; send https as a CONNECT. Nothing was sent.")
}
cut := len(after)
for _, sep := range []string{"/", "?", "#"} {
if i := strings.Index(after, sep); i >= 0 && i < cut {
cut = i
}
}
var ok bool
if host, port, ok = parseEgressAuthority(after[:cut], 80); !ok {
return egressRequest{}, egressBad(egressNoHost)
}
path, _, _ := strings.Cut(after[cut:], "#")
if !strings.HasPrefix(path, "/") {
path = "/" + path
}
authority = fmtEgressAuthority(host, port, 80)
target = "http://" + authority + path
} else {
if !strings.HasPrefix(rawTarget, "/") && rawTarget != "*" {
return egressRequest{}, egressBad(egressNoHost)
}
hostHdr := ""
if len(hosts) == 1 {
hostHdr = hosts[0]
}
var ok bool
if host, port, ok = parseEgressAuthority(hostHdr, 80); !ok {
return egressRequest{}, egressBad(egressNoHost)
}
authority = fmtEgressAuthority(host, port, 80)
target = rawTarget
}
// Body framing (RFC 9112 section 6). Every reading two parsers could disagree
// on is refused: both length headers, an unknown or doubled coding, a coding
// on HTTP/1.0, Content-Length values that differ.
req := egressRequest{host: host, port: port}
te, cl := values("transfer-encoding"), values("content-length")
if len(te) > 0 {
var codings []string
for _, v := range te {
for _, c := range strings.Split(v, ",") {
codings = append(codings, strings.ToLower(strings.TrimSpace(c)))
}
}
if len(cl) > 0 || version == "HTTP/1.0" || len(codings) != 1 || codings[0] != "chunked" {
return egressRequest{}, egressBad("The request's body length is ambiguous (Transfer-Encoding with " +
"Content-Length, on HTTP/1.0, or a coding other than chunked), so the bastion sent nothing.")
}
req.chunked = true
} else if len(cl) > 0 {
seen := map[string]bool{}
for _, v := range cl {
for _, c := range strings.Split(v, ",") {
seen[strings.TrimSpace(c)] = true
}
}
var one string
for k := range seen {
one = k
}
n, err := strconv.ParseInt(one, 10, 64)
if len(seen) != 1 || one == "" || len(one) > 18 || err != nil || n < 0 || strings.TrimLeft(one, "0123456789") != "" {
return egressRequest{}, egressBad("The request's Content-Length is not one number, so the bastion sent nothing.")
}
req.length = n
}
named := map[string]bool{}
for _, v := range values("connection") {
for _, t := range strings.Split(v, ",") {
named[strings.ToLower(strings.TrimSpace(t))] = true
}
}
var b strings.Builder
b.WriteString(method + " " + target + " " + version + "\r\n")
b.WriteString("Host: " + authority + "\r\n")
for _, h := range headers {
lk := strings.ToLower(h.name)
if egressHopByHop[lk] || named[lk] || lk == "host" || lk == "content-length" || lk == "transfer-encoding" {
continue
}
b.WriteString(h.name + ": " + h.value + "\r\n")
}
if req.chunked {
b.WriteString("Transfer-Encoding: chunked\r\n")
} else if len(cl) > 0 {
b.WriteString("Content-Length: " + strconv.FormatInt(req.length, 10) + "\r\n")
}
b.WriteString("Connection: close\r\n\r\n")
req.head = []byte(b.String())
return req, nil
}
// forwardEgressBody copies the one request body from src to dst, framed as the
// rebuilt head says. A chunked body goes out re-encoded (plain sizes, no
// extensions, trailers read and dropped), so the destination parses exactly the
// body this parser found.
func forwardEgressBody(dst io.Writer, src *bufio.Reader, req egressRequest) error {
if !req.chunked {
if req.length == 0 {
return nil
}
_, err := io.CopyN(dst, src, req.length)
return err
}
cw := httputil.NewChunkedWriter(dst)
if _, err := io.Copy(cw, httputil.NewChunkedReader(src)); err != nil {
return err
}
// The trailer section, read to its blank line within a bound, and dropped.
read := 0
for {
line, err := src.ReadSlice('\n')
read += len(line)
if err != nil || read > egressTrailerLimit {
return errors.New("chunked body: bad trailer section")
}
if string(line) == "\r\n" || string(line) == "\n" {
break
}
}
if err := cw.Close(); err != nil {
return err
}
_, err := io.WriteString(dst, "\r\n")
return err
}
// oneEgressExchange writes the one rebuilt request and its body to upstream,
// and carries the destination's answer back until the destination closes.
// Nothing the Host sends after the body is forwarded: its first byte closes
// both sides.
func oneEgressExchange(clientReader *bufio.Reader, client, upstream net.Conn, req egressRequest) {
answered := make(chan struct{})
go func() {
_, _ = io.Copy(client, upstream)
close(answered)
_ = client.Close()
}()
defer func() {
_ = upstream.Close()
<-answered
}()
if _, err := upstream.Write(req.head); err != nil {
return
}
if err := forwardEgressBody(upstream, clientReader, req); err != nil {
return
}
// The request is complete. A further byte is a second request on this
// connection: never forwarded, and the connection closes. End of stream
// from the Host is passed on, as the splice did.
go func() {
var one [1]byte
if n, _ := clientReader.Read(one[:]); n > 0 {
_ = upstream.Close()
_ = client.Close()
return
}
if cw, ok := upstream.(interface{ CloseWrite() error }); ok {
_ = cw.CloseWrite()
}
}()
<-answered
}
+17 -16
View File
@@ -7,8 +7,10 @@ package main
// The egress lane on a local (unanchored) box. It refuses this machine and // The egress lane on a local (unanchored) box. It refuses this machine and
// its link-local addresses, so a host cannot reach this machine's services // its link-local addresses, so a host cannot reach this machine's services
// (ollama above all) around the switches that govern them, and it keeps the // (ollama above all) around the switches that govern them, and, since
// person's LAN. An anchored box is unchanged: the public internet only. // 2026-10-26, the network around it as well: the LAN and the mesh
// (egress_network_test.go). An anchored box is unchanged: the public internet
// only.
import ( import (
"bufio" "bufio"
@@ -57,13 +59,14 @@ func localEgressFixture(t *testing.T) {
"self.example.test": {"192.168.50.7"}, "self.example.test": {"192.168.50.7"},
"nas.example.test": {"192.168.50.30"}, "nas.example.test": {"192.168.50.30"},
"dual.example.test": {"2001:db8::8", "198.51.100.8"}, "dual.example.test": {"2001:db8::8", "198.51.100.8"},
"site.example.test": {"198.51.100.9"},
}) })
} }
// MUTATION: restore the early return for an unanchored box in egressVettedAddrs. // MUTATION: restore the early return for an unanchored box in egressVettedAddrs.
// RESULT: red — 127.0.0.1:11434 is handed to the dial, and a name is dialled // RESULT: red — 127.0.0.1:11434 is handed to the dial, and a name is dialled
// as a name instead of the address that was vetted. // as a name instead of the address that was vetted.
func TestLocalEgressRefusesThisMachineAndKeepsTheLAN(t *testing.T) { func TestLocalEgressRefusesThisMachineAndTheLAN(t *testing.T) {
localEgressFixture(t) localEgressFixture(t)
ctx := context.Background() ctx := context.Background()
for _, hp := range []string{ for _, hp := range []string{
@@ -75,24 +78,22 @@ func TestLocalEgressRefusesThisMachineAndKeepsTheLAN(t *testing.T) {
"203.0.113.5:80", // a range a `local` route delivers here "203.0.113.5:80", // a range a `local` route delivers here
"ollama.example.test:11434", "mixed.example.test:443", "self.example.test:11434", "ollama.example.test:11434", "mixed.example.test:443", "self.example.test:11434",
"unknown.example.test:443", "unknown.example.test:443",
// the network around this machine (egress_network_test.go has the rest)
"192.168.50.20:8080", "10.1.2.3:443", "172.16.4.4:22", "[fd12:3456::9]:80", "100.64.0.9:80",
"nas.example.test:443",
} { } {
if addr, why := egressVettedAddr(ctx, hp); why == "" || addr != "" { if addr, why := egressVettedAddr(ctx, hp); why == "" || addr != "" {
t.Errorf("%s must be refused on a local box, got dial %q", hp, addr) t.Errorf("%s must be refused on a local box, got dial %q", hp, addr)
} }
} }
for hp, want := range map[string][]string{ for hp, want := range map[string][]string{
"192.168.50.20:8080": {"192.168.50.20:8080"},
"10.1.2.3:443": {"10.1.2.3:443"},
"172.16.4.4:22": {"172.16.4.4:22"},
"[fd12:3456::9]:80": {"[fd12:3456::9]:80"},
"100.64.0.9:80": {"100.64.0.9:80"},
"198.51.100.8:443": {"198.51.100.8:443"}, "198.51.100.8:443": {"198.51.100.8:443"},
"nas.example.test:443": {"192.168.50.30:443"}, "site.example.test:443": {"198.51.100.9:443"},
"dual.example.test:443": {"[2001:db8::8]:443", "198.51.100.8:443"}, "dual.example.test:443": {"[2001:db8::8]:443", "198.51.100.8:443"},
} { } {
got, why := egressVettedAddrs(ctx, hp) got, why := egressVettedAddrs(ctx, hp)
if why != "" || !reflect.DeepEqual(got, want) { if why != "" || !reflect.DeepEqual(got, want) {
t.Errorf("%s is the LAN or the internet, so a local box dials %v: got %v %q", hp, want, got, why) t.Errorf("%s is the internet, so a local box dials %v: got %v %q", hp, want, got, why)
} }
} }
} }
@@ -114,8 +115,8 @@ func TestAnchoredEgressStillRefusesTheLAN(t *testing.T) {
} }
// Through the lane itself, with a real dialer: a request for this machine is // Through the lane itself, with a real dialer: a request for this machine is
// answered 403 and never reaches the listener; a LAN request is dialled at its // answered 403 and never reaches the listener; a public name is dialled at
// vetted address; a dual-stack name falls back to its second address. // its vetted address; a dual-stack name falls back to its second address.
func TestLocalEgressLaneNeverDialsThisMachine(t *testing.T) { func TestLocalEgressLaneNeverDialsThisMachine(t *testing.T) {
localEgressFixture(t) localEgressFixture(t)
ln, err := net.Listen("tcp", "127.0.0.1:0") ln, err := net.Listen("tcp", "127.0.0.1:0")
@@ -164,11 +165,11 @@ func TestLocalEgressLaneNeverDialsThisMachine(t *testing.T) {
asked = append(asked, addr) asked = append(asked, addr)
return nil, errors.New("stop here") return nil, errors.New("stop here")
} }
if line := ask(recording, "CONNECT nas.example.test:443 HTTP/1.1\r\n\r\n"); !strings.HasPrefix(line, "HTTP/1.1 502") { if line := ask(recording, "CONNECT site.example.test:443 HTTP/1.1\r\n\r\n"); !strings.HasPrefix(line, "HTTP/1.1 502") {
t.Fatalf("a LAN destination is dialled (and this dial fails): %q", line) t.Fatalf("a public destination is dialled (and this dial fails): %q", line)
} }
if !reflect.DeepEqual(asked, []string{"192.168.50.30:443"}) { if !reflect.DeepEqual(asked, []string{"198.51.100.9:443"}) {
t.Fatalf("the vetted LAN address, never the name: %v", asked) t.Fatalf("the vetted address, never the name: %v", asked)
} }
asked = nil asked = nil
+76
View File
@@ -0,0 +1,76 @@
//go:build !windows
// Tagged !windows: it borrows localEgressFixture and answerEgressNames
// (egress_local_test.go).
package main
// The egress lane on a local (unanchored) box refuses the network around the
// machine as well as the machine itself: the LAN, the mesh and every other
// address that is not the public internet, by address and by any address a
// name resolves to. The Host's egress gate holds every dial to the same rule;
// a person's own machines are reached through their approved lanes.
import (
"bufio"
"context"
"errors"
"io"
"net"
"reflect"
"strings"
"testing"
"time"
)
// MUTATION: drop the local-box internalIP loop at the end of egressVettedAddrs.
// RESULT: red — 192.168.50.20, nas.example.test and the rest are handed to the
// dial, and the lane dials 192.168.50.30.
func TestLocalEgressRefusesTheLANAndTheMesh(t *testing.T) {
localEgressFixture(t)
answerEgressNames(t, map[string][]string{
"nas.example.test": {"192.168.50.30"},
"half.example.test": {"198.51.100.8", "10.1.2.3"},
"mesh.example.test": {"fd12:3456::9"},
"nat.example.test": {"64:ff9b::a01:203"}, // NAT64 for 10.1.2.3
"pub.example.test": {"198.51.100.8"},
})
ctx := context.Background()
for _, hp := range []string{
"192.168.50.20:8080", "10.1.2.3:443", "172.16.4.4:22", "[fd12:3456::9]:80", "100.64.0.9:80",
"198.18.0.1:80", "[64:ff9b::a01:203]:443",
"nas.example.test:443", "half.example.test:443", "mesh.example.test:80", "nat.example.test:443",
} {
if addrs, why := egressVettedAddrs(ctx, hp); why == "" || addrs != nil || !strings.Contains(why, "approved lanes") {
t.Errorf("%s is not the public internet and must be refused on a local box, got dial %v %q", hp, addrs, why)
}
}
if got, why := egressVettedAddrs(ctx, "pub.example.test:443"); why != "" || !reflect.DeepEqual(got, []string{"198.51.100.8:443"}) {
t.Fatalf("the public internet passes at its vetted address: %v %q", got, why)
}
// Through the lane itself: 403, and nothing is dialled.
var asked []string
recording := func(_ context.Context, _, addr string) (net.Conn, error) {
asked = append(asked, addr)
return nil, errors.New("stop here")
}
for _, req := range []string{
"CONNECT nas.example.test:443 HTTP/1.1\r\n\r\n",
"GET http://10.1.2.3/ HTTP/1.1\r\nHost: 10.1.2.3\r\n\r\n",
"CONNECT half.example.test:443 HTTP/1.1\r\n\r\n",
} {
srv, cli := net.Pipe()
go handleEgressConn(srv, recording, "home.host.example")
_ = cli.SetDeadline(time.Now().Add(5 * time.Second))
_, _ = io.WriteString(cli, req)
line, _ := bufio.NewReader(cli).ReadString('\n')
cli.Close()
if !strings.HasPrefix(line, "HTTP/1.1 403") {
t.Errorf("a local box must refuse %q, answered %q", req, line)
}
}
if len(asked) != 0 {
t.Fatalf("nothing on the network around this machine is dialled: %v", asked)
}
}
+393
View File
@@ -0,0 +1,393 @@
//go:build !windows
package main
import (
"bufio"
"bytes"
"context"
"errors"
"fmt"
"io"
"net"
"net/http"
"strings"
"sync"
"testing"
"time"
)
// The bastion's egress lanes carry ONE plain-HTTP request per connection, read
// the way the Host's egress gate reads it, and every refusal says
// why in its body.
//
// MUTATIONS, each red:
// - handleEgressConn writes head.String() to upstream and splices, as before:
// TestEgressPlainHTTPCarriesOneRequestPerConnection, ...SendsTheHostItChecked;
// - oneEgressExchange's watcher forwards instead of closing: ...OneRequest...;
// - parseEgressAuthority accepts "@" or cuts the authority at "/" only:
// TestEgressPlainHTTPReadsTheHostAsTheGateDoes;
// - writeEgressRefusal writes Content-Length: 0: TestEgressRefusalsCarryABody.
// recordingOrigin stands in for a destination that keeps a connection open
// after answering (it ignores Connection: close, as a careless server may),
// and records every byte one connection carried to it.
type recordingOrigin struct {
ln net.Listener
conns chan string
// silent: never answer, so the lane never holds an unread answer when it
// closes (a close with unread data is a reset, which can cost the origin
// bytes it has not read yet).
silent bool
}
func newSilentOrigin(t *testing.T) *recordingOrigin {
o := newRecordingOrigin(t)
o.silent = true
return o
}
func newRecordingOrigin(t *testing.T) *recordingOrigin {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
o := &recordingOrigin{ln: ln, conns: make(chan string, 16)}
t.Cleanup(func() { _ = ln.Close() })
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go o.serve(c)
}
}()
return o
}
func (o *recordingOrigin) serve(c net.Conn) {
defer c.Close()
var mu sync.Mutex
var got bytes.Buffer
answered := false
buf := make([]byte, 4096)
for {
_ = c.SetReadDeadline(time.Now().Add(800 * time.Millisecond))
n, err := c.Read(buf)
mu.Lock()
got.Write(buf[:n])
if !o.silent && !answered && bytes.Contains(got.Bytes(), []byte("\r\n\r\n")) {
answered = true
_, _ = io.WriteString(c, "HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
}
mu.Unlock()
if err != nil {
break
}
}
o.conns <- got.String()
}
// next is everything the origin's next connection carried, once it ended.
func (o *recordingOrigin) next(t *testing.T) string {
t.Helper()
s, ok := o.nextWithin(8 * time.Second)
if !ok {
t.Fatal("the destination never saw its connection end")
}
return s
}
func (o *recordingOrigin) nextWithin(d time.Duration) (string, bool) {
select {
case s := <-o.conns:
return s, true
case <-time.After(d):
return "", false
}
}
func (o *recordingOrigin) addr() string { return o.ln.Addr().String() }
// A keep-alive connection carries one request to the destination, never two:
// neither a request sent after the first answer (a client reusing the
// connection) nor one pipelined behind the first.
func TestEgressPlainHTTPCarriesOneRequestPerConnection(t *testing.T) {
first := "GET http://198.51.100.30/first HTTP/1.1\r\nHost: 198.51.100.30\r\n\r\n"
second := "GET http://198.51.100.30/second HTTP/1.1\r\nHost: 198.51.100.30\r\n\r\n"
t.Run("reused after the answer", func(t *testing.T) {
o := newRecordingOrigin(t)
var asked []string
cli, cleanup := dialHandler(t, redirectDial(o.addr(), &asked))
defer cleanup()
_, _ = io.WriteString(cli, first)
br := bufio.NewReader(cli)
resp, err := http.ReadResponse(br, nil)
if err != nil {
t.Fatalf("the first request got no answer: %v", err)
}
body, _ := io.ReadAll(resp.Body)
if string(body) != "ok" {
t.Fatalf("the first answer was %q", body)
}
_, _ = io.WriteString(cli, second)
got := o.next(t)
if strings.Contains(got, "/second") {
t.Fatalf("a second request on one connection reached the destination unvetted:\n%s", got)
}
if !strings.Contains(got, "\r\nConnection: close\r\n") {
t.Fatalf("the request went out without Connection: close:\n%s", got)
}
_ = cli.SetReadDeadline(time.Now().Add(3 * time.Second))
if rest, _ := io.ReadAll(br); len(rest) != 0 {
t.Fatalf("the lane answered the second request: %q", rest)
}
if len(asked) != 1 {
t.Fatalf("one connection, one dial: %v", asked)
}
})
t.Run("pipelined", func(t *testing.T) {
o := newRecordingOrigin(t)
var asked []string
cli, cleanup := dialHandler(t, redirectDial(o.addr(), &asked))
defer cleanup()
_, _ = io.WriteString(cli, first+second)
got := o.next(t)
if !strings.Contains(got, "GET http://198.51.100.30/first HTTP/1.1\r\n") {
t.Fatalf("the first request did not reach the destination:\n%s", got)
}
if strings.Contains(got, "/second") {
t.Fatalf("a pipelined second request reached the destination unvetted:\n%s", got)
}
})
}
// The destination reads the host the lane vetted, in the one Host header; hop
// headers, those Connection names and a proxy credential stay on this hop.
func TestEgressPlainHTTPSendsTheHostItChecked(t *testing.T) {
o := newRecordingOrigin(t)
var asked []string
cli, cleanup := dialHandler(t, redirectDial(o.addr(), &asked))
defer cleanup()
_, _ = io.WriteString(cli, "GET http://198.51.100.31/p?q=1 HTTP/1.1\r\n"+
"Host: inside.example.test\r\nProxy-Authorization: Basic eDp5\r\n"+
"Connection: keep-alive, X-Hop\r\nX-Hop: 1\r\nX-Keep: 2\r\n\r\n")
got := o.next(t)
head, _, _ := strings.Cut(got, "\r\n\r\n")
want := "GET http://198.51.100.31/p?q=1 HTTP/1.1\r\nHost: 198.51.100.31\r\nX-Keep: 2\r\nConnection: close"
if head != want {
t.Fatalf("the destination got\n%q\nwant\n%q", head, want)
}
if len(asked) != 1 || asked[0] != "198.51.100.31:80" {
t.Fatalf("the lane must dial the address it vetted: %v", asked)
}
}
// stubEgressNames answers names without DNS and records each name exactly as
// the lane asked for it.
func stubEgressNames(t *testing.T, m map[string]string) *[]string {
t.Helper()
var mu sync.Mutex
var asked []string
old := egressResolve
t.Cleanup(func() { egressResolve = old })
egressResolve = func(_ context.Context, host string) ([]net.IPAddr, error) {
mu.Lock()
asked = append(asked, host)
mu.Unlock()
ip, ok := m[host]
if !ok {
return nil, errors.New("no such host")
}
return []net.IPAddr{{IP: net.ParseIP(ip)}}, nil
}
return &asked
}
// The host is read as the gate reads it: the authority ends at the first "/",
// "?" or "#", is lower-cased and loses one trailing dot; userinfo, an
// unbracketed IPv6 literal, a bad port, https in absolute form, two Host
// headers and an ambiguous body length are refused with 400 and a body, before
// anything is resolved or dialled.
func TestEgressPlainHTTPReadsTheHostAsTheGateDoes(t *testing.T) {
for _, c := range []struct {
req, wantName, wantLine string
}{
{"GET http://public.example.test?@10.0.0.1/ HTTP/1.1\r\n\r\n", "public.example.test",
"GET http://public.example.test/?@10.0.0.1/ HTTP/1.1"},
{"GET http://public.example.test#@10.0.0.1/ HTTP/1.1\r\n\r\n", "public.example.test",
"GET http://public.example.test/ HTTP/1.1"},
{"GET http://Public.Example.Test./x HTTP/1.1\r\n\r\n", "public.example.test",
"GET http://public.example.test/x HTTP/1.1"},
{"GET /x HTTP/1.1\r\nHost: Public.Example.Test:8080\r\n\r\n", "public.example.test",
"GET /x HTTP/1.1"},
} {
o := newRecordingOrigin(t)
names := stubEgressNames(t, map[string]string{"public.example.test": "198.51.100.40"})
var asked []string
cli, cleanup := dialHandler(t, redirectDial(o.addr(), &asked))
_, _ = io.WriteString(cli, c.req)
got, _ := o.nextWithin(4 * time.Second)
cleanup()
if len(*names) != 1 || (*names)[0] != c.wantName {
t.Errorf("%q: the lane resolved %q; want only %q", c.req, *names, c.wantName)
}
if line, _, _ := strings.Cut(got, "\r\n"); line != c.wantLine {
t.Errorf("%q: the destination got the request line %q; want %q", c.req, line, c.wantLine)
}
}
for _, req := range []string{
"GET http://user:pw@public.example.test/ HTTP/1.1\r\n\r\n",
"GET http://198.51.100.41:80@public.example.test/ HTTP/1.1\r\n\r\n",
"GET http://2001:db8::1/ HTTP/1.1\r\n\r\n",
"GET http://public.example.test:80x/ HTTP/1.1\r\n\r\n",
"GET https://public.example.test/ HTTP/1.1\r\n\r\n",
"GET / HTTP/1.1\r\nHost: public.example.test\r\nHost: 198.51.100.41\r\n\r\n",
"GET / HTTP/1.1\r\n\r\n",
"POST http://public.example.test/ HTTP/1.1\r\nContent-Length: 3\r\nTransfer-Encoding: chunked\r\n\r\n",
"POST http://public.example.test/ HTTP/1.1\r\nContent-Length: 3\r\nContent-Length: 4\r\n\r\n",
"POST http://public.example.test/ HTTP/1.1\r\nTransfer-Encoding: gzip, chunked\r\n\r\n",
"GET http://public.example.test/ HTTP/1.1\r\n X-Folded: 1\r\n\r\n",
"CONNECT user@public.example.test:443 HTTP/1.1\r\n\r\n",
"CONNECT 2001:db8::1:443 HTTP/1.1\r\n\r\n",
"CONNECT [198.51.100.41]:443 HTTP/1.1\r\n\r\n",
} {
names := stubEgressNames(t, map[string]string{"public.example.test": "198.51.100.40"})
var asked []string
cli, cleanup := dialHandler(t, redirectDial("127.0.0.1:9", &asked))
_, _ = io.WriteString(cli, req)
resp, err := http.ReadResponse(bufio.NewReader(cli), nil)
if err != nil {
t.Errorf("%q: no answer: %v", req, err)
cleanup()
continue
}
body, _ := io.ReadAll(resp.Body)
cleanup()
if resp.StatusCode != 400 || len(body) == 0 || int64(len(body)) != resp.ContentLength {
t.Errorf("%q: answered %d with body %q; want 400 with a body saying why", req, resp.StatusCode, body)
}
if len(*names) != 0 || len(asked) != 0 {
t.Errorf("%q: refused requests resolve and dial nothing; resolved %q, dialled %q", req, *names, asked)
}
}
}
// A request body goes out framed as its head says, chunked bodies re-encoded,
// and nothing after it.
func TestEgressPlainHTTPBodyIsFramed(t *testing.T) {
for _, c := range []struct{ req, wantBody string }{
{"POST http://198.51.100.42/u HTTP/1.1\r\nHost: 198.51.100.42\r\nContent-Length: 5\r\n\r\nhello" +
"GET http://198.51.100.42/smuggled HTTP/1.1\r\n\r\n", "hello"},
{"POST http://198.51.100.42/u HTTP/1.1\r\nHost: 198.51.100.42\r\nTransfer-Encoding: chunked\r\n\r\n" +
"5;ext=1\r\nhello\r\n0\r\nX-Trailer: 1\r\n\r\nGET /smuggled HTTP/1.1\r\n\r\n", "5\r\nhello\r\n0\r\n\r\n"},
} {
o := newSilentOrigin(t)
var asked []string
cli, cleanup := dialHandler(t, redirectDial(o.addr(), &asked))
_, _ = io.WriteString(cli, c.req)
got := o.next(t)
cleanup()
_, body, _ := strings.Cut(got, "\r\n\r\n")
if body != c.wantBody {
t.Errorf("the destination got the body %q; want %q", body, c.wantBody)
}
}
}
// A refusal says why in its body, not only in a header: a 403 from the vetting
// and a 502 from a dial that found no answer, on both request forms, and the
// coder-run proxy's refusals too.
func TestEgressRefusalsCarryABody(t *testing.T) {
failDial := func(ctx context.Context, network, addr string) (net.Conn, error) {
return nil, fmt.Errorf("no route")
}
okDial := func(ctx context.Context, network, addr string) (net.Conn, error) {
return nil, fmt.Errorf("must not dial %s", addr)
}
check := func(name string, resp *http.Response, err error, want int) {
t.Helper()
if err != nil {
t.Errorf("%s: no answer: %v", name, err)
return
}
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode != want || len(body) < 10 || int64(len(body)) != resp.ContentLength {
t.Errorf("%s: answered %d with body %q (Content-Length %d); want %d with a short body saying why",
name, resp.StatusCode, body, resp.ContentLength, want)
}
}
for _, c := range []struct {
name string
dial egressDialer
req string
want int
}{
{"plain HTTP to this machine", okDial, "GET http://127.0.0.1:9/ HTTP/1.1\r\n\r\n", 403},
{"CONNECT to this machine", okDial, "CONNECT 127.0.0.1:9 HTTP/1.1\r\n\r\n", 403},
{"plain HTTP, no answer", failDial, "GET http://198.51.100.43/ HTTP/1.1\r\n\r\n", 502},
{"CONNECT, no answer", failDial, "CONNECT 198.51.100.43:443 HTTP/1.1\r\n\r\n", 502},
} {
cli, cleanup := dialHandler(t, c.dial)
_, _ = io.WriteString(cli, c.req)
resp, err := http.ReadResponse(bufio.NewReader(cli), nil)
check(c.name, resp, err, c.want)
cleanup()
}
oldDial := coderDialOut
t.Cleanup(func() { coderDialOut = oldDial })
coderDialOut = failDial
for _, c := range []struct {
name, req string
want int
}{
{"coder run to a private address", "GET http://10.0.0.1/ HTTP/1.1\r\nHost: 10.0.0.1\r\n\r\n", 403},
{"coder run, no answer", "GET http://93.184.216.34/ HTTP/1.1\r\nHost: 93.184.216.34\r\n\r\n", 502},
} {
boxEnd, runEnd := net.Pipe()
var refused int64
go handleCoderEgressConn(boxEnd, nil, &refused)
_ = runEnd.SetDeadline(time.Now().Add(5 * time.Second))
_, _ = io.WriteString(runEnd, c.req)
resp, err := http.ReadResponse(bufio.NewReader(runEnd), nil)
check(c.name, resp, err, c.want)
_ = runEnd.Close()
}
}
// The coder-run proxy carries one request per connection too: the run's own
// processes are the sender there.
func TestCoderEgressCarriesOneRequestPerConnection(t *testing.T) {
o := newRecordingOrigin(t)
oldDial := coderDialOut
t.Cleanup(func() { coderDialOut = oldDial })
coderDialOut = func(ctx context.Context, network, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, network, o.addr())
}
boxEnd, runEnd := net.Pipe()
var refused int64
go handleCoderEgressConn(boxEnd, nil, &refused)
_ = runEnd.SetDeadline(time.Now().Add(8 * time.Second))
_, _ = io.WriteString(runEnd, "GET http://93.184.216.34/first HTTP/1.1\r\nHost: 93.184.216.34\r\n\r\n")
br := bufio.NewReader(runEnd)
resp, err := http.ReadResponse(br, nil)
if err != nil {
t.Fatalf("the first request got no answer: %v", err)
}
_, _ = io.ReadAll(resp.Body)
_, _ = io.WriteString(runEnd, "GET http://93.184.216.34/second HTTP/1.1\r\nHost: inside.example.test\r\n\r\n")
got := o.next(t)
_ = runEnd.Close()
if strings.Contains(got, "/second") || strings.Contains(got, "inside.example.test") {
t.Fatalf("a second request on one connection reached the destination unvetted:\n%s", got)
}
if !strings.Contains(got, "\r\nConnection: close\r\n") {
t.Fatalf("the request went out without Connection: close:\n%s", got)
}
}
+47 -12
View File
@@ -14,35 +14,60 @@ import (
const egressWorkerCount = 4 const egressWorkerCount = 4
// egressRefusedWait is how long a worker waits before it parks again after a
// job it did not take (malformed, refused) or after its WebSocket closed while
// parked. A job it took parks the worker again at once, so the person's own
// traffic is never slowed by it; a Host that answers every worker with a job
// this machine refuses costs it at most egressWorkerCount dials, handshakes
// and refusals a second, not as many as the Host can send. The serve lane's
// workers wait the same (serveRefusedWait).
var egressRefusedWait = time.Second
// wsConn adapts a worker WebSocket into a net.Conn so the existing handleEgressConn drives // wsConn adapts a worker WebSocket into a net.Conn so the existing handleEgressConn drives
// it unchanged. The host frames the proxied client stream as binary messages: 'D'+payload = // it unchanged. The host frames the proxied client stream as binary messages: 'D'+payload =
// data, 'E' = end-of-stream. Reads surface the host's bytes; writes go back as 'D' frames; // data, 'E' = end-of-stream. Reads surface the host's bytes; writes go back as 'D' frames;
// Close/CloseWrite send 'E'. // Close/CloseWrite send 'E'.
//
// A 'D' frame is STREAMED into the caller's buffer, never read whole: the host picks a
// frame's length, and one frame of any length must cost this machine no more memory
// than the read it is asked for.
type wsConn struct { type wsConn struct {
ws *websocket.Conn ws *websocket.Conn
rbuf []byte cur io.Reader // the rest of the 'D' frame being read, or nil
wmu sync.Mutex wmu sync.Mutex
} }
func (c *wsConn) Read(p []byte) (int, error) { func (c *wsConn) Read(p []byte) (int, error) {
for len(c.rbuf) == 0 { for {
mt, data, err := c.ws.ReadMessage() if c.cur != nil {
n, err := c.cur.Read(p)
if err == io.EOF {
c.cur = nil
err = nil
}
if n > 0 || err != nil {
return n, err
}
continue
}
mt, r, err := c.ws.NextReader()
if err != nil { if err != nil {
return 0, err return 0, err
} }
if mt != websocket.BinaryMessage || len(data) == 0 { if mt != websocket.BinaryMessage {
continue continue // NextReader discards what is left of it
} }
switch data[0] { var tag [1]byte
if _, err := io.ReadFull(r, tag[:]); err != nil {
continue // an empty frame
}
switch tag[0] {
case 'D': case 'D':
c.rbuf = data[1:] c.cur = r
case 'E': case 'E':
return 0, io.EOF return 0, io.EOF
} }
} }
n := copy(p, c.rbuf)
c.rbuf = c.rbuf[n:]
return n, nil
} }
func (c *wsConn) Write(p []byte) (int, error) { func (c *wsConn) Write(p []byte) (int, error) {
@@ -80,19 +105,25 @@ func (c *wsConn) SetWriteDeadline(t time.Time) error { return c.ws.SetWriteDeadl
// bastion's tunnel IP) and serves one egress job per worker, reconnecting after each. Each // bastion's tunnel IP) and serves one egress job per worker, reconnecting after each. Each
// worker authenticates with the bastion's bearer, then blocks until the host dispatches a // worker authenticates with the bastion's bearer, then blocks until the host dispatches a
// request; handleEgressConn then dials it out the USER's connection (dialOut) and relays. // request; handleEgressConn then dials it out the USER's connection (dialOut) and relays.
// A worker reconnects at once after a job it took, and after egressRefusedWait after one
// it refused or a close while parked. The func returned waits until every worker has
// exited, which is once pctx is done and each one's job has ended.
// //
// pctx is the pairing's context (run.go): when the anchor revokes a pairing the // pctx is the pairing's context (run.go): when the anchor revokes a pairing the
// workers must exit, not re-dial a closed tunnel every 3s for the process's life. // workers must exit, not re-dial a closed tunnel every 3s for the process's life.
func runEgressWorkers(pctx context.Context, self string, wsURL string, tlsCfg *tls.Config, func runEgressWorkers(pctx context.Context, self string, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer string, dialHost func(ctx context.Context) (net.Conn, error), bearer string,
dialOut egressDialer, n int) { dialOut egressDialer, n int) (wait func()) {
dialer := &websocket.Dialer{ dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) }, NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg, TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second, HandshakeTimeout: 20 * time.Second,
} }
var wg sync.WaitGroup
for i := 0; i < n; i++ { for i := 0; i < n; i++ {
wg.Add(1)
go func() { go func() {
defer wg.Done()
for pctx.Err() == nil { for pctx.Err() == nil {
ctx, cancel := context.WithTimeout(pctx, 25*time.Second) ctx, cancel := context.WithTimeout(pctx, 25*time.Second)
ws, _, err := dialer.DialContext(ctx, wsURL, http.Header{}) ws, _, err := dialer.DialContext(ctx, wsURL, http.Header{})
@@ -108,11 +139,15 @@ func runEgressWorkers(pctx context.Context, self string, wsURL string, tlsCfg *t
} }
stopKA := armWSKeepalive(ws) // ping/read-deadline: a dead tunnel unblocks the park stopKA := armWSKeepalive(ws) // ping/read-deadline: a dead tunnel unblocks the park
unpark := context.AfterFunc(pctx, func() { _ = ws.Close() }) unpark := context.AfterFunc(pctx, func() { _ = ws.Close() })
handleEgressConn(&wsConn{ws: ws}, dialOut, self) // blocks until a job, serves it, then returns took := handleEgressConn(&wsConn{ws: ws}, dialOut, self) // blocks until a job, serves it, then returns
unpark() unpark()
stopKA() stopKA()
_ = ws.Close() _ = ws.Close()
if !took {
sleepCtx(pctx, egressRefusedWait)
}
} }
}() }()
} }
return wg.Wait
} }
+166
View File
@@ -0,0 +1,166 @@
//go:build !windows
// The egress worker pool (egress_relay.go: runEgressWorkers) against a
// stand-in Host's /bastion-egress: how fast workers park again after a job
// they took and after one they refused. Uses localEgressFixture
// (egress_local_test.go), so it carries the same build tag.
package main
import (
"context"
"io"
"log"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/gorilla/websocket"
)
// egressHost is a Host's /bastion-egress as the pool meets it: every worker
// that parks is counted and handed the job job(n) (n is its place in line),
// and what the bastion answered is read to its end.
type egressHost struct {
parked atomic.Int32
ok atomic.Int32 // answers that began "HTTP/1.1 200"
}
func startEgressHost(t *testing.T, job func(n int32) string) (*egressHost, string, func(context.Context) (net.Conn, error)) {
t.Helper()
h := &egressHost{}
up := websocket.Upgrader{}
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
c, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer c.Close()
if _, _, err := c.ReadMessage(); err != nil { // BEARER
return
}
n := h.parked.Add(1)
if err := c.WriteMessage(websocket.BinaryMessage, append([]byte{'D'}, job(n)...)); err != nil {
return
}
_ = c.SetReadDeadline(time.Now().Add(5 * time.Second))
var answer strings.Builder
for {
_, m, err := c.ReadMessage()
if err != nil || len(m) == 0 || m[0] == 'E' {
break
}
answer.Write(m[1:])
}
if strings.HasPrefix(answer.String(), "HTTP/1.1 200") {
h.ok.Add(1)
}
}))
srv.Config.ErrorLog = log.New(io.Discard, "", 0)
srv.Start()
t.Cleanup(srv.Close)
addr := srv.Listener.Addr().String()
dial := func(ctx context.Context) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "tcp", addr)
}
return h, "ws://" + addr + "/bastion-egress", dial
}
// runEgressPool runs egressWorkerCount workers for the test's life, and at its
// end stops them and waits until each has exited: a worker still vetting a job
// would otherwise read the fixture's stubs while the test's cleanup puts them
// back.
func runEgressPool(t *testing.T, wsURL string, dialHost func(context.Context) (net.Conn, error), dialOut egressDialer) {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
wait := runEgressWorkers(ctx, "home.host.example", wsURL, nil, dialHost, "test-bearer", dialOut, egressWorkerCount)
t.Cleanup(func() {
cancel()
done := make(chan struct{})
go func() { wait(); close(done) }()
select {
case <-done:
case <-time.After(10 * time.Second):
t.Error("the egress workers did not exit 10 s after the pool stopped")
}
})
}
// SECURITY (this machine's resources): a Host that answers every parked egress
// worker with a job this machine refuses (a request for this machine, a line
// that is no request) cannot make it dial, handshake and refuse in a loop as
// fast as the Host answers. Each worker that refused waits before it parks
// again, so the pool costs at most egressWorkerCount dials a second.
//
// MUTATION: in runEgressWorkers, park again at once whatever handleEgressConn
// reports (as the pool did).
// RESULT: red — hundreds of workers park in the 2 s window.
func TestARefusingHostCannotSpinTheEgressWorkers(t *testing.T) {
localEgressFixture(t)
var dialled atomic.Int32
dialOut := func(ctx context.Context, network, addr string) (net.Conn, error) {
dialled.Add(1)
return nil, io.EOF
}
h, wsURL, dialHost := startEgressHost(t, func(n int32) string {
if n%2 == 0 {
return "NOT-A-REQUEST\r\n"
}
return "CONNECT localhost:443 HTTP/1.1\r\nHost: localhost:443\r\n\r\n"
})
runEgressPool(t, wsURL, dialHost, dialOut)
time.Sleep(2 * time.Second)
// egressWorkerCount park at the start, then each worker once a second:
// about 12 in 2 s. A pool that parks again at once makes hundreds.
n := h.parked.Load()
t.Logf("%d workers parked in 2 s", n)
if limit := int32(egressWorkerCount * 4); n > limit {
t.Fatalf("%d egress workers parked in 2 s on a Host that refuses every job (at most %d): the pool re-dials in a loop", n, limit)
}
if n < egressWorkerCount {
t.Fatalf("only %d workers parked: the pool did not run, so the count proves nothing", n)
}
if d := dialled.Load(); d != 0 {
t.Fatalf("a refused job was dialled out %d times", d)
}
}
// The person's own traffic is not slowed: a worker that took a job parks again
// at once, so 40 jobs through 4 workers finish in a fraction of the time one
// wait per job would take (40 / 4 x egressRefusedWait = 10 s).
//
// MUTATION: in runEgressWorkers, wait egressRefusedWait after every job.
// RESULT: red — 40 jobs need about 10 s.
func TestEgressWorkersParkAgainAtOnceAfterAJobTheyTook(t *testing.T) {
localEgressFixture(t)
dialOut := func(ctx context.Context, network, addr string) (net.Conn, error) {
if addr != "198.51.100.9:443" {
t.Errorf("dialled %q, not the address the request named", addr)
}
ours, theirs := net.Pipe()
_ = theirs.Close() // the destination accepts and ends at once
return ours, nil
}
h, wsURL, dialHost := startEgressHost(t, func(int32) string {
return "CONNECT 198.51.100.9:443 HTTP/1.1\r\nHost: 198.51.100.9:443\r\n\r\n"
})
start := time.Now()
runEgressPool(t, wsURL, dialHost, dialOut)
const jobs = 40
deadline := start.Add(10 * time.Second)
for h.ok.Load() < jobs && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
took := time.Since(start)
t.Logf("%d accepted jobs answered in %v", h.ok.Load(), took)
if got := h.ok.Load(); got < jobs {
t.Fatalf("only %d of %d accepted jobs were answered 200 in %v", got, jobs, took)
}
if limit := 3 * egressRefusedWait; took > limit {
t.Fatalf("%d accepted jobs took %v (over %v): a worker that took a job waited before it parked again", jobs, took, limit)
}
}
+50 -11
View File
@@ -252,16 +252,39 @@ func harnessGate(self, dcnDir string, spec harnessSpec) string {
// `BEARER <bearer> lane=session`); a harness worker refuses it so a shell can never hold // `BEARER <bearer> lane=session`); a harness worker refuses it so a shell can never hold
// an agent worker for hours. // an agent worker for hours.
func handleHarnessConn(ws *websocket.Conn, self, dcnDir string) { func handleHarnessConn(ws *websocket.Conn, self, dcnDir string) {
mt, data, err := ws.ReadMessage() // The spec is read up to harnessSpecMaxBytes (lane_bounds.go) and refused
if err != nil || mt != websocket.BinaryMessage || len(data) == 0 { // past it; a frame of any other tag is not read at all.
tag, body, tooBig, err := readLaneJob(ws, func(t byte) int64 {
if t == 'H' {
return harnessSpecMaxBytes
}
return 0
})
if err != nil {
return return
} }
if data[0] == 'S' { if tag == 'S' {
sendHarness(ws, 'E', []byte("this is a harness worker; sessions go to a worker that said lane=session\n")) sendHarness(ws, 'E', []byte("this is a harness worker; sessions go to a worker that said lane=session\n"))
sendHarnessExit(ws, 1) sendHarnessExit(ws, 1)
return return
} }
if data[0] != 'H' { if tag != 'H' {
return
}
if tooBig {
// Gated like a spec that does not parse (below): a pairing the caps
// refuse learns nothing more from a frame that was too large.
if why := capsAllow(self, "harness:run"); why != "" {
b, _ := json.Marshal(map[string]any{"ok": false, "error": why})
sendHarness(ws, 'J', b)
sendHarnessExit(ws, 126)
return
}
b, _ := json.Marshal(map[string]any{"ok": false, "error": fmt.Sprintf(
"the harness spec is more than the %d MiB a job on this lane may be; nothing of it was read as a job",
harnessSpecMaxBytes>>20)})
sendHarness(ws, 'J', b)
sendHarnessExit(ws, 1)
return return
} }
// The gate depends on WHAT is asked: a service-slot job (service.go) is the // The gate depends on WHAT is asked: a service-slot job (service.go) is the
@@ -270,7 +293,7 @@ func handleHarnessConn(ws *websocket.Conn, self, dcnDir string) {
// before its parse error is reported, exactly as it always was, so an // before its parse error is reported, exactly as it always was, so an
// unauthorized pairing learns nothing from a malformed frame. // unauthorized pairing learns nothing from a malformed frame.
var spec harnessSpec var spec harnessSpec
perr := json.Unmarshal(data[1:], &spec) perr := json.Unmarshal(body, &spec)
gateOp := "harness:run" gateOp := "harness:run"
if perr == nil && spec.Mode == "service" { if perr == nil && spec.Mode == "service" {
gateOp = serviceGateOp(spec.ServiceOp) gateOp = serviceGateOp(spec.ServiceOp)
@@ -332,12 +355,10 @@ func handleHarnessConn(ws *websocket.Conn, self, dcnDir string) {
go func() { b, c := runHarness(spec); resCh <- harnessRes{b, c} }() go func() { b, c := runHarness(spec); resCh <- harnessRes{b, c} }()
closed := make(chan struct{}) closed := make(chan struct{})
go func() { go func() {
for { // Nothing the host sends during a run is read as anything: each frame
if _, _, err := ws.ReadMessage(); err != nil { // host close or drop // is dropped as it arrives (lane_bounds.go), never held whole.
_ = drainLane(ws) // returns on a host close or drop
close(closed) close(closed)
return
}
}
}() }()
ticker := time.NewTicker(8 * time.Second) ticker := time.NewTicker(8 * time.Second)
defer ticker.Stop() defer ticker.Stop()
@@ -951,7 +972,10 @@ func driveHarnessAgent(rt, dir string, spec harnessSpec, maxMin int) (agentRes m
return nil, fmt.Errorf("harness /run: %v", err) return nil, fmt.Errorf("harness /run: %v", err)
} }
defer resp.Body.Close() defer resp.Body.Close()
rb, _ := io.ReadAll(resp.Body) rb, err := readHarnessRunReply(resp.Body)
if err != nil {
return nil, err
}
var res map[string]interface{} var res map[string]interface{}
if err := json.Unmarshal(rb, &res); err != nil { if err := json.Unmarshal(rb, &res); err != nil {
return nil, fmt.Errorf("harness /run bad json (HTTP %d): %s", resp.StatusCode, truncate(string(rb), 200)) return nil, fmt.Errorf("harness /run bad json (HTTP %d): %s", resp.StatusCode, truncate(string(rb), 200))
@@ -959,6 +983,21 @@ func driveHarnessAgent(rt, dir string, spec harnessSpec, maxMin int) (agentRes m
return res, nil return res, nil
} }
// readHarnessRunReply reads the agent container's answer to /run. The
// container writes it, and it goes back to the host in one lane message, so it
// is read to the most that message carries (uvicornWSMaxSize) and no further:
// a reply that never ends must not grow in this machine's memory.
func readHarnessRunReply(body io.Reader) ([]byte, error) {
rb, err := io.ReadAll(io.LimitReader(body, uvicornWSMaxSize+1))
if err != nil {
return nil, fmt.Errorf("harness /run: reading the reply: %v", err)
}
if len(rb) > uvicornWSMaxSize {
return nil, fmt.Errorf("harness /run: the reply is more than the %d MiB the lane to the host carries", uvicornWSMaxSize>>20)
}
return rb, nil
}
// harnessAgentUID is the user the harness image runs its agent as, the owner // harnessAgentUID is the user the harness image runs its agent as, the owner
// of its home /home/privatae, where the credential folder is mounted. // of its home /home/privatae, where the credential folder is mounted.
const harnessAgentUID = 10005 const harnessAgentUID = 10005
+4 -4
View File
@@ -1010,15 +1010,15 @@ func TestAnchoredEgressRefusesLoopbackAndPrivateDestinations(t *testing.T) {
if addr, why := egressVettedAddr(ctx, "localhost:80"); why == "" || addr != "" { if addr, why := egressVettedAddr(ctx, "localhost:80"); why == "" || addr != "" {
t.Fatalf("a refused name yields no address: %q %q", addr, why) t.Fatalf("a refused name yields no address: %q %q", addr, why)
} }
// A local box refuses this machine too, and keeps its LAN // A local box refuses this machine too, and the network around it
// (egress_local_test.go has the rest). // (egress_local_test.go and egress_network_test.go have the rest).
t.Setenv("XDG_CONFIG_HOME", t.TempDir()) t.Setenv("XDG_CONFIG_HOME", t.TempDir())
stubOwnAddrs(t, loopbackOnly, localRange()) stubOwnAddrs(t, loopbackOnly, localRange())
if why := egressInternalRefusal(ctx, "127.0.0.1:11434"); why == "" { if why := egressInternalRefusal(ctx, "127.0.0.1:11434"); why == "" {
t.Fatal("a local box must refuse its own ollama on the egress lane") t.Fatal("a local box must refuse its own ollama on the egress lane")
} }
if addr, why := egressVettedAddr(ctx, "192.168.1.1:80"); why != "" || addr != "192.168.1.1:80" { if addr, why := egressVettedAddr(ctx, "192.168.1.1:80"); why == "" || addr != "" {
t.Fatalf("a local box keeps its LAN: %q %q", addr, why) t.Fatalf("a local box refuses its LAN on the egress lane: %q %q", addr, why)
} }
} }
+29 -4
View File
@@ -133,18 +133,39 @@ func inferenceFilterNeeded(self string, lane inferenceLane, scope modelScope) bo
// //
// With nothing to enforce it is the original raw splice, byte for byte. // With nothing to enforce it is the original raw splice, byte for byte.
func proxyInference(in net.Conn, upstream string, lane inferenceLane, self string) { func proxyInference(in net.Conn, upstream string, lane inferenceLane, self string) {
proxyInferenceVia(in, dialUpstream(upstream), lane, self, spliceConnVia)
}
// upstreamDial opens the connection to the service behind a forward.
type upstreamDial func() (net.Conn, error)
// dialUpstream is the forwards' dial: TCP to the address, 5 s.
func dialUpstream(upstream string) upstreamDial {
return func() (net.Conn, error) { return net.DialTimeout("tcp", upstream, 5*time.Second) }
}
// proxyInferenceVia is proxyInference with the dial, and the pass-through for
// a connection with nothing to enforce, in the caller's hands. The serve lane
// (servelane.go) uses it so its connections take the same decision and the
// same filter as the tunnel's, while the dial goes to the address it vetted
// and is watched by its idle bound.
func proxyInferenceVia(in net.Conn, dial upstreamDial, lane inferenceLane, self string, pass func(net.Conn, upstreamDial)) {
if inferenceFilterNeeded(self, lane, pairingModelScope(self)) { if inferenceFilterNeeded(self, lane, pairingModelScope(self)) {
serveInferenceFiltered(in, upstream, lane, self) serveInferenceFilteredVia(in, dial, lane, self)
return return
} }
spliceConn(in, upstream) pass(in, dial)
} }
// spliceConn is what the forwards did before this file existed, unchanged and // spliceConn is what the forwards did before this file existed, unchanged and
// in one place now that three call sites share it. // in one place now that three call sites share it.
func spliceConn(in net.Conn, upstream string) { func spliceConn(in net.Conn, upstream string) {
spliceConnVia(in, dialUpstream(upstream))
}
func spliceConnVia(in net.Conn, dial upstreamDial) {
defer in.Close() defer in.Close()
out, err := net.DialTimeout("tcp", upstream, 5*time.Second) out, err := dial()
if err != nil { if err != nil {
return return
} }
@@ -163,8 +184,12 @@ func spliceConn(in net.Conn, upstream string) {
// grant on every request below, so there is nowhere for a caller — including a // grant on every request below, so there is nowhere for a caller — including a
// test — to hand it one the grant does not say. // test — to hand it one the grant does not say.
func serveInferenceFiltered(in net.Conn, upstream string, lane inferenceLane, self string) { func serveInferenceFiltered(in net.Conn, upstream string, lane inferenceLane, self string) {
serveInferenceFilteredVia(in, dialUpstream(upstream), lane, self)
}
func serveInferenceFilteredVia(in net.Conn, dial upstreamDial, lane inferenceLane, self string) {
defer in.Close() defer in.Close()
out, err := net.DialTimeout("tcp", upstream, 5*time.Second) out, err := dial()
if err != nil { if err != nil {
writeFilterError(in, nil, http.StatusBadGateway, "the service behind this lane is not answering") writeFilterError(in, nil, http.StatusBadGateway, "the service behind this lane is not answering")
return return
+56
View File
@@ -0,0 +1,56 @@
package main
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// install.ps1 has no Windows runner here, so its -Pair shape is read from the
// file: the code is checked before anything is downloaded, reaches `pair -`
// on standard input and never an argument, exit 3 is success, and the
// install with no flag still ends the way it did.
func TestInstallPs1PairShape(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "deploy", "install.ps1"))
if err != nil {
t.Fatal(err)
}
s := string(raw)
for _, want := range []string{
"param([switch]$Networked, [string]$Pair)",
"$Pair | & $exe pair -",
"elseif ($rc -eq 3)",
"Start-ScheduledTask -TaskName 'PrivataeBastion'",
"Restart-Service -Name 'PrivataeBastion'",
`Write-Host "Open a NEW terminal, then: privatae-bastion pair <token>"`,
} {
if !strings.Contains(s, want) {
t.Errorf("install.ps1 lacks %q", want)
}
}
for _, bad := range []string{"pair $Pair", "pair -- $Pair", "-ArgumentList $Pair", "Write-Host $Pair", "$Pair\"", "'$Pair'"} {
if strings.Contains(s, bad) {
t.Errorf("install.ps1 must never put the code in an argument or the output: %q", bad)
}
}
check, fetch := strings.Index(s, "StartsWith('skp_')"), strings.Index(s, `Write-Host "Fetching manifest..."`)
if check < 0 || fetch < 0 || check > fetch {
t.Error("the pairing code must be checked before the manifest is fetched")
}
}
// The -Pair block itself, run by pwsh against a stand-in bastion
// (testdata/install_ps1_pair.ps1). Skipped where pwsh is not installed.
func TestInstallPs1PairBlockRunsUnderPwsh(t *testing.T) {
pwsh, err := exec.LookPath("pwsh")
if err != nil {
t.Skip("pwsh is not installed: install.ps1's -Pair block is checked only by TestInstallPs1PairShape here")
}
out, err := exec.Command(pwsh, "-NoProfile", "-NonInteractive", "-File",
filepath.Join("testdata", "install_ps1_pair.ps1"), filepath.Join("..", "deploy", "install.ps1")).CombinedOutput()
if err != nil || !strings.HasPrefix(strings.TrimSpace(string(out)), "ok:") {
t.Fatalf("%v\n%s", err, out)
}
}
+358
View File
@@ -3,10 +3,13 @@
package main package main
import ( import (
"crypto/sha256"
"encoding/hex"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"regexp" "regexp"
"strings"
"testing" "testing"
) )
@@ -74,3 +77,358 @@ func TestInstallShKeepsAPairedLocalMachineLocal(t *testing.T) {
t.Fatal("a machine with no pairing gets the networked default") t.Fatal("a machine with no pairing gets the networked default")
} }
} }
// installShPrivatePIDs is what a test that reaches install.sh's
// start_bastion runs under. start_bastion stops every copy of this user's
// bastion it can see in /proc, and on a machine that runs a real bastion as
// this user (root on a server) that would be the real one. So the test must
// see only its own processes: nil when it already runs in a PID namespace of
// its own (CI's container), the unshare prefix that makes one when it can,
// and a skip otherwise.
//
// The machine's own PID namespace is the kernel's initial one, whose number
// is fixed (PROC_PID_INIT_INO, 0xEFFFFFFC); any other is a namespace of its
// own. NSpid in /proc/self/status cannot tell: inside a container /proc is
// the container's, so it shows one level there too.
func installShPrivatePIDs(t *testing.T) []string {
t.Helper()
ns, err := os.Readlink("/proc/self/ns/pid")
if err == nil && ns != "pid:[4026531836]" {
return nil
}
if err == nil && os.Geteuid() == 0 {
if u, err := exec.LookPath("unshare"); err == nil {
wrap := []string{u, "--pid", "--fork", "--mount-proc"}
if exec.Command(wrap[0], append(wrap[1:], "true")...).Run() == nil {
return wrap
}
}
}
t.Skip("install.sh's start_bastion stops every copy of this user's bastion; run this test in a PID namespace of its own (unshare --pid --fork --mount-proc, as root)")
return nil
}
// pairShSetup is a stand-in machine for install.sh: a log directory, a
// home, and a PATH directory of stand-ins for systemctl, loginctl and sleep,
// each logging what it was given. unit is how the stand-in systemd reports
// the user unit: "steady" (one process, running) or "flapping" (a new
// process on every look, auto-restarting).
type pairShSetup struct {
dir, logs, bin, path, home string
}
func newPairShSetup(t *testing.T) *pairShSetup {
t.Helper()
d := t.TempDir()
s := &pairShSetup{dir: d, logs: filepath.Join(d, "log"), bin: filepath.Join(d, "bin"), path: filepath.Join(d, "path"), home: filepath.Join(d, "home")}
for _, p := range []string{s.logs, s.bin, s.path, s.home} {
if err := os.MkdirAll(p, 0o700); err != nil {
t.Fatal(err)
}
}
s.write(t, filepath.Join(s.path, "systemctl"), `#!/bin/sh
echo "$*" >> "$LOGS/systemctl"
case "$*" in
*"show -p MainPID --value"*)
if [ "$UNIT" = flapping ]; then n=$(cat "$LOGS/n" 2>/dev/null || echo 100); n=$((n + 1)); echo "$n" > "$LOGS/n"; echo "$n"; else echo 4242; fi ;;
*"show -p ActiveState --value"*) if [ "$UNIT" = flapping ]; then echo activating; else echo active; fi ;;
*"show -p SubState --value"*) if [ "$UNIT" = flapping ]; then echo auto-restart; else echo running; fi ;;
esac
exit 0
`)
s.write(t, filepath.Join(s.path, "loginctl"), "#!/bin/sh\necho \"$*\" >> \"$LOGS/loginctl\"\nexit \"$LINGER_RC\"\n")
s.write(t, filepath.Join(s.path, "sleep"), "#!/bin/sh\nexit 0\n")
return s
}
func (s *pairShSetup) write(t *testing.T, p, body string) {
t.Helper()
if err := os.WriteFile(p, []byte(body), 0o700); err != nil {
t.Fatal(err)
}
}
func (s *pairShSetup) env(extra ...string) []string {
return append([]string{"PATH=" + s.path + ":/usr/bin:/bin", "HOME=" + s.home, "XDG_CONFIG_HOME=" + filepath.Join(s.home, ".config"),
"LOGS=" + s.logs, "TMPDIR=" + s.dir}, extra...)
}
// standInCopy puts a copy of this test binary at dir/privatae-bastion: run
// with installShStandInRun it is a bastion that waits to be stopped, under
// the program name install.sh looks for.
func standInCopy(t *testing.T, dir string) string {
t.Helper()
self, err := os.Executable()
if err != nil {
t.Fatal(err)
}
b, err := os.ReadFile(self)
if err != nil {
t.Fatal(err)
}
p := filepath.Join(dir, "privatae-bastion")
if err := os.WriteFile(p, b, 0o755); err != nil {
t.Fatal(err)
}
return p
}
// pairShHarness runs install.sh's pairing functions against a stand-in
// bastion, systemctl and loginctl. It returns the run's output, the log
// directory and its error. With copyRunning, a copy of the bastion from another
// path is running when the functions start, and log/copy says whether it
// still was when they ended.
func pairShHarness(t *testing.T, root, pairRC, lingerRC string) (string, string, error) {
t.Helper()
return pairShHarnessWith(t, root, pairRC, lingerRC, "steady", false)
}
func pairShHarnessWith(t *testing.T, root, pairRC, lingerRC, unit string, copyRunning bool) (string, string, error) {
t.Helper()
wrap := installShPrivatePIDs(t)
s := newPairShSetup(t)
fake := filepath.Join(s.bin, "privatae-bastion")
s.write(t, fake, `#!/bin/sh
echo "HOME=$HOME $*" >> "$LOGS/argv"
case "$1" in
pair) cat > "$LOGS/stdin"; exit "$PAIR_RC" ;;
status) echo "status: paired" ;;
esac
exit 0
`)
copyPath := ""
if copyRunning {
other := filepath.Join(s.dir, "other-install")
if err := os.MkdirAll(other, 0o700); err != nil {
t.Fatal(err)
}
copyPath = standInCopy(t, other)
}
fns := installShFunc(t, "pair_as") + installShFunc(t, "bastion_copy") + installShFunc(t, "stop_user_copies") +
installShFunc(t, "user_unit_stays_up") + installShFunc(t, "start_bastion") + installShFunc(t, "pair_and_start")
script := fns + `
if [ -n "$COPY" ]; then
"$COPY" ` + installShStandInRun + ` >/dev/null 2>&1 &
copy=$!
i=0
while ! bastion_copy "$copy" && [ "$i" -lt 100 ]; do i=$((i + 1)); /bin/sleep 0.05; done
bastion_copy "$copy" || { echo "the stand-in copy never started" >&2; exit 97; }
fi
rc=0
pair_and_start "$1" "$2" "$3" linux || rc=$?
if [ -n "$COPY" ]; then
if bastion_copy "$copy"; then echo alive > "$LOGS/copy"; kill -9 "$copy"; else echo gone > "$LOGS/copy"; fi
fi
exit "$rc"
`
args := append(append([]string{}, wrap...), "sh", "-c", script, "sh", fake, "skp_INVENTEDCODE", root)
cmd := exec.Command(args[0], args[1:]...)
cmd.Env = s.env("PAIR_RC="+pairRC, "LINGER_RC="+lingerRC, "UNIT="+unit, "COPY="+copyPath)
out, err := cmd.CombinedOutput()
return string(out), s.logs, err
}
func readPairShLog(t *testing.T, dir, name string) string {
t.Helper()
b, _ := os.ReadFile(filepath.Join(dir, name))
return string(b)
}
// The pairing functions hand the code to `pair -` on standard input, never
// as an argument, then write and start a systemd user unit, ask for
// lingering, and end with one plain sentence. Exit 3 (already paired with
// this Host) is success.
func TestInstallShPairsOverStdinAndStartsAUserUnit(t *testing.T) {
out, logs, err := pairShHarness(t, "0", "0", "0")
if err != nil {
t.Fatalf("%v\n%s", err, out)
}
argv := readPairShLog(t, logs, "argv")
if !strings.Contains(argv, " pair -\n") || strings.Contains(argv, "skp_") {
t.Fatalf("the code must reach `pair -` on stdin and never an argument: %q", argv)
}
if got := readPairShLog(t, logs, "stdin"); got != "skp_INVENTEDCODE\n" {
t.Fatalf("pair's standard input: %q", got)
}
home := strings.TrimPrefix(strings.Fields(argv)[0], "HOME=")
unit, err := os.ReadFile(filepath.Join(home, ".config", "systemd", "user", "privatae-bastion.service"))
if err != nil || !regexp.MustCompile(`(?m)^ExecStart=\S+/privatae-bastion run$`).Match(unit) || !strings.Contains(string(unit), "WantedBy=default.target") {
t.Fatalf("the user unit: %v\n%s", err, unit)
}
sc := readPairShLog(t, logs, "systemctl")
for _, want := range []string{"--user daemon-reload", "--user enable privatae-bastion", "--user restart privatae-bastion"} {
if !strings.Contains(sc, want+"\n") {
t.Fatalf("systemctl was not asked to %q: %q", want, sc)
}
}
if !strings.Contains(readPairShLog(t, logs, "loginctl"), "enable-linger ") {
t.Fatal("lingering was not asked for")
}
if !strings.Contains(out, "status: paired") || !strings.HasSuffix(strings.TrimSpace(out),
"This machine is paired. The bastion runs as your systemd user service privatae-bastion, and starts at boot.") {
t.Fatalf("the ending: %s", out)
}
out, _, err = pairShHarness(t, "0", "3", "1")
if err != nil || !strings.Contains(out, "already paired with this Host") || !strings.Contains(out, "sudo loginctl enable-linger") {
t.Fatalf("exit 3 is success, and a refused linger says how to get it: %v\n%s", err, out)
}
out, logs, err = pairShHarness(t, "0", "1", "0")
if err == nil || strings.Contains(readPairShLog(t, logs, "systemctl"), "restart") || !strings.Contains(out, "pairing did not finish") {
t.Fatalf("a failed pair must stop before starting anything: %v\n%s", err, out)
}
out, logs, err = pairShHarness(t, "1", "0", "0")
if err != nil || !strings.Contains(readPairShLog(t, logs, "argv"), "HOME=/root ") ||
readPairShLog(t, logs, "systemctl") != "restart privatae-bastion\n" || strings.Contains(out, "user service") {
t.Fatalf("as root it pairs as the service's state and restarts the system service: %v\n%s\n%s", err, out, readPairShLog(t, logs, "systemctl"))
}
}
// Linux, as yourself: a copy of the bastion already running outside the
// user unit (from a terminal, or another path) is stopped before the unit
// starts, so the unit's copy is the one that serves; and a unit that does not
// stay up is said plainly, with a failure, never reported as running.
func TestInstallShStopsAnOutsideCopyAndChecksTheUnitStaysUp(t *testing.T) {
out, logs, err := pairShHarnessWith(t, "0", "0", "0", "steady", true)
if err != nil {
t.Fatalf("%v\n%s", err, out)
}
if got := strings.TrimSpace(readPairShLog(t, logs, "copy")); got != "gone" {
t.Fatalf("the copy running outside the unit must be stopped, it is %q\n%s", got, out)
}
if !strings.Contains(out, "Stopped 1 copy(ies) of the bastion running outside its service") {
t.Fatalf("the stop is said: %s", out)
}
sc := readPairShLog(t, logs, "systemctl")
if i, j := strings.Index(sc, "--user stop privatae-bastion\n"), strings.Index(sc, "--user restart privatae-bastion\n"); i < 0 || j < i {
t.Fatalf("the unit is stopped before the copies and started after: %q", sc)
}
if !strings.HasSuffix(strings.TrimSpace(out), "This machine is paired. The bastion runs as your systemd user service privatae-bastion, and starts at boot.") {
t.Fatalf("the ending: %s", out)
}
out, _, err = pairShHarnessWith(t, "0", "0", "0", "flapping", false)
if err == nil || !strings.Contains(out, "is not staying up") || !strings.Contains(out, "activating/auto-restart") ||
!strings.Contains(out, "journalctl --user -u privatae-bastion") || strings.Contains(out, "The bastion runs as your systemd user service") {
t.Fatalf("a unit that restarts must be reported as not running, and fail: %v\n%s", err, out)
}
}
// The whole of install.sh, the card's way: the code in PRIVATAE_PAIR_CODE
// and no argument. Every program it starts (stand-ins for uname, id, curl,
// printf, systemctl, loginctl and sleep, and the downloaded bastion) records
// its own arguments, the command line of the shell that started it, and
// whether the code is in its environment. The code must reach `pair -` on
// standard input and appear in none of those. Nothing leaves this machine:
// curl is a stand-in serving a manifest and a stand-in bastion.
func TestInstallShTakesTheCodeOnlyFromTheEnvironment(t *testing.T) {
wrap := installShPrivatePIDs(t)
s := newPairShSetup(t)
const code = "skp_eyJ2IjoxLCJ0IjoiSU5WRU5URUQtRU5WLUNPREUifQ"
record := `echo "$(basename "$0") argv=[$*] parent=[$(tr '\0' ' ' < /proc/$PPID/cmdline)] env=[${PRIVATAE_PAIR_CODE+set}] envcode=[$(env | grep -c "` + code + `")]" >> "$LOGS/procs"
`
bastion := "#!/bin/sh\n" + record + `case "$1" in
pair) cat > "$LOGS/stdin" ;;
status) echo "status: paired" ;;
esac
exit 0
`
s.write(t, filepath.Join(s.dir, "bastion.bin"), bastion)
sum := sha256.Sum256([]byte(bastion))
s.write(t, filepath.Join(s.dir, "version.json"), "{\n \"linux-amd64\": {\n \"url\": \"https://dl.privatae.ai/dl/privatae-bastion-linux-amd64?v=0.0.0\",\n \"sha256\": \""+hex.EncodeToString(sum[:])+"\"\n }\n}\n")
stand := map[string]string{
"uname": `case "$1" in -s) echo Linux ;; -m) echo x86_64 ;; *) echo Linux ;; esac`,
"id": `case "$1" in -u) echo 1000 ;; -un) echo tester ;; *) echo "uid=1000(tester)" ;; esac`,
"curl": `for a; do o=$a; done; case "$2" in */version.json) cp "$STAND/version.json" "$o" ;; *) cp "$STAND/bastion.bin" "$o" ;; esac`,
"printf": `exec /usr/bin/printf "$@"`,
}
for name, body := range stand {
s.write(t, filepath.Join(s.path, name), "#!/bin/sh\n"+record+body+"\n")
}
for _, name := range []string{"systemctl", "loginctl", "sleep"} {
p := filepath.Join(s.path, name)
b, _ := os.ReadFile(p)
s.write(t, p, strings.Replace(string(b), "#!/bin/sh\n", "#!/bin/sh\n"+record, 1))
}
sh, err := filepath.Abs(filepath.Join("..", "deploy", "install.sh"))
if err != nil {
t.Fatal(err)
}
args := append(append([]string{}, wrap...), "sh", sh)
cmd := exec.Command(args[0], args[1:]...)
cmd.Env = s.env("PRIVATAE_PAIR_CODE="+code, "STAND="+s.dir, "LINGER_RC=0", "UNIT=steady")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("%v\n%s", err, out)
}
if got := readPairShLog(t, s.logs, "stdin"); got != code+"\n" {
t.Fatalf("pair's standard input: %q\n%s", got, out)
}
procs := readPairShLog(t, s.logs, "procs")
for _, who := range []string{"uname argv=[-s]", "curl argv=[", "privatae-bastion argv=[pair -]", "privatae-bastion argv=[status]", "systemctl argv=[--user restart privatae-bastion]"} {
if !strings.Contains(procs, who) {
t.Fatalf("no record of %q: the stand-ins did not run as expected\n%s\n%s", who, procs, out)
}
}
if strings.Contains(procs, code) || strings.Contains(procs, "INVENTED-ENV-CODE") {
t.Fatalf("the code is in a program's arguments or its shell's command line:\n%s", procs)
}
if strings.Contains(procs, "env=[set]") || !strings.Contains(procs, "envcode=[0]") || strings.Contains(procs, "envcode=[1]") {
t.Fatalf("a program the installer started inherited the code in its environment:\n%s", procs)
}
if strings.Contains(string(out), code) {
t.Fatalf("the installer printed the code:\n%s", out)
}
if !strings.HasSuffix(strings.TrimSpace(string(out)), "https://privatae.ai/bastion/LICENSE.txt") || !strings.Contains(string(out), "This machine is paired.") {
t.Fatalf("the ending: %s", out)
}
}
// A PRIVATAE_PAIR_CODE that is empty or not a whole pairing code is refused
// before anything is downloaded, and never echoed; so is any argument, a
// code given as one (the removed --pair, or the code alone) included.
//
// The script runs with stand-ins for id (not root) and curl (records that it
// was reached, fetches nothing), so an installer whose refusal broke can
// neither reach the network nor install anything as root on the machine
// running the test: it fails here instead.
func TestInstallShRefusesABadPairCodeBeforeDownloading(t *testing.T) {
const code = "skp_eyJ2IjoxfQ"
stand := t.TempDir()
for name, body := range map[string]string{
"id": `case "$1" in -u) echo 1000 ;; -un) echo tester ;; *) echo "uid=1000(tester)" ;; esac`,
"curl": `echo "REACHED-THE-DOWNLOAD" >&2; exit 1`,
} {
if err := os.WriteFile(filepath.Join(stand, name), []byte("#!/bin/sh\n"+body+"\n"), 0o700); err != nil {
t.Fatal(err)
}
}
for _, c := range []struct {
env []string
args []string
}{
{[]string{"PRIVATAE_PAIR_CODE="}, nil},
{[]string{"PRIVATAE_PAIR_CODE=skp_"}, nil},
{[]string{"PRIVATAE_PAIR_CODE=skb_aG9tZS5zZWNyZXQteA"}, nil},
{[]string{"PRIVATAE_PAIR_CODE=" + code + " trailing"}, nil},
{[]string{"PRIVATAE_PAIR_CODE=" + code + "%3D"}, nil},
{nil, []string{"--pair", code}},
{nil, []string{"--pair=" + code}},
{nil, []string{code}},
{nil, []string{"--networkd"}},
} {
home := t.TempDir()
cmd := exec.Command("sh", append([]string{filepath.Join("..", "deploy", "install.sh")}, c.args...)...)
cmd.Env = append([]string{"PATH=" + stand + ":/usr/bin:/bin", "HOME=" + home, "TMPDIR=" + home}, c.env...)
out, err := cmd.CombinedOutput()
ee, ok := err.(*exec.ExitError)
if !ok || ee.ExitCode() != 2 || strings.Contains(string(out), "Fetching manifest") || strings.Contains(string(out), "REACHED-THE-DOWNLOAD") {
t.Fatalf("%v %v: want exit 2 before the download, got %v\n%s", c.env, c.args, err, out)
}
if strings.Contains(string(out), code) || strings.Contains(string(out), "skb_") || strings.Contains(string(out), "networkd") {
t.Fatalf("%v %v: the refusal echoes what it was given: %s", c.env, c.args, out)
}
}
}
+122
View File
@@ -0,0 +1,122 @@
package main
// What a paired host sends down a worker lane is read at a size this machine
// chose, never at one the host chose: the host picks a frame's length, and a
// frame of any length must not be held whole in this machine's memory. The
// egress lane (egress_relay.go) and the context socket (contextsock.go) follow
// the same rule.
//
// - A job's first frame (the H spec on a harness worker, the S spec on a
// session worker) is read up to its lane's bound and refused past it, by
// name, before any of it is parsed.
// - What a lane carries after that is streamed: terminal input into the pty
// (session.go), the replica's bytes into its socket (mirror.go). Frames that
// nothing reads (a harness run and a root command, which read nothing after
// their spec) are dropped as they arrive.
import (
"errors"
"fmt"
"io"
"net/http"
"github.com/gorilla/websocket"
)
const (
// harnessSpecMaxBytes bounds the H frame. Its bulk is the worktree tar
// (worktree_b64). The largest tree this lane hands back is
// maxReturnedTreeBytes (9 MiB, so 12 MiB in base64) inside one message of
// uvicornWSMaxSize (16 MiB), the most the host takes in (harness.go). A host
// that sends that tree into its next run needs those 12 MiB, plus the
// prompt and the run's credentials. Twice the lane's message size holds
// that with 20 MiB to spare.
harnessSpecMaxBytes = 2 * uvicornWSMaxSize
// sessionSpecMaxBytes bounds the S frame: a signed grant (op ids and the
// person's own words), ids, a public key, and for session:root:exec the
// command, itself held to rootExecCommandMax (64 KiB), which is at most
// 384 KiB even when JSON escapes every byte as \u00XX. 1 MiB is the bound
// Go's own HTTP server puts on a request head.
sessionSpecMaxBytes = http.DefaultMaxHeaderBytes
// sessionInputMaxBytes bounds one frame on an open terminal. The host relays
// what the person's browser sent it, one message each, and its own server
// takes a message up to uvicornWSMaxSize; the tag is one byte more. Input is
// streamed into the pty, so this caps a frame's length, not memory.
sessionInputMaxBytes = uvicornWSMaxSize + 1
// sessionResizeMaxBytes bounds the body of an R frame, {"cols":N,"rows":N}.
sessionResizeMaxBytes = 4 << 10
// heartbeatReplyMaxBytes bounds the host's answer to a heartbeat. It
// carries directives and the lists of grants this box should hold, each
// entry an id and a signature of about 130 bytes, so 4 MiB is tens of
// thousands of grants. Nothing in it is bulk: an update names a version,
// and the binary comes from the release manifest, not from this reply.
heartbeatReplyMaxBytes = 4 << 20
)
// errLaneNotJob is a first frame that is not a binary frame with a tag.
var errLaneNotJob = errors.New("the first frame is not a job")
// readLaneJob reads a worker's first frame: its tag, and its body up to
// maxFor(tag) bytes. A body past its bound is never kept: tooBig reports it,
// and body is nil. A tag the caller does not take should get a bound of 0;
// its body is then not read at all.
func readLaneJob(ws *websocket.Conn, maxFor func(tag byte) int64) (tag byte, body []byte, tooBig bool, err error) {
mt, r, err := ws.NextReader()
if err != nil {
return 0, nil, false, err
}
if mt != websocket.BinaryMessage {
return 0, nil, false, errLaneNotJob
}
var t [1]byte
if _, err := io.ReadFull(r, t[:]); err != nil {
return 0, nil, false, errLaneNotJob // an empty frame
}
max := maxFor(t[0])
if max <= 0 {
return t[0], nil, false, nil
}
body, err = io.ReadAll(io.LimitReader(r, max+1))
if err != nil {
return t[0], nil, false, err
}
if int64(len(body)) > max {
return t[0], nil, true, nil
}
return t[0], body, false, nil
}
// drainLane reads and drops every frame until the socket fails, holding none
// of them, and returns that failure. A lane that reads nothing after its job
// still has to read, to answer pings and to notice the host going away.
func drainLane(ws *websocket.Conn) error {
for {
_, r, err := ws.NextReader()
if err != nil {
return err
}
if _, err := io.Copy(io.Discard, r); err != nil {
return err
}
}
}
// readHeartbeatReply reads the host's answer to a heartbeat up to
// heartbeatReplyMaxBytes. ok is false for a reply past the bound or one that
// failed to arrive: such a beat is treated as unanswered, so none of its
// directives runs and it does not count as the host answering.
func readHeartbeatReply(resp *http.Response) (body []byte, ok bool) {
body, err := io.ReadAll(io.LimitReader(resp.Body, heartbeatReplyMaxBytes+1))
if err != nil {
return nil, false
}
if len(body) > heartbeatReplyMaxBytes {
fmt.Printf("heartbeat: the host's reply is more than %d MiB; ignored\n", heartbeatReplyMaxBytes>>20)
return nil, false
}
return body, true
}
+126
View File
@@ -0,0 +1,126 @@
package main
// The helpers lane_bounds.go and harness.go added for the reads that were
// inline before (the heartbeat reply in run.go, the agent's /run reply in
// driveHarnessAgent, the drains in harness.go and rootexec.go). They have no
// counterpart on main to run against.
import (
"bytes"
"io"
"net/http"
"net/http/httptest"
"runtime"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/gorilla/websocket"
)
// boxLaneHost stands up the HOST side of one worker socket (host runs on the
// upgraded server end) and returns the box side, dialled.
func boxLaneHost(t *testing.T, host func(c *websocket.Conn)) *websocket.Conn {
t.Helper()
up := websocket.Upgrader{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
c, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer c.Close()
host(c)
}))
t.Cleanup(srv.Close)
ws, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(srv.URL, "http"), nil)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = ws.Close() })
return ws
}
// endless is a body that never ends; served counts what was taken from it.
type endless struct{ served *atomic.Int64 }
func (e endless) Read(p []byte) (int, error) {
for i := range p {
p[i] = 'z'
}
e.served.Add(int64(len(p)))
return len(p), nil
}
func TestHeartbeatReplyIsReadOnlyToItsBound(t *testing.T) {
var served atomic.Int64
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
_, _ = io.Copy(w, io.LimitReader(endless{&served}, 64<<20))
}))
defer srv.Close()
resp, err := http.Get(srv.URL)
if err != nil {
t.Fatal(err)
}
body, ok := readHeartbeatReply(resp)
resp.Body.Close()
if ok || body != nil {
t.Fatalf("a 64 MiB heartbeat reply was taken (%d bytes)", len(body))
}
small := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"ok":true,"dcn_grants":{"stores":[]}}`))
}))
defer small.Close()
resp, err = http.Get(small.URL)
if err != nil {
t.Fatal(err)
}
body, ok = readHeartbeatReply(resp)
resp.Body.Close()
if !ok || !strings.Contains(string(body), "dcn_grants") {
t.Fatalf("an ordinary reply was not taken: ok=%v %q", ok, body)
}
}
func TestHarnessRunReplyIsReadOnlyToItsBound(t *testing.T) {
var served atomic.Int64
if _, err := readHarnessRunReply(endless{&served}); err == nil || !strings.Contains(err.Error(), "more than the 16 MiB") {
t.Fatalf("an endless /run reply: %v", err)
}
if n := served.Load(); n > 17<<20 {
t.Fatalf("an endless /run reply was read to %d MiB", n>>20)
}
big := append([]byte(`{"ok":true,"pad":"`), bytes.Repeat([]byte{'a'}, 15<<20)...)
big = append(big, `"}`...)
if rb, err := readHarnessRunReply(bytes.NewReader(big)); err != nil || len(rb) != len(big) {
t.Fatalf("a 15 MiB reply the lane carries: %d bytes, %v", len(rb), err)
}
}
// A drained lane holds no frame whole, and ends when the host goes away.
func TestDrainLaneHoldsNoFrame(t *testing.T) {
const frame = 64 << 20
payload := bytes.Repeat([]byte{'d'}, frame)
start := make(chan struct{})
box := boxLaneHost(t, func(c *websocket.Conn) {
<-start
_ = c.WriteMessage(websocket.BinaryMessage, payload)
})
runtime.GC()
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
close(start)
done := make(chan error, 1)
go func() { done <- drainLane(box) }()
select {
case <-done:
case <-time.After(20 * time.Second):
t.Fatal("drainLane did not return after the host went away")
}
runtime.ReadMemStats(&after)
if grew := after.TotalAlloc - before.TotalAlloc; grew > 8<<20 {
t.Fatalf("draining a %d MiB frame allocated %d MiB", frame>>20, grew>>20)
}
}
+481
View File
@@ -0,0 +1,481 @@
//go:build !windows
package main
// What a paired host sends down the harness, session, root-exec and mirror
// lanes, and what a coder run sends its egress proxy, is read at the size this
// machine chose (lane_bounds.go). Every test here drives only code that main
// already has, so each one also runs against main, where it fails.
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"runtime"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
"github.com/gorilla/websocket"
)
// hostSendsAJob is a host that writes one job frame while it reads the box's
// replies, and reports whether its write completed and what came back.
type hostResult struct {
werr error
frames map[byte][]byte
}
func hostSendsAJob(frame []byte, out chan<- hostResult) func(c *websocket.Conn) {
return hostSendsAJobAs(func(c *websocket.Conn) error {
return c.WriteMessage(websocket.BinaryMessage, frame)
}, out)
}
// writeInPieces writes msg as ONE message carried by many small frames: the
// writer flushes a continuation frame each time its 4 KiB buffer fills
// (gorilla's default write buffer), so a bound that counted a frame instead
// of the message would let all of it through.
func writeInPieces(msg []byte) func(c *websocket.Conn) error {
return func(c *websocket.Conn) error {
w, err := c.NextWriter(websocket.BinaryMessage)
if err != nil {
return err
}
for off := 0; off < len(msg); off += 1 << 10 {
if _, err := w.Write(msg[off:min(off+1<<10, len(msg))]); err != nil {
return err
}
}
return w.Close()
}
}
func hostSendsAJobAs(write func(c *websocket.Conn) error, out chan<- hostResult) func(c *websocket.Conn) {
return func(c *websocket.Conn) {
got := map[byte][]byte{}
readDone := make(chan struct{})
go func() {
defer close(readDone)
_ = c.SetReadDeadline(time.Now().Add(20 * time.Second))
for {
_, d, err := c.ReadMessage()
if err != nil || len(d) == 0 {
return
}
got[d[0]] = append(got[d[0]], d[1:]...)
if d[0] == 'X' {
return
}
}
}()
werr := write(c)
<-readDone
out <- hostResult{werr, got}
}
}
// jsonFrame is tag + a JSON object whose field `pad` carries n bytes of 'a'.
func jsonFrame(tag byte, head string, n int) []byte {
b := make([]byte, 0, len(head)+n+16)
b = append(b, tag)
b = append(b, head...)
b = append(b, `,"pad":"`...)
b = append(b, bytes.Repeat([]byte{'a'}, n)...)
b = append(b, `"}`...)
return b
}
// A harness spec far past the bound is not read to its end: the box answers
// that it is too large, names the bound, and closes, so the host's write of
// the rest fails. On main the box read all of it into memory, then ran it.
func TestHarnessWorkerStopsReadingASpecPastItsBound(t *testing.T) {
const frame = 96 << 20
out := make(chan hostResult, 1)
box := boxLaneHost(t, hostSendsAJob(jsonFrame('H', `{"harness_id":"big","mode":"echo","prompt":"ping"`, frame), out))
handleHarnessConn(box, "", "")
_ = box.Close()
r := <-out
if r.werr == nil {
t.Fatalf("the box read the whole %d MiB spec (it replied %.200q); it must stop at its bound", frame>>20, r.frames['J'])
}
if !strings.Contains(string(r.frames['J']), "more than the 32 MiB") || string(r.frames['X']) != "1" {
t.Fatalf("want a refusal naming the 32 MiB bound and exit 1: J=%.200q X=%q", r.frames['J'], r.frames['X'])
}
}
// The bound counts the whole message, not one frame of it: a spec past the
// bound, sent as one message in thousands of 4 KiB frames, is refused the same
// way and not read to its end. On main the box read all of it.
func TestHarnessWorkerBoundCountsEveryFrameOfTheSpec(t *testing.T) {
const frame = 40 << 20
out := make(chan hostResult, 1)
spec := jsonFrame('H', `{"harness_id":"pieces","mode":"echo","prompt":"ping"`, frame)
box := boxLaneHost(t, hostSendsAJobAs(writeInPieces(spec), out))
handleHarnessConn(box, "", "")
_ = box.Close()
r := <-out
if r.werr == nil {
t.Fatalf("the box read the whole %d MiB spec sent in small frames (it replied %.200q); it must stop at its bound", frame>>20, r.frames['J'])
}
if !strings.Contains(string(r.frames['J']), "more than the 32 MiB") || string(r.frames['X']) != "1" {
t.Fatalf("want a refusal naming the 32 MiB bound and exit 1: J=%.200q X=%q", r.frames['J'], r.frames['X'])
}
}
// The same for a session spec sent in small frames.
func TestSessionWorkerBoundCountsEveryFrameOfTheSpec(t *testing.T) {
env, _, _ := sessionTestEnv(t, []string{sessionOpOpen}, nil)
withSink(t)
const frame = 64 << 20
out := make(chan hostResult, 1)
spec := jsonFrame('S', `{"op":"session:open","session_id":"sess-pieces"`, frame)
box := boxLaneHost(t, hostSendsAJobAs(writeInPieces(spec), out))
handleSessionWorkerConn(box, "", env)
_ = box.Close()
r := <-out
if r.werr == nil {
t.Fatalf("the box read the whole %d MiB session spec sent in small frames (E=%q X=%q); it must stop at its bound",
frame>>20, r.frames['E'], r.frames['X'])
}
if !strings.Contains(string(r.frames['E']), "more than the 1024 KiB") || string(r.frames['X']) != "1" {
t.Fatalf("want a refusal naming the bound and exit 1: E=%q X=%q", r.frames['E'], r.frames['X'])
}
}
// A spec the lane can carry still runs: a tree as large as the most this lane
// hands back (12 MiB in base64), with room on top, past the 16 MiB a host
// takes in.
func TestHarnessWorkerTakesALargeSpecTheLaneCarries(t *testing.T) {
out := make(chan hostResult, 1)
box := boxLaneHost(t, hostSendsAJob(jsonFrame('H', `{"harness_id":"tree","mode":"echo","prompt":"ping","worktree_b64":"dGFy"`, 20<<20), out))
handleHarnessConn(box, "", "")
r := <-out
if r.werr != nil {
t.Fatalf("the host could not send a 20 MiB spec: %v", r.werr)
}
var res map[string]any
if err := json.Unmarshal(r.frames['J'], &res); err != nil || res["ok"] != true || res["echo"] != "ping" {
t.Fatalf("a 20 MiB spec did not run: J=%.300q X=%q", r.frames['J'], r.frames['X'])
}
}
// The same on a session worker: an S frame far past its 1 MiB bound is
// refused by name and not read to its end.
func TestSessionWorkerStopsReadingASpecPastItsBound(t *testing.T) {
env, _, _ := sessionTestEnv(t, []string{sessionOpOpen}, nil)
withSink(t) // a box that wrongly took the spec opens no real shell
const frame = 64 << 20
out := make(chan hostResult, 1)
box := boxLaneHost(t, hostSendsAJob(jsonFrame('S', `{"op":"session:open","session_id":"sess-big"`, frame), out))
handleSessionWorkerConn(box, "", env)
_ = box.Close()
r := <-out
if r.werr == nil {
t.Fatalf("the box read the whole %d MiB session spec (E=%q X=%q); it must stop at its bound",
frame>>20, r.frames['E'], r.frames['X'])
}
if !strings.Contains(string(r.frames['E']), "more than the 1024 KiB") || string(r.frames['X']) != "1" {
t.Fatalf("want a refusal naming the bound and exit 1: E=%q X=%q", r.frames['E'], r.frames['X'])
}
}
// A session spec near its bound (a grant, and room the size of a root command
// escaped six times over) still opens a terminal.
func TestSessionWorkerOpensASpecNearItsBound(t *testing.T) {
env, terms, sig := sessionTestEnv(t, []string{sessionOpOpen}, nil)
withFake(t)
// Here the test is the host: it takes the server end, and the box runs on
// the dialled end.
hostEnd := make(chan *websocket.Conn, 1)
hold := make(chan struct{})
t.Cleanup(func() { close(hold) })
boxEnd := boxLaneHost(t, func(c *websocket.Conn) { hostEnd <- c; <-hold })
b := <-hostEnd
head := fmt.Sprintf(`{"op":"session:open","session_id":"sess-near","grant_terms_b64":%q,"grant_sig_b64":%q`, terms, sig)
done := make(chan struct{})
go func() { defer close(done); handleSessionWorkerConn(boxEnd, "", env) }()
if err := b.WriteMessage(websocket.BinaryMessage, jsonFrame('S', head, 900<<10)); err != nil {
t.Fatal(err)
}
if err := b.WriteMessage(websocket.BinaryMessage, []byte("Ihello\n")); err != nil {
t.Fatal(err)
}
got := readUntil(t, b, 'O', 5*time.Second)
if string(got['O']) != "hello\n" {
t.Fatalf("a 900 KiB spec did not open a terminal: %q", got)
}
_ = b.Close()
<-done
}
// sinkProc is a terminal that counts what is typed into it and prints nothing.
type sinkProc struct {
n atomic.Int64
killed chan struct{}
once sync.Once
}
func (p *sinkProc) Read(b []byte) (int, error) { <-p.killed; return 0, io.EOF }
func (p *sinkProc) Write(b []byte) (int, error) { p.n.Add(int64(len(b))); return len(b), nil }
func (p *sinkProc) Resize(c, r int) error { return nil }
func (p *sinkProc) Wait() int { <-p.killed; return 0 }
func (p *sinkProc) Kill() { p.once.Do(func() { close(p.killed) }) }
func (p *sinkProc) User() string { return "sink" }
func (p *sinkProc) Shell() string { return "/bin/sink" }
func withSink(t *testing.T) *sinkProc {
t.Helper()
sp := &sinkProc{killed: make(chan struct{})}
old := sessionSpawn
sessionSpawn = func(op, shell string, cols, rows int) (sessionProc, error) { return sp, nil }
t.Cleanup(func() { sessionSpawn = old; sp.Kill() })
return sp
}
func waitFor(d time.Duration, cond func() bool) bool {
end := time.Now().Add(d)
for time.Now().Before(end) {
if cond() {
return true
}
time.Sleep(5 * time.Millisecond)
}
return cond()
}
// Terminal input is streamed into the pty: a 16 MiB paste, the most a host
// relays in one frame, all arrives, and the box never holds it whole.
func TestSessionInputIsStreamedIntoTheTerminal(t *testing.T) {
env, terms, sig := sessionTestEnv(t, []string{sessionOpOpen}, nil)
sp := withSink(t)
ws := sessionDial(t, env)
sendSpec(t, ws, sessionSpec{Op: sessionOpOpen, GrantB64: terms, SigB64: sig, SessionID: "sess-paste"})
const paste = 16 << 20
frame := append([]byte{'I'}, bytes.Repeat([]byte{'x'}, paste)...)
runtime.GC()
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
if err := ws.WriteMessage(websocket.BinaryMessage, frame); err != nil {
t.Fatal(err)
}
if !waitFor(20*time.Second, func() bool { return sp.n.Load() >= paste }) {
t.Fatalf("the terminal got %d of %d bytes", sp.n.Load(), paste)
}
runtime.ReadMemStats(&after)
if grew := after.TotalAlloc - before.TotalAlloc; grew > 8<<20 {
t.Fatalf("typing a %d MiB paste allocated %d MiB: the frame was held whole", paste>>20, grew>>20)
}
}
// One frame of input past the most a host relays ends the session, and the
// terminal never receives all of it.
func TestSessionInputFramePastItsBoundEndsTheSession(t *testing.T) {
env, terms, sig := sessionTestEnv(t, []string{sessionOpOpen}, nil)
sp := withSink(t)
ws := sessionDial(t, env)
sendSpec(t, ws, sessionSpec{Op: sessionOpOpen, GrantB64: terms, SigB64: sig, SessionID: "sess-huge"})
const frame = 17 << 20
go func() {
_ = ws.WriteMessage(websocket.BinaryMessage, append([]byte{'I'}, bytes.Repeat([]byte{'x'}, frame)...))
}()
ended := func() bool {
select {
case <-sp.killed:
return true
default:
return false
}
}
if !waitFor(20*time.Second, ended) {
t.Fatalf("a %d MiB input frame did not end the session; the terminal got %d bytes", frame>>20, sp.n.Load())
}
if n := sp.n.Load(); n >= frame {
t.Fatalf("the terminal received all %d bytes of a frame past the bound", n)
}
}
// The mirror forwarder streams a host frame into the replica's socket: reading
// 4 KiB of a 64 MiB frame does not need the frame held whole.
func TestMirrorForwarderStreamsAHostFrame(t *testing.T) {
const frame = 64 << 20
payload := bytes.Repeat([]byte{'m'}, frame)
start := make(chan struct{})
up := websocket.Upgrader{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
c, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer c.Close()
_, _, _ = c.ReadMessage() // the BEARER hello
<-start
_ = c.WriteMessage(websocket.BinaryMessage, payload)
time.Sleep(3 * time.Second)
}))
defer srv.Close()
addr := srv.Listener.Addr().String()
dialer := &websocket.Dialer{NetDialContext: func(ctx context.Context, n, a string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "tcp", addr)
}}
boxEnd, replica := net.Pipe()
defer replica.Close()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go pipeMirrorConn(ctx, boxEnd, dialer, "ws://"+addr+"/", "b")
buf := make([]byte, 4096)
runtime.GC()
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
close(start)
_ = replica.SetReadDeadline(time.Now().Add(20 * time.Second))
if _, err := io.ReadFull(replica, buf); err != nil || buf[0] != 'm' {
t.Fatalf("the replica read: %v", err)
}
runtime.ReadMemStats(&after)
if grew := after.TotalAlloc - before.TotalAlloc; grew > 8<<20 {
t.Fatalf("passing 4 KiB of a %d MiB host frame to the replica allocated %d MiB: the frame was held whole",
frame>>20, grew>>20)
}
}
// The forwarder carries at most eight connections at once. It listens on a
// fixed port (5433), so the test needs that port free: run it in its own
// network namespace on a machine where something already holds it.
func TestMirrorForwarderCapsConnectionsInFlight(t *testing.T) {
probe, err := net.Listen("tcp", "127.0.0.1:5433")
if err != nil {
t.Skipf("127.0.0.1:5433 is in use here (%v); run this test in its own network namespace", err)
}
_ = probe.Close()
var live, most atomic.Int64
release := make(chan struct{})
up := websocket.Upgrader{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
c, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer c.Close()
n := live.Add(1)
for {
m := most.Load()
if n <= m || most.CompareAndSwap(m, n) {
break
}
}
<-release
live.Add(-1)
}))
defer srv.Close()
defer close(release)
addr := srv.Listener.Addr().String()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go serveMirrorForward(ctx, "127.0.0.1", "127.0.0.0/8", "ws://"+addr+"/", nil,
func(ctx context.Context) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, "tcp", addr) }, "b")
var conns []net.Conn
defer func() {
for _, c := range conns {
_ = c.Close()
}
}()
if !waitFor(5*time.Second, func() bool {
c, err := net.Dial("tcp", "127.0.0.1:5433")
if err == nil {
conns = append(conns, c)
}
return err == nil
}) {
t.Fatal("the forwarder never listened")
}
for i := 1; i < 12; i++ {
c, err := net.Dial("tcp", "127.0.0.1:5433")
if err != nil {
t.Fatal(err)
}
conns = append(conns, c)
}
waitFor(3*time.Second, func() bool { return most.Load() >= 12 })
if m := most.Load(); m != 8 {
t.Fatalf("12 local connections made %d WebSockets to the host at once; want 8", m)
}
}
// A coder run's egress proxy reads a request head under a bound: a request
// line that never ends is not read to its end.
func TestCoderEgressHeadIsBounded(t *testing.T) {
boxEnd, runEnd := net.Pipe()
var refused int64
done := make(chan struct{})
go func() { defer close(done); handleCoderEgressConn(boxEnd, nil, &refused) }()
chunk := bytes.Repeat([]byte{'a'}, 64<<10)
_, _ = runEnd.Write([]byte("GET http://example.invalid/"))
sent := 0
var werr error
for sent < 8<<20 {
_ = runEnd.SetWriteDeadline(time.Now().Add(5 * time.Second))
var n int
n, werr = runEnd.Write(chunk)
sent += n
if werr != nil {
break
}
}
_ = runEnd.Close()
<-done
if werr == nil {
t.Fatalf("the proxy read %d MiB of a request line with no end; it must stop at a bound", sent>>20)
}
}
// A coder run's proxy carries at most 1024 connections at once; the next is
// answered 503 at once, never left waiting.
func TestCoderEgressCapsConnectionsInFlight(t *testing.T) {
dir := t.TempDir()
run := "lane-bounds-test"
if err := coderListenEgress(run, dir, os.Getuid()); err != nil {
t.Fatal(err)
}
defer coderCloseEgress(run)
sock := filepath.Join(dir, coderEgressSock)
var conns []net.Conn
defer func() {
for _, c := range conns {
_ = c.Close()
}
}()
for i := 0; i < 1024; i++ {
c, err := net.Dial("unix", sock)
if err != nil {
if errors.Is(err, syscall.EMFILE) || errors.Is(err, syscall.ENFILE) {
t.Skipf("this process may not open %d sockets: %v", 2*1024, err)
}
t.Fatal(err)
}
conns = append(conns, c)
}
// Every one of them is idle in its head, so the proxy holds them all.
time.Sleep(200 * time.Millisecond)
extra, err := net.Dial("unix", sock)
if err != nil {
t.Fatal(err)
}
conns = append(conns, extra)
_ = extra.SetReadDeadline(time.Now().Add(3 * time.Second))
line, _ := io.ReadAll(io.LimitReader(extra, 256))
if !strings.HasPrefix(string(line), "HTTP/1.1 503") {
t.Fatalf("connection 1025 of one run got %q; want a 503 at once", line)
}
}
+197
View File
@@ -0,0 +1,197 @@
package main
// What the host sends down a lane is read at the size the bastion asks for,
// never at the size the host chooses: one frame of any length must not be
// held whole in this machine's memory, and an egress request head that never
// ends must not be either.
import (
"bytes"
"context"
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"runtime"
"strconv"
"strings"
"testing"
"time"
"github.com/gorilla/websocket"
)
func TestEgressLaneDoesNotHoldAWholeHostFrame(t *testing.T) {
const frame = 64 << 20
payload := make([]byte, frame+1)
payload[0] = 'D'
for i := 1; i < len(payload); i++ {
payload[i] = 'a'
}
start := make(chan struct{})
up := websocket.Upgrader{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ws, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer ws.Close()
<-start
_ = ws.WriteMessage(websocket.BinaryMessage, payload)
time.Sleep(2 * time.Second)
}))
defer srv.Close()
ws, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(srv.URL, "http"), nil)
if err != nil {
t.Fatal(err)
}
defer ws.Close()
c := &wsConn{ws: ws}
buf := make([]byte, 4096)
runtime.GC()
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
close(start)
n, err := c.Read(buf)
runtime.ReadMemStats(&after)
if err != nil || n == 0 || buf[0] != 'a' {
t.Fatalf("read: n=%d err=%v", n, err)
}
if grew := after.TotalAlloc - before.TotalAlloc; grew > 8<<20 {
t.Fatalf("reading 4 KiB of a %d MiB host frame allocated %d MiB: the frame was held whole", frame>>20, grew>>20)
}
}
func TestEgressRequestHeadIsBounded(t *testing.T) {
bastionEnd, hostEnd := net.Pipe()
done := make(chan struct{})
go func() {
defer close(done)
handleEgressConn(bastionEnd, func(ctx context.Context, network, addr string) (net.Conn, error) {
return nil, errors.New("no dial in this test")
}, "test")
}()
chunk := []byte(strings.Repeat("a", 64<<10))
_, _ = hostEnd.Write([]byte("GET http://example.invalid/"))
sent := 0
var werr error
for sent < 8<<20 {
_ = hostEnd.SetWriteDeadline(time.Now().Add(5 * time.Second))
var n int
n, werr = hostEnd.Write(chunk)
sent += n
if werr != nil {
break
}
}
_ = hostEnd.Close()
<-done
if werr == nil {
t.Fatalf("the bastion read %d MiB of a request line with no end; it must stop at a bound", sent>>20)
}
}
// Streaming a frame must not change what the reader sees: every byte of every
// 'D' frame in order, with empty frames, text frames and unknown tags skipped,
// a frame boundary never visible, and 'E' as the end, read through a buffer
// far smaller than the frames.
func TestEgressLaneFramesReadInOrderThroughASmallBuffer(t *testing.T) {
var want []byte
frames := [][]byte{{'D'}, {}, {'X', 'z', 'z'}}
for i, size := range []int{1, 7, 4096, 70000, 3} {
f := []byte{'D'}
for j := 0; j < size; j++ {
f = append(f, byte('a'+(i+j)%26))
}
want = append(want, f[1:]...)
frames = append(frames, f, []byte{'D'})
}
up := websocket.Upgrader{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ws, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer ws.Close()
_ = ws.WriteMessage(websocket.TextMessage, []byte("Dnot data"))
for _, f := range frames {
_ = ws.WriteMessage(websocket.BinaryMessage, f)
}
_ = ws.WriteMessage(websocket.BinaryMessage, []byte{'E'})
time.Sleep(time.Second)
}))
defer srv.Close()
ws, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(srv.URL, "http"), nil)
if err != nil {
t.Fatal(err)
}
defer ws.Close()
c := &wsConn{ws: ws}
var got []byte
buf := make([]byte, 13)
for {
n, err := c.Read(buf)
if n == 0 && err == nil {
t.Fatal("Read returned 0, nil")
}
got = append(got, buf[:n]...)
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("read: %v", err)
}
}
if !bytes.Equal(got, want) {
t.Fatalf("read %d bytes, want %d; equal prefix %d", len(got), len(want), commonPrefix(got, want))
}
}
func commonPrefix(a, b []byte) int {
i := 0
for i < len(a) && i < len(b) && a[i] == b[i] {
i++
}
return i
}
// A large upload through the lane still arrives whole: the head bound applies
// to the head alone, and the bytes the reader buffered past it go upstream
// first, in order.
func TestEgressLargeUploadPassesTheHeadBound(t *testing.T) {
bastionEnd, hostEnd := net.Pipe()
upBastion, upFar := net.Pipe()
done := make(chan struct{})
go func() {
defer close(done)
handleEgressConn(bastionEnd, func(ctx context.Context, network, addr string) (net.Conn, error) {
return upBastion, nil
}, "")
}()
body := make([]byte, 3*egressMaxHeadBytes)
for i := range body {
body[i] = byte(i % 251)
}
head := "POST http://198.51.100.20/upload HTTP/1.1\r\nHost: 198.51.100.20\r\nContent-Length: " +
strconv.Itoa(len(body)) + "\r\n\r\n"
// Head and body in ONE write, so the reader holds body bytes when the head ends.
go func() {
_, _ = hostEnd.Write(append([]byte(head), body...))
}()
// The lane rebuilds a plain-HTTP head (egress_http.go): this request comes
// out the same, with `Connection: close` added as its last header.
want := head[:len(head)-len("\r\n")] + "Connection: close\r\n\r\n"
got := make([]byte, len(want)+len(body))
_ = upFar.SetReadDeadline(time.Now().Add(20 * time.Second))
if _, err := io.ReadFull(upFar, got); err != nil {
t.Fatalf("upstream got %v", err)
}
if string(got[:len(want)]) != want || !bytes.Equal(got[len(want):], body) {
t.Fatalf("upstream did not receive the head (%q) and the %d-byte body intact",
got[:len(want)], len(body))
}
_ = hostEnd.Close()
_ = upFar.Close()
<-done
}
+3 -2
View File
@@ -493,7 +493,7 @@ func startLaneHostOn(t *testing.T, raw net.Listener, cert tls.Certificate, clien
} }
} }
} }
for _, p := range []string{"/bastion-harness", "/bastion-mgmt", "/bastion-egress"} { for _, p := range []string{"/bastion-harness", "/bastion-mgmt", "/bastion-egress", "/bastion-serve"} {
mux.HandleFunc(p, ws) mux.HandleFunc(p, ws)
} }
srv := &http.Server{Handler: mux, ErrorLog: log.New(io.Discard, "", 0)} srv := &http.Server{Handler: mux, ErrorLog: log.New(io.Discard, "", 0)}
@@ -622,7 +622,8 @@ func TestTheUnixLaneCarriesTheBearerAndTheSameProtocol(t *testing.T) {
h.mu.Unlock() h.mu.Unlock()
if maxPark != harnessWorkerCount || count["/bastion-harness"] < harnessWorkerCount || if maxPark != harnessWorkerCount || count["/bastion-harness"] < harnessWorkerCount ||
count["/bastion-harness session"] != sessionWorkerCount || count["/bastion-harness session"] != sessionWorkerCount ||
count["/bastion-mgmt"] != mgmtWorkerCount || count["/bastion-egress"] != egressWorkerCount { count["/bastion-mgmt"] != mgmtWorkerCount || count["/bastion-egress"] != egressWorkerCount ||
count["/bastion-serve"] != serveWorkerCount {
t.Fatalf("the worker pools over the socket: %v, parked at once %d", count, maxPark) t.Fatalf("the worker pools over the socket: %v, parked at once %d", count, maxPark)
} }
select { select {
+2 -1
View File
@@ -417,7 +417,8 @@ func TestTheTCPLaneCarriesTheCertTheBearerAndTheSameProtocol(t *testing.T) {
h.mu.Unlock() h.mu.Unlock()
if maxPark != harnessWorkerCount || count["/bastion-harness"] < harnessWorkerCount || if maxPark != harnessWorkerCount || count["/bastion-harness"] < harnessWorkerCount ||
count["/bastion-harness session"] != sessionWorkerCount || count["/bastion-harness session"] != sessionWorkerCount ||
count["/bastion-mgmt"] != mgmtWorkerCount || count["/bastion-egress"] != egressWorkerCount { count["/bastion-mgmt"] != mgmtWorkerCount || count["/bastion-egress"] != egressWorkerCount ||
count["/bastion-serve"] != serveWorkerCount {
t.Fatalf("the worker pools over the port: %v, parked at once %d", count, maxPark) t.Fatalf("the worker pools over the port: %v, parked at once %d", count, maxPark)
} }
for _, n := range sni { for _, n := range sni {
+19 -1
View File
@@ -37,6 +37,13 @@ Usage:
that CA alone (name host.cseries.internal), the lanes go to that CA alone (name host.cseries.internal), the lanes go to
the lane port the Host names, and the CA is pinned for life. the lane port the Host names, and the CA is pinned for life.
No coder runs over this pairing No coder runs over this pairing
privatae-bastion pair <pairing code skp_…>
the same pairing from the one code a Cloud Host's card gives:
it carries the enroll address and the CA's fingerprint, and
the CA is taken from the enroll port only under that
fingerprint. On a local machine that already serves a Host,
it asks here whether to add this one as well. Exit 3: already
paired with this Host, nothing changed
privatae-bastion run bring up the tunnel + lanes for EVERY paired host privatae-bastion run bring up the tunnel + lanes for EVERY paired host
(a loopback control port per host for the verbs below: (a loopback control port per host for the verbs below:
127.0.0.1:7878, then 7879, …) 127.0.0.1:7878, then 7879, …)
@@ -188,7 +195,9 @@ func takeHostFlag(args []string) (string, []string) {
} }
const pairUsage = "usage: privatae-bastion pair [--replace] [--redeem-url unix:///<socket> | --redeem-url https://<host>:<port>/<path> --host-ca <pem file>] <token> | -\n" + const pairUsage = "usage: privatae-bastion pair [--replace] [--redeem-url unix:///<socket> | --redeem-url https://<host>:<port>/<path> --host-ca <pem file>] <token> | -\n" +
" - reads the token from standard input, so it stays out of the process list and shell history" " privatae-bastion pair <pairing code skp_…> | -\n" +
" - reads the token or pairing code from standard input, so it stays out of the process list and shell history\n" +
" exit status: 0 paired, 3 already paired with this Host (nothing changed), anything else a failure"
// parsePairArgs reads `pair`'s flags in any order: --replace, --redeem-url // parsePairArgs reads `pair`'s flags in any order: --replace, --redeem-url
// for a Host with no platform (pair_unix.go, pair_tcp.go) and --host-ca for // for a Host with no platform (pair_unix.go, pair_tcp.go) and --host-ca for
@@ -258,6 +267,15 @@ func main() {
os.Exit(2) os.Exit(2)
} }
err = cmdPair(token, redeemURL, hostCA, replace) err = cmdPair(token, redeemURL, hostCA, replace)
// Already paired with this Host, nothing changed: its own exit status,
// so the installers (install.ps1 -Pair, install.sh with
// PRIVATAE_PAIR_CODE) report "already connected".
var already *alreadyPairedError
if errors.As(err, &already) {
auditFlush(auditFlushOnExit)
fmt.Fprintln(os.Stderr, already)
os.Exit(pairExitAlreadyPaired)
}
case "run": case "run":
err = cmdRun() err = cmdRun()
case "observe:baseline": case "observe:baseline":
+12
View File
@@ -36,8 +36,13 @@ import (
"os" "os"
"strings" "strings"
"testing" "testing"
"time"
) )
// installShStandInRun is the argument that makes this test binary stand in
// for a running bastion (TestMain).
const installShStandInRun = "install-sh-stand-in-run"
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
// The op sandbox re-execs its own binary (ops_exec_linux.go); under // The op sandbox re-execs its own binary (ops_exec_linux.go); under
// `go test` that binary is this one, so the child must be answered here // `go test` that binary is this one, so the child must be answered here
@@ -62,6 +67,13 @@ func TestMain(m *testing.M) {
if len(os.Args) >= 3 && os.Args[1] == rootExecConnectSubcommand { if len(os.Args) >= 3 && os.Args[1] == rootExecConnectSubcommand {
os.Exit(rootExecConnectMain(os.Args[2:])) os.Exit(rootExecConnectMain(os.Args[2:]))
} }
// install_sh_test.go runs a copy of this binary named privatae-bastion as
// a bastion left running outside its service: it only waits to be
// stopped, and never starts the suite.
if len(os.Args) >= 2 && os.Args[1] == installShStandInRun {
time.Sleep(2 * time.Minute)
os.Exit(0)
}
// Likewise when git runs this binary as the bastion's credential helper // Likewise when git runs this binary as the bastion's credential helper
// (claims_git_test.go): answer as the helper, with the config dir the // (claims_git_test.go): answer as the helper, with the config dir the
// test named, and never start the suite. // test named, and never start the suite.
+26 -5
View File
@@ -491,6 +491,12 @@ func ensureMirrorNet(rt, name string) (string, string, error) {
// container and a BASE_BACKUP of the user's memory. Nothing here is defence in // container and a BASE_BACKUP of the user's memory. Nothing here is defence in
// depth; this is the depth. // depth; this is the depth.
// //
// mirrorForwardMaxConns bounds the connections the forwarder carries at once.
// The replica holds at most two (`pg_basebackup -X stream` while it seeds; one
// walreceiver after), so four times that leaves room for a diagnostic session
// and a reconnect that overlaps the connection it replaces.
const mirrorForwardMaxConns = 8
// One WebSocket per accepted connection, because `pg_basebackup -X stream` // One WebSocket per accepted connection, because `pg_basebackup -X stream`
// holds two replication connections at once and multiplexing them would mean // holds two replication connections at once and multiplexing them would mean
// inventing a framing layer for no gain. // inventing a framing layer for no gain.
@@ -507,6 +513,11 @@ func serveMirrorForward(ctx context.Context, gw, subnet, wsURL string,
allowed = n allowed = n
} }
} }
// Connections carried at once, across both listeners. Loopback is let in
// for diagnostics, so any local user can connect, and each connection is a
// WebSocket to the host and two pumps here; past the cap one is closed at
// once and its client tries again.
slots := make(chan struct{}, mirrorForwardMaxConns)
for _, addr := range []string{ for _, addr := range []string{
net.JoinHostPort(gw, strconv.Itoa(mirrorForwardPort)), net.JoinHostPort(gw, strconv.Itoa(mirrorForwardPort)),
net.JoinHostPort("127.0.0.1", strconv.Itoa(mirrorForwardPort)), net.JoinHostPort("127.0.0.1", strconv.Itoa(mirrorForwardPort)),
@@ -527,7 +538,15 @@ func serveMirrorForward(ctx context.Context, gw, subnet, wsURL string,
_ = c.Close() _ = c.Close()
continue continue
} }
go pipeMirrorConn(ctx, c, dialer, wsURL, bearer) select {
case slots <- struct{}{}:
go func() {
defer func() { <-slots }()
pipeMirrorConn(ctx, c, dialer, wsURL, bearer)
}()
default:
_ = c.Close()
}
} }
}(ln) }(ln)
} }
@@ -578,15 +597,17 @@ func pipeMirrorConn(ctx context.Context, local net.Conn, dialer *websocket.Diale
done <- struct{}{} done <- struct{}{}
}() }()
go func() { go func() {
// Each host frame is streamed into the replica's socket as it arrives,
// never held whole: the host picks a frame's length (lane_bounds.go).
for { for {
mt, data, err := ws.ReadMessage() mt, r, err := ws.NextReader()
if err != nil { if err != nil {
break break
} }
if mt != websocket.BinaryMessage || len(data) == 0 { if mt != websocket.BinaryMessage {
continue continue // NextReader drops what is left of it
} }
if _, werr := local.Write(data); werr != nil { if _, err := io.Copy(local, r); err != nil {
break break
} }
} }
+5 -3
View File
@@ -88,11 +88,13 @@ func setBoxMode(mode string) error {
} }
} }
if mode == modeLocal { if mode == modeLocal {
// Going back to local must not strand hosts: a local box serves ONE host, // Going back to local must not strand hosts: a local box serves a second
// so with several pairings the box would be lying about its own rule. // host only on a yes typed at this machine (pair_terminal.go), and these
// were paired under the networked rules, so with several pairings the box
// would serve hosts nobody here said yes to.
cfgs, err := loadPairings() cfgs, err := loadPairings()
if err == nil && len(cfgs) > 1 { if err == nil && len(cfgs) > 1 {
return fmt.Errorf("cannot switch to local mode: %d hosts are paired and a local bastion serves one — remove the others first", len(cfgs)) return fmt.Errorf("cannot switch to local mode: %d hosts are paired, and a local bastion adds a second host only on a yes typed at this machine — remove the others first, then pair them again here", len(cfgs))
} }
if currentAnchor() != nil { if currentAnchor() != nil {
return errors.New("cannot switch to local mode: this box is anchored (anchor.json) — anchoring is a networked-mode concept") return errors.New("cannot switch to local mode: this box is anchored (anchor.json) — anchoring is a networked-mode concept")
+29 -15
View File
@@ -19,7 +19,9 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
"slices"
"strings" "strings"
"sync"
"testing" "testing"
"time" "time"
) )
@@ -186,17 +188,29 @@ func TestFilterVerdictPolicy(t *testing.T) {
// fakeOllama answers the handful of routes the filter cares about and records // fakeOllama answers the handful of routes the filter cares about and records
// what actually reached it — the point of most of these assertions is that a // what actually reached it — the point of most of these assertions is that a
// refused request never arrives at all. // refused request never arrives at all. Its handler runs on the server's
// goroutines, beside the test's (and a pairing's beat probes it on its own),
// so the record is read through requests(), under the lock.
type fakeOllama struct { type fakeOllama struct {
srv *httptest.Server srv *httptest.Server
mu sync.Mutex
seen []string seen []string
} }
// requests is a copy of what reached the fake so far.
func (f *fakeOllama) requests() []string {
f.mu.Lock()
defer f.mu.Unlock()
return slices.Clone(f.seen)
}
func newFakeOllama(t *testing.T) *fakeOllama { func newFakeOllama(t *testing.T) *fakeOllama {
t.Helper() t.Helper()
f := &fakeOllama{} f := &fakeOllama{}
f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
f.seen = append(f.seen, r.Method+" "+r.URL.Path) f.seen = append(f.seen, r.Method+" "+r.URL.Path)
f.mu.Unlock()
switch r.URL.Path { switch r.URL.Path {
case "/api/tags": case "/api/tags":
_, _ = w.Write([]byte(`{"models":[{"name":"qwen4-embed:latest","size":42},` + _, _ = w.Write([]byte(`{"models":[{"name":"qwen4-embed:latest","size":42},` +
@@ -264,8 +278,8 @@ func TestFilteredForwardRefusesAnotherModelAndNeverForwardsIt(t *testing.T) {
if b := bodyOf(t, resp); !strings.Contains(b, "llama3:70b") || !strings.Contains(b, "qwen4-embed") { if b := bodyOf(t, resp); !strings.Contains(b, "llama3:70b") || !strings.Contains(b, "qwen4-embed") {
t.Fatalf("the refusal must name what was asked for and what is shared: %s", b) t.Fatalf("the refusal must name what was asked for and what is shared: %s", b)
} }
if len(up.seen) != 0 { if len(up.requests()) != 0 {
t.Fatalf("a refused request reached the upstream: %v", up.seen) t.Fatalf("a refused request reached the upstream: %v", up.requests())
} }
resp = roundTrip(t, addr, laneOllama, tMember, "POST", "/api/embed", resp = roundTrip(t, addr, laneOllama, tMember, "POST", "/api/embed",
@@ -273,8 +287,8 @@ func TestFilteredForwardRefusesAnotherModelAndNeverForwardsIt(t *testing.T) {
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusOK {
t.Fatalf("the shared model must work: %d", resp.StatusCode) t.Fatalf("the shared model must work: %d", resp.StatusCode)
} }
if len(up.seen) != 1 || up.seen[0] != "POST /api/embed" { if len(up.requests()) != 1 || up.requests()[0] != "POST /api/embed" {
t.Fatalf("the allowed request must reach the upstream once: %v", up.seen) t.Fatalf("the allowed request must reach the upstream once: %v", up.requests())
} }
} }
@@ -295,8 +309,8 @@ func TestFilteredForwardRefusesTheStoreMutatingHalf(t *testing.T) {
t.Fatalf("%s: status %d, want 403", p, resp.StatusCode) t.Fatalf("%s: status %d, want 403", p, resp.StatusCode)
} }
} }
if len(up.seen) != 0 { if len(up.requests()) != 0 {
t.Fatalf("a store-mutating request reached ollama: %v", up.seen) t.Fatalf("a store-mutating request reached ollama: %v", up.requests())
} }
// ...and using a model is untouched by that refusal. // ...and using a model is untouched by that refusal.
if resp := roundTrip(t, addr, laneOllama, tMember, "POST", "/api/embed", if resp := roundTrip(t, addr, laneOllama, tMember, "POST", "/api/embed",
@@ -344,8 +358,8 @@ func TestAnchoredBoxRefusesModelChangesThroughTheForwardForTheAnchor(t *testing.
t.Fatalf("%s for the anchor: status %d, want 403", p, resp.StatusCode) t.Fatalf("%s for the anchor: status %d, want 403", p, resp.StatusCode)
} }
} }
if len(up.seen) != 0 { if len(up.requests()) != 0 {
t.Fatalf("a store-mutating request from the anchor reached ollama: %v", up.seen) t.Fatalf("a store-mutating request from the anchor reached ollama: %v", up.requests())
} }
if resp := roundTrip(t, addr, laneOllama, tAnchor, "POST", "/api/embed", `{"model":"anything"}`); resp.StatusCode != http.StatusOK { if resp := roundTrip(t, addr, laneOllama, tAnchor, "POST", "/api/embed", `{"model":"anything"}`); resp.StatusCode != http.StatusOK {
t.Fatalf("the anchor still USES models: %d", resp.StatusCode) t.Fatalf("the anchor still USES models: %d", resp.StatusCode)
@@ -581,8 +595,8 @@ func TestOversizedBodyIsRefusedNotForwarded(t *testing.T) {
if resp.StatusCode != http.StatusForbidden { if resp.StatusCode != http.StatusForbidden {
t.Fatalf("a request naming no model must be refused: %d", resp.StatusCode) t.Fatalf("a request naming no model must be refused: %d", resp.StatusCode)
} }
if len(up.seen) != 0 { if len(up.requests()) != 0 {
t.Fatalf("it reached the upstream anyway: %v", up.seen) t.Fatalf("it reached the upstream anyway: %v", up.requests())
} }
} }
@@ -612,8 +626,8 @@ func TestAScopedPairingCannotRenameAModelIntoItsScope(t *testing.T) {
t.Fatalf("%s: the refusal must say why: %s", p, b) t.Fatalf("%s: the refusal must say why: %s", p, b)
} }
} }
if len(up.seen) != 0 { if len(up.requests()) != 0 {
t.Fatalf("a rename reached ollama: %v", up.seen) t.Fatalf("a rename reached ollama: %v", up.requests())
} }
} }
@@ -803,8 +817,8 @@ func TestAnEndlessHeaderIsCutOff(t *testing.T) {
t.Fatal("an endless header was read without limit") t.Fatal("an endless header was read without limit")
} }
client.Close() client.Close()
if len(up.seen) != 0 { if len(up.requests()) != 0 {
t.Fatalf("it reached the upstream: %v", up.seen) t.Fatalf("it reached the upstream: %v", up.requests())
} }
} }
+70 -28
View File
@@ -422,35 +422,50 @@ var (
modelDaemonMu sync.Mutex modelDaemonMu sync.Mutex
modelDaemonUp bool modelDaemonUp bool
modelDaemonAt time.Time modelDaemonAt time.Time
modelDaemonAddr string // the address the cached answer is for
) )
// modelDaemonReachable probes the SAME address the rest of this binary uses // modelDaemonReachable probes the SAME address the rest of this binary uses
// (localOllamaAddr, run.go:714-719) so the box can never advertise management // (localOllamaAddr, run.go:714-719) so the box can never advertise management
// of a daemon the forwards do not point at. // of a daemon the forwards do not point at.
func modelDaemonReachable() bool { func modelDaemonReachable() bool { return modelDaemonReachableAt(localOllamaAddr()) }
// modelDaemonReachableAt is modelDaemonReachable's probe of one address. The
// cached answer is for the address it was asked of, so a pairing whose
// address is the vetted one (modelDaemonTarget) never reads another's answer.
func modelDaemonReachableAt(addr string) bool {
modelDaemonMu.Lock() modelDaemonMu.Lock()
defer modelDaemonMu.Unlock() defer modelDaemonMu.Unlock()
if !modelDaemonAt.IsZero() && time.Since(modelDaemonAt) < modelDaemonCacheFor { if !modelDaemonAt.IsZero() && modelDaemonAddr == addr && time.Since(modelDaemonAt) < modelDaemonCacheFor {
return modelDaemonUp return modelDaemonUp
} }
cl := &http.Client{Timeout: 3 * time.Second} cl := &http.Client{Timeout: 3 * time.Second}
resp, err := cl.Get("http://" + localOllamaAddr() + "/api/tags") resp, err := cl.Get("http://" + addr + "/api/tags")
up := false up := false
if err == nil { if err == nil {
up = resp.StatusCode == http.StatusOK up = resp.StatusCode == http.StatusOK
resp.Body.Close() resp.Body.Close()
} }
modelDaemonUp, modelDaemonAt = up, time.Now() modelDaemonUp, modelDaemonAt, modelDaemonAddr = up, time.Now(), addr
return up return up
} }
// resetModelDaemonCache exists for the tests. // resetModelDaemonCache exists for the tests.
func resetModelDaemonCache() { func resetModelDaemonCache() {
modelDaemonMu.Lock() modelDaemonMu.Lock()
modelDaemonUp, modelDaemonAt = false, time.Time{} modelDaemonUp, modelDaemonAt, modelDaemonAddr = false, time.Time{}, ""
modelDaemonMu.Unlock() modelDaemonMu.Unlock()
} }
// modelSelf is the pairing a model:* job came from, or "" when the job was
// driven without one (modelDaemonTarget then keeps OLLAMA_ADDR, as before).
func modelSelf(ws *websocket.Conn) string {
if a := mgmtAccountOf(ws); a != nil {
return a.self
}
return ""
}
// modelSwapCapable answers the heartbeat's question: could this box have a // modelSwapCapable answers the heartbeat's question: could this box have a
// model placed on it BY THIS PAIRING. Two conditions, and the caps one is // model placed on it BY THIS PAIRING. Two conditions, and the caps one is
// first so a pairing outside the family costs no probe — the rule the // first so a pairing outside the family costs no probe — the rule the
@@ -459,13 +474,19 @@ func modelSwapCapable(self string) bool {
if capsAllow(self, modelGateOp) != "" { if capsAllow(self, modelGateOp) != "" {
return false return false
} }
return modelDaemonReachable() // A Host with no platform is offered management only of a model server on
// this machine, and a refused one is not probed at all (modelDaemonTarget).
addr, why := modelDaemonTarget(self)
if why != "" {
return false
}
return modelDaemonReachableAt(addr)
} }
// modelPostJSON posts one JSON body and returns (status, body). The body is // modelPostJSON posts one JSON body and returns (status, body). The body is
// read whatever the status is: a streaming ollama reports a FAILED pull inside // read whatever the status is: a streaming ollama reports a FAILED pull inside
// a 200 (measured, see the header), so the caller needs both. // a 200 (measured, see the header), so the caller needs both.
func modelPostJSON(method, path string, payload any, timeout time.Duration) (int, []byte, error) { func modelPostJSON(addr, method, path string, payload any, timeout time.Duration) (int, []byte, error) {
var rdr io.Reader var rdr io.Reader
if payload != nil { if payload != nil {
b, err := json.Marshal(payload) b, err := json.Marshal(payload)
@@ -474,7 +495,7 @@ func modelPostJSON(method, path string, payload any, timeout time.Duration) (int
} }
rdr = bytes.NewReader(b) rdr = bytes.NewReader(b)
} }
req, err := http.NewRequest(method, "http://"+localOllamaAddr()+path, rdr) req, err := http.NewRequest(method, "http://"+addr+path, rdr)
if err != nil { if err != nil {
return 0, nil, err return 0, nil, err
} }
@@ -528,8 +549,8 @@ func modelCosine(a, b []float64) float64 {
return dot / (math.Sqrt(na) * math.Sqrt(nb)) return dot / (math.Sqrt(na) * math.Sqrt(nb))
} }
func modelEmbedOnce(model, text string) ([]float64, error) { func modelEmbedOnce(addr, model, text string) ([]float64, error) {
status, body, err := modelPostJSON(http.MethodPost, "/api/embed", status, body, err := modelPostJSON(addr, http.MethodPost, "/api/embed",
map[string]any{"model": model, "input": []string{text}}, modelProbeTimeout) map[string]any{"model": model, "input": []string{text}}, modelProbeTimeout)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -549,8 +570,8 @@ func modelEmbedOnce(model, text string) ([]float64, error) {
return out.Embeddings[0], nil return out.Embeddings[0], nil
} }
func modelGenerateOnce(model, prompt string) (string, error) { func modelGenerateOnce(addr, model, prompt string) (string, error) {
status, body, err := modelPostJSON(http.MethodPost, "/api/generate", status, body, err := modelPostJSON(addr, http.MethodPost, "/api/generate",
map[string]any{"model": model, "prompt": prompt, "stream": false, "think": false}, modelProbeTimeout) map[string]any{"model": model, "prompt": prompt, "stream": false, "think": false}, modelProbeTimeout)
if err != nil { if err != nil {
return "", err return "", err
@@ -573,12 +594,12 @@ func modelGenerateOnce(model, prompt string) (string, error) {
// probeEmbedModel proves the model EMBEDS: the same text twice must come back // probeEmbedModel proves the model EMBEDS: the same text twice must come back
// as one answer, and a different text must move it. Returns the measured width // as one answer, and a different text must move it. Returns the measured width
// and "" , or 0 and the reason it is not usable. // and "" , or 0 and the reason it is not usable.
func probeEmbedModel(model string) (int, string) { func probeEmbedModel(addr, model string) (int, string) {
v1, err := modelEmbedOnce(model, modelProbeTextA) v1, err := modelEmbedOnce(addr, model, modelProbeTextA)
if err != nil { if err != nil {
return 0, "it was downloaded but does not answer as an embedding model: " + err.Error() return 0, "it was downloaded but does not answer as an embedding model: " + err.Error()
} }
v2, err := modelEmbedOnce(model, modelProbeTextA) v2, err := modelEmbedOnce(addr, model, modelProbeTextA)
if err != nil { if err != nil {
return 0, "it answered once and then failed: " + err.Error() return 0, "it answered once and then failed: " + err.Error()
} }
@@ -587,7 +608,7 @@ func probeEmbedModel(model string) (int, string) {
return 0, fmt.Sprintf("it gave two different answers to the SAME text (similarity %.4f) — "+ return 0, fmt.Sprintf("it gave two different answers to the SAME text (similarity %.4f) — "+
"nothing routed at it would be reproducible", same) "nothing routed at it would be reproducible", same)
} }
v3, err := modelEmbedOnce(model, modelProbeTextB) v3, err := modelEmbedOnce(addr, model, modelProbeTextB)
if err != nil { if err != nil {
return 0, "the control text failed: " + err.Error() return 0, "the control text failed: " + err.Error()
} }
@@ -604,15 +625,15 @@ func probeEmbedModel(model string) (int, string) {
// probeGenerateModel proves the model GENERATES: a non-empty answer, and a // probeGenerateModel proves the model GENERATES: a non-empty answer, and a
// different prompt must change it (the same constant-answer trap). // different prompt must change it (the same constant-answer trap).
func probeGenerateModel(model string) string { func probeGenerateModel(addr, model string) string {
a, err := modelGenerateOnce(model, modelProbePromptA) a, err := modelGenerateOnce(addr, model, modelProbePromptA)
if err != nil { if err != nil {
return "it was downloaded but does not answer as a text model: " + err.Error() return "it was downloaded but does not answer as a text model: " + err.Error()
} }
if strings.TrimSpace(a) == "" { if strings.TrimSpace(a) == "" {
return "it answered with nothing at all" return "it answered with nothing at all"
} }
b, err := modelGenerateOnce(model, modelProbePromptB) b, err := modelGenerateOnce(addr, model, modelProbePromptB)
if err != nil { if err != nil {
return "it answered once and then failed: " + err.Error() return "it answered once and then failed: " + err.Error()
} }
@@ -704,7 +725,12 @@ func modelReply(ws *websocket.Conn, v any) {
// here — the two together are what makes an honest proposal possible: the host // here — the two together are what makes an honest proposal possible: the host
// can see a model is present without being able to claim it may remove it. // can see a model is present without being able to claim it may remove it.
func handleModelList(ws *websocket.Conn, root string, _ []string) { func handleModelList(ws *websocket.Conn, root string, _ []string) {
status, body, err := modelPostJSON(http.MethodGet, "/api/tags", nil, modelListTimeout) addr, why := modelDaemonTarget(modelSelf(ws))
if why != "" {
modelFail(ws, "model:list: "+why)
return
}
status, body, err := modelPostJSON(addr, http.MethodGet, "/api/tags", nil, modelListTimeout)
if err != nil { if err != nil {
modelFail(ws, "model:list: no model daemon answering on this machine: "+err.Error()) modelFail(ws, "model:list: no model daemon answering on this machine: "+err.Error())
return return
@@ -974,12 +1000,12 @@ func modelPullRunningNote(j *modelPullJob) string {
// runBackupFetch updates job.Got inside its read loop (backupfetch.go:309-311): // runBackupFetch updates job.Got inside its read loop (backupfetch.go:309-311):
// progress that only exists when the transfer ends is not progress. nil is // progress that only exists when the transfer ends is not progress. nil is
// allowed — the direct-call tests pass it. // allowed — the direct-call tests pass it.
func modelPullStreaming(model string, maxBytes int64, note func(want, got int64)) (int64, error) { func modelPullStreaming(addr, model string, maxBytes int64, note func(want, got int64)) (int64, error) {
body, err := json.Marshal(map[string]any{"model": model, "stream": true}) body, err := json.Marshal(map[string]any{"model": model, "stream": true})
if err != nil { if err != nil {
return 0, err return 0, err
} }
httpReq, err := http.NewRequest(http.MethodPost, "http://"+localOllamaAddr()+"/api/pull", httpReq, err := http.NewRequest(http.MethodPost, "http://"+addr+"/api/pull",
bytes.NewReader(body)) bytes.NewReader(body))
if err != nil { if err != nil {
return 0, err return 0, err
@@ -1109,6 +1135,17 @@ func modelPullStart(ws *websocket.Conn, root string, req modelReq) {
return return
} }
// WHERE THE PULL GOES, decided once and held for the download and its
// probe: a Host with no platform pulls only onto a model server on this
// machine (modelDaemonTarget). Refused before the approval is read, so a
// pull that cannot go here costs the person nothing. For such a pairing a
// name in OLLAMA_ADDR is looked up here; nothing is dialled.
addr, why := modelDaemonTarget(modelSelf(ws))
if why != "" {
modelRefuse(ws, req.Model, why)
return
}
// THE APPROVAL GATE. Nothing above this line has touched the network; nothing // THE APPROVAL GATE. Nothing above this line has touched the network; nothing
// below it runs until the user has said yes to THIS proposal on this machine. // below it runs until the user has said yes to THIS proposal on this machine.
// The async split moved the download off this frame and deliberately did NOT // The async split moved the download off this frame and deliberately did NOT
@@ -1152,7 +1189,7 @@ func modelPullStart(ws *websocket.Conn, root string, req modelReq) {
snap := *job snap := *job
modelPullJobsMu.Unlock() modelPullJobsMu.Unlock()
go runModelPull(job, root, req) go runModelPull(job, root, req, addr)
modelReply(ws, snap) modelReply(ws, snap)
} }
@@ -1167,7 +1204,7 @@ func modelPullStart(ws *websocket.Conn, root string, req modelReq) {
// bytes on this disk", which is true the moment the download succeeds. A model // bytes on this disk", which is true the moment the download succeeds. A model
// that downloaded but failed its probe must stay droppable, or a bad pull would // that downloaded but failed its probe must stay droppable, or a bad pull would
// leave dead weight on the user's machine that nothing is allowed to remove. // leave dead weight on the user's machine that nothing is allowed to remove.
func runModelPull(job *modelPullJob, root string, req modelReq) { func runModelPull(job *modelPullJob, root string, req modelReq, addr string) {
set := func(f func(j *modelPullJob)) { set := func(f func(j *modelPullJob)) {
modelPullJobsMu.Lock() modelPullJobsMu.Lock()
f(job) f(job)
@@ -1201,7 +1238,7 @@ func runModelPull(job *modelPullJob, root string, req modelReq) {
// The 200-with-an-error-in-the-body case that stream:false was chosen for is // The 200-with-an-error-in-the-body case that stream:false was chosen for is
// handled either way: modelBodyError reads EVERY line, so a failure on the // handled either way: modelBodyError reads EVERY line, so a failure on the
// last line of a stream is caught exactly as before. // last line of a stream is caught exactly as before.
total, err := modelPullStreaming(req.Model, req.MaxBytes, total, err := modelPullStreaming(addr, req.Model, req.MaxBytes,
func(want, got int64) { set(func(j *modelPullJob) { j.Want, j.Got = want, got }) }) func(want, got int64) { set(func(j *modelPullJob) { j.Want, j.Got = want, got }) })
if err != nil { if err != nil {
finish("failed", func(j *modelPullJob) { finish("failed", func(j *modelPullJob) {
@@ -1242,9 +1279,9 @@ func runModelPull(job *modelPullJob, root string, req modelReq) {
var dim int var dim int
var why string var why string
if req.Role == "embed" { if req.Role == "embed" {
dim, why = probeEmbedModel(req.Model) dim, why = probeEmbedModel(addr, req.Model)
} else { } else {
why = probeGenerateModel(req.Model) why = probeGenerateModel(addr, req.Model)
} }
if why != "" { if why != "" {
finish("done", func(j *modelPullJob) { finish("done", func(j *modelPullJob) {
@@ -1284,7 +1321,12 @@ func handleModelDrop(ws *websocket.Conn, root string, args []string) {
"not remove it. Only a model this host pulled here is its to drop.") "not remove it. Only a model this host pulled here is its to drop.")
return return
} }
status, body, err := modelPostJSON(http.MethodDelete, "/api/delete", addr, why := modelDaemonTarget(modelSelf(ws))
if why != "" {
modelFail(ws, "model:drop: "+why)
return
}
status, body, err := modelPostJSON(addr, http.MethodDelete, "/api/delete",
map[string]any{"model": req.Model}, modelListTimeout) map[string]any{"model": req.Model}, modelListTimeout)
if err != nil { if err != nil {
modelFail(ws, "model:drop: the model daemon did not answer: "+err.Error()) modelFail(ws, "model:drop: the model daemon did not answer: "+err.Error())
+11 -6
View File
@@ -286,7 +286,8 @@ func TestNetworksInsideThisMachineAreThisMachine(t *testing.T) {
} }
// The local egress lane on an unanchored box: an inside address, by address // The local egress lane on an unanchored box: an inside address, by address
// and by name, is refused as this machine; the LAN passes. // and by name, is refused as this machine; the LAN is refused as the network
// around it; the public internet passes.
t.Setenv("XDG_CONFIG_HOME", t.TempDir()) t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if currentAnchor() != nil || boxMode() != modeLocal { if currentAnchor() != nil || boxMode() != modeLocal {
t.Fatal("the namespaced run must be a local, unanchored box") t.Fatal("the namespaced run must be a local, unanchored box")
@@ -301,12 +302,16 @@ func TestNetworksInsideThisMachineAreThisMachine(t *testing.T) {
t.Errorf("local egress: %s is on a network inside this machine and must be refused as this machine, got %v %q", hp, got, why) t.Errorf("local egress: %s is on a network inside this machine and must be refused as this machine, got %v %q", hp, got, why)
} }
} }
for hp, want := range map[string]string{"10.77.0.9:445": "10.77.0.9:445", "192.168.50.9:631": "192.168.50.9:631", for _, hp := range []string{"10.77.0.9:445", "192.168.50.9:631", "nas.example.test:445", "192.168.60.9:445",
"nas.example.test:445": "10.77.0.9:445", "198.51.100.9:443": "198.51.100.9:443", "10.88.0.9:80", "10.89.0.9:80"} {
"[2001:db8:ff::9]:443": "[2001:db8:ff::9]:443", "192.168.60.9:445": "192.168.60.9:445", if got, why := egressVettedAddrs(ctx, hp); !strings.Contains(why, "approved lanes") {
"10.88.0.9:80": "10.88.0.9:80", "10.89.0.9:80": "10.89.0.9:80", "[2001:db8:88::9]:80": "[2001:db8:88::9]:80"} { t.Errorf("local egress: %s is the LAN and must be refused as the network around this machine, got %v %q", hp, got, why)
}
}
for hp, want := range map[string]string{"198.51.100.9:443": "198.51.100.9:443",
"[2001:db8:ff::9]:443": "[2001:db8:ff::9]:443", "[2001:db8:88::9]:80": "[2001:db8:88::9]:80"} {
if got, why := egressVettedAddrs(ctx, hp); why != "" || len(got) != 1 || got[0] != want { if got, why := egressVettedAddrs(ctx, hp); why != "" || len(got) != 1 || got[0] != want {
t.Errorf("local egress: %s is the LAN and passes as %s: %v %q", hp, want, got, why) t.Errorf("local egress: %s leaves this machine for the internet and passes as %s: %v %q", hp, want, got, why)
} }
} }
// A coder run's proxy refuses the public range on an inside network too // A coder run's proxy refuses the public range on an inside network too
+6
View File
@@ -9,6 +9,7 @@ import (
"errors" "errors"
"net" "net"
"strings" "strings"
"sync"
"testing" "testing"
) )
@@ -33,8 +34,13 @@ func stubOwnAddrs(t *testing.T, ifaces []string, route func(net.IP) (routeKind,
} }
asked := &[]string{} asked := &[]string{}
if route != nil { if route != nil {
// Locked: a lane's workers vet their jobs at once, each on its own
// goroutine (egress_relay_test.go).
var mu sync.Mutex
ownRouteLookup = func(ip net.IP) (routeKind, error) { ownRouteLookup = func(ip net.IP) (routeKind, error) {
mu.Lock()
*asked = append(*asked, ip.String()) *asked = append(*asked, ip.String())
mu.Unlock()
return route(ip) return route(ip)
} }
} else { } else {
+149 -14
View File
@@ -163,10 +163,12 @@ type redeemResp struct {
// mode rule, and the anchoring rule. Returns the existing pairing on a // mode rule, and the anchoring rule. Returns the existing pairing on a
// --replace and the state the new pairing file must carry. // --replace and the state the new pairing file must carry.
// //
// - LOCAL mode (mode.go): one host, full stop. A second `pair` is refused with // - LOCAL mode (mode.go): one host unless the person says otherwise. A
// a plain message; only --replace of the one host is allowed. This is the // second `pair` is refused here with localSecondHostError; `pair` itself
// entire protection for a single-user machine — a second host appearing // turns that into a question asked on this machine's terminal
// means someone else got a token onto the box. // (pair_terminal.go), and every other caller keeps the refusal. On a
// single-user machine a second host appearing unasked means someone else
// got a token onto the box, so only a yes typed here adds one.
// - NETWORKED + anchored (anchor.go): every pairing after the anchor is written // - NETWORKED + anchored (anchor.go): every pairing after the anchor is written
// PENDING. It gets nothing until the anchor host's grant arrives. // PENDING. It gets nothing until the anchor host's grant arrives.
// - --replace keeps the pairing's current state: rotating credentials is not // - --replace keeps the pairing's current state: rotating credentials is not
@@ -191,11 +193,14 @@ func pairPrecheck(cfgs []*Config, handle string, replace, forceActive bool) (exi
return existing, existing.State, nil return existing, existing.State, nil
} }
if boxMode() == modeLocal && len(cfgs) > 0 { if boxMode() == modeLocal && len(cfgs) > 0 {
return nil, "", fmt.Errorf("this bastion is in local mode and already serves %s — a local bastion serves one host. "+ return nil, "", &localSecondHostError{serving: cfgs[0].HostName}
"To rotate that host's credentials: `privatae-bastion pair --replace <token>`. "+
"To serve several hosts, reinstall as a networked bastion (install.sh --networked, root) and anchor it",
cfgs[0].HostName)
} }
return pairPrecheckState(forceActive)
}
// pairPrecheckState is the state a new pairing gets once pairPrecheck's walls
// are passed: active for claim-anchor, pending behind an anchor, else "".
func pairPrecheckState(forceActive bool) (*Config, string, error) {
if forceActive { if forceActive {
return nil, pairingActive, nil return nil, pairingActive, nil
} }
@@ -205,8 +210,73 @@ func pairPrecheck(cfgs []*Config, handle string, replace, forceActive bool) (exi
return nil, "", nil return nil, "", nil
} }
// localSecondHostError is pairPrecheck's refusal on a local machine that
// already serves a host. cmdPair asks the person instead (confirmAddHost).
type localSecondHostError struct{ serving string }
func (e *localSecondHostError) Error() string { return localOneHostRefusal(e.serving) }
// localOneHostRefusal says why a second host was not added to a local
// bastion and what the person can do, in this system's own words. A networked
// reinstall starts a bastion with its own state, so it says plainly that the
// host already here would have to be paired again.
func localOneHostRefusal(serving string) string {
networked := "reinstall it as root with install.sh --networked"
if runtime.GOOS == "windows" {
networked = "reinstall it from an Administrator PowerShell with install.ps1 -Networked"
}
return fmt.Sprintf("this bastion is in local mode and already serves %s, and it adds another Host only when you "+
"answer yes at this computer's terminal. To rotate %s's credentials instead: `privatae-bastion pair --replace <token>`. "+
"A machine that other people's Hosts pair to is a networked bastion: %s. That makes a new bastion with "+
"state of its own, so %s has to be paired with it again.", serving, serving, networked, serving)
}
// alreadyPairedError: the host a pair names is already paired here, and
// nothing was changed. `pair` exits pairExitAlreadyPaired on it, so an
// installer can tell "already connected" from a failure.
type alreadyPairedError struct{ host, msg string }
func (e *alreadyPairedError) Error() string { return e.msg }
// pairExitAlreadyPaired is `pair`'s exit status for alreadyPairedError.
const pairExitAlreadyPaired = 3
// sameNameOtherHost refuses, as a plain failure and never as
// alreadyPairedError, a pair whose Host has the name of prev but trusts
// another CA (wantCA, its fingerprint). A Cloud Host's name can repeat; its CA
// cannot, so that is another Host and never "already connected". nil when
// prev is nil or holds that same CA.
func sameNameOtherHost(prev *Config, wantCA string) error {
if prev == nil || caSHA256(prev.CACert) == wantCA {
return nil
}
return fmt.Errorf("this computer already serves a different Host named %s (its certificate is not this one's), "+
"and one bastion cannot serve two Hosts of the same name yet; nothing was paired", prev.HostName)
}
func alreadyPairedRefusal(host string) error {
return &alreadyPairedError{host: host, msg: fmt.Sprintf("already paired with %s — run `privatae-bastion pair --replace <token>` to rotate its credentials", host)}
}
// pairOpts are what `pair` brings beyond the token: where to redeem, the
// Host CA as a file or as a pairing code's fingerprint, and the person to ask
// before a local machine adds a second host (nil: refuse, as every caller but
// `pair` does).
type pairOpts struct {
redeemURL string
hostCAFile string
caSHA256 string
replace bool
forceActive bool
askSecond func(serving, newHost string) (bool, error)
}
func cmdPair(token, redeemURL, hostCAFile string, replace bool) error { func cmdPair(token, redeemURL, hostCAFile string, replace bool) error {
cfg, all, err := pairWithTokenVia(token, redeemURL, hostCAFile, replace, false) token, opts, err := pairOptsFor(token, redeemURL, hostCAFile, replace)
if err != nil {
return err
}
cfg, all, err := pairWith(token, opts)
if err != nil { if err != nil {
return err return err
} }
@@ -236,6 +306,26 @@ func cmdPair(token, redeemURL, hostCAFile string, replace bool) error {
return nil return nil
} }
// pairOptsFor is what `pair`'s arguments come to: the token to redeem and
// the options for it, the person at this machine to ask about a second host
// among them. A pairing code (pair_code.go) carries the redeem address and the
// Host CA's fingerprint itself, so it takes neither flag.
func pairOptsFor(token, redeemURL, hostCAFile string, replace bool) (string, pairOpts, error) {
opts := pairOpts{redeemURL: redeemURL, hostCAFile: hostCAFile, replace: replace, askSecond: confirmAddHost}
if !isPairCode(token) {
return token, opts, nil
}
if redeemURL != "" || hostCAFile != "" {
return "", opts, errors.New("a pairing code (skp_…) carries its own address and certificate; give it alone, without --redeem-url or --host-ca")
}
pc, err := decodePairCode(token)
if err != nil {
return "", opts, err
}
opts.redeemURL, opts.caSHA256 = pc.RedeemURL, pc.CASHA256
return pc.Token, opts, nil
}
// pairReport is what `pair` prints once the pairing stands. Whatever the form // pairReport is what `pair` prints once the pairing stands. Whatever the form
// (anchored just now, over TCP, over a unix socket, pending, or over the // (anchored just now, over TCP, over a unix socket, pending, or over the
// tunnel), it ends by saying to enrol a second factor when this pairing has // tunnel), it ends by saying to enrol a second factor when this pairing has
@@ -338,6 +428,14 @@ func pairWithTokenAt(token, redeemURL string, replace, forceActive bool) (*Confi
// redeem URL needs (pair_tcp.go): the Host's enroll port, trusted through // redeem URL needs (pair_tcp.go): the Host's enroll port, trusted through
// that CA alone. A unix:// or platform redeem takes no --host-ca. // that CA alone. A unix:// or platform redeem takes no --host-ca.
func pairWithTokenVia(token, redeemURL, hostCAFile string, replace, forceActive bool) (*Config, []*Config, error) { func pairWithTokenVia(token, redeemURL, hostCAFile string, replace, forceActive bool) (*Config, []*Config, error) {
return pairWith(token, pairOpts{redeemURL: redeemURL, hostCAFile: hostCAFile, replace: replace, forceActive: forceActive})
}
// pairWith is every pairing's one path. opts.caSHA256 set (a pairing code)
// takes the Host CA from the enroll port once the walls below are passed,
// kept only under that fingerprint (fetchPinnedHostCA).
func pairWith(token string, opts pairOpts) (*Config, []*Config, error) {
redeemURL, hostCAFile, replace, forceActive := opts.redeemURL, opts.hostCAFile, opts.replace, opts.forceActive
// ONE BASTION PER MACHINE, and this is the check that matters most. // ONE BASTION PER MACHINE, and this is the check that matters most.
// //
// It is FIRST on purpose. loadPairings below reaches pairingsDir -> MkdirAll, // It is FIRST on purpose. loadPairings below reaches pairingsDir -> MkdirAll,
@@ -367,12 +465,17 @@ func pairWithTokenVia(token, redeemURL, hostCAFile string, replace, forceActive
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
} }
if hostCAFile == "" { switch {
case opts.caSHA256 != "":
// A pairing code: the CA comes from the enroll port after the
// walls, under the code's fingerprint.
case hostCAFile == "":
return nil, nil, errors.New("--redeem-url https://… needs --host-ca <the Host's CA, PEM>: it is the only certificate the redeem trusts") return nil, nil, errors.New("--redeem-url https://… needs --host-ca <the Host's CA, PEM>: it is the only certificate the redeem trusts")
} default:
if hostCA, err = loadHostCA(hostCAFile); err != nil { if hostCA, err = loadHostCA(hostCAFile); err != nil {
return nil, nil, err return nil, nil, err
} }
}
tcpRedeem, transport = u, transportTCP tcpRedeem, transport = u, transportTCP
default: default:
s, err := parseUnixRedeemURL(redeemURL) s, err := parseUnixRedeemURL(redeemURL)
@@ -397,12 +500,41 @@ func pairWithTokenVia(token, redeemURL, hostCAFile string, replace, forceActive
return nil, nil, err return nil, nil, err
} }
existing, state, err := pairPrecheck(cfgs, handle, replace, forceActive) existing, state, err := pairPrecheck(cfgs, handle, replace, forceActive)
var second *localSecondHostError
if errors.As(err, &second) && opts.askSecond != nil {
yes, aerr := opts.askSecond(second.serving, handle)
if aerr != nil {
return nil, nil, aerr
}
if !yes {
return nil, nil, fmt.Errorf("nothing was paired: %s stays the only Host this computer serves", second.serving)
}
existing, state, err = pairPrecheckState(forceActive)
}
// The CA this pair trusts, as a fingerprint: the pairing code's, or that
// of the --host-ca file. "" for a platform or unix-socket pairing, whose
// CA comes back in the answer.
wantCA := opts.caSHA256
if wantCA == "" && hostCA != "" {
wantCA = caSHA256(hostCA)
}
var already *alreadyPairedError
if errors.As(err, &already) && wantCA != "" {
if refusal := sameNameOtherHost(pairingFor(cfgs, handle), wantCA); refusal != nil {
return nil, nil, refusal
}
}
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
} }
if err := replaceKeepsTransport(existing, transport); err != nil { if err := replaceKeepsTransport(existing, transport); err != nil {
return nil, nil, err return nil, nil, err
} }
if tcpRedeem != nil && hostCA == "" {
if hostCA, err = fetchPinnedHostCA(tcpRedeem, opts.caSHA256); err != nil {
return nil, nil, err
}
}
if tcpRedeem != nil { if tcpRedeem != nil {
if err := tcpPinRefusal(existing, hostCA); err != nil { if err := tcpPinRefusal(existing, hostCA); err != nil {
return nil, nil, err return nil, nil, err
@@ -438,7 +570,7 @@ func pairWithTokenVia(token, redeemURL, hostCAFile string, replace, forceActive
if prev := pairingFor(cfgs, rr.hostName()); prev != nil && validHostName(rr.hostName()) { if prev := pairingFor(cfgs, rr.hostName()); prev != nil && validHostName(rr.hostName()) {
existing = prev existing = prev
if !replace { if !replace {
return nil, nil, fmt.Errorf("already paired with %s — run `privatae-bastion pair --replace <token>` to rotate its credentials", prev.HostName) return nil, nil, alreadyPairedRefusal(prev.HostName)
} }
if err := replaceKeepsTransport(existing, transportUnix); err != nil { if err := replaceKeepsTransport(existing, transportUnix); err != nil {
return nil, nil, err return nil, nil, err
@@ -463,7 +595,10 @@ func pairWithTokenVia(token, redeemURL, hostCAFile string, replace, forceActive
if prev := pairingFor(cfgs, rr.hostName()); prev != nil && validHostName(rr.hostName()) { if prev := pairingFor(cfgs, rr.hostName()); prev != nil && validHostName(rr.hostName()) {
existing = prev existing = prev
if !replace { if !replace {
return nil, nil, fmt.Errorf("already paired with %s — run `privatae-bastion pair --replace <token>` to rotate its credentials", prev.HostName) if refusal := sameNameOtherHost(prev, caSHA256(hostCA)); refusal != nil {
return nil, nil, refusal
}
return nil, nil, alreadyPairedRefusal(prev.HostName)
} }
if err := replaceKeepsTransport(existing, transportTCP); err != nil { if err := replaceKeepsTransport(existing, transportTCP); err != nil {
return nil, nil, err return nil, nil, err
@@ -531,7 +666,7 @@ func pairWithTokenVia(token, redeemURL, hostCAFile string, replace, forceActive
if prev := pairingFor(cfgs, rr.HostName); prev != nil { if prev := pairingFor(cfgs, rr.HostName); prev != nil {
existing = prev existing = prev
if !replace { if !replace {
return nil, nil, fmt.Errorf("already paired with %s — run `privatae-bastion pair --replace <token>` to rotate its credentials", prev.HostName) return nil, nil, alreadyPairedRefusal(prev.HostName)
} }
} }
opSecret, err := genSecret(24) opSecret, err := genSecret(24)
+254
View File
@@ -0,0 +1,254 @@
package main
// pair_code.go: the pairing code a Host's "Set up a bastion" card hands out,
// so pairing a machine is one pasted line.
//
// privatae-bastion pair skp_… or … | privatae-bastion pair -
//
// The code is the single-use token plus the two things the person used to copy
// by hand: the Host's enroll address and its CA. The CA does not travel; its
// fingerprint does. The bastion takes the CA from the enroll port's own TLS
// chain and keeps it only when its SHA-256 is the one in the code, so the trust
// is the same as `--host-ca`: the fingerprint comes from the owner's own
// signed-in card, as the PEM did. From there the pairing is exactly
// `pair --redeem-url <u> --host-ca <that CA> <t>`.
//
// No error here repeats the code, for the reason pair_stdin.go gives: a log of
// a failed install must not hold the token.
import (
"bytes"
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/base64"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"net"
"net/url"
"strings"
"time"
)
// pairCodePrefix marks a pairing code; a bare token is skb_….
const pairCodePrefix = "skp_"
// pairCodeVersion is the one layout this bastion reads.
const pairCodeVersion = 1
// pairCode is a decoded pairing code: where to redeem, the Host CA's
// fingerprint, and the token.
type pairCode struct {
RedeemURL string // "u": the Host's enroll port, https://<host>:<port>/<path>
CASHA256 string // "ca": SHA-256 of the Host CA certificate's DER, lowercase hex
Token string // "t": the skb_ token
}
func isPairCode(s string) bool { return strings.HasPrefix(s, pairCodePrefix) }
// decodePairCode reads skp_<base64url(JSON)>: exactly the keys v, u, ca and t,
// v the one version this bastion knows, u an enroll URL parseTCPRedeemURL
// takes, ca 64 lowercase hex digits and t a token decodeToken takes.
func decodePairCode(code string) (pairCode, error) {
var pc pairCode
bad := func(why string) (pairCode, error) {
return pairCode{}, fmt.Errorf("the pairing code is not one this bastion can read (%s); copy the whole line from the card again", why)
}
raw, err := base64.RawURLEncoding.DecodeString(strings.TrimPrefix(code, pairCodePrefix))
if err != nil {
return bad("it is not base64url")
}
var fields map[string]json.RawMessage
if err := json.Unmarshal(raw, &fields); err != nil {
return bad("it holds no JSON object")
}
// A map keeps the last of two same-named keys, so "exactly these four"
// is counted on the object as written: a key given twice is an extra key.
if n, err := jsonObjectKeyCount(raw); err != nil || n != len(fields) {
return bad("it names a field twice")
}
if len(fields) != 4 {
return bad(fmt.Sprintf("it has %d fields, and a pairing code has 4", len(fields)))
}
var v int
if err := json.Unmarshal(fields["v"], &v); err != nil {
return bad("its version is missing")
}
if v != pairCodeVersion {
return pairCode{}, fmt.Errorf("the pairing code is version %d, and this bastion reads version %d: update the bastion and pair again", v, pairCodeVersion)
}
for k, dst := range map[string]*string{"u": &pc.RedeemURL, "ca": &pc.CASHA256, "t": &pc.Token} {
if err := json.Unmarshal(fields[k], dst); err != nil || *dst == "" {
return bad("its " + k + " field is missing")
}
}
if _, err := parseTCPRedeemURL(pc.RedeemURL); err != nil {
return bad("its address is not an https enroll address")
}
if len(pc.CASHA256) != 64 || strings.ToLower(pc.CASHA256) != pc.CASHA256 {
return bad("its certificate fingerprint is not 64 lowercase hex digits")
}
if _, err := hex.DecodeString(pc.CASHA256); err != nil {
return bad("its certificate fingerprint is not hex")
}
if !strings.HasPrefix(pc.Token, "skb_") {
return bad("it carries no pairing token")
}
if _, _, err := decodeToken(pc.Token); err != nil {
return bad("its pairing token is malformed")
}
return pc, nil
}
// jsonObjectKeyCount counts the keys of the JSON object raw as written,
// repeats included. raw has already decoded as an object.
func jsonObjectKeyCount(raw []byte) (int, error) {
dec := json.NewDecoder(bytes.NewReader(raw))
if _, err := dec.Token(); err != nil { // the opening brace
return 0, err
}
n := 0
for dec.More() {
if _, err := dec.Token(); err != nil { // the key
return 0, err
}
var v json.RawMessage
if err := dec.Decode(&v); err != nil {
return 0, err
}
n++
}
return n, nil
}
// pairAnswerTimeout is how long the enroll port has to answer the first
// connection. A Cloud Host's pairing port drops every address but the one the
// card opened it to, so a pair from anywhere else gets no answer at all, and
// without this it would wait for the system's own connect timeout, minutes.
// A variable so a test can shorten it; nothing else assigns it.
var pairAnswerTimeout = 10 * time.Second
// pairDial dials the enroll port. A variable so a test can stand in a port
// that never answers; nothing else assigns it.
var pairDial = (&net.Dialer{}).DialContext
// pairWindowError is what a pair says when the Host's enroll port did not
// answer: the plain reason, which is nearly always the pairing window. A type
// of its own so the redeem's HTTP client, which wraps every dial error in the
// request's URL, can hand it on as it is.
type pairWindowError struct{ host string }
func (e *pairWindowError) Error() string {
return fmt.Sprintf("%s's pairing port didn't answer within %s. A Cloud Host opens it only to the address "+
"your browser used when you opened \"Set up a bastion\", for 15 minutes. Pair from that network, or "+
"on the card choose \"Pairing a machine on another network?\" and give this machine's address. Nothing was paired.",
e.host, pairAnswerTimeout)
}
// dialEnrollPort opens a TCP connection to the redeem URL's host and port
// within pairAnswerTimeout, or says why not in plain words.
func dialEnrollPort(ctx context.Context, u *url.URL) (net.Conn, error) {
addr := u.Host
if u.Port() == "" {
addr = net.JoinHostPort(u.Hostname(), "443")
}
ctx, cancel := context.WithTimeout(ctx, pairAnswerTimeout)
defer cancel()
c, err := pairDial(ctx, "tcp", addr)
if err == nil {
return c, nil
}
var ne net.Error
if errors.Is(err, context.DeadlineExceeded) || (errors.As(err, &ne) && ne.Timeout()) {
return nil, &pairWindowError{host: u.Hostname()}
}
return nil, fmt.Errorf("could not reach %s's pairing port: %v; nothing was paired", u.Hostname(), err)
}
// fetchPinnedHostCA takes the Host CA from the enroll port's TLS chain and
// returns it as PEM, only when it is a CA whose SHA-256 over its DER is
// wantSHA and the port's own certificate verifies under it alone for
// tcpServerName, the check hostCATLS makes on every later connection. No
// system roots are consulted at any point.
func fetchPinnedHostCA(u *url.URL, wantSHA string) (string, error) {
conn, err := dialEnrollPort(context.Background(), u)
if err != nil {
return "", err
}
defer conn.Close()
var caPEM string
tc := &tls.Config{
ServerName: tcpServerName, MinVersion: tls.VersionTLS12,
// The chain is checked by verifyPinnedChain below, against the pinned
// CA alone; the system roots must play no part, so the default
// verification is the thing turned off here.
InsecureSkipVerify: true,
VerifyConnection: func(cs tls.ConnectionState) error {
p, err := verifyPinnedChain(cs.PeerCertificates, wantSHA)
caPEM = p
return err
},
}
_ = conn.SetDeadline(time.Now().Add(30 * time.Second))
tlsConn := tls.Client(conn, tc)
if err := tlsConn.Handshake(); err != nil {
return "", fmt.Errorf("%s's pairing port: %v; nothing was paired", u.Hostname(), err)
}
_ = tlsConn.Close()
return caPEM, nil
}
// verifyPinnedChain picks the pinned CA out of a presented chain and checks
// the leaf against it. The leaf is chain[0]; the CA is any later certificate
// whose SHA-256 is wantSHA.
func verifyPinnedChain(chain []*x509.Certificate, wantSHA string) (string, error) {
if len(chain) == 0 {
return "", errors.New("the Host's pairing port presented no certificate")
}
var ca *x509.Certificate
sawCA := false
for _, c := range chain[1:] {
if !c.IsCA {
continue
}
sawCA = true
sum := sha256.Sum256(c.Raw)
if hex.EncodeToString(sum[:]) == wantSHA {
ca = c
break
}
}
if ca == nil {
if sawCA {
return "", errors.New("the Host's pairing port presented a certificate authority other than the one this pairing code names — refusing")
}
return "", errors.New("the Host's pairing port did not present its certificate authority, so this pairing code cannot be checked — refusing (the Host needs a newer release, or pair with --host-ca)")
}
roots := x509.NewCertPool()
roots.AddCert(ca)
inter := x509.NewCertPool()
for _, c := range chain[1:] {
if c != ca {
inter.AddCert(c)
}
}
if _, err := chain[0].Verify(x509.VerifyOptions{Roots: roots, Intermediates: inter, DNSName: tcpServerName}); err != nil {
return "", fmt.Errorf("the Host's pairing port certificate is not signed by the certificate authority this pairing code names: %v — refusing", err)
}
return string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: ca.Raw})), nil
}
// caSHA256 is the fingerprint a pairing code carries for a PEM CA: SHA-256 of
// the first certificate's DER, lowercase hex. "" when the PEM holds none.
func caSHA256(caPEM string) string {
certs := pemCerts(caPEM)
if len(certs) == 0 {
return ""
}
sum := sha256.Sum256(certs[0].Raw)
return hex.EncodeToString(sum[:])
}
+626
View File
@@ -0,0 +1,626 @@
//go:build linux
// The pairing code a Cloud Host's card hands out (pair_code.go): one string
// carrying the token, the enroll address and the Host CA's fingerprint, and
// the question a local machine asks before it serves a second Host
// (pair_terminal.go).
//
// Uses the fixtures of local_host_linux_test.go and local_host_tcp_linux_test.go
// (newTestCA, pairingBox, startTCPEnrollHost, homeToken), so it carries the same
// build tag.
package main
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"math/big"
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
)
// makePairCode builds a code the way a Host's card does: skp_ and base64url,
// no padding, of the JSON object fields.
func makePairCode(t *testing.T, fields map[string]any) string {
t.Helper()
raw, err := json.Marshal(fields)
if err != nil {
t.Fatal(err)
}
return pairCodePrefix + base64.RawURLEncoding.EncodeToString(raw)
}
func pairCodeFingerprint(ca *testCA) string {
sum := sha256.Sum256(ca.cert.Raw)
return hex.EncodeToString(sum[:])
}
// chainWithCA is the enroll port's certificate as a Host from this release
// serves it: the leaf, then the CA.
func chainWithCA(t *testing.T, ca *testCA) tls.Certificate {
t.Helper()
c := ca.serverCert(t, tcpServerName)
c.Certificate = append(c.Certificate, ca.cert.Raw)
return c
}
func goodCodeFields(url, fp string) map[string]any {
return map[string]any{"v": 1, "u": url, "ca": fp, "t": homeToken()}
}
// ── decoding ────────────────────────────────────────────────────────────────
func TestAPairingCodeDecodes(t *testing.T) {
fp := strings.Repeat("ab", 32)
code := makePairCode(t, goodCodeFields("https://maple.example-id.cloud.example:8810/v1/local/redeem", fp))
pc, err := decodePairCode(code)
if err != nil {
t.Fatal(err)
}
if pc.RedeemURL != "https://maple.example-id.cloud.example:8810/v1/local/redeem" || pc.CASHA256 != fp || pc.Token != homeToken() {
t.Fatalf("decoded: %+v", pc)
}
token, opts, err := pairOptsFor(code, "", "", false)
if err != nil || token != homeToken() || opts.redeemURL != pc.RedeemURL || opts.caSHA256 != fp || opts.hostCAFile != "" || opts.askSecond == nil {
t.Fatalf("pair's options from a code: %q %+v %v", token, opts, err)
}
if _, _, err := pairOptsFor(code, "", "host-ca.pem", false); err == nil {
t.Fatal("a code carries its own certificate: --host-ca beside it must be refused")
}
if tok, opts, err := pairOptsFor(homeToken(), "", "", false); err != nil || tok != homeToken() || opts.caSHA256 != "" {
t.Fatalf("a bare token passes through as it was: %q %+v %v", tok, opts, err)
}
}
// A code this bastion cannot read is refused with what was wrong, never
// with its content: a log of a failed install must not hold the token.
func TestAPairingCodeIsReadStrictly(t *testing.T) {
fp := strings.Repeat("0f", 32)
u := "https://maple.example-id.cloud.example:8810/v1/local/redeem"
with := func(edit func(m map[string]any)) string {
m := goodCodeFields(u, fp)
edit(m)
return makePairCode(t, m)
}
for name, code := range map[string]string{
"garbled": "skp_%%%not-base64%%%",
"not JSON": pairCodePrefix + base64.RawURLEncoding.EncodeToString([]byte("secret-x home")),
"wrong version": with(func(m map[string]any) { m["v"] = 2 }),
"version as text": with(func(m map[string]any) { m["v"] = "1" }),
"extra key": with(func(m map[string]any) { m["x"] = "y" }),
"missing key": with(func(m map[string]any) { delete(m, "ca") }),
"another key": with(func(m map[string]any) { delete(m, "t"); m["token"] = homeToken() }),
"http address": with(func(m map[string]any) { m["u"] = "http://maple.example-id.cloud.example:8810/v1/local/redeem" }),
"address with user": with(func(m map[string]any) { m["u"] = "https://a@maple.example-id.cloud.example/v1/local/redeem" }),
"short fingerprint": with(func(m map[string]any) { m["ca"] = fp[:63] }),
"upper fingerprint": with(func(m map[string]any) { m["ca"] = strings.ToUpper(fp) }),
"not hex": with(func(m map[string]any) { m["ca"] = strings.Repeat("zz", 32) }),
"no skb_ token": with(func(m map[string]any) { m["t"] = "secret-x" }),
"malformed token": with(func(m map[string]any) { m["t"] = "skb_bm9kb3Q" }), // base64url("nodot")
} {
_, err := decodePairCode(code)
if err == nil {
t.Errorf("%s: refused nothing", name)
continue
}
if strings.Contains(err.Error(), "secret-x") || strings.Contains(err.Error(), homeToken()) {
t.Errorf("%s: the refusal repeats the token: %v", name, err)
}
}
_, err := decodePairCode(with(func(m map[string]any) { m["v"] = 2 }))
if err == nil || !strings.Contains(err.Error(), "update the bastion") {
t.Fatalf("a newer code says to update the bastion: %v", err)
}
}
// ── the pinned CA ───────────────────────────────────────────────────────────
// SECURITY: the CA is taken from the enroll port's chain only when its
// SHA-256 is the code's, and the port's own certificate must verify under it
// for tcpServerName. A chain without the CA, another CA, or a leaf another CA
// signed are refused before the token is spent, and nothing is saved.
func TestAPairingCodePinsTheHostCAByFingerprint(t *testing.T) {
ca := newTestCA(t, "maple host CA")
impostor := newTestCA(t, "impostor")
leafByImpostor := impostor.serverCert(t, tcpServerName)
leafByImpostor.Certificate = append(leafByImpostor.Certificate, ca.cert.Raw)
for name, c := range map[string]struct {
serve tls.Certificate
fp string
want string
}{
"no CA in the chain": {ca.serverCert(t, tcpServerName), pairCodeFingerprint(ca), "did not present"},
"another CA in the chain": {chainWithCA(t, impostor), pairCodeFingerprint(ca), "other than"},
"another fingerprint": {chainWithCA(t, ca), pairCodeFingerprint(impostor), "other than"},
"a leaf another CA signed": {leafByImpostor, pairCodeFingerprint(ca), "not signed by"},
"the CA, under another name": {func() tls.Certificate {
c := ca.serverCert(t, "maple.example")
c.Certificate = append(c.Certificate, ca.cert.Raw)
return c
}(), pairCodeFingerprint(ca), "not signed by"},
} {
t.Run(name, func(t *testing.T) {
pairingBox(t)
h := startTCPEnrollHost(t, ca, c.serve, "127.0.0.1:18821")
token, opts, err := pairOptsFor(makePairCode(t, goodCodeFields(h.url, c.fp)), "", "", false)
if err != nil {
t.Fatal(err)
}
_, _, err = pairWith(token, opts)
if err == nil || !strings.Contains(err.Error(), c.want) {
t.Fatalf("want a refusal naming %q, got %v", c.want, err)
}
if h.hits.Load() != 0 {
t.Fatal("the token went to an enroll port whose CA was never checked")
}
if cfgs, _ := loadPairings(); len(cfgs) != 0 {
t.Fatalf("a refused pairing saved %d pairings", len(cfgs))
}
})
}
}
// A good code pairs exactly as --redeem-url u --host-ca <that CA> t does: the
// same TCP pairing, the same CA pinned.
func TestAPairingCodePairsAsTheHostCAFileDoes(t *testing.T) {
pairingBox(t)
ca := newTestCA(t, "maple host CA")
h := startTCPEnrollHost(t, ca, chainWithCA(t, ca), "127.0.0.1:18822")
token, opts, err := pairOptsFor(makePairCode(t, goodCodeFields(h.url, pairCodeFingerprint(ca))), "", "", false)
if err != nil {
t.Fatal(err)
}
cfg, _, err := pairWith(token, opts)
if err != nil {
t.Fatal(err)
}
if !cfg.tcpTransport() || !sameCAs(cfg.CACert, ca.pem) || cfg.LaneAddr != "127.0.0.1:18822" || cfg.HostName != "home.example.test" {
t.Fatalf("the pairing: transport %q lane %q host %q", cfg.Transport, cfg.LaneAddr, cfg.HostName)
}
h.mu.Lock()
req, sni := h.last, h.sni
h.mu.Unlock()
if req.Secret != "secret-x" || req.Handle != "home" || sni != tcpServerName {
t.Fatalf("the redeem is today's: secret %q handle %q sni %q", req.Secret, req.Handle, sni)
}
// The same code again: already paired with this Host, nothing changed,
// and the error is the one `pair` exits 3 on.
before, _ := os.ReadFile(filepath.Join(mustPairingsDir(t), "home.example.test.json"))
token, opts, _ = pairOptsFor(makePairCode(t, goodCodeFields(h.url, pairCodeFingerprint(ca))), "", "", false)
_, _, err = pairWith(token, opts)
var already *alreadyPairedError
if !errors.As(err, &already) {
t.Fatalf("pairing the same Host again must be alreadyPairedError (exit %d), got %v", pairExitAlreadyPaired, err)
}
after, _ := os.ReadFile(filepath.Join(mustPairingsDir(t), "home.example.test.json"))
if string(before) != string(after) || h.hits.Load() != 1 {
t.Fatal("an already-paired Host must change nothing and spend no token")
}
// Another Host of the same name (Cloud names repeat): a different CA, so
// never "already connected".
other := newTestCA(t, "another maple")
h2 := startTCPEnrollHost(t, other, chainWithCA(t, other), "127.0.0.1:18823")
token, opts, _ = pairOptsFor(makePairCode(t, goodCodeFields(h2.url, pairCodeFingerprint(other))), "", "", false)
_, _, err = pairWith(token, opts)
if err == nil || errors.As(err, &already) || !strings.Contains(err.Error(), "different Host named") {
t.Fatalf("a same-named Host under another CA is a plain refusal, not exit 3: %v", err)
}
}
func mustPairingsDir(t *testing.T) string {
t.Helper()
d, err := pairingsDir()
if err != nil {
t.Fatal(err)
}
return d
}
// ── an enroll port that does not answer ─────────────────────────────────────
// A Cloud Host's pairing port drops every address but the one the card opened
// it to. A pair from elsewhere says so within pairAnswerTimeout, in plain
// words, instead of waiting for the system's connect timeout. Both the code
// and --host-ca paths.
func TestAnEnrollPortThatDoesNotAnswerNamesThePairingWindow(t *testing.T) {
pairingBox(t)
oldTimeout, oldDial := pairAnswerTimeout, pairDial
t.Cleanup(func() { pairAnswerTimeout, pairDial = oldTimeout, oldDial })
pairAnswerTimeout = 150 * time.Millisecond
pairDial = func(ctx context.Context, _, _ string) (net.Conn, error) {
<-ctx.Done()
return nil, ctx.Err()
}
u := "https://maple.example-id.cloud.example:8810/v1/local/redeem"
ca := newTestCA(t, "maple host CA")
check := func(name string, err error, took time.Duration) {
t.Helper()
var we *pairWindowError
if !errors.As(err, &we) {
t.Fatalf("%s: want the pairing-window message, got %v", name, err)
}
msg := err.Error()
if !strings.Contains(msg, "maple.example-id.cloud.example's pairing port didn't answer") ||
!strings.Contains(msg, "15 minutes") || !strings.Contains(msg, "another network") || !strings.Contains(msg, "Nothing was paired") {
t.Fatalf("%s: the message: %s", name, msg)
}
if took > 2*time.Second {
t.Fatalf("%s: took %s, the timeout is %s", name, took, pairAnswerTimeout)
}
}
start := time.Now()
token, opts, _ := pairOptsFor(makePairCode(t, goodCodeFields(u, pairCodeFingerprint(ca))), "", "", false)
_, _, err := pairWith(token, opts)
check("pairing code", err, time.Since(start))
start = time.Now()
_, _, err = pairWithTokenVia(homeToken(), u, writeCA(t, ca), false, false)
check("--host-ca", err, time.Since(start))
}
// ── a local machine that already serves a Host ──────────────────────────────
// fakeTerminal stands in a person at this machine: their answer, and what
// the question said. answer "" with noTerm set is a run with no terminal.
type fakeTerminal struct {
answer string
noTerm bool
asked strings.Builder
opened int
}
func (f *fakeTerminal) use(t *testing.T) {
t.Helper()
old := openTerminal
t.Cleanup(func() { openTerminal = old })
openTerminal = func() (io.Reader, io.Writer, func(), error) {
f.opened++
if f.noTerm {
return nil, nil, nil, errNoTerminal
}
return strings.NewReader(f.answer), &f.asked, func() {}, nil
}
}
func otherToken() string { return "skb_b3RoZXIuc2VjcmV0LXk" } // base64url("other.secret-y")
// SECURITY: a local machine that serves a Host adds another only on a yes
// typed at this machine's terminal. No terminal, or any answer but y or yes,
// refuses before the token is spent, and the first pairing is untouched.
// Every caller but `pair` (claim-anchor, the pairing op) keeps the old
// refusal: it has no person to ask.
func TestALocalMachineAddsASecondHostOnlyOnAYesTypedHere(t *testing.T) {
pairingBox(t)
ca := newTestCA(t, "home host CA")
home := startTCPEnrollHost(t, ca, ca.serverCert(t, tcpServerName), "127.0.0.1:18831")
if _, _, err := pairWithTokenVia(homeToken(), home.url, writeCA(t, ca), false, false); err != nil {
t.Fatal(err)
}
first, _ := os.ReadFile(filepath.Join(mustPairingsDir(t), "home.example.test.json"))
otherCA := newTestCA(t, "other host CA")
other := startTCPEnrollHost(t, otherCA, chainWithCA(t, otherCA), "127.0.0.1:18832")
other.set(func(h *tcpEnrollHost) {
h.edit = func(ans map[string]any) { ans["host_handle"], ans["host_name"] = "other", "other.example.test" }
})
code := makePairCode(t, map[string]any{"v": 1, "u": other.url, "ca": pairCodeFingerprint(otherCA), "t": otherToken()})
pairOther := func() error {
token, opts, err := pairOptsFor(code, "", "", false)
if err != nil {
t.Fatal(err)
}
_, _, err = pairWith(token, opts)
return err
}
untouched := func(why string) {
t.Helper()
now, _ := os.ReadFile(filepath.Join(mustPairingsDir(t), "home.example.test.json"))
cfgs, _ := loadPairings()
if string(now) != string(first) {
t.Fatalf("%s: the first pairing changed", why)
}
if why != "yes" && (len(cfgs) != 1 || other.hits.Load() != 0) {
t.Fatalf("%s: %d pairings, %d redeems", why, len(cfgs), other.hits.Load())
}
}
// No terminal: refused, and it says where to run it instead.
ft := &fakeTerminal{noTerm: true}
ft.use(t)
if err := pairOther(); err == nil || !strings.Contains(err.Error(), "terminal window on this computer") {
t.Fatalf("no terminal must refuse in plain words: %v", err)
}
untouched("no terminal")
// Anything but y or yes.
for _, a := range []string{"\n", "n\n", "no\n", "yess\n", "sure\n", ""} {
ft := &fakeTerminal{answer: a}
ft.use(t)
if err := pairOther(); err == nil || !strings.Contains(err.Error(), "stays the only Host") {
t.Fatalf("answer %q must refuse: %v", a, err)
}
if !strings.Contains(ft.asked.String(), "already serves your Host home.example.test. Add other as well?") {
t.Fatalf("the question: %q", ft.asked.String())
}
untouched("answer " + a)
}
// The callers with no person to ask keep the refusal, and ask nobody.
ft = &fakeTerminal{answer: "y\n"}
ft.use(t)
var second *localSecondHostError
if _, _, err := pairWithTokenVia(otherToken(), other.url, writeCA(t, otherCA), false, false); !errors.As(err, &second) {
t.Fatalf("pairWithTokenVia (no person to ask) must keep the refusal: %v", err)
}
if ft.opened != 0 {
t.Fatal("a caller with no person to ask opened the terminal")
}
untouched("no asker")
// A typed yes adds it beside the first.
ft = &fakeTerminal{answer: "Yes\n"}
ft.use(t)
if err := pairOther(); err != nil {
t.Fatal(err)
}
cfgs, _ := loadPairings()
if len(cfgs) != 2 || pairingFor(cfgs, "other.example.test") == nil {
t.Fatalf("after yes: %d pairings", len(cfgs))
}
untouched("yes")
}
// secondHostStdinChild set to 1 makes this test binary the child of
// TestTheSecondHostAnswerIsNeverReadFromStandardInput.
const secondHostStdinChild = "PRIVATAE_TEST_SECOND_HOST_STDIN_CHILD"
// SECURITY: the answer to the second-Host question comes from the controlling
// terminal, never from standard input. The installers hand the pairing code to
// `pair -` on standard input, so a "y" piped in after the code must not add a
// Host. The real openTerminal runs in a child with no controlling terminal (a
// session of its own) and "y" waiting on its standard input: it must find no
// terminal, and leave standard input unread.
func TestTheSecondHostAnswerIsNeverReadFromStandardInput(t *testing.T) {
if os.Getenv(secondHostStdinChild) == "1" {
yes, err := confirmAddHost("home.example.test", "other")
rest, _ := io.ReadAll(os.Stdin)
fmt.Printf("child: yes=%v refused=%v stdin-left=%q\n", yes, err != nil && strings.Contains(err.Error(), "no terminal"), rest)
os.Exit(0)
}
self, err := os.Executable()
if err != nil {
t.Fatal(err)
}
cmd := exec.Command(self, "-test.run=^TestTheSecondHostAnswerIsNeverReadFromStandardInput$")
cmd.Env = append(os.Environ(), secondHostStdinChild+"=1")
cmd.Stdin = strings.NewReader("y\n")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("the child: %v\n%s", err, out)
}
if !strings.Contains(string(out), `child: yes=false refused=true stdin-left="y\n"`) {
t.Fatalf("a y on standard input must neither answer the question nor be read: %s", out)
}
}
// The refusal names this system's own way to a networked bastion, and says
// that the Host already here would have to be paired again.
func TestTheLocalRefusalSaysWhatANetworkedReinstallCosts(t *testing.T) {
msg := localOneHostRefusal("home.example.test")
if !strings.Contains(msg, "install.sh --networked") || strings.Contains(msg, "install.ps1") {
t.Fatalf("on linux it names install.sh: %s", msg)
}
if !strings.Contains(msg, "home.example.test has to be paired with it again") || !strings.Contains(msg, "answer yes at this computer's terminal") {
t.Fatalf("the refusal: %s", msg)
}
}
// ── what the Host's card actually makes ─────────────────────────────────────
// Made by the C-Series Host's own pairing_code (bastion_enroll.py) from the CA
// below and an invented token, for an invented address: compact JSON with the
// keys v, u, ca and t in base64url without padding. The bastion must read
// exactly that, and the fingerprint must be the one caSHA256 computes from
// the same CA, or a real code would never pin.
const hostMadePairCode = "skp_eyJ2IjoxLCJ1IjoiaHR0cHM6Ly9tYXBsZS5leGFtcGxlLWlkLmNsb3VkLmV4YW1wbGU6ODgxMC92MS9sb2NhbC9yZWRlZW0iLCJjYSI6IjcyMzNkOWU4ODU5MzIyZjE1YTJiNmMzYWIyMmZiNjMxNGI4M2I0Njg2ZmNhYmZiYjRmNGU1NDE5NDA1MTFkMDciLCJ0Ijoic2tiX2JXRndiR1V1YzJWamNtVjBMV2x1ZG1WdWRHVmtMVEF3TURFIn0"
const hostMadePairCodeCA = `-----BEGIN CERTIFICATE-----
MIIBTTCB9KADAgECAgIQkjAKBggqhkjOPQQDAjAjMSEwHwYDVQQDDBhleGFtcGxl
IGludmVudGVkIGhvc3QgQ0EwIBcNMjYwMTAxMDAwMDAwWhgPMjEyNTEyMDgwMDAw
MDBaMCMxITAfBgNVBAMMGGV4YW1wbGUgaW52ZW50ZWQgaG9zdCBDQTBZMBMGByqG
SM49AgEGCCqGSM49AwEHA0IABA32JMrh4sMMLvNNoSBq5FgXFgHd/2n71Qk9ayje
Kj9rdt+Q3CFYWgtCqFHc6ZLUfXexKkdWvnET3px2hGqZWbKjFjAUMBIGA1UdEwEB
/wQIMAYBAf8CAQAwCgYIKoZIzj0EAwIDSAAwRQIhAIQ6sKtiqxcAQTPq7+toiEl0
qrWb4CKKB2UtvcQoC9YwAiBYGKbmvG9ZszE8X7uKUGexjP9x6lDOwqfmnLoruuVg
Kg==
-----END CERTIFICATE-----
`
func TestACodeTheHostMadeDecodesAndPins(t *testing.T) {
pc, err := decodePairCode(hostMadePairCode)
if err != nil {
t.Fatal(err)
}
if pc.RedeemURL != "https://maple.example-id.cloud.example:8810/v1/local/redeem" ||
pc.Token != "skb_bWFwbGUuc2VjcmV0LWludmVudGVkLTAwMDE" { // base64url("maple.secret-invented-0001")
t.Fatalf("decoded: %+v", pc)
}
if got := caSHA256(hostMadePairCodeCA); got != pc.CASHA256 {
t.Fatalf("the Host's fingerprint %s is not caSHA256 of its CA (%s)", pc.CASHA256, got)
}
if h, _, err := decodeToken(pc.Token); err != nil || h != "maple" {
t.Fatalf("the token's handle: %q %v", h, err)
}
}
// SECURITY: a key given twice is an extra key. A map would keep only the
// last, so two codes that differ in which token they carry could both read as
// "exactly v, u, ca and t".
func TestAPairingCodeWithAKeyGivenTwiceIsRefused(t *testing.T) {
fp := strings.Repeat("0f", 32)
u := "https://maple.example-id.cloud.example:8810/v1/local/redeem"
raw := `{"v":1,"u":"` + u + `","ca":"` + fp + `","t":"` + otherToken() + `","t":"` + homeToken() + `"}`
if _, err := decodePairCode(pairCodePrefix + base64.RawURLEncoding.EncodeToString([]byte(raw))); err == nil {
t.Fatal("a code naming t twice was read")
}
once := `{"v":1,"u":"` + u + `","ca":"` + fp + `","t":"` + homeToken() + `"}`
if _, err := decodePairCode(pairCodePrefix + base64.RawURLEncoding.EncodeToString([]byte(once))); err != nil {
t.Fatalf("the same code with t once: %v", err)
}
}
// pinnedChainCert is a certificate for the pinned-chain cases: signed by
// parent (self-signed when parent is nil), a CA or not, valid from nb to na.
func pinnedChainCert(t *testing.T, cn string, isCA bool, nb, na time.Time, parent *x509.Certificate, parentKey *ecdsa.PrivateKey, dns ...string) (*x509.Certificate, *ecdsa.PrivateKey) {
t.Helper()
key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
tmpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: cn},
NotBefore: nb, NotAfter: na, DNSNames: dns, BasicConstraintsValid: true, IsCA: isCA}
if isCA {
tmpl.KeyUsage = x509.KeyUsageCertSign | x509.KeyUsageDigitalSignature
} else {
tmpl.KeyUsage, tmpl.ExtKeyUsage = x509.KeyUsageDigitalSignature, []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}
}
signer, signKey := tmpl, key
if parent != nil {
signer, signKey = parent, parentKey
}
der, err := x509.CreateCertificate(rand.Reader, tmpl, signer, &key.PublicKey, signKey)
if err != nil {
t.Fatal(err)
}
c, _ := x509.ParseCertificate(der)
return c, key
}
// SECURITY: the cases of the pinned chain the TLS tests do not reach. A
// certificate with the code's fingerprint that is not a CA, and a CA that has
// expired, are refused; the pinned CA anywhere after the leaf, beside other
// certificates, is the one returned.
func TestThePinnedChainTakesOnlyAValidCAWithTheFingerprint(t *testing.T) {
now := time.Now()
fpOf := func(c *x509.Certificate) string { s := sha256.Sum256(c.Raw); return hex.EncodeToString(s[:]) }
ca, caKey := pinnedChainCert(t, "maple host CA", true, now.Add(-time.Hour), now.Add(24*time.Hour), nil, nil)
leaf, _ := pinnedChainCert(t, "leaf", false, now.Add(-time.Hour), now.Add(time.Hour), ca, caKey, tcpServerName)
other, _ := pinnedChainCert(t, "another CA", true, now.Add(-time.Hour), now.Add(24*time.Hour), nil, nil)
got, err := verifyPinnedChain([]*x509.Certificate{leaf, other, ca}, fpOf(ca))
if err != nil || caSHA256(got) != fpOf(ca) {
t.Fatalf("the pinned CA after another certificate: %v", err)
}
// A self-signed certificate that is not a CA, signing the leaf.
notCA, notCAKey := pinnedChainCert(t, "not a CA", false, now.Add(-time.Hour), now.Add(24*time.Hour), nil, nil)
leaf2, _ := pinnedChainCert(t, "leaf", false, now.Add(-time.Hour), now.Add(time.Hour), notCA, notCAKey, tcpServerName)
if _, err := verifyPinnedChain([]*x509.Certificate{leaf2, notCA}, fpOf(notCA)); err == nil {
t.Fatal("a certificate with the fingerprint that is not a CA was pinned")
}
// An expired CA, with a leaf it signed that is still in its dates.
old, oldKey := pinnedChainCert(t, "expired host CA", true, now.Add(-48*time.Hour), now.Add(-time.Hour), nil, nil)
leaf3, _ := pinnedChainCert(t, "leaf", false, now.Add(-2*time.Hour), now.Add(time.Hour), old, oldKey, tcpServerName)
if _, err := verifyPinnedChain([]*x509.Certificate{leaf3, old}, fpOf(old)); err == nil {
t.Fatal("an expired CA was pinned")
}
if _, err := verifyPinnedChain([]*x509.Certificate{leaf}, fpOf(ca)); err == nil {
t.Fatal("a chain of the leaf alone was accepted")
}
}
// Exit 3 means the same Host, so the same CA, on every TCP pairing: a
// --host-ca pair to a Host of a name already paired here under another CA is
// a plain failure, as the pairing code's is.
func TestHostCAFileWithASameNamedHostUnderAnotherCAIsNotAlreadyPaired(t *testing.T) {
pairingBox(t)
ca := newTestCA(t, "home host CA")
h := startTCPEnrollHost(t, ca, ca.serverCert(t, tcpServerName), "127.0.0.1:18841")
if _, _, err := pairWithTokenVia(homeToken(), h.url, writeCA(t, ca), false, false); err != nil {
t.Fatal(err)
}
var already *alreadyPairedError
if _, _, err := pairWithTokenVia(homeToken(), h.url, writeCA(t, ca), false, false); !errors.As(err, &already) {
t.Fatalf("the same Host under its own CA is already paired (exit 3): %v", err)
}
other := newTestCA(t, "another home")
h2 := startTCPEnrollHost(t, other, other.serverCert(t, tcpServerName), "127.0.0.1:18842")
_, _, err := pairWithTokenVia(homeToken(), h2.url, writeCA(t, other), false, false)
if err == nil || errors.As(err, &already) || !strings.Contains(err.Error(), "different Host named") {
t.Fatalf("a same-named Host under another CA is a plain refusal, not exit 3: %v", err)
}
if h2.hits.Load() != 0 {
t.Fatal("the refusal spent the other Host's token")
}
}
// SECURITY: the name in the question is the token's handle, which whoever
// made the token chose. A control character or escape sequence in it reaches
// the person's screen escaped, so it cannot rewrite the question.
func TestTheSecondHostQuestionEscapesTheTokensName(t *testing.T) {
ft := &fakeTerminal{answer: "n\n"}
ft.use(t)
if _, err := confirmAddHost("home.example.test", "evil\r\x1b[2K\x1b[1Ainnocent"); err != nil {
t.Fatal(err)
}
q := ft.asked.String()
if strings.ContainsAny(q, "\x1b\r") {
t.Fatalf("the question carries the handle's control bytes raw: %q", q)
}
if !strings.Contains(q, `Add "evil\r\x1b[2K\x1b[1Ainnocent" as well?`) {
t.Fatalf("the question: %q", q)
}
ft = &fakeTerminal{answer: "n\n"}
ft.use(t)
if _, err := confirmAddHost("home.example.test", "maple"); err != nil || !strings.Contains(ft.asked.String(), "Add maple as well?") {
t.Fatalf("a plain name is shown as it is: %q %v", ft.asked.String(), err)
}
}
// The name the redeem answers with is checked again after the redeem (the
// token's handle can differ from it). There too, the same name under another
// CA is a plain failure, never exit 3.
func TestARedeemAnsweringAPairedNameUnderAnotherCAIsNotAlreadyPaired(t *testing.T) {
pairingBox(t)
ca := newTestCA(t, "home host CA")
h := startTCPEnrollHost(t, ca, ca.serverCert(t, tcpServerName), "127.0.0.1:18843")
if _, _, err := pairWithTokenVia(homeToken(), h.url, writeCA(t, ca), false, false); err != nil {
t.Fatal(err)
}
other := newTestCA(t, "another home")
h2 := startTCPEnrollHost(t, other, chainWithCA(t, other), "127.0.0.1:18844")
// otherToken's handle is "other"; the Host answers home.example.test.
token, opts, err := pairOptsFor(makePairCode(t, map[string]any{"v": 1, "u": h2.url, "ca": pairCodeFingerprint(other), "t": otherToken()}), "", "", false)
if err != nil {
t.Fatal(err)
}
opts.askSecond = func(string, string) (bool, error) { return true, nil }
_, _, err = pairWith(token, opts)
var already *alreadyPairedError
if err == nil || errors.As(err, &already) || !strings.Contains(err.Error(), "different Host named home.example.test") {
t.Fatalf("a redeem naming a paired Host under another CA is a plain refusal, not exit 3: %v", err)
}
if cfgs, _ := loadPairings(); len(cfgs) != 1 {
t.Fatalf("%d pairings after the refusal", len(cfgs))
}
}
+8
View File
@@ -149,11 +149,19 @@ func redeemOverTCP(u *url.URL, caPEM string, body []byte) (tcpRedeemResp, error)
if err != nil { if err != nil {
return rr, fmt.Errorf("--host-ca: %w", err) return rr, fmt.Errorf("--host-ca: %w", err)
} }
// The dial is dialEnrollPort's: an enroll port that does not answer in
// pairAnswerTimeout gets the pairing window's plain message, not a wait
// for the system's connect timeout.
httpc := &http.Client{Timeout: 30 * time.Second, Transport: &http.Transport{ httpc := &http.Client{Timeout: 30 * time.Second, Transport: &http.Transport{
Proxy: nil, TLSClientConfig: tc, ForceAttemptHTTP2: false, Proxy: nil, TLSClientConfig: tc, ForceAttemptHTTP2: false,
DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { return dialEnrollPort(ctx, u) },
}, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} }, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
resp, err := httpc.Post(u.String(), "application/json", bytes.NewReader(body)) resp, err := httpc.Post(u.String(), "application/json", bytes.NewReader(body))
if err != nil { if err != nil {
var we *pairWindowError
if errors.As(err, &we) {
return rr, we
}
return rr, fmt.Errorf("redeem: %w", err) return rr, fmt.Errorf("redeem: %w", err)
} }
defer resp.Body.Close() defer resp.Body.Close()
+66
View File
@@ -0,0 +1,66 @@
package main
// pair_terminal.go: the one question `pair` asks a person, on a local machine
// that already serves a Host: whether to add another.
//
// A local bastion used to serve one Host, full stop, because on a single-user
// machine a second Host appearing meant someone else had got a token onto the
// box. A person who uses two Hosts (a Cloud machine beside the platform one)
// could not pair the second at all. Now the second is added when the person at
// this machine says so, and only then: the answer is read from the controlling
// terminal, never from standard input, so a token piped in by a script or an
// installer (which feeds the pairing code through standard input) cannot add a
// Host by itself. With no terminal there is nobody to ask, and the pair is
// refused.
import (
"bufio"
"errors"
"fmt"
"io"
"strconv"
"strings"
)
// errNoTerminal: this process has no controlling terminal to ask on.
var errNoTerminal = errors.New("no terminal")
// openTerminal opens the controlling terminal for one question: a reader for
// the answer, a writer for the question, and a close. A variable so a test can
// stand in a person; nothing else assigns it.
var openTerminal = openControllingTerminal
// terminalSafeName is a Host name as the question shows it. newHost is the
// handle the token carries, which nothing has checked yet: whoever made the
// token chose it, and a control character or an escape sequence in it could
// rewrite the question on the person's screen. A plain host name is shown as
// it is; anything else is quoted, every such byte escaped.
func terminalSafeName(name string) string {
if validHostName(name) {
return name
}
return strconv.QuoteToASCII(name)
}
// confirmAddHost asks whether to add newHost beside serving, the Host this
// local machine already serves. True only for a typed y or yes.
func confirmAddHost(serving, newHost string) (bool, error) {
in, out, closeTerm, err := openTerminal()
if err != nil {
return false, fmt.Errorf("this computer already serves your Host %s, and adding another needs a yes typed "+
"at this computer, but this run has no terminal to ask in. Run the same command in a terminal "+
"window on this computer; nothing was paired", serving)
}
defer closeTerm()
fmt.Fprintf(out, "This computer already serves your Host %s. Add %s as well? It gets a pairing of its own,\n"+
"and `privatae-bastion ui` shows what each Host may do here. [y/N] ", terminalSafeName(serving), terminalSafeName(newHost))
line, err := bufio.NewReader(io.LimitReader(in, 256)).ReadString('\n')
if err != nil && !errors.Is(err, io.EOF) {
return false, fmt.Errorf("could not read the answer (%v); nothing was paired", err)
}
switch strings.ToLower(strings.TrimSpace(line)) {
case "y", "yes":
return true, nil
}
return false, nil
}
+19
View File
@@ -0,0 +1,19 @@
//go:build !windows
package main
import (
"io"
"os"
)
// openControllingTerminal opens /dev/tty, this process's controlling terminal
// whatever its standard input is. It fails with no terminal (a service, a
// pipe from `ssh host cmd` without -t, cron).
func openControllingTerminal() (io.Reader, io.Writer, func(), error) {
f, err := os.OpenFile("/dev/tty", os.O_RDWR, 0)
if err != nil {
return nil, nil, nil, errNoTerminal
}
return f, f, func() { _ = f.Close() }, nil
}
+26
View File
@@ -0,0 +1,26 @@
//go:build windows
package main
import (
"io"
"os"
)
// openControllingTerminal opens the console this process is attached to,
// CONIN$ for the answer and CONOUT$ for the question, whatever its standard
// input is: install.ps1 pipes the pairing code into `pair -`, and the person
// still answers in the PowerShell window. It fails with no console (a
// service, a scheduled task with no window).
func openControllingTerminal() (io.Reader, io.Writer, func(), error) {
in, err := os.OpenFile("CONIN$", os.O_RDWR, 0)
if err != nil {
return nil, nil, nil, errNoTerminal
}
out, err := os.OpenFile("CONOUT$", os.O_RDWR, 0)
if err != nil {
_ = in.Close()
return nil, nil, nil, errNoTerminal
}
return in, out, func() { _ = in.Close(); _ = out.Close() }, nil
}
+5 -1
View File
@@ -29,7 +29,11 @@ const piiForwardPort = 18081
// The bearer-authed surface is proxied opaquely — this client never // The bearer-authed surface is proxied opaquely — this client never
// holds the PII_API_KEY. // holds the PII_API_KEY.
func piiProbe() (bool, bool) { func piiProbe() (bool, bool) {
a := piiTarget() return piiProbeAt(piiTarget())
}
// piiProbeAt is piiProbe's probe of one address.
func piiProbeAt(a string) (bool, bool) {
if a == "" { if a == "" {
return false, false return false, false
} }
+2 -5
View File
@@ -686,11 +686,8 @@ func handleRootExec(ws *websocket.Conn, spec sessionSpec, self string, env sessi
closed := make(chan struct{}) closed := make(chan struct{})
go func() { go func() {
defer close(closed) defer close(closed)
for { // Each frame is dropped as it arrives (lane_bounds.go), never held whole.
if _, _, err := ws.ReadMessage(); err != nil { _ = drainLane(ws)
return
}
}
}() }()
done := make(chan int, 1) done := make(chan int, 1)
go func() { done <- proc.Wait() }() go func() { done <- proc.Wait() }()
+36 -7
View File
@@ -617,8 +617,11 @@ func runPairing(ctx context.Context, cfg *Config) error {
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+cfg.Bearer) req.Header.Set("Authorization", "Bearer "+cfg.Bearer)
if resp, err := upstream.Do(req); err == nil { if resp, err := upstream.Do(req); err == nil {
body, _ := io.ReadAll(resp.Body) body, ok := readHeartbeatReply(resp)
resp.Body.Close() resp.Body.Close()
if !ok {
return // a reply past its bound is no answer (lane_bounds.go)
}
if resp.StatusCode < 300 { if resp.StatusCode < 300 {
// last_ok per pairing (anchor.go): the anchor rotation's // last_ok per pairing (anchor.go): the anchor rotation's
// silence path reads this — 14 days with no 2xx from the // silence path reads this — 14 days with no 2xx from the
@@ -727,14 +730,19 @@ func unixLaneDialer(cfg *Config) (func(ctx context.Context) (net.Conn, error), *
// parked harness workers, sessions, git, the heartbeat and what its reply // parked harness workers, sessions, git, the heartbeat and what its reply
// directs. Absent, because they listen on the tunnel's stack or move the // directs. Absent, because they listen on the tunnel's stack or move the
// tunnel: the ollama, LLM and PII forwards, the service slots, and endpoint // tunnel: the ollama, LLM and PII forwards, the service slots, and endpoint
// rotation and failover. A unix pairing's coder run has its control plane on // rotation and failover. The model servers the forwards reach are served
// a socket in RunsDir (coder_socket.go), not a tunnel port; a tcp pairing has // over the serve lane instead (servelane.go), by service name. A unix
// no coder runs. // pairing's coder run has its control plane on a socket in RunsDir
// (coder_socket.go), not a tunnel port; a tcp pairing has no coder runs. The
// heartbeat advertises only what runs here (agents.go, beatLanes).
func runUnixPairing(ctx context.Context, cfg *Config) error { func runUnixPairing(ctx context.Context, cfg *Config) error {
logf := func(format string, a ...any) { fmt.Printf("["+cfg.HostName+"] "+format, a...) } logf := func(format string, a ...any) { fmt.Printf("["+cfg.HostName+"] "+format, a...) }
if !cfg.active() { if !cfg.active() {
return errPairingNotActive return errPairingNotActive
} }
// Before any lane: model management for this pairing (modelDaemonTarget)
// reaches only a model server on this machine, as the serve lane does.
defer markLocalPairing(cfg.HostName)()
dialHost, laneTLS, err := localLaneDialer(cfg) dialHost, laneTLS, err := localLaneDialer(cfg)
if err != nil { if err != nil {
return err return err
@@ -762,6 +770,13 @@ func runUnixPairing(ctx context.Context, cfg *Config) error {
logf("egress workers up — outbound exits THIS machine's internet via %s on %s\n", wsURL, lane) logf("egress workers up — outbound exits THIS machine's internet via %s on %s\n", wsURL, lane)
runEgressWorkers(ctx, cfg.HostName, wsURL, laneTLS.Clone(), dialHost, cfg.Bearer, realDialer.DialContext, egressWorkerCount) runEgressWorkers(ctx, cfg.HostName, wsURL, laneTLS.Clone(), dialHost, cfg.Bearer, realDialer.DialContext, egressWorkerCount)
}() }()
// The serve lane: the model servers set on this machine, which the tunnel
// forwards would carry, by service name and never by address.
go func() {
wsURL := fmt.Sprintf("wss://%s:8443/bastion-serve", cfg.HostName)
logf("serve workers up — the model servers set on this machine, by name, via %s on %s\n", wsURL, lane)
runServeWorkers(ctx, cfg.HostName, cfg.dcnDir(), wsURL, laneTLS.Clone(), dialHost, cfg.Bearer, serveWorkerCount)
}()
go func() { go func() {
wsURL := fmt.Sprintf("wss://%s:8443/bastion-mgmt", cfg.HostName) wsURL := fmt.Sprintf("wss://%s:8443/bastion-mgmt", cfg.HostName)
if n := sweepStaleScratch(cfg.contextRoot()); n > 0 { if n := sweepStaleScratch(cfg.contextRoot()); n > 0 {
@@ -841,8 +856,11 @@ func runUnixPairing(ctx context.Context, cfg *Config) error {
if err != nil { if err != nil {
return return
} }
body, _ := io.ReadAll(resp.Body) body, ok := readHeartbeatReply(resp)
resp.Body.Close() resp.Body.Close()
if !ok {
return // a reply past its bound is no answer (lane_bounds.go)
}
if resp.StatusCode < 300 { if resp.StatusCode < 300 {
recordPairingOK(cfg.HostName) recordPairingOK(cfg.HostName)
} }
@@ -890,7 +908,7 @@ func heartbeatPayload(cfg *Config, dev interface{ IpcGet() (string, error) }, ho
for k, v := range pairingEnforcing(cfg.HostName) { for k, v := range pairingEnforcing(cfg.HostName) {
enforcing[k] = v enforcing[k] = v
} }
return buildHeartbeatBody(cfg.HostName, hostname, runtime.GOOS, clientVersion, wgPub, cfg.Fingerprint, cfg.BastionTunnelIP, cfg.Endpoint, return buildHeartbeatBodyOn(beatLanesFor(cfg), cfg.HostName, hostname, runtime.GOOS, clientVersion, wgPub, cfg.Fingerprint, cfg.BastionTunnelIP, cfg.Endpoint,
rotation.isUnsaved(cfg), currentMirrorReport(cfg.HostName), enforcing, rotation.isUnsaved(cfg), currentMirrorReport(cfg.HostName), enforcing,
// Read fresh on every beat: a host paired a minute ago must // Read fresh on every beat: a host paired a minute ago must
// appear as a peer on the NEXT beat, not after a restart. // appear as a peer on the NEXT beat, not after a restart.
@@ -1124,6 +1142,13 @@ func ollamaModels(tunnelIP string) (bool, []string) {
addrs = append(addrs, net.JoinHostPort(tunnelIP, "11434")) addrs = append(addrs, net.JoinHostPort(tunnelIP, "11434"))
} }
addrs = append(addrs, "127.0.0.1:11434") addrs = append(addrs, "127.0.0.1:11434")
return ollamaModelsAt(addrs)
}
// ollamaModelsAt is ollamaModels' probe of the addresses given, in order. A
// local pairing's beat asks it about the one address its serve lane dials
// (servelane.go), so what the beat says is served is what the lane reaches.
func ollamaModelsAt(addrs []string) (bool, []string) {
cl := &http.Client{Timeout: 3 * time.Second} cl := &http.Client{Timeout: 3 * time.Second}
for _, a := range addrs { for _, a := range addrs {
resp, err := cl.Get("http://" + a + "/api/tags") resp, err := cl.Get("http://" + a + "/api/tags")
@@ -1172,7 +1197,11 @@ const llmForwardPort = 18080
// and the operator setting the env IS the opt-in. Same 200-is-alive rule as // and the operator setting the env IS the opt-in. Same 200-is-alive rule as
// /api/tags: a live server with an unparseable list still counts. // /api/tags: a live server with an unparseable list still counts.
func openAICompatModels() (bool, []string) { func openAICompatModels() (bool, []string) {
a := llmTarget() return openAICompatModelsAt(llmTarget())
}
// openAICompatModelsAt is openAICompatModels' probe of one address.
func openAICompatModelsAt(a string) (bool, []string) {
if a == "" { if a == "" {
return false, nil return false, nil
} }
+721
View File
@@ -0,0 +1,721 @@
package main
// The serve lane (/bastion-serve): how a Host with no platform reaches the
// model servers on this machine.
//
// Over the tunnel the Host dials a forward on the tunnel's stack
// (serveOllamaForward and serveLLMForward in run.go, servePIIForward in
// pii.go). A local pairing (a Host on this machine's socket, or in a VM on
// it over TCP) has no tunnel, so those forwards are absent. This lane turns
// the direction round, as egress does: this machine parks worker WebSockets
// on the Host, and the Host hands each one connection.
//
// THE HOST NAMES A SERVICE, NEVER AN ADDRESS. Its first frame on a worker is
// 'S' + {"service":"ollama"}. The address that service reaches is the one set
// on this machine (OLLAMA_ADDR, PRIVATAE_LLM_ADDR, PRIVATAE_PII_ADDR), read by
// the functions the tunnel forwards read, and it must be this machine: a
// server on another computer on the LAN is refused. Nothing the Host sends
// can name, change or widen a target. The bastion answers 'A' and carries
// the connection, or 'E' + a reason and closes, having dialled nothing.
//
// After 'A' the worker is a wsConn ('D' + bytes, 'E' for the end, both ways)
// handed to the forwards' own code: ollama and llm through proxyInference's
// decision and filter (the model scope, /api/pull and /api/delete refused
// without `models`, the list rewrite, its size bounds), pii through the PII
// scope and a pass-through. The `inference` family is checked on every
// connection, as inferenceRefuser does at a forward's accept.
//
// Bounds, per pairing: 16 streams in flight across the services, the 17th
// answered 'E' busy; a stream with no byte either way for 10 minutes is
// closed, on every service; a response's length is not capped, because a
// long generation is the person's own traffic.
import (
"context"
"crypto/tls"
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/gorilla/websocket"
)
const (
// serveWorkerCount workers stay parked per pairing. A replacement is
// dialled as soon as one takes a job, so a long stream never holds up the
// next request.
serveWorkerCount = 2
// serveMaxInFlight is the streams one pairing may hold open here at once.
// It protects this machine and must never be what slows the person down:
// ollama runs one request per model at a time and queues 512 more, and the
// Host's own client holds at most 10 connections per server, so a request
// waits in the server's queue before it waits here.
serveMaxInFlight = 16
// serveHelloMax bounds the 'S' frame. A service name and its JSON fit in a
// fraction of it; anything longer is not a request this lane reads.
serveHelloMax = 1024
// serveTargetLookup bounds resolving a target set by name (localhost:11434).
serveTargetLookup = 2 * time.Second
)
// serveIdleTimeout closes a stream that has carried no byte either way for
// this long. The filter's own read bound (filteredIdleTimeout) covers a request
// being read; this covers the rest: a server that answers headers and then
// nothing, and a pass-through, which has no bound of its own. A var so a test
// can shorten it.
var serveIdleTimeout = filteredIdleTimeout
// serveService is one service the lane carries: its name on the wire, where
// its target is read, which forward's code serves it.
type serveService struct {
name string
envVar string // named in a refusal, so the person knows what to set
target func() string
filter bool // through proxyInference's decision and filter
lane inferenceLane // the filter's lane, when filter
}
// serveServices is every service the lane knows, in the order the beat lists
// them. A name not in it is refused before anything is read or dialled.
var serveServices = []serveService{
{name: "ollama", envVar: "OLLAMA_ADDR", target: localOllamaAddr, filter: true, lane: laneOllama},
{name: "llm", envVar: "PRIVATAE_LLM_ADDR", target: llmTarget, filter: true, lane: laneLLM},
{name: "pii", envVar: "PRIVATAE_PII_ADDR", target: piiTarget},
}
func serveServiceNamed(name string) (serveService, bool) {
for _, s := range serveServices {
if s.name == name {
return s, true
}
}
return serveService{}, false
}
// serveLookupIP resolves a target set by name. A seam so a test can answer.
var serveLookupIP = func(ctx context.Context, host string) ([]net.IPAddr, error) {
return net.DefaultResolver.LookupIPAddr(ctx, host)
}
// serveVetTarget checks the address set on this machine for a service and
// returns the addresses to dial, or the reason it is refused. Every address
// the target names must be this machine: loopback, an address on its
// interfaces, or one the kernel delivers here (ownaddr.go), which takes in a
// server in a container on a network made on this machine. A server on
// another computer is refused: a device the bastion does not run on is the
// device lane's problem, solved there with pinning. 0.0.0.0 and :: are listen
// addresses, and dialling one reaches this machine, so each is dialled as
// loopback of its family. The addresses dialled are the ones checked, never
// the name again.
//
// Every address checked is returned, in the resolver's order, and the dial
// tries them in turn (serveDial, servePick), as the OS dialler does with a
// name: localhost answers ::1 first on most systems, and ollama listens on
// 127.0.0.1 only by default, so dialling the first address alone fails where
// a plain dial of the name reaches it.
func serveVetTarget(target string) ([]string, string) {
host, port, err := net.SplitHostPort(target)
if err != nil {
return nil, fmt.Sprintf("%q is not host:port", target)
}
if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 {
return nil, fmt.Sprintf("%q has no port between 1 and 65535", target)
}
var ips []net.IP
if ip := net.ParseIP(host); ip != nil {
ips = []net.IP{ip}
} else {
ctx, cancel := context.WithTimeout(context.Background(), serveTargetLookup)
as, err := serveLookupIP(ctx, host)
cancel()
if err != nil || len(as) == 0 {
return nil, fmt.Sprintf("%q could not be resolved on this machine", host)
}
for _, a := range as {
ips = append(ips, a.IP)
}
}
for i, ip := range ips {
switch {
case ip.IsUnspecified() && ip.To4() != nil:
ips[i] = net.IPv4(127, 0, 0, 1)
case ip.IsUnspecified():
ips[i] = net.IPv6loopback
}
}
for _, ip := range ips {
if ip.IsLoopback() {
continue
}
own, err := ownDestination([]net.IP{ip})
if err != nil {
return nil, fmt.Sprintf("this machine could not tell whether %s is its own address (%v)", ip, err)
}
if !own {
return nil, fmt.Sprintf("%s is not this machine; the lane reaches only servers on the computer it runs on", ip)
}
}
out := make([]string, 0, len(ips))
for _, ip := range ips {
a := net.JoinHostPort(ip.String(), port)
if !slices.Contains(out, a) {
out = append(out, a)
}
}
return out, ""
}
// serveDialTimeout bounds one dial of a service's addresses, all of them
// together, as the forwards' dial (dialUpstream) bounds its one address.
const serveDialTimeout = 5 * time.Second
// serveDial dials the checked addresses in turn (dialEach: the next starts
// when one fails, or has not answered within the Happy Eyeballs delay) and
// returns the first that answers, with the address it was dialled at. Only
// these addresses are dialled.
func serveDial(addrs []string, timeout time.Duration) (net.Conn, string, error) {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
var mu sync.Mutex
at := map[net.Conn]string{}
d := &net.Dialer{}
c, err := dialEach(ctx, func(ctx context.Context, network, a string) (net.Conn, error) {
c, err := d.DialContext(ctx, network, a)
if err == nil {
mu.Lock()
at[c] = a
mu.Unlock()
}
return c, err
}, addrs)
if err != nil {
return nil, "", err
}
mu.Lock()
defer mu.Unlock()
return c, at[c], nil
}
// servePick is the one address an HTTP client here is given for a target
// whose check returned several (the beat's probes, model management): the
// first that accepts a connection, tried in turn as serveDial does, or the
// first checked when none does, so the request fails as it would have. One
// address is returned as it is, with nothing dialled.
func servePick(addrs []string) string {
if len(addrs) == 1 {
return addrs[0]
}
c, a, err := serveDial(addrs, serveTargetLookup)
if err != nil {
return addrs[0]
}
_ = c.Close()
return a
}
// serveTargetFor is the addresses a service dials here, or why it does not.
// The beat and the lane both ask it, so what the beat says is served is what
// the lane reaches (the rule env.go states for the forwards).
func serveTargetFor(s serveService) ([]string, string) {
t := s.target()
if t == "" {
return nil, "unavailable: no " + s.name + " server is set on this machine (" + s.envVar + ")"
}
addrs, why := serveVetTarget(t)
if why != "" {
return nil, "refused: the " + s.name + " server set on this machine (" + s.envVar + ") is refused: " + why
}
return addrs, ""
}
// serveInFlight counts each pairing's open streams, keyed by host name, so a
// pairing restarted while its streams still run counts them.
var serveInFlight sync.Map // string -> *atomic.Int32
func serveAcquire(self string) bool {
v, _ := serveInFlight.LoadOrStore(self, new(atomic.Int32))
n := v.(*atomic.Int32)
for {
cur := n.Load()
if cur >= serveMaxInFlight {
return false
}
if n.CompareAndSwap(cur, cur+1) {
return true
}
}
}
func serveRelease(self string) {
if v, ok := serveInFlight.Load(self); ok {
v.(*atomic.Int32).Add(-1)
}
}
// serveHello is the 'S' frame's JSON. Fields this build does not know are
// ignored, so a Host may add one without breaking an older bastion.
type serveHello struct {
Service string `json:"service"`
}
// serveVerdict decides one job before anything is dialled: the service, the
// family, the PII scope, the target. It returns the service and the addresses,
// or the reason the job is refused.
func serveVerdict(self, name string) (serveService, []string, string) {
s, ok := serveServiceNamed(name)
if !ok {
return s, nil, "refused: " + strconv.Quote(truncate(name, 64)) + " is not a service this lane carries " +
"(ollama, llm or pii). The lane takes a service's name and reaches the server set on this machine for it; " +
"it never takes an address."
}
if why := capsAllow(self, "inference"); why != "" {
return s, nil, "refused: " + why
}
if s.name == "pii" && !pairingModelScope(self).allowsPII() {
return s, nil, "refused: this pairing is scoped to named models and the PII sterilizer was not one of them " +
"(approve with 'pii' in the model list to share it)"
}
addrs, why := serveTargetFor(s)
if why != "" {
return s, nil, why
}
return s, addrs, ""
}
// serveJob serves the one job a parked worker is given. taken is called once
// the job is accepted (its stream counted, before 'A'), so the pool parks a
// replacement while this one works. It reports whether a job was accepted. A
// refused job (malformed, refused, busy) reports false, and its worker waits
// before it parks again: the replacement is not dialled at once, so a Host
// that answers every worker with a frame it knows is refused cannot make this
// machine dial, handshake and write an audit line in a loop as fast as it
// answers.
func serveJob(ws *websocket.Conn, self, dcnDir string, taken func()) bool {
mt, r, err := ws.NextReader()
if err != nil {
return false
}
refuse := func(service, why string) {
_ = ws.WriteMessage(websocket.BinaryMessage, append([]byte{'E'}, why...))
serveAudit(dcnDir, map[string]any{"service": service, "decision": "refused", "reason": why})
}
raw, err := io.ReadAll(io.LimitReader(r, serveHelloMax+1))
if err != nil {
return false
}
var hello serveHello
if mt != websocket.BinaryMessage || len(raw) == 0 || raw[0] != 'S' || len(raw) > serveHelloMax ||
json.Unmarshal(raw[1:], &hello) != nil {
// Malformed: answered, not audited (as a malformed terminal request is not).
_ = ws.WriteMessage(websocket.BinaryMessage, []byte("Erefused: the first frame on this lane is 'S' and "+
"{\"service\":\"<name>\"}, at most 1024 bytes"))
return false
}
s, addrs, why := serveVerdict(self, hello.Service)
if why != "" {
refuse(truncate(hello.Service, 64), why)
return false
}
if !serveAcquire(self) {
refuse(s.name, fmt.Sprintf("busy: this pairing already has %d streams open on this machine, "+
"which is as many as it may hold; try again when one ends", serveMaxInFlight))
return false
}
defer serveRelease(self)
taken()
if err := ws.WriteMessage(websocket.BinaryMessage, []byte{'A'}); err != nil {
return true
}
w := newServeWatch(serveIdleTimeout)
in := w.wrap(&wsConn{ws: ws})
var reachedMu sync.Mutex
reached := ""
dial := func() (net.Conn, error) {
out, a, err := serveDial(addrs, serveDialTimeout)
if err != nil {
return nil, err
}
reachedMu.Lock()
reached = a
reachedMu.Unlock()
return w.wrap(out), nil
}
if s.filter {
proxyInferenceVia(in, dial, s.lane, self, servePass)
} else {
servePass(in, dial)
}
w.stop()
fromHost, toHost := w.counts()
reachedMu.Lock()
target := reached
reachedMu.Unlock()
if target == "" {
target = strings.Join(addrs, " ") // none answered, or none was dialled
}
serveAudit(dcnDir, map[string]any{"service": s.name, "target": target, "decision": "served",
"bytes_from_host": fromHost, "bytes_to_host": toHost, "idle_closed": w.idleClosed()})
return true
}
// servePass is the lane's pass-through: egress's splice, which keeps a
// half-close (the Host's 'E' ends what it sends, and the answer still comes
// back), where the tunnel's spliceConn ends both directions with the first.
func servePass(in net.Conn, dial upstreamDial) {
out, err := dial()
if err != nil {
_ = in.Close()
return
}
splice(in, in, out)
}
// serveWatch is the idle bound of one stream. Both ends are wrapped; any byte
// read or written on either one counts, and once none has for the bound,
// both are closed, which unblocks whichever copy or read is waiting.
type serveWatch struct {
bound time.Duration
last atomic.Int64 // unix nanoseconds of the last byte
fromHost atomic.Int64
toHost atomic.Int64
mu sync.Mutex
conns []net.Conn
host net.Conn
done chan struct{}
once sync.Once
idled atomic.Bool
}
func newServeWatch(bound time.Duration) *serveWatch {
w := &serveWatch{bound: bound, done: make(chan struct{})}
w.last.Store(time.Now().UnixNano())
go w.run()
return w
}
func (w *serveWatch) run() {
tick := w.bound / 4
if tick < 10*time.Millisecond {
tick = 10 * time.Millisecond
}
if tick > 5*time.Second {
tick = 5 * time.Second
}
t := time.NewTicker(tick)
defer t.Stop()
for {
select {
case <-w.done:
return
case <-t.C:
if time.Since(time.Unix(0, w.last.Load())) >= w.bound {
w.idled.Store(true)
w.mu.Lock()
for _, c := range w.conns {
_ = c.Close()
}
w.mu.Unlock()
return
}
}
}
}
func (w *serveWatch) stop() { w.once.Do(func() { close(w.done) }) }
func (w *serveWatch) idleClosed() bool { return w.idled.Load() }
func (w *serveWatch) counts() (fromHost, toHost int64) { return w.fromHost.Load(), w.toHost.Load() }
// wrap watches c. The first conn wrapped is the Host's: what is read from it
// came from the Host, what is written to it goes back.
func (w *serveWatch) wrap(c net.Conn) net.Conn {
w.mu.Lock()
defer w.mu.Unlock()
if w.host == nil {
w.host = c
}
w.conns = append(w.conns, c)
return &serveWatched{Conn: c, w: w, host: w.host == c}
}
type serveWatched struct {
net.Conn
w *serveWatch
host bool
}
func (c *serveWatched) Read(p []byte) (int, error) {
n, err := c.Conn.Read(p)
if n > 0 {
c.w.last.Store(time.Now().UnixNano())
if c.host {
c.w.fromHost.Add(int64(n))
}
}
return n, err
}
func (c *serveWatched) Write(p []byte) (int, error) {
n, err := c.Conn.Write(p)
if n > 0 {
c.w.last.Store(time.Now().UnixNano())
if c.host {
c.w.toHost.Add(int64(n))
}
}
return n, err
}
// CloseWrite passes a half-close through, so splice can end one direction.
func (c *serveWatched) CloseWrite() error {
if cw, ok := c.Conn.(interface{ CloseWrite() error }); ok {
return cw.CloseWrite()
}
return c.Conn.Close()
}
// serveAudit writes one line per job to the pairing's audit log, the file
// dcnAudit writes, and to the system log.
func serveAudit(dir string, rec map[string]any) {
if dir == "" {
return
}
rec["ts"] = time.Now().UTC().Format(time.RFC3339)
rec["op"] = "serve"
raw, _ := json.Marshal(rec)
auditToSystemLog(auditFilePairingD, auditPairingOfDir(dir), raw)
if err := os.MkdirAll(dir, 0o700); err != nil {
return
}
fh, err := os.OpenFile(filepath.Join(dir, auditFilePairingD), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600)
if err != nil {
return
}
defer fh.Close()
_, _ = fh.Write(append(raw, '\n'))
}
// serveRetryMax is the longest a worker waits before dialling again while the
// Host answers the lane's upgrade and refuses it. A Host that predates the
// lane does that (an unknown WebSocket route is a 404, or a 403 from a server
// that closes it before accepting), and two workers dialling it every 3 s
// for good would be work for nothing on the person's machine. A Host that
// cannot be reached at all (restarting) is dialled every 3 s, as egress is.
var serveRetryMax = time.Minute
// serveRefusedWait is how long a worker waits before it parks again after its
// WebSocket closed while parked, or after its job was refused. With
// serveWorkerCount workers, a Host that refuses or misuses every job costs
// this machine at most that many dials a second.
var serveRefusedWait = time.Second
// localPairings counts the runs of each pairing with a Host that has no
// platform (runUnixPairing), by host name, while they run. A count, not a
// flag, so a run that is still stopping when the same pairing starts again
// cannot unmark the new one.
var localPairings sync.Map // string -> *atomic.Int32
// markLocalPairing records that self runs as a pairing with a Host that has no
// platform, until the returned func is called.
func markLocalPairing(self string) func() {
v, _ := localPairings.LoadOrStore(self, new(atomic.Int32))
n := v.(*atomic.Int32)
n.Add(1)
var once sync.Once
return func() { once.Do(func() { n.Add(-1) }) }
}
// isLocalPairing reports whether self is running as a pairing with a Host that
// has no platform.
func isLocalPairing(self string) bool {
v, ok := localPairings.Load(self)
return ok && v.(*atomic.Int32).Load() > 0
}
// modelDaemonTarget is the address model management (modelswap.go: the
// model-swap probe in the beat, model:list, model:pull and its probes,
// model:drop) dials for the pairing self, or the reason it dials nothing.
// For a Host with no platform it is the serve lane's own ollama target, under
// the same rule: OLLAMA_ADDR (else loopback) only when every address it names
// is this machine, and the address dialled is one of those checked: the first
// that answers, tried in turn (servePick). A bastion serves only the computer
// it runs on, and managing the models on another computer is serving it. A
// tunnel pairing keeps OLLAMA_ADDR wherever it points, as its forwards do.
func modelDaemonTarget(self string) (string, string) {
t := localOllamaAddr()
if !isLocalPairing(self) {
return t, ""
}
addrs, why := serveVetTarget(t)
if why != "" {
s, _ := serveServiceNamed("ollama")
serveTargetNotice(self, s, t, why)
return "", "refused: the model server set on this machine (OLLAMA_ADDR) is not used for this Host: " + why
}
return servePick(addrs), ""
}
// serveTargetNoticed is the last refusal said at the machine for each pairing
// and service, so the console carries it once, and again only when it changes.
var serveTargetNoticed sync.Map // "<self>\x00<service>" -> string
// serveTargetNotice says at this machine, in plain words, that the server set
// here for a service is not used for the Host with no platform, and why. A
// refusal that only the Host hears leaves the person who set the address with
// nothing on their own screen to explain why the model is missing.
func serveTargetNotice(self string, s serveService, target, why string) {
key := self + "\x00" + s.name
// The beat and a model op can refuse the same target at once; only the one
// that records this reason says it.
for {
prev, loaded := serveTargetNoticed.LoadOrStore(key, why)
if !loaded {
break
}
if prev.(string) == why {
return
}
if serveTargetNoticed.CompareAndSwap(key, prev, why) {
break
}
}
fmt.Printf("[%s] %s=%s is not used for the Host on this computer: %s. Set %s to a server on this "+
"computer, or unset it.\n", self, s.envVar, target, why, s.envVar)
}
// serveTargetNoticeClear forgets what was said for a service that is fine
// again, so a later refusal is said again.
func serveTargetNoticeClear(self string, s serveService) {
serveTargetNoticed.Delete(self + "\x00" + s.name)
}
// runServeWorkers keeps n workers parked on the Host's /bastion-serve for the
// pairing self, each serving one job (serveJob). A worker that takes a job
// starts its replacement at once and ends with the job; one whose WebSocket
// dies while parked dials again. pctx is the pairing's context: a revoked
// pairing's workers exit.
func runServeWorkers(pctx context.Context, self, dcnDir, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer string, n int) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
var spawn func()
spawn = func() {
go func() {
wait := 3 * time.Second
for pctx.Err() == nil {
ctx, cancel := context.WithTimeout(pctx, 25*time.Second)
ws, resp, err := dialer.DialContext(ctx, wsURL, http.Header{})
cancel()
if err != nil {
if resp != nil {
sleepCtx(pctx, wait)
wait = min(2*wait, serveRetryMax)
} else {
sleepCtx(pctx, 3*time.Second)
}
continue
}
wait = 3 * time.Second
if err := ws.WriteMessage(websocket.TextMessage, []byte("BEARER "+bearer)); err != nil {
_ = ws.Close()
sleepCtx(pctx, time.Second)
continue
}
stopKA := armWSKeepalive(ws)
unpark := context.AfterFunc(pctx, func() { _ = ws.Close() })
took := serveJob(ws, self, dcnDir, spawn)
unpark()
stopKA()
_ = ws.Close()
if took {
return
}
// Closed while parked (a Host restart), or the job refused: not
// straight back, so a Host that closes or misuses every worker is
// not dialled in a loop.
sleepCtx(pctx, serveRefusedWait)
}
}()
}
for i := 0; i < n; i++ {
spawn()
}
}
// laneServeReport is what a local pairing's beat says of its serve lane.
type laneServeReport struct {
served []string // lane_serves
models []string // ollama's, filtered to the scope
llmModels []string // the llm server's, filtered to the scope
piiOK bool
}
// laneServes is what this pairing serves over the lane right now, for the
// beat: each service whose target is set and is this machine, whose server
// answers its probe, and whose family the pairing holds. A scoped pairing
// with no model left in a server's list is not served that server, as the
// tunnel's beat withholds a lane with nothing in it.
func laneServes(self string) laneServeReport {
rep := laneServeReport{served: []string{}, models: []string{}, llmModels: []string{}}
if capsAllow(self, "inference") != "" {
return rep
}
scope := pairingModelScope(self)
for _, s := range serveServices {
t := s.target()
if t == "" {
continue
}
addrs, why := serveVetTarget(t)
if why != "" {
serveTargetNotice(self, s, t, why)
continue
}
serveTargetNoticeClear(self, s)
addr := servePick(addrs)
switch s.name {
case "ollama":
ok, models := ollamaModelsAt([]string{addr})
models = scope.filterModels(models)
if !ok || (scope.scoped && len(models) == 0) {
continue
}
if models != nil {
rep.models = models
}
case "llm":
ok, models := openAICompatModelsAt(addr)
models = scope.filterModels(models)
if !ok || (scope.scoped && len(models) == 0) {
continue
}
if models != nil {
rep.llmModels = models
}
case "pii":
if !scope.allowsPII() {
continue
}
if _, healthy := piiProbeAt(addr); !healthy {
continue
}
rep.piiOK = true
}
rep.served = append(rep.served, s.name)
}
return rep
}
+131
View File
@@ -0,0 +1,131 @@
//go:build linux
// The serve lane dials a target set by name at each address it checked, in
// turn (servelane.go: serveVetTarget, serveDial, servePick). Uses the fixtures
// of servelane_linux_test.go, modelscope_test.go and modelswap_test.go.
package main
import (
"bufio"
"context"
"fmt"
"io"
"net"
"net/http"
"slices"
"strings"
"testing"
"time"
)
// A target set by name is dialled at every address it was checked for, in
// turn, as a plain dial of the name is. OLLAMA_ADDR=localhost:<port>, a
// resolver that answers ::1 first (as most do) and a server listening on
// 127.0.0.1 only (ollama's default): a job on the lane, the beat and model
// management each reach it. The resolver's two addresses are both this
// machine's and are the only ones there are to dial.
//
// MUTATION: serveVetTarget returns only the first address it checked (as it
// did), or serveJob dials only addrs[0].
// RESULT: red — the job's request gets no answer (its one dial, to [::1], is
// refused), the beat has no ollama in lane_serves, and model management is
// handed [::1].
func TestTheServeLaneDialsEachCheckedAddressInTurn(t *testing.T) {
up := newFakeOllama(t)
listening := up.srv.Listener.Addr().String()
if !strings.HasPrefix(listening, "127.0.0.1:") {
t.Fatalf("the stand-in listens on %s, not 127.0.0.1 only, so this case proves nothing", listening)
}
port := portOf(up.srv.Listener)
if c, err := net.DialTimeout("tcp", net.JoinHostPort("::1", port), time.Second); err == nil {
_ = c.Close()
t.Fatalf("something answers on [::1]:%s, so the first address would not fail and this case proves nothing", port)
}
old := serveLookupIP
t.Cleanup(func() { serveLookupIP = old })
serveLookupIP = func(_ context.Context, host string) ([]net.IPAddr, error) {
if host != "localhost" {
return nil, fmt.Errorf("no such host")
}
return []net.IPAddr{{IP: net.IPv6loopback}, {IP: net.ParseIP("127.0.0.1")}}, nil
}
t.Setenv("OLLAMA_ADDR", "localhost:"+port)
t.Setenv("PRIVATAE_LLM_ADDR", "")
t.Setenv("PRIVATAE_PII_ADDR", "")
resetModelDaemonCache()
h, cfg := serveLaneBox(t, "dual.example.test")
runInBackground(t, cfg)
if got := strsOf(h.beat(t)["lane_serves"]); !slices.Contains(got, "ollama") {
t.Errorf("the beat does not say ollama is served: lane_serves %q", got)
}
ws := h.take(t)
if got := ask(t, ws, "ollama"); got != "A" {
t.Fatalf("the lane must accept ollama: %q", got)
}
conn := laneConn(ws)
req, _ := http.NewRequest(http.MethodGet, "http://serve/api/tags", nil)
req.Close = true
if err := req.Write(conn); err != nil {
t.Fatal(err)
}
_ = conn.SetReadDeadline(time.Now().Add(10 * time.Second))
resp, err := http.ReadResponse(bufio.NewReader(conn), req)
if err != nil {
t.Fatalf("no answer over the lane (only [::1] was dialled?): %v", err)
}
b, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK || !strings.Contains(string(b), "qwen4-embed") {
t.Fatalf("over the lane: %d %s", resp.StatusCode, b)
}
if addr, why := modelDaemonTarget(cfg.HostName); addr != listening || why != "" {
t.Errorf("model management is handed %q (%q), want the address that answers, %s", addr, why, listening)
}
out, errOut, code, _ := driveModelOp(t, cfg.HostName, "model:list", nil, t.TempDir())
if code != 0 || !strings.Contains(out, "qwen4-embed") {
t.Errorf("model:list: exit %d, %q %q", code, out, errOut)
}
for _, s := range up.requests() {
if !strings.HasPrefix(s, "GET ") {
t.Errorf("the stand-in was asked %q", s)
}
}
}
// servePick returns the first checked address that accepts, and the first
// checked when none does; it never returns an address it was not given.
func TestServePickTakesTheFirstCheckedAddressThatAnswers(t *testing.T) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer ln.Close()
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
_ = c.Close()
}
}()
port := portOf(ln)
v6 := net.JoinHostPort("::1", port)
if c, err := net.DialTimeout("tcp", v6, time.Second); err == nil {
_ = c.Close()
t.Fatalf("something answers on %s, so this case proves nothing", v6)
}
if got := servePick([]string{v6, ln.Addr().String()}); got != ln.Addr().String() {
t.Errorf("pick %q, want %s", got, ln.Addr())
}
_ = ln.Close()
if got := servePick([]string{v6, "127.0.0.1:" + port}); got != v6 {
t.Errorf("with nothing answering, pick %q, want the first checked, %s", got, v6)
}
if got := servePick([]string{"127.0.0.1:" + port}); got != "127.0.0.1:"+port {
t.Errorf("one address: %q", got)
}
}
+992
View File
@@ -0,0 +1,992 @@
//go:build linux
// The serve lane (servelane.go) over a tcp pairing: a Host in a VM on this
// machine reaches the model servers set here by service name, through the
// forwards' own filter, within the lane's bounds, and the pairing's beat says
// what it serves.
//
// Uses the fixtures of local_host_linux_test.go and local_host_tcp_linux_test.go
// (newTestCA, tcpPairingFor, laneListener, pairingBox...), so it carries the
// same build tag.
package main
import (
"bufio"
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"fmt"
"io"
"log"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/gorilla/websocket"
)
// ── fixtures ────────────────────────────────────────────────────────────────
// serveHost is a Host's lane port as the serve lane meets it: the lane
// port's mutual TLS, every worker route kept parked, each beat recorded
// whole, and each /bastion-serve worker handed to the test once it has sent
// its hello.
type serveHost struct {
mu sync.Mutex
beats []map[string]any
hellos []string
workers chan *websocket.Conn
}
func startServeHost(t *testing.T, raw net.Listener, cert tls.Certificate, clientCA *testCA) *serveHost {
t.Helper()
h := &serveHost{workers: make(chan *websocket.Conn, 64)}
pool := x509.NewCertPool()
pool.AddCert(clientCA.cert)
ln := tls.NewListener(raw, &tls.Config{Certificates: []tls.Certificate{cert}, ClientCAs: pool,
ClientAuth: tls.RequireAndVerifyClientCert})
up := websocket.Upgrader{}
mux := http.NewServeMux()
mux.HandleFunc("/api/network/bastions/heartbeat", func(w http.ResponseWriter, r *http.Request) {
var m map[string]any
_ = json.NewDecoder(r.Body).Decode(&m)
h.mu.Lock()
h.beats = append(h.beats, m)
h.mu.Unlock()
_, _ = w.Write([]byte("{}"))
})
mux.HandleFunc("/bastion-serve", func(w http.ResponseWriter, r *http.Request) {
c, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
_, hello, err := c.ReadMessage()
if err != nil {
c.Close()
return
}
h.mu.Lock()
h.hellos = append(h.hellos, string(hello))
h.mu.Unlock()
h.workers <- c // the test owns it from here
})
park := func(w http.ResponseWriter, r *http.Request) {
c, err := up.Upgrade(w, r, nil)
if err != nil {
return
}
defer c.Close()
for {
if _, _, err := c.ReadMessage(); err != nil {
return
}
}
}
for _, p := range []string{"/bastion-harness", "/bastion-mgmt", "/bastion-egress"} {
mux.HandleFunc(p, park)
}
srv := &http.Server{Handler: mux, ErrorLog: log.New(io.Discard, "", 0)}
go func() { _ = srv.Serve(ln) }()
t.Cleanup(func() {
_ = srv.Close()
for {
select {
case c := <-h.workers:
c.Close()
default:
return
}
}
})
return h
}
// take is the next parked serve worker.
func (h *serveHost) take(t *testing.T) *websocket.Conn {
t.Helper()
select {
case c := <-h.workers:
t.Cleanup(func() { c.Close() })
return c
case <-time.After(10 * time.Second):
t.Fatal("no /bastion-serve worker parked on the Host within 10 s")
}
return nil
}
// ask sends the 'S' frame for service and returns the bastion's answer: "A",
// or "E" and the reason.
func ask(t *testing.T, c *websocket.Conn, service string) string {
t.Helper()
b, _ := json.Marshal(map[string]string{"service": service})
return askRaw(t, c, websocket.BinaryMessage, append([]byte{'S'}, b...))
}
func askRaw(t *testing.T, c *websocket.Conn, mt int, frame []byte) string {
t.Helper()
if err := c.WriteMessage(mt, frame); err != nil {
t.Fatalf("send the first frame: %v", err)
}
_ = c.SetReadDeadline(time.Now().Add(10 * time.Second))
_, m, err := c.ReadMessage()
_ = c.SetReadDeadline(time.Time{})
if err != nil {
t.Fatalf("no answer to the first frame: %v", err)
}
return string(m)
}
// beat is the first beat the Host received, waiting up to 10 s.
func (h *serveHost) beat(t *testing.T) map[string]any {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
for time.Now().Before(deadline) {
h.mu.Lock()
if len(h.beats) > 0 {
b := h.beats[0]
h.mu.Unlock()
return b
}
h.mu.Unlock()
time.Sleep(20 * time.Millisecond)
}
t.Fatal("no heartbeat reached the Host within 10 s")
return nil
}
// runInBackground runs the pairing until the test ends.
func runInBackground(t *testing.T, cfg *Config) {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- runPairing(ctx, cfg) }()
t.Cleanup(func() {
cancel()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Error("the pairing did not stop")
}
// A stopped (revoked) pairing's streams end with it: each one's worker
// is closed. Waited for here too because a stream's last act is its
// audit line, in the pairing's directory, which the test then removes.
deadline := time.Now().Add(5 * time.Second)
for {
v, ok := serveInFlight.Load(cfg.HostName)
if !ok || v.(*atomic.Int32).Load() == 0 {
return
}
if time.Now().After(deadline) {
t.Errorf("%d streams still open 5 s after the pairing stopped", v.(*atomic.Int32).Load())
return
}
time.Sleep(10 * time.Millisecond)
}
})
}
// serveLaneBox is an unanchored local machine with a tcp pairing to a
// stand-in Host on 127.0.0.1, running. host keeps each test's pairing (and its
// stream count) its own.
func serveLaneBox(t *testing.T, host string) (*serveHost, *Config) {
t.Helper()
pairingBox(t)
laneTestProcess(t)
ca := newTestCA(t, "home host CA")
ln := laneListener(t)
h := startServeHost(t, ln, ca.serverCert(t, tcpServerName), ca)
cfg := tcpPairingFor(t, ca, host, ln.Addr().String())
if err := cfg.save(); err != nil {
t.Fatal(err)
}
return h, cfg
}
// laneConn is the Host's end of an accepted job: 'D' frames both ways, 'E'
// for the end, as the Host's relay frames a spliced connection.
func laneConn(c *websocket.Conn) net.Conn { return &wsConn{ws: c} }
// chatRequest writes one streaming chat request on conn and returns the reader
// its response is read from.
func chatRequest(t *testing.T, conn net.Conn, model string) (*http.Response, *http.Request) {
t.Helper()
body := fmt.Sprintf(`{"model":%q,"stream":true,"messages":[{"role":"user","content":"hello"}]}`, model)
req, _ := http.NewRequest(http.MethodPost, "http://serve/v1/chat/completions", strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
if err := req.Write(conn); err != nil {
t.Fatalf("write the request: %v", err)
}
_ = conn.SetReadDeadline(time.Now().Add(15 * time.Second))
resp, err := http.ReadResponse(bufio.NewReader(conn), req)
if err != nil {
t.Fatalf("read the response: %v", err)
}
return resp, req
}
// ── 1. a chat streams ───────────────────────────────────────────────────────
// streamingServer is a stand-in OpenAI-compatible server. Its chat answer is
// three chunks; after the first it waits until the test says the Host has
// read it, and records whether it had to give up waiting.
type streamingServer struct {
srv *httptest.Server
sawFirst chan struct{}
gated atomic.Bool
seen atomic.Int32
}
func newStreamingServer(t *testing.T) *streamingServer {
t.Helper()
s := &streamingServer{sawFirst: make(chan struct{})}
s.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.seen.Add(1)
switch r.URL.Path {
case "/v1/chat/completions":
case "/api/tags":
_, _ = w.Write([]byte(`{"models":[{"name":"test-model:latest"}]}`))
return
default:
_, _ = w.Write([]byte(`{"ok":true}`))
return
}
_, _ = io.Copy(io.Discard, r.Body)
w.Header().Set("Content-Type", "text/event-stream")
fl := w.(http.Flusher)
_, _ = io.WriteString(w, "data: {\"chunk\":1}\n\n")
fl.Flush()
select {
case <-s.sawFirst:
s.gated.Store(true)
case <-time.After(10 * time.Second):
}
_, _ = io.WriteString(w, "data: {\"chunk\":2}\n\n")
fl.Flush()
_, _ = io.WriteString(w, "data: {\"chunk\":3}\n\n")
fl.Flush()
_, _ = io.WriteString(w, "data: [DONE]\n\n")
}))
t.Cleanup(s.srv.Close)
return s
}
// SECURITY (and the lane's purpose): over a tcp pairing, the Host asks for a
// service by name and a chat streams from the server set here. The Host reads
// chunk 1 before the server writes chunk 3, so nothing in the path holds a
// response back: not the lane's framing, not the pass-through an unscoped
// pairing gets, not the filter a pairing without `models` gets.
//
// MUTATION: drop the serve workers from runUnixPairing.
// RESULT: red — no /bastion-serve worker ever parks on the Host.
func TestTheServeLaneStreamsAChatOverATCPPairing(t *testing.T) {
for _, c := range []struct {
name, service, env string
modelsOff bool
}{
{"llm, passed through", "llm", "PRIVATAE_LLM_ADDR", false},
{"ollama, through the filter", "ollama", "OLLAMA_ADDR", true},
} {
t.Run(c.name, func(t *testing.T) {
s := newStreamingServer(t)
t.Setenv(c.env, s.srv.Listener.Addr().String())
h, cfg := serveLaneBox(t, "stream-"+c.service+".example.test")
if c.modelsOff {
if err := localDenySet(cfg.HostName, "models", true); err != nil {
t.Fatal(err)
}
if !inferenceFilterNeeded(cfg.HostName, laneOllama, pairingModelScope(cfg.HostName)) {
t.Fatal("with models off the ollama lane must be filtered, or this case proves nothing")
}
}
runInBackground(t, cfg)
ws := h.take(t)
if got := ask(t, ws, c.service); got != "A" {
t.Fatalf("the lane must accept %s: %q", c.service, got)
}
conn := laneConn(ws)
resp, _ := chatRequest(t, conn, "test-model")
if resp.StatusCode != http.StatusOK {
t.Fatalf("the chat answered %d", resp.StatusCode)
}
br := bufio.NewReader(resp.Body)
var got []string
first := false
for {
line, err := br.ReadString('\n')
if strings.HasPrefix(line, "data: ") {
got = append(got, strings.TrimSpace(line))
if !first && strings.Contains(line, `"chunk":1`) {
first = true
close(s.sawFirst)
}
}
if err != nil || strings.Contains(line, "[DONE]") {
break
}
}
if !first {
t.Fatalf("the stream never carried chunk 1: %q", got)
}
if !s.gated.Load() {
t.Fatal("the server wrote chunk 3 before the Host read chunk 1: the lane held the stream back")
}
want := []string{`data: {"chunk":1}`, `data: {"chunk":2}`, `data: {"chunk":3}`, `data: [DONE]`}
if !slices.Equal(got, want) {
t.Fatalf("the stream arrived as %q", got)
}
})
}
}
// ── 2. a service, never an address ──────────────────────────────────────────
// SECURITY: the Host names a service and the target is the one set on this
// machine. A first frame naming an address, a name the lane does not carry, a
// frame that is not 'S', a target set to a computer that is not this one, and
// a service with no target each answer 'E' with a reason, and nothing is
// dialled: a listener at the address named gets no connection.
//
// MUTATION: in serveVetTarget, skip the ownDestination check.
// RESULT: red — ollama with OLLAMA_ADDR on another computer is accepted.
func TestTheServeLaneTakesAServiceNameNeverAnAddress(t *testing.T) {
named, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer named.Close()
var reached atomic.Int32
go func() {
for {
c, err := named.Accept()
if err != nil {
return
}
reached.Add(1)
c.Close()
}
}()
t.Setenv("OLLAMA_ADDR", "192.0.2.10:11434") // TEST-NET-1: never this machine
t.Setenv("PRIVATAE_LLM_ADDR", "")
t.Setenv("PRIVATAE_PII_ADDR", "198.51.100.7:"+portOf(named))
h, cfg := serveLaneBox(t, "names.example.test")
runInBackground(t, cfg)
for _, c := range []struct {
name string
frame []byte
mt int
want []string
}{
{"an address for a service", sFrame(named.Addr().String()), websocket.BinaryMessage,
[]string{"Erefused: ", "never takes an address"}},
{"a name the lane does not carry", sFrame("nas"), websocket.BinaryMessage,
[]string{"Erefused: ", `"nas" is not a service`}},
{"a URL for a service", sFrame("http://" + named.Addr().String() + "/api/tags"), websocket.BinaryMessage,
[]string{"Erefused: ", "is not a service"}},
{"a frame that is not S", []byte("D" + "GET / HTTP/1.1\r\n\r\n"), websocket.BinaryMessage,
[]string{"Erefused: ", "the first frame on this lane is 'S'"}},
{"a text frame", []byte(`S{"service":"ollama"}`), websocket.TextMessage,
[]string{"Erefused: ", "the first frame on this lane is 'S'"}},
{"an S frame past its bound", append(sFrame("ollama"), make([]byte, serveHelloMax)...), websocket.BinaryMessage,
[]string{"Erefused: ", "at most 1024 bytes"}},
{"ollama set to another computer", sFrame("ollama"), websocket.BinaryMessage,
[]string{"Erefused: ", "OLLAMA_ADDR", "192.0.2.10 is not this machine"}},
{"pii set to another computer", sFrame("pii"), websocket.BinaryMessage,
[]string{"Erefused: ", "PRIVATAE_PII_ADDR", "198.51.100.7 is not this machine"}},
{"llm with nothing set", sFrame("llm"), websocket.BinaryMessage,
[]string{"Eunavailable: ", "PRIVATAE_LLM_ADDR"}},
} {
ws := h.take(t)
got := askRaw(t, ws, c.mt, c.frame)
for _, w := range c.want {
if !strings.Contains(got, w) {
t.Errorf("%s: the answer %q does not carry %q", c.name, got, w)
}
}
if !strings.HasPrefix(got, "E") {
t.Errorf("%s: answered %q, not E", c.name, got)
}
// Refused means closed: nothing after the E.
_ = ws.SetReadDeadline(time.Now().Add(3 * time.Second))
if _, m, err := ws.ReadMessage(); err == nil {
t.Errorf("%s: the worker carried on after its refusal: %q", c.name, m)
}
}
time.Sleep(200 * time.Millisecond)
if n := reached.Load(); n != 0 {
t.Fatalf("a refused job dialled the address it named (%d connections)", n)
}
}
func sFrame(service string) []byte {
b, _ := json.Marshal(map[string]string{"service": service})
return append([]byte{'S'}, b...)
}
func portOf(ln net.Listener) string {
_, p, _ := net.SplitHostPort(ln.Addr().String())
return p
}
// A target set by name is resolved and each address checked; the addresses
// dialled are the ones checked, every one of them, in the resolver's order.
// 0.0.0.0 is a listen address and dials as loopback (once, if loopback is
// there already). This machine's own interface address passes.
func TestServeVetTargetChecksEveryAddressItWouldDial(t *testing.T) {
old := serveLookupIP
t.Cleanup(func() { serveLookupIP = old })
serveLookupIP = func(_ context.Context, host string) ([]net.IPAddr, error) {
switch host {
case "localhost":
return []net.IPAddr{{IP: net.ParseIP("127.0.0.1")}, {IP: net.ParseIP("::1")}}, nil
case "v6first.example.test":
return []net.IPAddr{{IP: net.ParseIP("::1")}, {IP: net.ParseIP("127.0.0.1")}, {IP: net.ParseIP("0.0.0.0")}}, nil
case "mixed.example.test":
return []net.IPAddr{{IP: net.ParseIP("127.0.0.1")}, {IP: net.ParseIP("192.0.2.10")}}, nil
}
return nil, fmt.Errorf("no such host")
}
for target, want := range map[string][]string{
"127.0.0.1:11434": {"127.0.0.1:11434"},
"[::1]:8080": {"[::1]:8080"},
"0.0.0.0:11434": {"127.0.0.1:11434"},
"[::]:11434": {"[::1]:11434"},
"localhost:11434": {"127.0.0.1:11434", "[::1]:11434"},
"v6first.example.test:80": {"[::1]:80", "127.0.0.1:80"},
"mixed.example.test:11434": nil,
"nowhere.example.test:80": nil,
"192.0.2.10:11434": nil,
"198.51.100.9:80": nil,
"127.0.0.1": nil,
"127.0.0.1:0": nil,
"127.0.0.1:99999": nil,
} {
got, why := serveVetTarget(target)
if !slices.Equal(got, want) || (want == nil) != (why != "") {
t.Errorf("%q: dial %q (%q), want %q", target, got, why, want)
}
}
ips, err := ownInterfaceIPs()
if err != nil {
t.Fatal(err)
}
for _, ip := range ips {
if ip.IsLoopback() || ip.IsLinkLocalUnicast() {
continue
}
a := net.JoinHostPort(ip.String(), "11434")
if got, why := serveVetTarget(a); !slices.Equal(got, []string{a}) {
t.Errorf("this machine's own address %s is refused: %q", a, why)
}
}
}
// ── 3. the forwards' walls, unchanged ───────────────────────────────────────
// SECURITY: the lane keeps every wall the tunnel forwards keep. With
// `inference` switched off at this machine, a job is refused before any dial.
// With `models` off, /api/pull and /api/delete are refused 403 and never reach
// ollama, while a chat does. On an anchored machine they are refused to the
// anchor itself. A pairing scoped to a model is refused another one with 403,
// which never reaches the server, and is served its own.
func TestTheServeLaneKeepsTheForwardsWalls(t *testing.T) {
t.Run("inference off at the machine", func(t *testing.T) {
// A raw listener, so a dial by the serve job is seen even though the
// Host sends it nothing: a connection that does not open with a beat's
// probe (GET /api/tags, which any pairing's beat in this process may
// send to OLLAMA_ADDR) is the job's.
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { ln.Close() })
var jobDials atomic.Int32
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
defer c.Close()
_ = c.SetReadDeadline(time.Now().Add(2 * time.Second))
b := make([]byte, len("GET /api/tags "))
if _, err := io.ReadFull(c, b); err != nil || string(b) != "GET /api/tags " {
jobDials.Add(1)
}
}()
}
}()
t.Setenv("OLLAMA_ADDR", ln.Addr().String())
h, cfg := serveLaneBox(t, "walls-off.example.test")
if err := localDenySet(cfg.HostName, "inference", true); err != nil {
t.Fatal(err)
}
runInBackground(t, cfg)
got := ask(t, h.take(t), "ollama")
if !strings.HasPrefix(got, "Erefused: ") {
t.Fatalf("with inference off the job must be refused: %q", got)
}
time.Sleep(2500 * time.Millisecond)
if n := jobDials.Load(); n != 0 {
t.Fatalf("a refused job dialled ollama (%d connections)", n)
}
})
pullAndDelete := func(t *testing.T, h *serveHost, up *fakeOllama) {
t.Helper()
for _, p := range []string{"/api/pull", "/api/delete"} {
ws := h.take(t)
if got := ask(t, ws, "ollama"); got != "A" {
t.Fatalf("%s: the job must be accepted, and refused by the filter: %q", p, got)
}
conn := laneConn(ws)
req, _ := http.NewRequest(http.MethodPost, "http://serve"+p, strings.NewReader(`{"model":"test-model"}`))
if err := req.Write(conn); err != nil {
t.Fatal(err)
}
_ = conn.SetReadDeadline(time.Now().Add(10 * time.Second))
resp, err := http.ReadResponse(bufio.NewReader(conn), req)
if err != nil {
t.Fatalf("%s: %v", p, err)
}
b, _ := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusForbidden || !strings.Contains(string(b), "models") {
t.Fatalf("%s over the lane: %d %s", p, resp.StatusCode, b)
}
}
for _, s := range up.requests() {
if strings.Contains(s, "/api/pull") || strings.Contains(s, "/api/delete") {
t.Fatalf("a refused store request reached ollama: %q", up.requests())
}
}
}
t.Run("models off at the machine", func(t *testing.T) {
up := newFakeOllama(t)
t.Setenv("OLLAMA_ADDR", up.srv.Listener.Addr().String())
h, cfg := serveLaneBox(t, "walls-models.example.test")
if err := localDenySet(cfg.HostName, "models", true); err != nil {
t.Fatal(err)
}
runInBackground(t, cfg)
pullAndDelete(t, h, up)
})
t.Run("an anchored machine, as its anchor", func(t *testing.T) {
up := newFakeOllama(t)
t.Setenv("OLLAMA_ADDR", up.srv.Listener.Addr().String())
h, cfg := serveLaneBox(t, "walls-anchor.example.test")
p, _ := modePath()
if err := os.WriteFile(p, []byte(`{"mode":"networked"}`), 0o600); err != nil {
t.Fatal(err)
}
if err := writeAnchor("Acme IT", cfg.HostName); err != nil {
t.Fatal(err)
}
if currentAnchor() == nil || capsAllow(cfg.HostName, "inference") != "" {
t.Fatal("the pairing must be this machine's anchor, holding inference, or this case proves nothing")
}
runInBackground(t, cfg)
pullAndDelete(t, h, up)
})
t.Run("a pairing scoped to one model", func(t *testing.T) {
priv := setupBox(t)
withMarkerAt(t, filepath.Join(t.TempDir(), "machine.json"))
laneTestProcess(t)
up := newFakeOllama(t)
t.Setenv("PRIVATAE_LLM_ADDR", up.srv.Listener.Addr().String())
ca := newTestCA(t, "home host CA")
ln := laneListener(t)
h := startServeHost(t, ln, ca.serverCert(t, tcpServerName), ca)
member := tcpPairingFor(t, ca, tMember, ln.Addr().String())
member.State = pairingPending
if err := member.save(); err != nil {
t.Fatal(err)
}
syncGrant(t, priv, tMember, []string{"inference"}, []string{"qwen4-embed"})
cfgs, err := loadPairings()
if err != nil {
t.Fatal(err)
}
cfg := pairingFor(cfgs, tMember)
if cfg == nil || !cfg.active() || !cfg.tcpTransport() {
t.Fatalf("the member's tcp pairing must be active after the grant: %+v", cfg)
}
runInBackground(t, cfg)
for model, want := range map[string]int{"llama3:70b": http.StatusForbidden, "qwen4-embed": http.StatusOK} {
ws := h.take(t)
if got := ask(t, ws, "llm"); got != "A" {
t.Fatalf("the member holds inference, so the job is accepted: %q", got)
}
resp, _ := chatRequest(t, laneConn(ws), model)
b, _ := io.ReadAll(resp.Body)
if resp.StatusCode != want {
t.Fatalf("%s over the scoped lane: %d %s", model, resp.StatusCode, b)
}
}
chats := 0
for _, s := range up.requests() {
if s == "POST /v1/chat/completions" {
chats++
}
}
if chats != 1 {
t.Fatalf("only the scoped model's request may reach the server: %q", up.requests())
}
})
}
// ── 4. the bounds ───────────────────────────────────────────────────────────
// SECURITY: one pairing holds at most 16 streams open here. Sixteen run at
// once; the 17th is answered 'E' busy with the reason and dials nothing; once
// one ends, the next is accepted.
func TestTheServeLaneHoldsSixteenStreamsAndRefusesTheSeventeenth(t *testing.T) {
var held, opened atomic.Int32
release := make(chan struct{})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/chat/completions" {
_, _ = w.Write([]byte(`{}`)) // the beat's probe
return
}
opened.Add(1)
held.Add(1)
defer held.Add(-1)
w.Header().Set("Content-Type", "text/event-stream")
_, _ = io.WriteString(w, "data: {\"chunk\":1}\n\n")
w.(http.Flusher).Flush()
select {
case <-release:
case <-r.Context().Done():
}
}))
t.Cleanup(func() { close(release); srv.Close() })
t.Setenv("PRIVATAE_LLM_ADDR", srv.Listener.Addr().String())
h, cfg := serveLaneBox(t, "sixteen.example.test")
runInBackground(t, cfg)
var streams []*websocket.Conn
for i := 0; i < serveMaxInFlight; i++ {
ws := h.take(t)
if got := ask(t, ws, "llm"); got != "A" {
t.Fatalf("stream %d of %d must be accepted: %q", i+1, serveMaxInFlight, got)
}
resp, _ := chatRequest(t, laneConn(ws), "test-model")
if resp.StatusCode != http.StatusOK {
t.Fatalf("stream %d: %d", i+1, resp.StatusCode)
}
streams = append(streams, ws)
}
if n := held.Load(); n != serveMaxInFlight {
t.Fatalf("the server holds %d streams, want %d at once", n, serveMaxInFlight)
}
got := ask(t, h.take(t), "llm")
if !strings.HasPrefix(got, "Ebusy: ") || !strings.Contains(got, "16 streams") {
t.Fatalf("the 17th stream must be answered busy with the reason: %q", got)
}
if n := opened.Load(); n != serveMaxInFlight {
t.Fatalf("the refused 17th reached the server (%d requests)", n)
}
// One ends; the next is accepted.
streams[0].Close()
deadline := time.Now().Add(10 * time.Second)
for {
ws := h.take(t)
got := ask(t, ws, "llm")
if got == "A" {
break
}
if !strings.HasPrefix(got, "Ebusy: ") || time.Now().After(deadline) {
t.Fatalf("after a stream ended the next must be accepted: %q", got)
}
time.Sleep(50 * time.Millisecond)
}
}
// SECURITY: a stream with no byte either way for the idle bound is closed,
// both ends, on a pass-through and through the filter: a server that answers
// headers and then nothing does not hold the stream (or this machine's slot)
// for ever. A stream that keeps sending, for several times the bound, is not
// cut: the bound is on silence, never on a response's length.
func TestTheServeLaneClosesAStreamIdleForTheBound(t *testing.T) {
old := serveIdleTimeout
serveIdleTimeout = 400 * time.Millisecond
t.Cleanup(func() { serveIdleTimeout = old })
// silent answers headers and then holds the connection, recording when
// this machine closes it.
silentClosed := make(chan time.Time, 4)
silent, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { silent.Close() })
go func() {
for {
c, err := silent.Accept()
if err != nil {
return
}
go func(c net.Conn) {
defer c.Close()
br := bufio.NewReader(c)
req, err := http.ReadRequest(br)
if err != nil {
return
}
if req.URL.Path != "/v1/chat/completions" {
// The beat's probes get an ordinary answer.
_, _ = io.WriteString(c, "HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\n{}")
return
}
_, _ = io.WriteString(c, "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nTransfer-Encoding: chunked\r\n\r\n")
_, _ = io.Copy(io.Discard, br)
silentClosed <- time.Now()
}(c)
}
}()
// steady sends a chunk every 100 ms for 1.6 s: four times the bound.
steady := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/chat/completions" {
_, _ = w.Write([]byte(`{}`))
return
}
w.Header().Set("Content-Type", "text/event-stream")
for i := 1; i <= 16; i++ {
fmt.Fprintf(w, "data: {\"chunk\":%d}\n\n", i)
w.(http.Flusher).Flush()
time.Sleep(100 * time.Millisecond)
}
_, _ = io.WriteString(w, "data: [DONE]\n\n")
}))
t.Cleanup(steady.Close)
for _, c := range []struct {
name, service, env string
modelsOff bool
}{
{"a pass-through", "llm", "PRIVATAE_LLM_ADDR", false},
{"through the filter", "ollama", "OLLAMA_ADDR", true},
} {
t.Run(c.name, func(t *testing.T) {
t.Setenv(c.env, silent.Addr().String())
h, cfg := serveLaneBox(t, "idle-"+c.service+".example.test")
if c.modelsOff {
if err := localDenySet(cfg.HostName, "models", true); err != nil {
t.Fatal(err)
}
}
runInBackground(t, cfg)
ws := h.take(t)
if got := ask(t, ws, c.service); got != "A" {
t.Fatalf("accept: %q", got)
}
conn := laneConn(ws)
resp, _ := chatRequest(t, conn, "test-model")
start := time.Now()
_ = conn.SetReadDeadline(time.Now().Add(10 * time.Second))
_, err := io.Copy(io.Discard, resp.Body)
took := time.Since(start)
if err == nil {
t.Fatal("a silent stream must end in a close, not a complete answer")
}
if took < 300*time.Millisecond || took > 3*time.Second {
t.Fatalf("the silent stream was closed after %v; the bound is %v", took, serveIdleTimeout)
}
select {
case <-silentClosed:
case <-time.After(3 * time.Second):
t.Fatal("the server's end of the silent stream was left open")
}
t.Setenv(c.env, steady.Listener.Addr().String())
ws = h.take(t)
if got := ask(t, ws, c.service); got != "A" {
t.Fatalf("accept: %q", got)
}
resp, _ = chatRequest(t, laneConn(ws), "test-model")
b, err := io.ReadAll(resp.Body)
if err != nil || !strings.Contains(string(b), `"chunk":16`) || !strings.Contains(string(b), "[DONE]") {
t.Fatalf("a stream that keeps sending was cut: %v %q", err, b)
}
})
}
}
// ── 5. the beat says what is served ─────────────────────────────────────────
func capsOf(m map[string]any) []string {
var out []string
if l, ok := m["capabilities"].([]any); ok {
for _, c := range l {
out = append(out, fmt.Sprint(c))
}
}
return out
}
func strsOf(v any) []string {
out := []string{}
if l, ok := v.([]any); ok {
for _, c := range l {
out = append(out, fmt.Sprint(c))
}
}
return out
}
// SECURITY (an honest beat): a tcp pairing's beat lists in lane_serves what
// its serve lane carries, and advertises a model lane only when its service
// is there. With ollama answering here, llm unset and pii set to another
// computer, the beat carries lane_serves ["ollama"] and `embed` with its
// models, and no `llm-openai`, `pii`, pii_port, llm_port, `services` or
// `coders`, none of which runs on a tcp pairing. With all three served, all
// three are listed and advertised, pii_ok with them and pii_port still not. A
// unix pairing's beat lists its lane too and keeps `coders` (its runs have a
// socket); a tunnel pairing's beat has no lane_serves.
//
// MUTATION: have heartbeatPayload call buildHeartbeatBody (the tunnel's beat).
// RESULT: red — no lane_serves, and pii and services advertised with nothing
// serving them.
func TestATCPPairingsBeatAdvertisesOnlyWhatItServes(t *testing.T) {
up := newFakeOllama(t)
t.Setenv("OLLAMA_ADDR", up.srv.Listener.Addr().String())
t.Setenv("PRIVATAE_LLM_ADDR", "")
t.Setenv("PRIVATAE_PII_ADDR", "192.0.2.10:9")
stubServiceRuntime(t, "docker")
stubCoders(t)
h, cfg := serveLaneBox(t, "beat.example.test")
coderOn(t, 2)
if !servicesCapable(cfg.HostName) || coderHeartbeat(cfg.HostName) == nil {
t.Fatal("the slots and coder runs must be available on this machine, or their absence proves nothing")
}
runInBackground(t, cfg)
beat := h.beat(t)
caps := capsOf(beat)
if got := strsOf(beat["lane_serves"]); !slices.Equal(got, []string{"ollama"}) {
t.Fatalf("lane_serves: %q (the beat: %v)", got, beat)
}
if !slices.Contains(caps, "embed") || !slices.Contains(strsOf(beat["models"]), "qwen4-embed:latest") {
t.Fatalf("ollama is served, so embed and its models are advertised: %q %v", caps, beat["models"])
}
for _, c := range []string{"llm-openai", "pii", serviceCapability, coderCapability} {
if slices.Contains(caps, c) {
t.Errorf("a tcp pairing advertises %q, which nothing serves on it: %q", c, caps)
}
}
for _, k := range []string{"pii_port", "pii_ok", "services", "coders"} {
if _, ok := beat[k]; ok {
t.Errorf("a tcp pairing's beat carries %q: %v", k, beat[k])
}
}
if beat["llm_port"] != float64(0) {
t.Errorf("no port listens for the llm server on a tcp pairing: llm_port %v", beat["llm_port"])
}
// All three served.
health := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"status":"ok"}`))
}))
t.Cleanup(health.Close)
t.Setenv("PRIVATAE_LLM_ADDR", up.srv.Listener.Addr().String())
t.Setenv("PRIVATAE_PII_ADDR", health.Listener.Addr().String())
var all map[string]any
if err := json.Unmarshal([]byte(heartbeatPayload(cfg, nil, "box", "pub")), &all); err != nil {
t.Fatal(err)
}
if got := strsOf(all["lane_serves"]); !slices.Equal(got, []string{"ollama", "llm", "pii"}) {
t.Fatalf("lane_serves with all three set: %q", got)
}
caps = capsOf(all)
for _, c := range []string{"embed", "llm-openai", "pii"} {
if !slices.Contains(caps, c) {
t.Errorf("%q is served, so it is advertised: %q", c, caps)
}
}
if all["pii_ok"] != true || all["pii_port"] != nil || !slices.Contains(strsOf(all["llm_models"]), "qwen4-embed") {
t.Errorf("pii_ok %v, pii_port %v, llm_models %v", all["pii_ok"], all["pii_port"], all["llm_models"])
}
// A unix pairing: its lane, and its coder runs (a socket each).
unix := *cfg
unix.Transport, unix.LaneAddr, unix.LaneSocket, unix.RunsDir = transportUnix, "", "/run/privatae-host/lane.sock", "/run/privatae-host/runs"
var u map[string]any
_ = json.Unmarshal([]byte(heartbeatPayload(&unix, nil, "box", "pub")), &u)
if _, ok := u["lane_serves"]; !ok || slices.Contains(capsOf(u), serviceCapability) || !slices.Contains(capsOf(u), coderCapability) {
t.Errorf("a unix pairing: lane_serves %v, caps %q", u["lane_serves"], capsOf(u))
}
// A tunnel pairing: the forwards, no lane.
tunnel := *cfg
tunnel.Transport, tunnel.LaneAddr = "", ""
var tb map[string]any
_ = json.Unmarshal([]byte(heartbeatPayload(&tunnel, nil, "box", "pub")), &tb)
if _, ok := tb["lane_serves"]; ok || tb["pii_port"] != float64(piiForwardPort) {
t.Errorf("a tunnel pairing: lane_serves %v, pii_port %v", tb["lane_serves"], tb["pii_port"])
}
}
// The lane writes one audit line per job: the service, the address dialled
// and the bytes each way for a served one; the reason for a refused one.
func TestTheServeLaneAuditsEachJob(t *testing.T) {
up := newFakeOllama(t)
t.Setenv("OLLAMA_ADDR", up.srv.Listener.Addr().String())
h, cfg := serveLaneBox(t, "audit.example.test")
runInBackground(t, cfg)
ws := h.take(t)
if got := ask(t, ws, "ollama"); got != "A" {
t.Fatalf("accept: %q", got)
}
conn := laneConn(ws)
req, _ := http.NewRequest(http.MethodGet, "http://serve/api/tags", nil)
req.Close = true
_ = req.Write(conn)
resp, err := http.ReadResponse(bufio.NewReader(conn), req)
if err != nil {
t.Fatal(err)
}
_, _ = io.ReadAll(resp.Body)
ws.Close()
if got := ask(t, h.take(t), "10.0.0.2:11434"); !strings.HasPrefix(got, "E") {
t.Fatalf("refuse: %q", got)
}
path := filepath.Join(cfg.dcnDir(), "audit.jsonl")
deadline := time.Now().Add(5 * time.Second)
for {
raw, _ := os.ReadFile(path)
var served, refused map[string]any
for _, line := range strings.Split(strings.TrimSpace(string(raw)), "\n") {
var m map[string]any
if json.Unmarshal([]byte(line), &m) != nil || m["op"] != "serve" {
continue
}
switch m["decision"] {
case "served":
served = m
case "refused":
refused = m
}
}
if served != nil && refused != nil {
if served["service"] != "ollama" || served["target"] != up.srv.Listener.Addr().String() ||
served["bytes_from_host"].(float64) <= 0 || served["bytes_to_host"].(float64) <= 0 {
t.Fatalf("the served line: %v", served)
}
if !strings.Contains(fmt.Sprint(refused["reason"]), "never takes an address") {
t.Fatalf("the refused line: %v", refused)
}
return
}
if time.Now().After(deadline) {
t.Fatalf("no served and refused lines in %s: %s", path, raw)
}
time.Sleep(50 * time.Millisecond)
}
}
+240
View File
@@ -0,0 +1,240 @@
//go:build linux
// Model management for a Host with no platform (modelswap.go through
// modelDaemonTarget): the model-swap probe, model:list, model:pull and
// model:drop reach only a model server on this machine, as the serve lane
// does, and the machine's console says why one is not used. Uses the
// fixtures of servelane_linux_test.go and modelswap_test.go.
package main
import (
"bytes"
"os"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/gorilla/websocket"
)
// captureStdout collects what this process prints until the test ends, and
// returns a func that reads it so far.
func captureStdout(t *testing.T) func() string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
old := os.Stdout
os.Stdout = w
var mu sync.Mutex
var buf bytes.Buffer
done := make(chan struct{})
go func() {
defer close(done)
b := make([]byte, 4096)
for {
n, err := r.Read(b)
if n > 0 {
mu.Lock()
buf.Write(b[:n])
mu.Unlock()
_, _ = old.Write(b[:n])
}
if err != nil {
return
}
}
}()
t.Cleanup(func() {
os.Stdout = old
_ = w.Close()
<-done
_ = r.Close()
})
return func() string {
mu.Lock()
defer mu.Unlock()
return buf.String()
}
}
// SECURITY: a tcp pairing (a Host with no platform) is not given management of
// a model server on another computer. With OLLAMA_ADDR on TEST-NET, model:list,
// an approved model:pull and model:drop of a model this Host placed are each
// refused with the reason, before anything is dialled or downloaded, and the
// machine's console says in plain words that OLLAMA_ADDR is not used and what
// to set. With OLLAMA_ADDR on this machine the same pairing lists its models.
//
// MUTATION: modelDaemonTarget returns localOllamaAddr() for every pairing.
// RESULT: red — model:list fails dialling 192.0.2.10, the approved pull
// starts a download job (exit 0), and no line is printed at the machine.
func TestModelManagementOnALocalPairingReachesOnlyThisMachine(t *testing.T) {
console := captureStdout(t)
t.Setenv("OLLAMA_ADDR", "192.0.2.10:11434") // TEST-NET-1: never this machine
t.Setenv("PRIVATAE_LLM_ADDR", "")
t.Setenv("PRIVATAE_PII_ADDR", "")
resetModelDaemonCache()
h, cfg := serveLaneBox(t, "models.example.test")
runInBackground(t, cfg)
beat := h.beat(t)
for _, c := range capsOf(beat) {
if c == modelSwapCapability {
t.Fatalf("a Host with no platform is offered management of a server on another computer: %q", capsOf(beat))
}
}
root := t.TempDir()
start := time.Now()
_, errOut, code, _ := driveModelOp(t, cfg.HostName, "model:list", nil, root)
if code == 0 || !strings.Contains(errOut, "OLLAMA_ADDR") || !strings.Contains(errOut, "192.0.2.10 is not this machine") {
t.Fatalf("model:list: exit %d, %q; want a refusal naming OLLAMA_ADDR and why", code, errOut)
}
if d := time.Since(start); d > 2*time.Second {
t.Errorf("model:list took %v: it waited on a dial instead of refusing first", d)
}
enrollFactor(t, root)
approval := proposeAndApprove(t, root, "nomic-embed-text", "embed", 1<<30, "")
_, errOut, code, _ = driveModelOp(t, cfg.HostName, "model:pull",
[]string{pullArg("nomic-embed-text", "embed", 1<<30, "", approval)}, root)
if code != 126 || !strings.Contains(errOut, "was not downloaded") || !strings.Contains(errOut, "192.0.2.10 is not this machine") {
t.Fatalf("model:pull: exit %d, %q; want 126 and a refusal saying why", code, errOut)
}
modelPullJobsMu.Lock()
for id, j := range modelPullJobs {
if j.Model == "nomic-embed-text" {
t.Errorf("a pull job %s was started for a server on another computer", id)
}
}
modelPullJobsMu.Unlock()
if err := recordPlaced(root, "placed-here", placedModel{PlacedAt: "now", Role: "embed"}); err != nil {
t.Fatal(err)
}
_, errOut, code, _ = driveModelOp(t, cfg.HostName, "model:drop", []string{`{"model":"placed-here"}`}, root)
if code == 0 || !strings.Contains(errOut, "192.0.2.10 is not this machine") {
t.Fatalf("model:drop: exit %d, %q; want a refusal saying why", code, errOut)
}
if !placedByThisHost(root, "placed-here") {
t.Error("a refused drop changed the ledger")
}
said := console()
if !strings.Contains(said, "["+cfg.HostName+"] OLLAMA_ADDR=192.0.2.10:11434 is not used for the Host on this computer") ||
!strings.Contains(said, "Set OLLAMA_ADDR to a server on this computer, or unset it.") {
t.Errorf("the machine's console does not say why OLLAMA_ADDR is not used:\n%s", said)
}
// Counted for this pairing only: a pairing from an earlier test whose beat
// is still ending can say it for itself.
if n := strings.Count(said, "["+cfg.HostName+"] OLLAMA_ADDR=192.0.2.10:11434 is not used"); n != 1 {
t.Errorf("the console said it %d times; once, until it changes", n)
}
// The control: a model server on this machine is listed for the same pairing.
newFakeDaemon(t, &fakeDaemon{tags: []string{"here-model"}})
out, errOut, code, _ := driveModelOp(t, cfg.HostName, "model:list", nil, root)
if code != 0 || !strings.Contains(out, "here-model") {
t.Fatalf("model:list of a server on this machine: exit %d, %q %q", code, out, errOut)
}
}
// The beat and a model op can refuse the same target at the same moment. The
// machine's console still says it once: the reason is recorded and said by one
// caller, not checked by each and then said by all.
//
// MUTATION: in serveTargetNotice, Load the last reason, then Store and print
// (as it did).
// RESULT: red — more than one line for 32 callers released together (the
// count varies from run to run).
func TestTheConsoleSaysARefusedTargetOnceUnderConcurrentCallers(t *testing.T) {
console := captureStdout(t)
s, _ := serveServiceNamed("ollama")
self := "notice-once.example.test"
t.Cleanup(func() { serveTargetNoticeClear(self, s) })
start := make(chan struct{})
var wg sync.WaitGroup
for i := 0; i < 32; i++ {
wg.Add(1)
go func() {
defer wg.Done()
<-start
serveTargetNotice(self, s, "192.0.2.10:11434", "192.0.2.10 is not this machine")
}()
}
close(start)
wg.Wait()
serveTargetNotice(self, s, "192.0.2.10:11434", "another reason")
deadline := time.Now().Add(2 * time.Second)
for !strings.Contains(console(), "another reason") && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
said := console()
if n := strings.Count(said, "["+self+"] OLLAMA_ADDR=192.0.2.10:11434 is not used"); n != 2 {
t.Fatalf("the console said it %d times; once for the first reason and once when it changed:\n%s", n, said)
}
}
// A tunnel pairing keeps OLLAMA_ADDR wherever it points, as its forwards do:
// the rule is for a Host with no platform.
func TestModelDaemonTargetKeepsATunnelPairingsAddress(t *testing.T) {
t.Setenv("OLLAMA_ADDR", "192.0.2.10:11434")
if addr, why := modelDaemonTarget("tunnel.example.test"); addr != "192.0.2.10:11434" || why != "" {
t.Fatalf("a tunnel pairing: %q %q", addr, why)
}
unmark := markLocalPairing("tunnel.example.test")
if addr, why := modelDaemonTarget("tunnel.example.test"); addr != "" || !strings.Contains(why, "192.0.2.10 is not this machine") {
t.Fatalf("the same name run as a local pairing: %q %q", addr, why)
}
unmark()
unmark() // once only
if isLocalPairing("tunnel.example.test") {
t.Fatal("still marked local after its run ended")
}
}
// SECURITY (this machine's resources): a Host that answers every parked serve
// worker with a job it knows is refused (a malformed frame, a name the lane
// does not carry) cannot make this machine dial, handshake and audit in a
// loop. Each refused worker waits before it parks again, so two workers cost
// at most a few dials a second.
//
// MUTATION: spawn the replacement when the first frame arrives (taken() before
// the hello is read), as the lane first did.
// RESULT: red — hundreds of workers park in the 2 s window.
func TestARefusingHostCannotSpinTheServeWorkers(t *testing.T) {
t.Setenv("OLLAMA_ADDR", "")
h, cfg := serveLaneBox(t, "spin.example.test")
runInBackground(t, cfg)
var answered atomic.Int32
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
var ws *websocket.Conn
select {
case ws = <-h.workers:
case <-time.After(time.Until(deadline)):
}
if ws == nil {
break
}
frame := sFrame("nas")
if answered.Load()%2 == 1 {
frame = []byte("Dnot a hello")
}
_ = ws.WriteMessage(websocket.BinaryMessage, frame)
_ = ws.SetReadDeadline(time.Now().Add(2 * time.Second))
_, _, _ = ws.ReadMessage()
ws.Close()
answered.Add(1)
}
// 2 parked at the start, then each worker once a second: about 6 in 2 s.
if n := answered.Load(); n > 10 {
t.Fatalf("%d serve workers parked in 2 s on a Host that refuses every job: the pool re-dials in a loop", n)
}
if answered.Load() < 2 {
t.Fatalf("only %d workers parked: the pool did not run, so the count proves nothing", answered.Load())
}
}
+274
View File
@@ -0,0 +1,274 @@
//go:build linux
package main
import (
"fmt"
"net"
"os"
"os/exec"
"runtime"
"strings"
"sync/atomic"
"syscall"
"testing"
"time"
"golang.org/x/sys/unix"
)
const serveNetnsEnv = "PRIVATAE_TEST_SERVE_NETNS"
// otherComputer is a computer on this machine's LAN, at 10.0.0.2 in a network
// namespace of its own, joined to this one by a veth that carries this
// machine's default route (so 10.0.0.2 is the LAN, not a container here). It
// listens on 80 and 11434 and counts what reaches it off the wire: IPv4
// packets addressed to it, TCP SYNs to any address (IPv4 and IPv6), and
// accepted connections.
type otherComputer struct {
toIt atomic.Int64
syns atomic.Int64
accepts atomic.Int64
stop atomic.Bool
}
// startOtherComputer makes the namespace on one OS thread locked for good
// (the thread ends with its goroutine), so every socket it opens and every
// ip(8) it runs are the other computer's, and this test's are not.
func startOtherComputer(t *testing.T, ipBin string) *otherComputer {
t.Helper()
oc := &otherComputer{}
t.Cleanup(func() { oc.stop.Store(true) })
errc := make(chan error, 1)
go func() {
runtime.LockOSThread()
// This test's namespace, held open before the thread leaves it: the
// veth's other end goes there. Not /proc/<pid>/ns/net, which is the
// main thread's, and the main thread may be this one.
here, err := os.Open("/proc/thread-self/ns/net")
if err != nil {
errc <- err
return
}
defer here.Close()
if err := unix.Unshare(unix.CLONE_NEWNET); err != nil {
errc <- fmt.Errorf("unshare: %w", err)
return
}
for _, a := range [][]string{
{"link", "add", "vB", "type", "veth", "peer", "name", "vA", "netns", "/proc/self/fd/3"},
{"link", "set", "lo", "up"},
{"addr", "add", "10.0.0.2/24", "dev", "vB"},
{"link", "set", "vB", "up"},
} {
cmd := exec.Command(ipBin, a...)
cmd.ExtraFiles = []*os.File{here}
if out, err := cmd.CombinedOutput(); err != nil {
errc <- fmt.Errorf("ip %s: %v %s", strings.Join(a, " "), err, out)
return
}
}
for _, port := range []string{"80", "11434"} {
ln, err := net.Listen("tcp", "10.0.0.2:"+port)
if err != nil {
errc <- err
return
}
go func() {
for !oc.stop.Load() {
c, err := ln.Accept()
if err != nil {
return
}
oc.accepts.Add(1)
c.Close()
}
}()
t.Cleanup(func() { ln.Close() })
}
ifi, err := net.InterfaceByName("vB")
if err != nil {
errc <- err
return
}
proto := int(htons16(unix.ETH_P_ALL))
fd, err := unix.Socket(unix.AF_PACKET, unix.SOCK_RAW, proto)
if err != nil {
errc <- fmt.Errorf("packet socket: %w", err)
return
}
if err := unix.Bind(fd, &unix.SockaddrLinklayer{Protocol: htons16(unix.ETH_P_ALL), Ifindex: ifi.Index}); err != nil {
unix.Close(fd)
errc <- fmt.Errorf("bind the packet socket: %w", err)
return
}
tv := unix.NsecToTimeval((200 * time.Millisecond).Nanoseconds())
_ = unix.SetsockoptTimeval(fd, unix.SOL_SOCKET, unix.SO_RCVTIMEO, &tv)
go oc.capture(fd)
errc <- nil
}()
if err := <-errc; err != nil {
t.Skipf("the other computer's namespace could not be made: %v", err)
}
return oc
}
func htons16(v uint16) uint16 { return v<<8 | v>>8 }
// capture counts the frames arriving on vB (its own replies are left out).
func (oc *otherComputer) capture(fd int) {
defer unix.Close(fd)
buf := make([]byte, 65536)
for !oc.stop.Load() {
n, from, err := unix.Recvfrom(fd, buf, 0)
if err != nil {
continue
}
if ll, ok := from.(*unix.SockaddrLinklayer); ok && ll.Pkttype == unix.PACKET_OUTGOING {
continue
}
f := buf[:n]
if len(f) < 14 {
continue
}
switch uint16(f[12])<<8 | uint16(f[13]) {
case 0x0800:
if len(f) < 34 {
continue
}
ihl := int(f[14]&0x0f) * 4
if net.IP(f[30:34]).Equal(net.IPv4(10, 0, 0, 2)) {
oc.toIt.Add(1)
}
if f[23] == 6 && len(f) >= 14+ihl+14 && isSYN(f[14+ihl+13]) {
oc.syns.Add(1)
}
case 0x86dd:
if len(f) >= 14+40+14 && f[20] == 6 && isSYN(f[14+40+13]) {
oc.syns.Add(1)
}
}
}
}
func isSYN(flags byte) bool { return flags&0x02 != 0 && flags&0x10 == 0 }
// SECURITY, IN A REAL NETWORK NAMESPACE: a job that names an address instead
// of a service, a name the lane does not carry, and each service set on this
// machine to a computer on its LAN (OLLAMA_ADDR, PRIVATAE_LLM_ADDR and
// PRIVATAE_PII_ADDR all at 10.0.0.2) each answer 'E', and the other computer
// sees no packet addressed to it and no SYN at all, from the lane or from the
// beat that probes what the lane serves, the beat's model-management probe, or
// model:list. A direct dial from this machine
// afterwards is seen, so the count is real. Runs the test binary again in a
// private user and network namespace; skipped where one cannot be made.
//
// MUTATION: in serveVetTarget, skip the ownDestination check.
// RESULT: red — the other computer sees the lane's SYN on 11434 and 80.
// MUTATION: modelDaemonTarget returns localOllamaAddr() for every pairing.
// RESULT: red — the beat's model-swap probe and model:list SYN 10.0.0.2:11434.
func TestTheServeLaneSendsNoPacketForAnAddressInARealNamespace(t *testing.T) {
if os.Getenv(serveNetnsEnv) != "1" {
if _, err := exec.LookPath("ip"); err != nil {
if _, err := os.Stat("/usr/sbin/ip"); err != nil {
t.Skip("needs ip(8)")
}
}
cmd := exec.Command(os.Args[0], "-test.run=^TestTheServeLaneSendsNoPacketForAnAddressInARealNamespace$",
"-test.v", "-test.count=1")
cmd.Env = append(os.Environ(), serveNetnsEnv+"=1")
cmd.SysProcAttr = &syscall.SysProcAttr{
Cloneflags: syscall.CLONE_NEWUSER | syscall.CLONE_NEWNET,
UidMappings: []syscall.SysProcIDMap{{ContainerID: 0, HostID: os.Getuid(), Size: 1}},
GidMappings: []syscall.SysProcIDMap{{ContainerID: 0, HostID: os.Getgid(), Size: 1}},
}
out, err := cmd.CombinedOutput()
if cmd.ProcessState == nil {
t.Skipf("no private user and network namespace here: %v", err)
}
t.Logf("in a private network namespace:\n%s", out)
if strings.Contains(string(out), "--- SKIP") {
t.Skip("the namespaced run could not set up its network")
}
if err != nil {
t.Fatalf("the namespaced run failed: %v", err)
}
if !strings.Contains(string(out), "--- PASS: TestTheServeLaneSendsNoPacketForAnAddressInARealNamespace") {
t.Fatal("the namespaced run did not run")
}
return
}
ipBin, err := exec.LookPath("ip")
if err != nil {
ipBin = "/usr/sbin/ip"
}
if out, err := exec.Command(ipBin, "link", "set", "lo", "up").CombinedOutput(); err != nil {
t.Skipf("ip link set lo up: %v %s", err, out)
}
oc := startOtherComputer(t, ipBin)
for _, a := range [][]string{
{"addr", "add", "10.0.0.1/24", "dev", "vA"},
{"link", "set", "vA", "up"},
// This machine's way out: 10.0.0.2 is its LAN (and its router), not a
// container on a veth of its own (ownaddr.go).
{"route", "add", "default", "via", "10.0.0.2", "dev", "vA"},
} {
if out, err := exec.Command(ipBin, a...).CombinedOutput(); err != nil {
t.Skipf("ip %s: %v %s", strings.Join(a, " "), err, out)
}
}
t.Setenv("OLLAMA_ADDR", "10.0.0.2:11434")
t.Setenv("PRIVATAE_LLM_ADDR", "10.0.0.2:80")
t.Setenv("PRIVATAE_PII_ADDR", "10.0.0.2:80")
h, cfg := serveLaneBox(t, "netns.example.test")
// `models` stays on: the model-management probe in the beat and model:list
// (modelswap.go, modelDaemonTarget) are held to this machine for a Host
// with no platform too, so they are in the count.
resetModelDaemonCache()
runInBackground(t, cfg)
beat := h.beat(t)
if got := strsOf(beat["lane_serves"]); len(got) != 0 {
t.Fatalf("nothing set here is this machine's, so nothing is served: %q", got)
}
for _, c := range capsOf(beat) {
if c == modelSwapCapability {
t.Fatalf("model management of the other computer is offered: %q", capsOf(beat))
}
}
if _, errOut, code, _ := driveModelOp(t, cfg.HostName, "model:list", nil, ""); code == 0 ||
!strings.Contains(errOut, "10.0.0.2 is not this machine") {
t.Errorf("model:list of the other computer: exit %d, %q", code, errOut)
}
for service, want := range map[string]string{
"10.0.0.2:80": "never takes an address",
"10.0.0.2:11434": "never takes an address",
"nas": "is not a service",
"ollama": "10.0.0.2 is not this machine",
"llm": "10.0.0.2 is not this machine",
"pii": "10.0.0.2 is not this machine",
} {
got := ask(t, h.take(t), service)
if !strings.HasPrefix(got, "Erefused: ") || !strings.Contains(got, want) {
t.Errorf("%q: answered %q, want E carrying %q", service, got, want)
}
}
time.Sleep(500 * time.Millisecond)
if a, s, c := oc.toIt.Load(), oc.syns.Load(), oc.accepts.Load(); a != 0 || s != 0 || c != 0 {
t.Fatalf("the other computer saw %d packets addressed to it, %d SYNs and %d connections", a, s, c)
}
// The count is real: a dial from this machine is seen.
c, err := net.DialTimeout("tcp", "10.0.0.2:80", 3*time.Second)
if err != nil {
t.Fatalf("the control dial to the other computer failed, so the count proves nothing: %v", err)
}
c.Close()
deadline := time.Now().Add(3 * time.Second)
for oc.toIt.Load() == 0 || oc.syns.Load() == 0 || oc.accepts.Load() == 0 {
if time.Now().After(deadline) {
t.Fatalf("the control dial was not counted: %d packets, %d SYNs, %d connections",
oc.toIt.Load(), oc.syns.Load(), oc.accepts.Load())
}
time.Sleep(20 * time.Millisecond)
}
}
+24 -1
View File
@@ -288,17 +288,40 @@ func serviceArgs(spec harnessSpec) []string {
// serviceOwnerRefusal: on an anchored box a member pairing may not replace or // serviceOwnerRefusal: on an anchored box a member pairing may not replace or
// stop a slot another host built. The anchor pairing may (its user is the // stop a slot another host built. The anchor pairing may (its user is the
// machine's admin), and an unanchored local box serves one host. // machine's admin). A local box has no anchor to be that admin, and serves a
// second host only on its person's yes (pair_terminal.go); once it serves two,
// each host changes only the slots it built. With one host there is nobody to
// keep apart (a host renamed since it built a slot still owns it), and
// pairings that cannot be read are treated as two.
func serviceOwnerRefusal(self string, r *serviceRecord) string { func serviceOwnerRefusal(self string, r *serviceRecord) string {
if r == nil || r.Owner == "" || r.Owner == self { if r == nil || r.Owner == "" || r.Owner == self {
return "" return ""
} }
if boxMode() == modeLocal {
if !localServesSeveralHosts() {
return ""
}
return "refused: slot '" + r.Name + "' was built by " + r.Owner + "; on a local machine that serves several Hosts each changes only the servers it built"
}
if why := anchorOnly(self, "changing a server another host built"); why != "" { if why := anchorOnly(self, "changing a server another host built"); why != "" {
return "refused: slot '" + r.Name + "' was built by " + r.Owner + "; " + why return "refused: slot '" + r.Name + "' was built by " + r.Owner + "; " + why
} }
return "" return ""
} }
// localServesSeveralHosts: this is a local machine that serves more than one
// Host (the second added on its person's yes, pair_terminal.go), so what one
// Host built or started here is kept from the others. Pairings that cannot be
// read count as several. False on a networked machine, whose anchor rule
// (anchorOnly) keeps its Hosts apart.
func localServesSeveralHosts() bool {
if boxMode() != modeLocal {
return false
}
cfgs, err := loadPairings()
return err != nil || len(cfgs) > 1
}
func serviceFail(res map[string]interface{}, msg string) ([]byte, int) { func serviceFail(res map[string]interface{}, msg string) ([]byte, int) {
res["ok"], res["error"] = false, msg res["ok"], res["error"] = false, msg
b, _ := json.Marshal(res) b, _ := json.Marshal(res)
+41 -2
View File
@@ -495,10 +495,24 @@ func TestServiceOwnerRuleOnAnAnchoredBox(t *testing.T) {
if why := serviceOwnerRefusal(tMember, &serviceRecord{Name: "forge", Owner: tMember}); why != "" { if why := serviceOwnerRefusal(tMember, &serviceRecord{Name: "forge", Owner: tMember}); why != "" {
t.Fatalf("the builder may change its own server: %s", why) t.Fatalf("the builder may change its own server: %s", why)
} }
// Local box: one host, no owner rule to apply. // Local box with one host: nobody to keep apart, as before.
t.Setenv("XDG_CONFIG_HOME", t.TempDir()) t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if err := (&Config{HostName: tMember, ListenAddr: "127.0.0.1:7878"}).save(); err != nil {
t.Fatal(err)
}
if why := serviceOwnerRefusal(tMember, r); why != "" { if why := serviceOwnerRefusal(tMember, r); why != "" {
t.Fatalf("a local bastion serves one host: %s", why) t.Fatalf("a local machine with one host applies no owner rule: %s", why)
}
// Local box serving a second host (added on its person's yes): each host
// changes only the servers it built.
if err := (&Config{HostName: tAnchor, ListenAddr: "127.0.0.1:7879"}).save(); err != nil {
t.Fatal(err)
}
if why := serviceOwnerRefusal(tMember, r); why == "" || !strings.Contains(why, "built by "+tAnchor) {
t.Fatalf("on a local machine serving two hosts one must not change the other's server: %q", why)
}
if why := serviceOwnerRefusal(tMember, &serviceRecord{Name: "forge", Owner: tMember}); why != "" {
t.Fatalf("on a local machine a host changes its own server: %s", why)
} }
} }
@@ -852,3 +866,28 @@ func TestSpliceUnixCarriesBothWays(t *testing.T) {
} }
_ = a.Close() _ = a.Close()
} }
// A coder run on a local machine that serves two Hosts is stopped only by
// the Host that started it, as a server is changed only by the Host that
// built it. With one Host there is nobody to keep apart.
func TestCoderRunOwnerRuleOnALocalBoxServingTwoHosts(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if err := (&Config{HostName: tMember, ListenAddr: "127.0.0.1:7878"}).save(); err != nil {
t.Fatal(err)
}
if why := coderOwnerRefusal(tMember, tAnchor); why != "" {
t.Fatalf("a local machine with one host applies no owner rule: %s", why)
}
if err := (&Config{HostName: tAnchor, ListenAddr: "127.0.0.1:7879"}).save(); err != nil {
t.Fatal(err)
}
if why := coderOwnerRefusal(tMember, tAnchor); why == "" || !strings.Contains(why, "started by "+tAnchor) {
t.Fatalf("on a local machine serving two hosts one must not stop the other's run: %q", why)
}
if why := coderOwnerRefusal(tMember, ""); why == "" {
t.Fatal("on a local machine serving two hosts a run of unknown owner proves nothing about self")
}
if why := coderOwnerRefusal(tMember, tMember); why != "" {
t.Fatalf("a host stops its own run: %s", why)
}
}
+31 -12
View File
@@ -113,13 +113,26 @@ func runSessionWorkers(pctx context.Context, self string, wsURL string, tlsCfg *
// session; an H frame is refused with a plain line (harness jobs belong on the // session; an H frame is refused with a plain line (harness jobs belong on the
// harness workers) so a host that routes wrongly learns it at once. // harness workers) so a host that routes wrongly learns it at once.
func handleSessionWorkerConn(ws *websocket.Conn, self string, senv sessionEnv) { func handleSessionWorkerConn(ws *websocket.Conn, self string, senv sessionEnv) {
mt, data, err := ws.ReadMessage() // The spec is read up to sessionSpecMaxBytes (lane_bounds.go) and refused
if err != nil || mt != websocket.BinaryMessage || len(data) == 0 { // past it; a frame of any other tag is not read at all.
tag, body, tooBig, err := readLaneJob(ws, func(t byte) int64 {
if t == 'S' {
return sessionSpecMaxBytes
}
return 0
})
if err != nil {
return return
} }
switch data[0] { switch tag {
case 'S': case 'S':
handleSessionConn(ws, data, self, senv) if tooBig {
sendSession(ws, 'E', []byte(fmt.Sprintf("the session spec is more than the %d KiB a session may ask with; nothing of it was read\n",
sessionSpecMaxBytes>>10)))
sendSessionExit(ws, 1)
return
}
handleSessionConn(ws, append([]byte{'S'}, body...), self, senv)
case 'H': case 'H':
sendSession(ws, 'E', []byte("this is a session worker (lane=session); harness jobs go to a harness worker\n")) sendSession(ws, 'E', []byte("this is a session worker (lane=session); harness jobs go to a harness worker\n"))
sendSessionExit(ws, 1) sendSessionExit(ws, 1)
@@ -573,23 +586,29 @@ func handleSessionConn(ws *websocket.Conn, first []byte, self string, env sessio
closed := make(chan struct{}) closed := make(chan struct{})
go func() { go func() {
defer close(closed) defer close(closed)
// Each frame is streamed (lane_bounds.go): input goes into the pty as it
// arrives, never held whole, and a frame is at most sessionInputMaxBytes.
ws.SetReadLimit(sessionInputMaxBytes)
for { for {
mt, data, err := ws.ReadMessage() mt, r, err := ws.NextReader()
if err != nil { if err != nil {
return return
} }
if mt != websocket.BinaryMessage || len(data) == 0 { if mt != websocket.BinaryMessage {
continue continue // NextReader drops what is left of it
} }
switch data[0] { var tag [1]byte
if _, err := io.ReadFull(r, tag[:]); err != nil {
continue // an empty frame
}
switch tag[0] {
case 'I': case 'I':
lastInput.Store(time.Now().UnixNano()) lastInput.Store(time.Now().UnixNano())
if len(data) > 1 { _, _ = io.Copy(proc, r)
_, _ = proc.Write(data[1:])
}
case 'R': case 'R':
var rz sessionResize var rz sessionResize
if json.Unmarshal(data[1:], &rz) == nil && rz.Cols > 0 && rz.Rows > 0 { b, _ := io.ReadAll(io.LimitReader(r, sessionResizeMaxBytes))
if json.Unmarshal(b, &rz) == nil && rz.Cols > 0 && rz.Rows > 0 {
_ = proc.Resize(rz.Cols, rz.Rows) _ = proc.Resize(rz.Cols, rz.Rows)
} }
} }
+58
View File
@@ -0,0 +1,58 @@
# Runs install.ps1's -Pair block against a stand-in bastion, with the Windows
# cmdlets it calls stood in by functions that log their call. pwsh on Linux:
# pwsh -NoProfile -File testdata/install_ps1_pair.ps1 ../deploy/install.ps1
# Exits 0 when every case holds, and prints the first that does not.
param([string]$Script)
$ErrorActionPreference = 'Stop'
$src = Get-Content -Raw $Script
$m = [regex]::Match($src, '(?ms)^if \(\$Pair\) \{\r?\n.*?^\}\r?\n')
if (-not $m.Success) { Write-Output 'FAIL: install.ps1 has no top-level if ($Pair) { … } block'; exit 1 }
$block = [scriptblock]::Create($m.Value)
$dir = Join-Path ([IO.Path]::GetTempPath()) ([guid]::NewGuid().ToString())
New-Item -ItemType Directory -Path $dir | Out-Null
$log = Join-Path $dir 'log'
$exe = Join-Path $dir 'privatae-bastion'
Set-Content -Path $exe -Value ("#!/bin/sh`necho ""ARGV `$*"" >> '$log'`n" +
"if [ ""`$1"" = pair ]; then sed 's/^/STDIN /' >> '$log'; exit `$FAKE_RC; fi`necho status-ok`n")
chmod 755 $exe
function Stop-ScheduledTask { $script:calls += 'stop-task' }
function Start-ScheduledTask { $script:calls += 'start-task' }
function Restart-Service { $script:calls += 'restart-service' }
function Start-Process { $script:calls += 'start-process' }
function Stop-Process { $script:calls += 'stop-process' }
function Get-Process { @() }
function Start-Sleep { }
function Write-Host { param($Object) $script:out += "$Object`n" }
function Run-Case([string]$rc, [bool]$networked) {
$script:calls = @(); $script:out = ''
if (Test-Path $log) { Remove-Item $log }
$env:FAKE_RC = $rc
$Pair = 'skp_INVENTEDCODE'; $Networked = $networked
$threw = $null
try { . $block } catch { $threw = $_.Exception.Message }
$l = if (Test-Path $log) { Get-Content -Raw $log } else { '' }
return @{ log = $l; calls = ($script:calls -join ','); out = $script:out; threw = $threw }
}
function Fail([string]$why) { Write-Output "FAIL: $why"; exit 1 }
$r = Run-Case '0' $false
if ($r.threw) { Fail "rc 0 threw: $($r.threw)" }
if ($r.log -notmatch '(?m)^ARGV pair -$' -or $r.log -match 'ARGV .*skp_') { Fail "the code must reach 'pair -' on stdin, never argv: $($r.log)" }
if ($r.log -notmatch '(?m)^STDIN skp_INVENTEDCODE\r?$') { Fail "pair's stdin: $($r.log)" }
if ($r.calls -ne 'stop-task,start-task') { Fail "per-user: stop, then start the logon task: $($r.calls)" }
if ($r.out -notmatch 'This computer is paired\. The bastion is running, and starts again each time you sign in to Windows\.') { Fail "the ending: $($r.out)" }
$r = Run-Case '3' $false
if ($r.threw -or $r.out -notmatch 'already paired with this Host') { Fail "exit 3 is success: $($r.threw) $($r.out)" }
$r = Run-Case '1' $false
if (-not $r.threw -or $r.calls -ne '') { Fail "a failed pair must throw before starting anything: $($r.threw) $($r.calls)" }
$r = Run-Case '0' $true
if ($r.threw -or $r.calls -ne 'restart-service' -or $r.out -notmatch 'Windows service PrivataeBastion') { Fail "networked restarts the service: $($r.threw) $($r.calls)" }
Remove-Item -Recurse -Force $dir
Write-Output 'ok: install.ps1 -Pair pairs over stdin, treats exit 3 as success, stops on a failure, restarts task or service'
+1 -1
View File
@@ -888,7 +888,7 @@ func (s *uiSetup) anchorChoose(w http.ResponseWriter, r *http.Request) {
Class: "bad", Class: "bad",
Command: setModeCommandHint(), Command: setModeCommandHint(),
How: adminHowTo() + ", then claim the anchor.", How: adminHowTo() + ", then claim the anchor.",
Why: "This machine is in local mode, where it serves one host and needs no anchor. " + Why: "This machine is in local mode, where it serves only the Hosts you said yes to here and needs no anchor. " +
"An anchor only means something on a networked bastion.", "An anchor only means something on a networked bastion.",
}) })
return return
+1 -1
View File
@@ -1 +1 @@
0.55.3 0.57.0
+89 -1
View File
@@ -12,14 +12,102 @@
# An input that breaks a target is written under client-go/testdata/fuzz/<target>/ # An input that breaks a target is written under client-go/testdata/fuzz/<target>/
# and the run stops there. Commit that file with the fix: from then on it is one # and the run stops there. Commit that file with the fix: from then on it is one
# of the seeds every `go test` runs. # of the seeds every `go test` runs.
#
# The one failure this script does not stop on is Go's own timer, and only on
# the Go releases that have the race behind it (go_has_fuzztime_race below).
# When -fuzztime runs out there is a moment in which the fuzzer's parent
# context has the deadline error and its child context does not yet. The check
# in Go's fuzz coordinator that drops a cancellation error compares the error
# only with the child's (internal/fuzz/fuzz.go, `err == fuzzCtx.Err()`), so the
# deadline is reported as a failure instead of a clean stop. Go's fix is in
# 1.27 and is due in 1.26.11. The run then ends with exactly
#
# --- FAIL: <target> (N.NNs)
# context deadline exceeded
#
# and nothing else, whatever the target does. A run is counted as a pass only
# when ALL of these hold; any other failure is a failure:
# - the go in PATH is a release with the race. On any other, including one
# whose version cannot be read, the timer report is a failure like any
# other, and deploy/test_fuzz.sh fails until this workaround is removed;
# - the output carries none of the lines Go prints for a failing input
# ("Failing input written to", "To re-run:", "fuzz: minimizing",
# "failure while testing seed corpus entry"), wherever they appear;
# - the output, less the fuzzer's progress lines and go test's closing
# FAIL lines, is those two lines and no more (a crash, a panic, a t.Fatal
# message, a seed that fails: each adds a line, so none of them can look
# like the timer);
# - no file under any testdata/fuzz directory of the module was added or
# changed during the run. A crash still being minimized when the race hits
# is written there by Go with only the deadline error as its message, so
# the directories are checked, not just the text. Go writes a failing
# input nowhere else: its build-cache corpus only ever receives inputs
# that widened coverage, which most passing runs add, so it is not a sign
# of a failure.
# deploy/test_fuzz.sh checks each of these, with planted failing inputs.
set -euo pipefail set -euo pipefail
HERE="$(cd "$(dirname "$0")/.." && pwd)" HERE="$(cd "$(dirname "$0")/.." && pwd)"
SECS="${1:-20}" SECS="${1:-20}"
cd "$HERE/client-go" cd "$HERE/client-go"
log="$(mktemp)"
trap 'rm -f "$log"' EXIT
# go_has_fuzztime_race: the go in PATH is a release that reports its own
# -fuzztime deadline as a failure. The module needs Go 1.26 or later
# (client-go/go.mod), and the fix is due in 1.26.11, so only 1.26.0 to 1.26.10
# qualify. When the image moves past them, remove go_has_fuzztime_race, its
# use below and the stand-in cases in deploy/test_fuzz.sh, which fails on such
# a Go to say so. If a later Go still shows the race, add it here instead.
go_has_fuzztime_race() {
case "$(go env GOVERSION 2>/dev/null || true)" in
go1.26.[0-9] | go1.26.10) return 0 ;;
esac
return 1
}
# Every file under any testdata/fuzz directory of the module, with its
# checksum, in a fixed order.
corpus_state() {
find . -path '*/testdata/fuzz/*' -type f -exec cksum {} + | LC_ALL=C sort -k3
}
# crash_reported: $log carries a line Go prints only for a failing input.
crash_reported() {
grep -q -i -E -e 'failing input written to' -e 'to re-run:' \
-e 'fuzz: minimizing' -e 'failure while testing seed corpus entry' "$log"
}
# deadline_only <target>: the output in $log is Go's timer report and nothing
# else. The fuzzer's progress lines and go test's closing "FAIL", "exit status"
# and "FAIL <package> <time>" lines are dropped; two lines must remain.
deadline_only() {
local rest
rest="$(grep -v -E -e '^fuzz: elapsed: ' -e '^FAIL$' -e '^exit status [0-9]+$' \
-e '^FAIL[[:space:]]+[^[:space:]]+[[:space:]]+[0-9.]+s$' "$log" || true)"
[ "$(printf '%s\n' "$rest" | wc -l)" -eq 2 ] || return 1
printf '%s\n' "$rest" | sed -n 1p | grep -qE "^--- FAIL: $1 \([0-9.]+s\)\$" || return 1
[ "$(printf '%s\n' "$rest" | sed -n 2p)" = " context deadline exceeded" ]
}
targets="$(go test -list '^Fuzz' . | grep '^Fuzz' || true)" targets="$(go test -list '^Fuzz' . | grep '^Fuzz' || true)"
[ -n "$targets" ] || { echo "no fuzz targets found" >&2; exit 1; } [ -n "$targets" ] || { echo "no fuzz targets found" >&2; exit 1; }
for t in $targets; do for t in $targets; do
echo "== $t (${SECS}s)" echo "== $t (${SECS}s)"
go test -run '^$' -fuzz "^${t}\$" -fuzztime "${SECS}s" . before="$(corpus_state)"
rc=0
go test -run '^$' -fuzz "^${t}\$" -fuzztime "${SECS}s" . 2>&1 | tee "$log" || rc=$?
[ "$rc" -ne 0 ] || continue
if [ "$(corpus_state)" = "$before" ] && ! crash_reported && deadline_only "$t"; then
if go_has_fuzztime_race; then
echo "== $t: Go reported its own -fuzztime deadline as a failure (a race in" \
"Go's fuzzer before 1.26.11); no input was written and nothing else" \
"failed, so this counts as a pass"
continue
fi
echo "== $t: Go $(go env GOVERSION 2>/dev/null || echo '(unknown)') is not a" \
"release with the fuzzer's deadline race, so this report is a failure" >&2
fi
exit "$rc"
done done
echo "every target ran ${SECS}s with no failing input" echo "every target ran ${SECS}s with no failing input"
+68 -3
View File
@@ -21,7 +21,17 @@
# OWN account keeps the same profile — but "Run as different user" does not, and # OWN account keeps the same profile — but "Run as different user" does not, and
# the bastion would then read an empty directory and refuse to see its pairings. # the bastion would then read an empty directory and refuse to see its pairings.
# This is the Windows shape of the same trap `sudo -H` exists for on Unix. # This is the Windows shape of the same trap `sudo -H` exists for on Unix.
param([switch]$Networked) #
# -Pair '<code>': pair as well, with the code from your Host's "Set up a
# bastion" card, and start the bastion so it serves now and at every sign-in.
# The one line the card gives is
# & ([scriptblock]::Create((irm https://privatae.ai/bastion/install.ps1))) -Pair 'skp_…'
# The code reaches the bastion on standard input, never as an argument, so it
# is not in the process list. On a machine whose bastion is already the
# Windows service (installed with -Networked, or by `npx cseries`), -Pair pairs
# that service from an Administrator PowerShell instead of installing a second
# bastion beside it.
param([switch]$Networked, [string]$Pair)
# #
# Brought into the repo (this file previously existed ONLY as bytes in the # Brought into the repo (this file previously existed ONLY as bytes in the
# edge webroot, so what customers piped to `iex` had no review history) and # edge webroot, so what customers piped to `iex` had no review history) and
@@ -35,6 +45,18 @@ $base = 'https://dl.privatae.ai'
$arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') {'arm64'} else {'amd64'} $arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') {'arm64'} else {'amd64'}
$key = "windows-$arch.exe" $key = "windows-$arch.exe"
# Checked before anything is downloaded. Never echoed: it holds the token.
if ($Pair -and -not $Pair.StartsWith('skp_')) {
throw "-Pair takes the pairing code from your Host's card, which starts skp_. Nothing was installed."
}
# A machine whose bastion already runs as the Windows service is paired
# through that service. A per-user install beside it would be a second
# bastion, which the service's own guard refuses to pair anyway.
if ($Pair -and -not $Networked -and (Get-Service -Name 'PrivataeBastion' -ErrorAction SilentlyContinue)) {
$Networked = $true
Write-Host "This computer's bastion runs as the Windows service PrivataeBastion; pairing that service."
}
if ($Networked) { if ($Networked) {
# Checked before anything is downloaded, so a refusal leaves nothing behind # Checked before anything is downloaded, so a refusal leaves nothing behind
# and is one clear sentence at the moment it can be acted on. # and is one clear sentence at the moment it can be acted on.
@@ -43,7 +65,7 @@ if ($Networked) {
if (-not $isAdmin) { if (-not $isAdmin) {
throw ("-Networked needs an elevated terminal. A networked bastion runs as a Windows service, " + throw ("-Networked needs an elevated terminal. A networked bastion runs as a Windows service, " +
"and whoever administers the machine is what stands behind it.`n" + "and whoever administers the machine is what stands behind it.`n" +
"Close this window, right-click your terminal, choose 'Run as administrator', and run it again.") "Close this window, right-click your terminal, choose 'Run as administrator', and run the same line again.")
} }
} }
@@ -223,11 +245,54 @@ if ($Networked) {
Write-Host "" Write-Host ""
Write-Host "The bastion runs as the Windows service 'PrivataeBastion' (LocalSystem). It starts" Write-Host "The bastion runs as the Windows service 'PrivataeBastion' (LocalSystem). It starts"
Write-Host "serving as soon as this machine is paired." Write-Host "serving as soon as this machine is paired."
if (-not $Pair) {
Write-Host "Next, anchor this machine to the Host that should govern it:" Write-Host "Next, anchor this machine to the Host that should govern it:"
Write-Host " privatae-bastion claim-anchor" Write-Host " privatae-bastion claim-anchor"
Write-Host " (add --host <name> if this machine already serves several Hosts)" Write-Host " (add --host <name> if this machine already serves several Hosts)"
Write-Host "Every host paired after the anchor waits, pending, until that Host approves it." Write-Host "Every host paired after the anchor waits, pending, until that Host approves it."
} else { }
} elseif (-not $Pair) {
Write-Host "Open a NEW terminal, then: privatae-bastion pair <token>" Write-Host "Open a NEW terminal, then: privatae-bastion pair <token>"
} }
if ($Pair) {
Write-Host ""
# The full path, never the name on PATH, which this window may not have
# picked up yet. The code goes on standard input: `pair -` reads its first
# line. `pair` exits 3 when this computer is already paired with that Host,
# which is success here too. On a computer that already serves another
# Host, `pair` asks in this window whether to add this one.
$Pair | & $exe pair -
$rc = $LASTEXITCODE
if ($rc -eq 0) { $done = 'This computer is paired.' }
elseif ($rc -eq 3) { $done = 'This computer was already paired with this Host; nothing changed there.' }
else { throw "Pairing did not finish (the reason is above). The bastion is installed; nothing else changed." }
# A running bastion reads a new pairing at its next start, so it is
# restarted, not just started.
if ($Networked) {
Restart-Service -Name 'PrivataeBastion'
$how = 'The bastion runs as the Windows service PrivataeBastion, and starts with Windows.'
} else {
# Every copy is stopped first, from wherever it was started (a terminal,
# an older install, a detached copy after a self-update): each would
# hold this machine's first control port, 127.0.0.1:7878, and the new
# one would fail to start beside it.
Stop-ScheduledTask -TaskName 'PrivataeBastion' -ErrorAction SilentlyContinue
Get-Process -Name 'privatae-bastion' -ErrorAction SilentlyContinue | ForEach-Object {
Stop-Process -Id $_.Id -Force -ErrorAction SilentlyContinue
}
try {
Start-ScheduledTask -TaskName 'PrivataeBastion'
$how = 'The bastion is running, and starts again each time you sign in to Windows.'
} catch {
Start-Process -FilePath $exe -ArgumentList 'run' -WindowStyle Hidden
$how = "The bastion is running until you sign out (its sign-in task could not be started: $($_.Exception.Message))."
}
}
Start-Sleep -Seconds 3
& $exe status
Write-Host ""
Write-Host "$done $how"
}
Write-Host "privatae-bastion is free software under the GNU GPL v3 or later: https://privatae.ai/bastion/LICENSE.txt" Write-Host "privatae-bastion is free software under the GNU GPL v3 or later: https://privatae.ai/bastion/LICENSE.txt"
+213 -2
View File
@@ -6,7 +6,33 @@
# sudo -H privatae-bastion pair <token> # sudo -H privatae-bastion pair <token>
# Run as your own user (a desktop), it installs the LOCAL bastion for that user. # Run as your own user (a desktop), it installs the LOCAL bastion for that user.
# #
# PRIVATAE_PAIR_CODE='<code>' pairs as well, with the code from your Host's "Set
# up a bastion" card, and starts the bastion so it serves now and after a
# restart. The one line the card gives is
# curl -fsSL https://privatae.ai/bastion/install.sh | PRIVATAE_PAIR_CODE='skp_…' sh
#
set -e set -e
# The pairing code comes from the environment, never from an argument: an
# argument sits in this shell's command line for the whole install, where any
# user of this machine can read it with ps, and one at the same address could
# redeem it first. A process's environment is readable only by its owner and
# root. It is read here, before this script starts any program, and unset at
# once, so no program it starts inherits it. Never echoed: it holds the token.
pair_code=${PRIVATAE_PAIR_CODE-}
pair_given=${PRIVATAE_PAIR_CODE+1}
unset PRIVATAE_PAIR_CODE
if [ -n "$pair_given" ]; then
# skp_ and base64url: anything else is a line that lost part of the code.
case "$pair_code" in
skp_*[!A-Za-z0-9_-]* | skp_) pair_code="" ;;
skp_?*) ;;
*) pair_code="" ;;
esac
if [ -z "$pair_code" ]; then
echo "ERROR: PRIVATAE_PAIR_CODE is not a pairing code. Copy the whole line from your Host's card again (the code starts skp_). Nothing was installed." >&2
exit 2
fi
fi
# dl.privatae.ai, not privatae.ai/bastion: the download host is served from each # dl.privatae.ai, not privatae.ai/bastion: the download host is served from each
# edge's OWN local disk, so installs survive losing the platform datacentre, and # edge's OWN local disk, so installs survive losing the platform datacentre, and
# DNS points it at the edge with the bandwidth for an install wave (~15MB a # DNS points it at the edge with the bandwidth for an install wave (~15MB a
@@ -33,7 +59,9 @@ for a in "$@"; do
case "$a" in case "$a" in
--networked) networked=1 ;; --networked) networked=1 ;;
--local) local=1 ;; --local) local=1 ;;
*) echo "unknown option: $a" >&2; exit 2 ;; # An unknown argument is never echoed: it may be a pairing code given in
# the wrong place, and a log of the failed install must not hold it.
*) echo "ERROR: unknown option. This installer takes --networked or --local; a pairing code goes in PRIVATAE_PAIR_CODE, as your Host's card shows. Nothing was installed." >&2; exit 2 ;;
esac esac
done done
if [ "$networked" = 1 ] && [ "$local" = 1 ]; then if [ "$networked" = 1 ] && [ "$local" = 1 ]; then
@@ -54,6 +82,179 @@ paired_local() {
return 1 return 1
} }
# pair_as ROOT OS BIN ARGS...: run the bastion as the state it serves. As
# root that is the service's HOME (the unit's /root, launchd's /var/root),
# whatever HOME `sudo` left behind; as yourself it is your own.
pair_as() {
pa_root=$1; pa_os=$2; shift 2
if [ "$pa_root" = 1 ]; then
if [ "$pa_os" = darwin ]; then HOME=/var/root "$@"; else HOME=/root "$@"; fi
else
"$@"
fi
}
# bastion_copy PID: 0 when PID is a process of this user running a program
# named privatae-bastion, from any path, including one replaced on disk since
# it started. A process that has exited, or another user's, is not.
bastion_copy() {
[ -O "/proc/$1" ] || return 1
bc_exe=$(readlink "/proc/$1/exe" 2>/dev/null) || return 1
case "${bc_exe##*/}" in
"privatae-bastion" | "privatae-bastion (deleted)") return 0 ;;
esac
return 1
}
# stop_user_copies: stop every copy of this user's bastion (Linux), as
# install.ps1 does on Windows: one left running in a terminal, from an older
# install or another path. `run` holds the machine's run lock, so the copy the
# service starts would only wait behind it, and that copy never reads the new
# pairing; an older build would fail on 127.0.0.1:7878 and restart for ever.
# Asked to stop, then killed after 10 seconds. Says on stderr how many.
stop_user_copies() {
suc_pids=""
for suc_d in /proc/[0-9]*; do
suc_pid=${suc_d#/proc/}
bastion_copy "$suc_pid" || continue
kill "$suc_pid" 2>/dev/null && suc_pids="$suc_pids $suc_pid"
done
[ -n "$suc_pids" ] || return 0
set -- $suc_pids
echo "Stopped $# copy(ies) of the bastion running outside its service (a terminal, or another install)." >&2
for suc_i in 1 2 3 4 5 6 7 8 9 10; do
suc_left=""
for suc_pid in $suc_pids; do bastion_copy "$suc_pid" && suc_left="$suc_left $suc_pid"; done
[ -n "$suc_left" ] || return 0
sleep 1
done
for suc_pid in $suc_left; do kill -9 "$suc_pid" 2>/dev/null || true; done
}
# user_unit_stays_up: 0 when the user unit privatae-bastion is running and
# has kept the same process across a few seconds, longer than its
# RestartSec, so a copy that fails at start and is restarted is caught.
# Otherwise says what systemd shows, on stderr.
user_unit_stays_up() {
sleep 1
uu_first=$(systemctl --user show -p MainPID --value privatae-bastion 2>/dev/null) || uu_first=""
sleep 4
uu_pid=$(systemctl --user show -p MainPID --value privatae-bastion 2>/dev/null) || uu_pid=""
uu_state=$(systemctl --user show -p ActiveState --value privatae-bastion 2>/dev/null) || uu_state=""
uu_sub=$(systemctl --user show -p SubState --value privatae-bastion 2>/dev/null) || uu_sub=""
if [ "$uu_state" = active ] && [ "$uu_sub" = running ] && [ -n "$uu_pid" ] && [ "$uu_pid" != 0 ] && [ "$uu_pid" = "$uu_first" ]; then
return 0
fi
echo "The bastion's user service privatae-bastion is not staying up (systemd shows it ${uu_state:-unknown}/${uu_sub:-unknown}, and it has restarted)." >&2
echo "Its log says why: journalctl --user -u privatae-bastion" >&2
return 1
}
# start_bastion BIN ROOT OS: start, or restart, the bastion so it serves every
# pairing now and starts again by itself. A running bastion reads a new
# pairing at its next start, so a restart is what a fresh pairing needs.
# Prints one plain sentence saying how it will run.
start_bastion() {
sb_bin=$1; sb_root=$2; sb_os=$3
if [ "$sb_root" = 1 ] && [ "$sb_os" = linux ] && command -v systemctl >/dev/null 2>&1; then
systemctl restart privatae-bastion || return 1
echo "The bastion runs as the system service privatae-bastion, and starts at boot."
return 0
fi
if [ "$sb_root" = 1 ] && [ "$sb_os" = darwin ] && [ -f /Library/LaunchDaemons/ai.privatae.bastion.plist ]; then
/bin/launchctl kickstart -k system/ai.privatae.bastion || return 1
echo "The bastion runs as the system service ai.privatae.bastion, and starts at boot."
return 0
fi
if [ "$sb_root" != 1 ] && [ "$sb_os" = linux ] && command -v systemctl >/dev/null 2>&1 && systemctl --user show-environment >/dev/null 2>&1; then
# A systemd user unit: yours, no root. It starts when you sign in; with
# lingering on, at boot and after you sign out as well.
sb_dir="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
mkdir -p "$sb_dir"
cat > "$sb_dir/privatae-bastion.service" <<UNIT
[Unit]
Description=Privatae bastion - serves this machine to your Hosts (tunnel + lanes)
[Service]
Type=simple
ExecStart=$sb_bin run
Restart=always
RestartSec=2
[Install]
WantedBy=default.target
UNIT
systemctl --user daemon-reload && systemctl --user enable privatae-bastion >/dev/null 2>&1 || return 1
# Its own copy first, then every other: what is left running after this
# is only what the unit starts.
systemctl --user stop privatae-bastion >/dev/null 2>&1 || true
stop_user_copies
systemctl --user restart privatae-bastion || return 1
user_unit_stays_up || return 1
if loginctl enable-linger "$(id -un)" >/dev/null 2>&1; then
echo "The bastion runs as your systemd user service privatae-bastion, and starts at boot."
else
echo "The bastion runs as your systemd user service privatae-bastion while you are signed in."
echo "To keep it running at boot and after you sign out: sudo loginctl enable-linger $(id -un)"
fi
return 0
fi
if [ "$sb_root" != 1 ] && [ "$sb_os" = darwin ]; then
# A LaunchAgent: yours, no root, started each time you log in.
sb_plist="$HOME/Library/LaunchAgents/ai.privatae.bastion.user.plist"
mkdir -p "$HOME/Library/LaunchAgents"
cat > "$sb_plist" <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>ai.privatae.bastion.user</string>
<key>ProgramArguments</key><array><string>$sb_bin</string><string>run</string></array>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
</dict>
</plist>
PLIST
/bin/launchctl bootout "gui/$(id -u)/ai.privatae.bastion.user" >/dev/null 2>&1 || true
/bin/launchctl bootstrap "gui/$(id -u)" "$sb_plist" || return 1
echo "The bastion runs as your LaunchAgent ai.privatae.bastion.user, and starts each time you log in."
return 0
fi
# No service manager this script can use: run it now, detached, once every
# other copy of it has stopped (stop_user_copies says why).
if [ "$sb_os" = linux ] && [ "$sb_root" != 1 ]; then stop_user_copies; fi
nohup "$sb_bin" run >/dev/null 2>&1 &
sb_pid=$!
sleep 3
if [ "$sb_os" = linux ] && ! bastion_copy "$sb_pid"; then
echo "The bastion stopped as soon as it started. Run it in a terminal to see why: $sb_bin run" >&2
return 1
fi
echo "The bastion is running, until this machine restarts. Start it again with: $sb_bin run"
}
# pair_and_start BIN CODE ROOT OS: pair with CODE, handed to `pair -` on
# standard input and never as an argument, where the process list would show
# it. Then start the bastion and show its status. `pair` exits 3 when this
# machine is already paired with that Host, which is success here too.
pair_and_start() {
ps_bin=$1; ps_code=$2; ps_root=$3; ps_os=$4
ps_rc=0
printf '%s\n' "$ps_code" | pair_as "$ps_root" "$ps_os" "$ps_bin" pair - || ps_rc=$?
case "$ps_rc" in
0) ps_done="This machine is paired." ;;
3) ps_done="This machine was already paired with this Host; nothing changed there." ;;
*) echo "ERROR: pairing did not finish (the reason is above). The bastion is installed; nothing else changed." >&2
return 1 ;;
esac
ps_how=$(start_bastion "$ps_bin" "$ps_root" "$ps_os") || {
echo "ERROR: $ps_done But the bastion did not start. Start it with: $ps_bin run" >&2
return 1
}
sleep 2
pair_as "$ps_root" "$ps_os" "$ps_bin" status || true
echo ""
echo "$ps_done $ps_how"
}
if [ "$root" = 1 ] && [ "$local" != 1 ]; then if [ "$root" = 1 ] && [ "$local" != 1 ]; then
case "$(uname -s)" in case "$(uname -s)" in
Linux) if command -v systemctl >/dev/null 2>&1; then networked=1; fi ;; Linux) if command -v systemctl >/dev/null 2>&1; then networked=1; fi ;;
@@ -199,7 +400,9 @@ UNIT
systemctl enable --now privatae-bastion >/dev/null 2>&1 || true systemctl enable --now privatae-bastion >/dev/null 2>&1 || true
echo "Service installed + enabled (it restarts until you pair, then connects automatically)." echo "Service installed + enabled (it restarts until you pair, then connects automatically)."
echo "" echo ""
if [ "$networked" = 1 ]; then if [ -n "$pair_code" ]; then
:
elif [ "$networked" = 1 ]; then
echo "Networked bastion. Pair it with YOUR host; the first host paired becomes its anchor:" echo "Networked bastion. Pair it with YOUR host; the first host paired becomes its anchor:"
echo " sudo -H privatae-bastion pair <token>" echo " sudo -H privatae-bastion pair <token>"
echo "Then enrol your second factor, so you can approve here:" echo "Then enrol your second factor, so you can approve here:"
@@ -224,6 +427,7 @@ elif [ "$darwin_service" = 1 ]; then
HOME=/var/root "$bin" set-mode networked || die "could not set networked mode" HOME=/var/root "$bin" set-mode networked || die "could not set networked mode"
"$bin" service install || die "could not register the system service" "$bin" service install || die "could not register the system service"
echo "" echo ""
if [ -z "$pair_code" ]; then
echo "Networked bastion, running as a system service (launchd, as root)." echo "Networked bastion, running as a system service (launchd, as root)."
echo "Pair it with YOUR host; the first host paired becomes its anchor:" echo "Pair it with YOUR host; the first host paired becomes its anchor:"
# The full path, not the name on PATH: /usr/local/bin may belong to the # The full path, not the name on PATH: /usr/local/bin may belong to the
@@ -234,6 +438,7 @@ elif [ "$darwin_service" = 1 ]; then
echo " sudo -H '$bin' stepup-enroll" echo " sudo -H '$bin' stepup-enroll"
echo "Every terminal and write waits for your approval, and every later host for your host's." echo "Every terminal and write waits for your approval, and every later host for your host's."
echo "(Already paired before this install? Anchor with: sudo -H '$bin' claim-anchor)" echo "(Already paired before this install? Anchor with: sudo -H '$bin' claim-anchor)"
fi
elif [ "$networked" = 1 ]; then elif [ "$networked" = 1 ]; then
die "--networked needs a system service manager: systemd on Linux, launchd on macOS" die "--networked needs a system service manager: systemd on Linux, launchd on macOS"
else else
@@ -243,7 +448,13 @@ else
echo "This machine is already paired as a LOCAL bastion, and reinstalling keeps it local." echo "This machine is already paired as a LOCAL bastion, and reinstalling keeps it local."
echo "To make it a networked server instead, run the installer again with --networked." echo "To make it a networked server instead, run the installer again with --networked."
fi fi
if [ -z "$pair_code" ]; then
echo "Now pair: privatae-bastion pair <token>" echo "Now pair: privatae-bastion pair <token>"
echo "Run it: privatae-bastion run (Linux non-root: 'loginctl enable-linger \$USER' + a --user unit for boot persistence)" echo "Run it: privatae-bastion run (Linux non-root: 'loginctl enable-linger \$USER' + a --user unit for boot persistence)"
fi fi
fi
if [ -n "$pair_code" ]; then
echo ""
pair_and_start "$bin" "$pair_code" "$root" "$os" || exit 1
fi
echo "privatae-bastion is free software under the GNU GPL v3 or later: https://privatae.ai/bastion/LICENSE.txt" echo "privatae-bastion is free software under the GNU GPL v3 or later: https://privatae.ai/bastion/LICENSE.txt"
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env bash
# Checks deploy/fuzz.sh: that it does not fail on Go's own -fuzztime report
# on a Go release with the fuzzer's deadline race, and that it never lets a
# real failure through as that report.
#
# bash deploy/test_fuzz.sh
#
# Part one runs the script against a stand-in `go` that prints what Go prints,
# so the timer report can be produced on demand and every way a real failure
# could be dressed up as it can be tried. Part two runs it against the real go
# on throwaway modules whose targets fail in the ways a real bug would: a
# planted failing seed, a panic and a test failure that both say "context
# deadline exceeded", found by the fuzzer itself. Part three fails when the go
# this runs with no longer has the race, so the workaround is removed rather
# than kept past its need. Every part runs in a temporary directory; nothing in
# the tree is touched.
set -euo pipefail
HERE="$(cd "$(dirname "$0")/.." && pwd)"
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
bad=0
# expect <name> <want: pass|fail> <must contain> <dir> <env...>: run the copy of
# fuzz.sh in <dir> and compare its exit status and output with what is wanted.
expect() {
local name="$1" want="$2" must="$3" dir="$4" rc=0 out
shift 4
out="$(cd "$dir" && env "$@" bash deploy/fuzz.sh 1 2>&1)" || rc=$?
if { [ "$want" = pass ] && [ "$rc" -ne 0 ]; } || { [ "$want" = fail ] && [ "$rc" -eq 0 ]; }; then
echo "NOT OK $name: wanted $want, exit status was $rc"
printf '%s\n' "$out" | sed 's/^/ /'
bad=1
elif [ -n "$must" ] && ! grep -qF -- "$must" <<<"$out"; then
echo "NOT OK $name: the output does not say: $must"
printf '%s\n' "$out" | sed 's/^/ /'
bad=1
else
echo "ok $name"
fi
}
# --- part one: a stand-in go ------------------------------------------------
stub="$tmp/stub"
mkdir -p "$stub/deploy" "$stub/client-go" "$tmp/bin"
cp "$HERE/deploy/fuzz.sh" "$stub/deploy/fuzz.sh"
cat >"$tmp/bin/go" <<'EOF'
#!/usr/bin/env bash
# Prints what `go test` printed in the failing CI run, by FAKE_MODE, or by
# FAKE_MODE_<target> for one target of several (FAKE_TARGETS). `go env
# GOVERSION` prints FAKE_GOVERSION, a release with the race unless set; set to
# "fail", the command fails.
if [ "$1 $2" = "env GOVERSION" ]; then
[ "${FAKE_GOVERSION-}" != fail ] || exit 1
printf '%s\n' "${FAKE_GOVERSION-go1.26.10}"; exit 0
fi
if [ "$1 $2" = "test -list" ]; then
printf '%s\n' ${FAKE_TARGETS:-FuzzOne}; printf 'ok \texample/pkg\t0.008s\n'; exit 0
fi
t= prev=
for a in "$@"; do
if [ "$prev" = -fuzz ]; then t="${a#^}"; t="${t%\$}"; fi
prev="$a"
done
m="FAKE_MODE_$t"
mode="${!m:-$FAKE_MODE}"
[ "$t" = FuzzOne ] && other=FuzzTwo || other=FuzzOne
echo "fuzz: elapsed: 0s, gathering baseline coverage: 11/11 completed, now fuzzing with 6 workers"
echo "fuzz: elapsed: 6s, execs: 4368 (1448/sec), new interesting: 1 (total: 12)"
timer() { printf -- '--- FAIL: %s (6.02s)\n context deadline exceeded\nFAIL\nexit status 1\nFAIL\texample/pkg\t6.025s\n' "$1"; }
crasher() { mkdir -p "${2:-testdata/fuzz}/$t"; printf 'go test fuzz v1\n[]byte("x")\n' >"${2:-testdata/fuzz}/$t/$1"; }
case "$mode" in
pass) printf 'PASS\nok \texample/pkg\t6.019s\n' ;;
timer) timer $t; exit 1 ;;
timer-new-file) crasher 1a2b3c4d; timer $t; exit 1 ;;
timer-sub-file) crasher 1a2b3c4d sub/testdata/fuzz; timer $t; exit 1 ;;
timer-changed) printf 'changed\n' >testdata/fuzz/$t/seed; timer $t; exit 1 ;;
timer-other) timer $other; exit 1 ;;
timer-message) crasher 1a2b3c4d
printf -- '--- FAIL: %s (6.02s)\n context deadline exceeded\n Failing input written to testdata/fuzz/%s/1a2b3c4d\nFAIL\nexit status 1\n' $t $t; exit 1 ;;
timer-minimize) echo "fuzz: minimizing 12-byte failing input file"; timer $t; exit 1 ;;
timer-glued) printf 'fuzz: elapsed: 6s, execs: 4370 (1448/sec)%s\n' "$FAKE_GLUE"; timer $t; exit 1 ;;
timer-own-line) printf -- '--- FAIL: %s (0.01s)\n fuzz_test.go:9: context deadline exceeded\nFAIL\nexit status 1\n' $t; exit 1 ;;
timer-panic-line) printf -- '--- FAIL: %s (6.02s)\n panic: context deadline exceeded\nFAIL\nexit status 1\n' $t; exit 1 ;;
fatal-then-timer) printf -- '--- FAIL: %s (6.02s)\n the verifier accepted bytes that were not signed\n context deadline exceeded\nFAIL\n' $t; exit 1 ;;
panic) crasher 1a2b3c4d
printf -- '--- FAIL: %s (0.40s)\n panic: planted [recovered]\n Failing input written to testdata/fuzz/%s/1a2b3c4d\nFAIL\nexit status 1\n' $t $t; exit 1 ;;
other) printf -- '--- FAIL: %s (0.40s)\n fuzzing process hung or terminated unexpectedly: exit status 2\nFAIL\n' $t; exit 1 ;;
esac
EOF
chmod +x "$tmp/bin/go"
stubrun() { # <name> <want> <must contain> <FAKE_MODE> [VAR=value ...]
rm -rf "$stub/client-go" && mkdir -p "$stub/client-go/testdata/fuzz/FuzzOne"
printf 'go test fuzz v1\n[]byte("seed")\n' >"$stub/client-go/testdata/fuzz/FuzzOne/seed"
expect "$1" "$2" "$3" "$stub" PATH="$tmp/bin:$PATH" FAKE_MODE="$4" "${@:5}"
}
stubrun "a run that passes" pass "every target ran" pass
stubrun "the timer report alone is a pass (the failing CI run's output)" pass "counts as a pass" timer
stubrun "the timer report on go1.26.0 is a pass" pass "counts as a pass" timer FAKE_GOVERSION=go1.26.0
stubrun "the timer report with a crasher file written is a failure" fail "" timer-new-file
stubrun "the timer report with a crasher in another package's corpus is a failure" fail "" timer-sub-file
stubrun "the timer report with a corpus file changed is a failure" fail "" timer-changed
stubrun "the timer report for another target is a failure" fail "" timer-other
stubrun "the timer report with Go's 'Failing input written' line is a failure" fail "" timer-message
stubrun "the timer report after 'minimizing' is a failure" fail "" timer-minimize
for glue in "Failing input written to| testdata/fuzz/FuzzOne/1a2b3c4d" "To re-run:|" \
"fuzz: minimizing| 12-byte failing input file" "failure while testing seed corpus entry|: FuzzOne/seed"; do
stubrun "the timer report with '${glue%%|*}' glued to a progress line is a failure" fail "" \
timer-glued FAKE_GLUE="${glue%%|*}${glue#*|}"
done
stubrun "a test's own 'file:line: context deadline exceeded' is a failure" fail "" timer-own-line
stubrun "a panic saying 'context deadline exceeded' is a failure" fail "" timer-panic-line
stubrun "a test's own message before the timer line is a failure" fail "" fatal-then-timer
stubrun "a panic with a crasher written is a failure" fail "" panic
stubrun "a worker that died is a failure" fail "" other
for v in go1.26.11 go1.27.0 "devel go1.28-0123abcd" "" fail; do
stubrun "the timer report on a Go without the race (${v:-empty version}) is a failure" fail "is not a release" timer FAKE_GOVERSION="$v"
done
stubrun "two targets, both with the timer report alone, pass" pass "every target ran" timer FAKE_TARGETS="FuzzOne FuzzTwo"
stubrun "two targets, the second with a real crash, is a failure" fail "panic: planted" timer \
FAKE_TARGETS="FuzzOne FuzzTwo" FAKE_MODE_FuzzTwo=panic
stubrun "two targets, the second printing the first's timer report, is a failure" fail "" pass \
FAKE_TARGETS="FuzzOne FuzzTwo" FAKE_MODE_FuzzTwo=timer-other
# --- part two: the real go, planted failing inputs ----------------------------
mkreal() { # <dir> <test file after its package clause>: a throwaway module
mkdir -p "$1/deploy" "$1/client-go"
cp "$HERE/deploy/fuzz.sh" "$1/deploy/fuzz.sh"
(cd "$1/client-go" && go mod init plantedfuzz >/dev/null 2>&1 &&
printf 'package plantedfuzz\n\n%s\n' "$2" >fuzz_test.go)
}
mkreal "$tmp/clean" 'import "testing"
func FuzzClean(f *testing.F) {
f.Add([]byte("ok"))
f.Fuzz(func(t *testing.T, b []byte) {})
}'
expect "the real go on a target that cannot fail passes" pass "every target ran" "$tmp/clean"
mkreal "$tmp/planted" 'import "testing"
func FuzzPlanted(f *testing.F) {
f.Add([]byte("ok"))
f.Fuzz(func(t *testing.T, b []byte) {
if string(b) == "BOOM" {
t.Fatalf("planted failure")
}
})
}'
mkdir -p "$tmp/planted/client-go/testdata/fuzz/FuzzPlanted"
printf 'go test fuzz v1\n[]byte("BOOM")\n' >"$tmp/planted/client-go/testdata/fuzz/FuzzPlanted/planted"
expect "the real go on a planted failing input fails" fail "planted failure" "$tmp/planted"
# Inputs the fuzzer finds itself: any input but the seed fails, so the first
# mutation is a crash, and its message is the deadline error's own text.
mkreal "$tmp/panics" 'import (
"context"
"testing"
)
func FuzzPanicsWithDeadline(f *testing.F) {
f.Add([]byte("ok"))
f.Fuzz(func(t *testing.T, b []byte) {
if string(b) != "ok" {
panic(context.DeadlineExceeded)
}
})
}'
expect "the real go on a panic saying 'context deadline exceeded' fails" fail \
"--- FAIL: FuzzPanicsWithDeadline" "$tmp/panics"
mkreal "$tmp/deadlinebug" 'import (
"context"
"testing"
)
func FuzzReturnsADeadlineError(f *testing.F) {
f.Add([]byte("ok"))
f.Fuzz(func(t *testing.T, b []byte) {
if string(b) != "ok" {
ctx, cancel := context.WithTimeout(context.Background(), 0)
defer cancel()
<-ctx.Done()
t.Fatal(ctx.Err())
}
})
}'
expect "the real go on a target failing with its own deadline error fails" fail \
"--- FAIL: FuzzReturnsADeadlineError" "$tmp/deadlinebug"
# The closest a target's own failure comes to the timer report: two lines,
# the second the deadline error, told apart only by its file:line prefix.
mkreal "$tmp/setup" 'import (
"context"
"testing"
)
func FuzzSetupTimesOut(f *testing.F) {
f.Fatal(context.DeadlineExceeded)
f.Fuzz(func(t *testing.T, b []byte) {})
}'
expect "the real go on a target whose setup fails with the deadline error fails" fail \
"fuzz_test.go:" "$tmp/setup"
# --- part three: the go this runs with still needs the workaround -------------
# fuzz.sh's own version check, given this go's version: when it no longer
# grants the timer report a pass, the race is fixed here and the workaround is
# due to be removed.
v="$(go env GOVERSION 2>/dev/null || echo fail)"
was="$bad"
bad=0
stubrun "this go ($v) is a release with the race fuzz.sh works around" pass "counts as a pass" \
timer FAKE_GOVERSION="$v"
if [ "$bad" -ne 0 ]; then
echo " $v does not have the fuzzer's deadline race: remove go_has_fuzztime_race and"
echo " its use from deploy/fuzz.sh and the timer cases from this file."
fi
bad=$((was | bad))
[ "$bad" -eq 0 ] || { echo "deploy/fuzz.sh: a check failed" >&2; exit 1; }
echo "deploy/fuzz.sh: every check passed"
+41 -4
View File
@@ -27,7 +27,7 @@ says so. Those parts are enforced only by the code as written.
| A Host on this machine gets the same walls | `local_host_linux_test.go`: `TestAUnixRedeemURLCannotPointAtATCPAddress` | Every network spelling of a redeem URL (a host and port, `localhost`, user info, another scheme, a relative, unclean or query-carrying path) is refused before the token is read; a TCP listener at the named address gets no connection and no file is written. | | A Host on this machine gets the same walls | `local_host_linux_test.go`: `TestAUnixRedeemURLCannotPointAtATCPAddress` | Every network spelling of a redeem URL (a host and port, `localhost`, user info, another scheme, a relative, unclean or query-carrying path) is refused before the token is read; a TCP listener at the named address gets no connection and no file is written. |
| | `local_host_linux_test.go`: `TestAUnixPairingPinsTheHostsCA` | A `--replace` whose answer brings another CA is refused and changes nothing; an operator certificate the CA did not sign, a lane socket or runs directory outside the shared directory, or an answer that is not transport `unix` saves nothing; a platform `--replace` of the pairing is refused before the platform is called. | | | `local_host_linux_test.go`: `TestAUnixPairingPinsTheHostsCA` | A `--replace` whose answer brings another CA is refused and changes nothing; an operator certificate the CA did not sign, a lane socket or runs directory outside the shared directory, or an answer that is not transport `unix` saves nothing; a platform `--replace` of the pairing is refused before the platform is called. |
| | `local_host_linux_test.go`: `TestTheUnixLaneRefusesAPeerWithoutThePinnedCA` | A lane server whose certificate another CA signed, or that names another host, is dialled and receives no request. | | | `local_host_linux_test.go`: `TestTheUnixLaneRefusesAPeerWithoutThePinnedCA` | A lane server whose certificate another CA signed, or that names another host, is dialled and receives no request. |
| | `local_host_linux_test.go`: `TestTheUnixLaneCarriesTheBearerAndTheSameProtocol` | A pairing with no bearer does not start and dials nothing. With one, the heartbeat and the first frame of every worker carry it; the pools are the tunnel's (two parked harness workers) and a parked worker serves a harness job over the socket. | | | `local_host_linux_test.go`: `TestTheUnixLaneCarriesTheBearerAndTheSameProtocol` | A pairing with no bearer does not start and dials nothing. With one, the heartbeat and the first frame of every worker carry it; the pools are the tunnel's (two parked harness workers), plus two parked serve workers, and a parked worker serves a harness job over the socket. |
| | `local_host_linux_test.go`: `TestTheCoderGrantRuleAppliesToAUnixPairing` | On an anchored machine a run-up over the unix lane without the standing coder grant is refused at the gate; with the approved grant it passes. | | | `local_host_linux_test.go`: `TestTheCoderGrantRuleAppliesToAUnixPairing` | On an anchored machine a run-up over the unix lane without the standing coder grant is refused at the gate; with the approved grant it passes. |
| | `local_host_linux_test.go`: `TestARunSocketIsTheOwningPairingsOnly` | The owner's run is a 0660 socket in the owner's runs directory, given to the `privatae-host` group, and reaches the run's control socket; another pairing's directory never gets it, another pairing cannot open it, and once the run is not the pairing's a connection gets nothing and the socket goes. | | | `local_host_linux_test.go`: `TestARunSocketIsTheOwningPairingsOnly` | The owner's run is a 0660 socket in the owner's runs directory, given to the `privatae-host` group, and reaches the run's control socket; another pairing's directory never gets it, another pairing cannot open it, and once the run is not the pairing's a connection gets nothing and the socket goes. |
| | `local_host_linux_test.go`: `TestARunSocketGoesAtRunDownAndAtTheWatchdogsStop` | The socket is removed at run-down and at the watchdog's stop and does not come back; with no `privatae-host` group it is root's and the reason is logged. | | | `local_host_linux_test.go`: `TestARunSocketGoesAtRunDownAndAtTheWatchdogsStop` | The socket is removed at run-down and at the watchdog's stop and does not come back; with no `privatae-host` group it is root's and the reason is logged. |
@@ -36,6 +36,22 @@ says so. Those parts are enforced only by the code as written.
| | `local_host_linux_test.go`: `TestAUnixPairingsRunStillCannotReachThisMachine` | A run a unix pairing owns gets 403 from its egress proxy for loopback, this machine's public address and a name for it. | | | `local_host_linux_test.go`: `TestAUnixPairingsRunStillCannotReachThisMachine` | A run a unix pairing owns gets 403 from its egress proxy for loopback, this machine's public address and a name for it. |
| | `local_host_linux_test.go`: `TestTheOwnersImagePrefix` | Run-up takes only `<prefix>harness-<engine>@sha256:<digest>` under the owner's prefix, and pull credentials only for its registry. | | | `local_host_linux_test.go`: `TestTheOwnersImagePrefix` | Run-up takes only `<prefix>harness-<engine>@sha256:<digest>` under the owner's prefix, and pull credentials only for its registry. |
| | `local_host_linux_test.go`: `TestALocalImageRunsByItsIDWithoutAPull` | A local image runs by its ID with no pull only when the image with that ID carries a tag of the named repository. | | | `local_host_linux_test.go`: `TestALocalImageRunsByItsIDWithoutAPull` | A local image runs by its ID with no pull only when the image with that ID carries a tag of the named repository. |
| A Host with no platform reaches your model servers by name, never by address | `servelane_netns_linux_test.go`: `TestTheServeLaneSendsNoPacketForAnAddressInARealNamespace` | In a private network namespace, with a second namespace at 10.0.0.2 on this machine's LAN (its default route) and every service set to it: an address sent as the service, a name the lane does not carry, and `ollama`, `llm` and `pii` each answer `E` with the reason, the beat lists nothing served, and the other computer sees no packet addressed to it, no SYN and no connection, from the lane, the beat, the beat's model-management probe or `model:list` (which is refused with the reason); a direct dial afterwards is seen. Skipped where a namespace cannot be made. |
| | `servelane_linux_test.go`: `TestTheServeLaneTakesAServiceNameNeverAnAddress` | Over a tcp pairing: an address, a URL or an unknown name sent as the service, a first frame that is not `S` or is text, an `S` frame past 1024 bytes, `ollama` and `pii` set to addresses that are not this machine, and `llm` with nothing set each answer `E` with the reason and nothing after it; a listener at the address named gets no connection. |
| | `servelane_linux_test.go`: `TestServeVetTargetChecksEveryAddressItWouldDial` | A target is dialled at the addresses checked, every one, in the resolver's order: loopback in both families, `0.0.0.0` and `::` as loopback, a name whose every answer is loopback (both of `localhost`'s, and `::1` first where the resolver puts it first), and each of this machine's own interface addresses pass; a name with one answer elsewhere, a name that does not resolve, a TEST-NET address, a target with no port or a port out of range are refused. |
| | `servelane_dial_linux_test.go`: `TestTheServeLaneDialsEachCheckedAddressInTurn` | With `OLLAMA_ADDR=localhost:<port>`, a resolver that answers `::1` first and a stand-in listening on 127.0.0.1 only (nothing on `[::1]`): a job on the lane is answered by the stand-in, the beat lists `ollama` in `lane_serves`, model management is handed 127.0.0.1 and `model:list` lists the stand-in's models. |
| | `servelane_dial_linux_test.go`: `TestServePickTakesTheFirstCheckedAddressThatAnswers` | Of the addresses checked, the one given to an HTTP client is the first that accepts a connection, the first checked when none does, and a lone address as it is; never one it was not given. |
| | `servelane_linux_test.go`: `TestTheServeLaneStreamsAChatOverATCPPairing` | Over a tcp pairing, a chat to a stand-in OpenAI-compatible server streams: the Host reads chunk 1 before the server writes chunk 3, passed through for `llm` and through the filter for `ollama` with `models` off. |
| | `servelane_linux_test.go`: `TestTheServeLaneKeepsTheForwardsWalls` | With `inference` off at the machine, a job is refused before ollama sees anything. With `models` off, and on an anchored machine to its anchor, `/api/pull` and `/api/delete` answer 403 and never reach ollama. A member scoped to one model is refused another with 403, which never reaches the server, and is served its own. |
| | `servelane_linux_test.go`: `TestTheServeLaneHoldsSixteenStreamsAndRefusesTheSeventeenth` | Sixteen streams are held open at once; the 17th answers `E` busy with the reason and never reaches the server; once one ends, the next is accepted. |
| | `servelane_linux_test.go`: `TestTheServeLaneClosesAStreamIdleForTheBound` | With the bound shortened, a server that answers headers and then nothing has its stream closed at the bound, both ends, passed through and through the filter; a stream sending a chunk every 100 ms for four times the bound arrives whole. |
| | `servelane_linux_test.go`: `TestATCPPairingsBeatAdvertisesOnlyWhatItServes` | A tcp pairing's beat with ollama answering, `llm` unset and `pii` set to another computer carries `lane_serves` `["ollama"]`, `embed` and its models, `llm_port` 0, and no `llm-openai`, `pii`, `pii_port`, `pii_ok`, service slots or coder runs, though the machine could hold both; with all three served it lists and advertises all three, with `pii_ok` and no `pii_port`. A unix pairing's beat lists its lane and keeps coder runs; a tunnel pairing's has no `lane_serves` and keeps `pii_port`. |
| | `servelane_linux_test.go`: `TestTheServeLaneAuditsEachJob` | A served job writes one audit line with the service, the address dialled and the bytes each way; a refused one writes its reason. |
| | `servelane_models_linux_test.go`: `TestModelManagementOnALocalPairingReachesOnlyThisMachine` | Over a tcp pairing with `OLLAMA_ADDR` on another computer: the beat offers no model management; `model:list`, an approved `model:pull` and `model:drop` of a model this Host placed are each refused with the reason, at once, with no download job and the ledger unchanged; the bastion prints once in its output which address is not used and what to set. With `OLLAMA_ADDR` on this machine the same pairing lists its models. |
| | `servelane_models_linux_test.go`: `TestTheConsoleSaysARefusedTargetOnceUnderConcurrentCallers` | Thirty-two callers refusing the same target at once print one line in the bastion's output, and a changed reason prints one more. |
| | `servelane_models_linux_test.go`: `TestModelDaemonTargetKeepsATunnelPairingsAddress` | A tunnel pairing's model management keeps `OLLAMA_ADDR` wherever it points; the same name running as a local pairing is refused it, and is not counted local once its run ends. |
| | `servelane_models_linux_test.go`: `TestARefusingHostCannotSpinTheServeWorkers` | A Host that answers every parked serve worker with a refused or malformed job gets at most about two workers a second, not a dial loop. |
| | not covered by a test | A unix pairing's serve jobs end to end: its pool is counted (`TestTheUnixLaneCarriesTheBearerAndTheSameProtocol`) and its beat read, and the jobs run the same code as a tcp pairing's. |
| No MCP endpoint | `claims_static_test.go`: `TestNoRouteOpOrDependencySpeaksMCP` | Every HTTP route on every listener is in a fixed table and none names MCP; no management op, import or module names MCP; no string in the code is a URL or route with an `/mcp` path, or names JSON-RPC. The agent clients' tool-server config files are named only in `mcpconfig.go`, which reads them, and nothing in the config reader imports or can reach a dialler, an HTTP client, a websocket or a process start. | | No MCP endpoint | `claims_static_test.go`: `TestNoRouteOpOrDependencySpeaksMCP` | Every HTTP route on every listener is in a fixed table and none names MCP; no management op, import or module names MCP; no string in the code is a URL or route with an `/mcp` path, or names JSON-RPC. The agent clients' tool-server config files are named only in `mcpconfig.go`, which reads them, and nothing in the config reader imports or can reach a dialler, an HTTP client, a websocket or a process start. |
| | `claims_static_test.go`: `TestMCPReaderCheckCatchesADialAndAStrayBasename` | The check above reports a config reader that starts a process, imports `net/http` or reaches a dialler through another function, a Windows reader that starts a process, opens a socket, loads a DLL by name or opens a remote registry, and a second file that names a client's config file, the servers key, an `/mcp` URL or an `/mcp` route. | | | `claims_static_test.go`: `TestMCPReaderCheckCatchesADialAndAStrayBasename` | The check above reports a config reader that starts a process, imports `net/http` or reaches a dialler through another function, a Windows reader that starts a process, opens a socket, loads a DLL by name or opens a remote registry, and a second file that names a client's config file, the servers key, an `/mcp` URL or an `/mcp` route. |
| Grants are verified here | `anchor_test.go`: `TestPairingGrantSyncWalls` | A pairing grant signed by another key, for another machine, naming another anchor, or expired is refused, and the pairing stays pending. | | Grants are verified here | `anchor_test.go`: `TestPairingGrantSyncWalls` | A pairing grant signed by another key, for another machine, naming another anchor, or expired is refused, and the pairing stays pending. |
@@ -133,13 +149,16 @@ says so. Those parts are enforced only by the code as written.
| | `ownaddr_test.go`: `TestCoderEgressRefusesAnAddressTheKernelDeliversHere` | A coder run is refused an address the kernel delivers here, in every spelling, and refused anything when the lookup fails; `allow_lan` stays the exception. | | | `ownaddr_test.go`: `TestCoderEgressRefusesAnAddressTheKernelDeliversHere` | A coder run is refused an address the kernel delivers here, in every spelling, and refused anything when the lookup fails; `allow_lan` stays the exception. |
| | `ownaddr_netns_linux_test.go`: `TestAnyIPDestinationsAreRefusedInARealNamespace` | In a private network namespace with `local 192.0.2.0/24` and `local 2001:db8:a::/64` on `lo`, plus a broadcast route (198.51.101.255), anycast routes (198.51.102.0/24, 2001:db8:d::/64) and a multicast route: the real route lookup reads each as this machine, and a unicast route or no route as not; a 0.0.0.0 listener answers on both local ranges. A coder run's proxy and the anchored egress lane each refuse those addresses (local, broadcast and anycast), the IPv4-mapped and NAT64 forms of each IPv4 one, and a name resolving to each, with the lane's this-machine reason (a name the lane could not resolve does not count), and pass the unicast and no-route addresses and a name resolving to each, dialling the vetted address; each answers 403 end to end with a real dialer, the proxy over its socket and the anchored lane through its connection handler as the anchor host, to CONNECT by address and by name and to plain HTTP. The backup lanes' URL check refuses the same addresses, forms and names as this machine's or its network's and passes the unicast and no-route ones, and their dial wall refuses the local, broadcast and anycast addresses. Nothing reaches the listener. Skipped where a namespace cannot be made. | | | `ownaddr_netns_linux_test.go`: `TestAnyIPDestinationsAreRefusedInARealNamespace` | In a private network namespace with `local 192.0.2.0/24` and `local 2001:db8:a::/64` on `lo`, plus a broadcast route (198.51.101.255), anycast routes (198.51.102.0/24, 2001:db8:d::/64) and a multicast route: the real route lookup reads each as this machine, and a unicast route or no route as not; a 0.0.0.0 listener answers on both local ranges. A coder run's proxy and the anchored egress lane each refuse those addresses (local, broadcast and anycast), the IPv4-mapped and NAT64 forms of each IPv4 one, and a name resolving to each, with the lane's this-machine reason (a name the lane could not resolve does not count), and pass the unicast and no-route addresses and a name resolving to each, dialling the vetted address; each answers 403 end to end with a real dialer, the proxy over its socket and the anchored lane through its connection handler as the anchor host, to CONNECT by address and by name and to plain HTTP. The backup lanes' URL check refuses the same addresses, forms and names as this machine's or its network's and passes the unicast and no-route ones, and their dial wall refuses the local, broadcast and anycast addresses. Nothing reaches the listener. Skipped where a namespace cannot be made. |
| | `ownaddr_netns_linux_test.go`: `TestOnlyAUnicastRouteLeavesThisMachine` | The kernel's route type is read so that only unicast leaves this machine; local, broadcast, anycast, multicast, unspec, nat and xresolve are this machine. | | | `ownaddr_netns_linux_test.go`: `TestOnlyAUnicastRouteLeavesThisMachine` | The kernel's route type is read so that only unicast leaves this machine; local, broadcast, anycast, multicast, unspec, nat and xresolve are this machine. |
| | `hostops_test.go`: `TestAnchoredEgressRefusesLoopbackAndPrivateDestinations` | On an anchored machine the egress lane refuses loopback, `localhost`, private, ULA, link-local, CGNAT and metadata addresses and dials the vetted address; a local machine refuses its own ollama and keeps its LAN. | | | `hostops_test.go`: `TestAnchoredEgressRefusesLoopbackAndPrivateDestinations` | On an anchored machine the egress lane refuses loopback, `localhost`, private, ULA, link-local, CGNAT and metadata addresses and dials the vetted address; a local machine refuses its own ollama and its LAN. |
| | `ownaddr_test.go`: `TestAnchoredEgressRefusesThisMachinesOwnAddresses` | On an anchored machine the egress lane refuses an address on this machine's interface and one a `local` route delivers here, in every spelling, and refuses when the lookup fails; a neighbour passes; a local machine refuses those addresses too. | | | `ownaddr_test.go`: `TestAnchoredEgressRefusesThisMachinesOwnAddresses` | On an anchored machine the egress lane refuses an address on this machine's interface and one a `local` route delivers here, in every spelling, and refuses when the lookup fails; a neighbour passes; a local machine refuses those addresses too. |
| | `egress_local_test.go`: `TestLocalEgressRefusesThisMachineAndKeepsTheLAN` | On a local machine the egress lane refuses loopback, `localhost`, unspecified, `0.0.0.0/8`, link-local and metadata addresses, this machine's own LAN address and AnyIP range, each also inside an IPv4-mapped, NAT64 or 6to4 address, and a name with one such answer; it dials private, ULA, CGNAT and public addresses, and a name's vetted addresses, never the name. | | | `egress_local_test.go`: `TestLocalEgressRefusesThisMachineAndTheLAN` | On a local machine the egress lane refuses loopback, `localhost`, unspecified, `0.0.0.0/8`, link-local and metadata addresses, this machine's own LAN address and AnyIP range, each also inside an IPv4-mapped, NAT64 or 6to4 address, private, ULA and CGNAT addresses, and a name with one such answer; it dials public addresses, and a name's vetted addresses, never the name. |
| | `ownaddr_inside_linux_test.go`: `TestNetworksInsideThisMachineAreThisMachine` | In a private network namespace with two bridges whose only port is a veth (one with an IPv4 and an IPv6 range, one with a public range), a bridge whose port is a dummy, a bridge with no port, a veth routed to directly, a range routed through a veth with this machine's own address on it as the gateway, a veth whose only default route is in another table, a bridge over a VLAN, a route over a dummy link, the network on three veths that carry this machine's main-table default routes (an IPv4 multipath default over two, an IPv6 default over the third: this machine as a container), addresses through those defaults' gateways and a route through a gateway on the dummy-port bridge (a LAN behind a router VM): the real route lookup reads the first seven as this machine and the rest as leaving; a local machine's egress lane refuses the inside addresses, by address and by name, and dials the others; a coder run's proxy refuses the public range on an inside network and dials a public address through the gateway; the backup lanes allow the IPv4 and IPv6 addresses through the gateway. | | | `egress_network_test.go`: `TestLocalEgressRefusesTheLANAndTheMesh` | On a local machine the egress lane refuses the network around it as an anchored one does: private, ULA, CGNAT and benchmarking addresses, a NAT64 address for one, and a name with any one such answer, each answered 403 through the lane with nothing dialled; a public name passes at its vetted address. |
| | `ownaddr_inside_linux_test.go`: `TestNetworksInsideThisMachineAreThisMachine` | In a private network namespace with two bridges whose only port is a veth (one with an IPv4 and an IPv6 range, one with a public range), a bridge whose port is a dummy, a bridge with no port, a veth routed to directly, a range routed through a veth with this machine's own address on it as the gateway, a veth whose only default route is in another table, a bridge over a VLAN, a route over a dummy link, the network on three veths that carry this machine's main-table default routes (an IPv4 multipath default over two, an IPv6 default over the third: this machine as a container), addresses through those defaults' gateways and a route through a gateway on the dummy-port bridge (a LAN behind a router VM): the real route lookup reads the first seven as this machine and the rest as leaving; a local machine's egress lane refuses the inside addresses, by address and by name, as this machine, refuses the private addresses beyond them as its network, and dials the public ones; a coder run's proxy refuses the public range on an inside network and dials a public address through the gateway; the backup lanes allow the IPv4 and IPv6 addresses through the gateway. |
| | `ownaddr_inside_linux_test.go`: `TestInsideMachineReadsTheLinkAndABridgesPorts` | A veth, and a bridge whose every port is a veth, a tap or a dummy (or that has none), keep a packet on this machine; a bridge with a network card, a VLAN or a bridge among its ports, a network card, loopback, a VPN, a tunnel and a macvlan do not. | | | `ownaddr_inside_linux_test.go`: `TestInsideMachineReadsTheLinkAndABridgesPorts` | A veth, and a bridge whose every port is a veth, a tap or a dummy (or that has none), keep a packet on this machine; a bridge with a network card, a VLAN or a bridge among its ports, a network card, loopback, a VPN, a tunnel and a macvlan do not. |
| | `egress_local_test.go`: `TestLocalEgressLaneNeverDialsThisMachine` | Through the lane on a local machine with a real dialer: a request for a loopback listener gets 403 over CONNECT and plain HTTP and never reaches it; a LAN name is dialled at its vetted address; a dual-stack name falls back to its second address. | | | `egress_local_test.go`: `TestLocalEgressLaneNeverDialsThisMachine` | Through the lane on a local machine with a real dialer: a request for a loopback listener gets 403 over CONNECT and plain HTTP and never reaches it; a LAN name is dialled at its vetted address; a dual-stack name falls back to its second address. |
| | `egress_dial_test.go`: `TestDialEachDoesNotWaitOutAnAddressThatGoesNowhere` | The dial both egress lanes share tries only the vetted addresses, in order; an address that never answers holds up the next for well under its share of the budget, and is cancelled once the next answers. | | | `egress_dial_test.go`: `TestDialEachDoesNotWaitOutAnAddressThatGoesNowhere` | The dial both egress lanes share tries only the vetted addresses, in order; an address that never answers holds up the next for well under its share of the budget, and is cancelled once the next answers. |
| | `egress_relay_test.go`: `TestARefusingHostCannotSpinTheEgressWorkers` | A Host that answers every parked egress worker with a job this machine refuses (a request for `localhost`, a line that is no request) gets at most four workers a second back, not a dial loop, and nothing is dialled out. |
| | `egress_relay_test.go`: `TestEgressWorkersParkAgainAtOnceAfterAJobTheyTook` | A worker that took a job parks again at once: 40 accepted jobs through the four workers are each answered 200 well inside the 10 s one wait per job would take. |
| | `egress_local_test.go`: `TestLocalModelsSwitchLeavesNoEgressRouteToOllama` | On a local machine with `models` switched off for the host, the ollama forward filters, and the egress lane refuses `/api/pull` to ollama on loopback, `localhost` and this machine's LAN address without a dial. | | | `egress_local_test.go`: `TestLocalModelsSwitchLeavesNoEgressRouteToOllama` | On a local machine with `models` switched off for the host, the ollama forward filters, and the egress lane refuses `/api/pull` to ollama on loopback, `localhost` and this machine's LAN address without a dial. |
| | `egress_local_test.go`: `TestAnchoredEgressStillRefusesTheLAN` | An anchored machine's egress lane still refuses the LAN addresses a local machine reaches, with its own wording, and passes the public internet. | | | `egress_local_test.go`: `TestAnchoredEgressStillRefusesTheLAN` | An anchored machine's egress lane still refuses the LAN addresses a local machine reaches, with its own wording, and passes the public internet. |
| | `egress_obsolete_test.go`: `TestEveryLaneRefusesTheDeprecatedIPv6Forms` | The anchored and the local egress lane each refuse IPv4-compatible (`::/96`, with a public, a loopback and a LAN address inside), IPv4-translated (`::ffff:0:0/96`) and site-local (`fec0::/10`) addresses, and a name with one such answer among public ones, and pass the public internet; a coder run's proxy refuses the same addresses. | | | `egress_obsolete_test.go`: `TestEveryLaneRefusesTheDeprecatedIPv6Forms` | The anchored and the local egress lane each refuse IPv4-compatible (`::/96`, with a public, a loopback and a LAN address inside), IPv4-translated (`::ffff:0:0/96`) and site-local (`fec0::/10`) addresses, and a name with one such answer among public ones, and pass the public internet; a coder run's proxy refuses the same addresses. |
@@ -155,6 +174,24 @@ says so. Those parts are enforced only by the code as written.
| A networked machine is anchored to its first host | `pair_anchor_test.go`: `TestPairAnchorsANetworkedMachineToItsFirstHost` | On a networked machine with no anchor, the first pairing made with machine authority anchors it and is active; a local machine, an anchored one (a new host stays pending), a second pairing, and a caller without machine authority are left as they were. `claim-anchor` afterwards succeeds and changes nothing. Called directly, not through `pair`'s redeem. | | A networked machine is anchored to its first host | `pair_anchor_test.go`: `TestPairAnchorsANetworkedMachineToItsFirstHost` | On a networked machine with no anchor, the first pairing made with machine authority anchors it and is active; a local machine, an anchored one (a new host stays pending), a second pairing, and a caller without machine authority are left as they were. `claim-anchor` afterwards succeeds and changes nothing. Called directly, not through `pair`'s redeem. |
| | `anchor_test.go`: `TestClaimAnchorWithNoToken` | `claim-anchor` with no token on a machine anchored to that host succeeds and changes nothing, the host named in full, by handle or not at all; when the anchor's pairing is pending or revoked it is refused and the state is left as it was; another host is refused without `--rotate`. | | | `anchor_test.go`: `TestClaimAnchorWithNoToken` | `claim-anchor` with no token on a machine anchored to that host succeeds and changes nothing, the host named in full, by handle or not at all; when the anchor's pairing is pending or revoked it is refused and the state is left as it was; another host is refused without `--rotate`. |
| A root reinstall keeps a paired local machine local | `install_sh_test.go`: `TestInstallShKeepsAPairedLocalMachineLocal` | `install.sh`'s check finds a pairing in a folder whose mode is local (or unset), in either folder it looks in; a networked folder, or one with no pairing, gets the networked default. The rest of the installer is not run by a test. | | A root reinstall keeps a paired local machine local | `install_sh_test.go`: `TestInstallShKeepsAPairedLocalMachineLocal` | `install.sh`'s check finds a pairing in a folder whose mode is local (or unset), in either folder it looks in; a networked folder, or one with no pairing, gets the networked default. The rest of the installer is not run by a test. |
| A pairing code pins the Host's CA by its fingerprint | `pair_code_linux_test.go`: `TestAPairingCodePinsTheHostCAByFingerprint` | A pairing code whose enroll port presents no CA, another CA, a CA under another fingerprint, a leaf another CA signed, or a leaf without the name `host.cseries.internal` is refused before the token is spent, and nothing is saved. |
| | `pair_code_linux_test.go`: `TestAPairingCodePairsAsTheHostCAFileDoes` | A good code makes the same TCP pairing `--redeem-url` and `--host-ca` make, with that CA pinned; the same code again is "already paired" (exit 3) and changes nothing; a Host of the same name under another CA is a plain refusal. |
| | `pair_code_linux_test.go`: `TestAPairingCodeIsReadStrictly` | A code that is garbled, another version, or has an extra, missing or malformed field is refused, and no refusal repeats the token. |
| | `pair_code_linux_test.go`: `TestACodeTheHostMadeDecodesAndPins` | A code made by the C-Series Host's own `pairing_code` decodes to its address and token, and its fingerprint is the one `caSHA256` computes from the same CA. |
| | `pair_code_linux_test.go`: `TestAPairingCodeWithAKeyGivenTwiceIsRefused` | A code that names a field twice is refused. |
| | `pair_code_linux_test.go`: `TestThePinnedChainTakesOnlyAValidCAWithTheFingerprint` | A certificate with the fingerprint that is not a CA, an expired CA, or a chain of the leaf alone is refused; the pinned CA after other certificates is the one kept. Called directly, not through TLS. |
| | `pair_code_linux_test.go`: `TestHostCAFileWithASameNamedHostUnderAnotherCAIsNotAlreadyPaired` | With `--host-ca` too, a Host of the same name under another CA is a plain refusal, not exit 3, and its token is not spent; the same Host under its own CA is exit 3. |
| | `pair_code_linux_test.go`: `TestARedeemAnsweringAPairedNameUnderAnotherCAIsNotAlreadyPaired` | When the redeem answers with a paired name the token's handle did not show, a Host of that name under another CA is a plain refusal, not exit 3, and nothing is saved. |
| A local machine adds a second host only on a yes typed there | `pair_code_linux_test.go`: `TestALocalMachineAddsASecondHostOnlyOnAYesTypedHere` | With no terminal, or any answer but y or yes, the second host is refused before its token is spent and the first pairing is unchanged; callers with no person to ask keep the refusal and open no terminal; a typed yes adds it beside the first. |
| | `pair_code_linux_test.go`: `TestTheSecondHostAnswerIsNeverReadFromStandardInput` | The answer is read from the controlling terminal, never from standard input: with no terminal and a y waiting on standard input, the question finds no terminal, refuses, and leaves standard input unread. |
| | `service_test.go`: `TestServiceOwnerRuleOnAnAnchoredBox` | On a local machine that serves two hosts, a host may not replace or stop a server the other built, and may change its own; with one host no owner rule applies. |
| | `service_test.go`: `TestCoderRunOwnerRuleOnALocalBoxServingTwoHosts` | On a local machine that serves two hosts, a host may not stop a coder run the other started, or one of unknown owner, and may stop its own; with one host nothing changes. |
| | `pair_code_linux_test.go`: `TestTheSecondHostQuestionEscapesTheTokensName` | The second-host question shows a token's name that is not a plain host name quoted, with its control bytes escaped. |
| The installers keep the pairing code out of every process's arguments | `install_sh_test.go`: `TestInstallShTakesTheCodeOnlyFromTheEnvironment` | `install.sh` takes the code only from `PRIVATAE_PAIR_CODE` and unsets it before it starts any program: no program it starts, the downloaded bastion included, has the code in its arguments, its shell's command line or its environment, and the code reaches `pair -` on standard input. Run on stand-in programs; nothing is downloaded. |
| | `install_sh_test.go`: `TestInstallShRefusesABadPairCodeBeforeDownloading` | An empty or partial `PRIVATAE_PAIR_CODE`, and any unknown argument (the removed `--pair` and a bare code included), are refused before the download and never echoed. |
| | `install_sh_test.go`: `TestInstallShPairsOverStdinAndStartsAUserUnit` | The pairing functions hand the code to `pair -` on standard input, then start the bastion; a failed pair starts nothing. Run on stand-in programs, not on a real systemd. |
| | `install_sh_test.go`: `TestInstallShStopsAnOutsideCopyAndChecksTheUnitStaysUp` | On Linux as yourself, a copy of the bastion running outside the user unit is stopped before the unit starts, and a unit that keeps restarting is reported as not running, with a failure. Run on a stand-in systemd, in a PID namespace of its own. |
| | `install_ps1_test.go`: `TestInstallPs1PairShape` | `install.ps1` pipes the code to `pair -`, never into an argument or the output, and checks it before downloading. Read from the file; `TestInstallPs1PairBlockRunsUnderPwsh` runs the block where pwsh is installed. Not run on Windows. |
| A terminal is checked here and logged | `session_test.go`: `TestSessionOpensWithoutAGrantOnALocalBox` | On a local machine a terminal with no grant and no approval opens and is audited. The same test calls `sessionPinOfferedKey` directly: the first key offered is pinned and a different one is refused. | | A terminal is checked here and logged | `session_test.go`: `TestSessionOpensWithoutAGrantOnALocalBox` | On a local machine a terminal with no grant and no approval opens and is audited. The same test calls `sessionPinOfferedKey` directly: the first key offered is pinned and a different one is refused. |
| | `session_test.go`: `TestSessionRefusedWithoutGrantIs126` | A grant that is sent is verified: a bad signature, an op not listed, or an op the grant denies is refused and audited. | | | `session_test.go`: `TestSessionRefusedWithoutGrantIs126` | A grant that is sent is verified: a bad signature, an op not listed, or an op the grant denies is refused and audited. |
| | `fuzz_grants_test.go`: `FuzzEachOpIsAllowedOnlyByItsOwnFamily` | On an anchored machine the family check (`capsAllow`) lets a host other than the anchor run an op, the session ops included, only if its pairing grant names that op's family. Called directly, not through the terminal handler. | | | `fuzz_grants_test.go`: `FuzzEachOpIsAllowedOnlyByItsOwnFamily` | On an anchored machine the family check (`capsAllow`) lets a host other than the anchor run an op, the session ops included, only if its pairing grant names that op's family. Called directly, not through the terminal handler. |
+3
View File
@@ -7,6 +7,9 @@
| `pair --replace <token>` | Re-pair an existing host: rotates its credentials, keeps its port, context store and DCN state. | | `pair --replace <token>` | Re-pair an existing host: rotates its credentials, keeps its port, context store and DCN state. |
| `pair [--replace] --redeem-url unix:///<socket> <token>` | Pair with a Host on this machine over its enroll socket. Its CA is pinned for life. See [A Host on this machine](host-on-this-machine.md). | | `pair [--replace] --redeem-url unix:///<socket> <token>` | Pair with a Host on this machine over its enroll socket. Its CA is pinned for life. See [A Host on this machine](host-on-this-machine.md). |
| `pair [--replace] --redeem-url https://<host>:<port>/<path> --host-ca <pem> <token>` | Pair with a Host in a VM on this machine over its enroll port, trusting that CA alone. Its CA is pinned for life. See [A Host in a VM on this machine](host-on-this-machine.md#a-host-in-a-vm-on-this-machine). | | `pair [--replace] --redeem-url https://<host>:<port>/<path> --host-ca <pem> <token>` | Pair with a Host in a VM on this machine over its enroll port, trusting that CA alone. Its CA is pinned for life. See [A Host in a VM on this machine](host-on-this-machine.md#a-host-in-a-vm-on-this-machine). |
| `pair <pairing code>` | Pair with the one code (`skp_…`) a Cloud Host's "Set up a bastion" card gives: it carries the enroll address and the fingerprint of the Host's CA, and the CA is taken from the enroll port only under that fingerprint. On a local machine that already serves a Host it asks, on this machine's terminal, whether to add this one; only a yes adds it. |
| | The card's one line installs the bastion, pairs it with that code and starts it: `curl -fsSL https://privatae.ai/bastion/install.sh \| PRIVATAE_PAIR_CODE='skp_…' sh` on macOS and Linux (the code goes in the environment, never in an argument, which the process list would show), and `& ([scriptblock]::Create((irm https://privatae.ai/bastion/install.ps1))) -Pair 'skp_…'` on Windows. Either hands the code to `pair -` on standard input. |
| | `pair` exits 0 when it paired, 3 when this machine is already paired with that Host (nothing changed), and anything else on a failure. |
| `run` | Bring up the tunnel and lanes for every paired host. | | `run` | Bring up the tunnel and lanes for every paired host. |
| `status` | Show every pairing and its tunnel status. | | `status` | Show every pairing and its tunnel status. |
| `ui` | Open the local console in your browser. | | `ui` | Open the local console in your browser. |
+31 -1
View File
@@ -99,4 +99,34 @@ are checked before the token is read.
Such a pairing has no WireGuard forwards (inference, PII, service slots) and Such a pairing has no WireGuard forwards (inference, PII, service slots) and
no coder runs: a run's control plane is a tunnel port or a socket in a unix no coder runs: a run's control plane is a tunnel port or a socket in a unix
pairing's runs directory, and a TCP pairing has neither, so its run-up is pairing's runs directory, and a TCP pairing has neither, so its run-up is
refused before any container starts. refused before any container starts. Its model servers ride the serve lane,
below.
## The model servers on this machine
A pairing over the socket or the lane port reaches the model servers on this
machine over the serve lane, `/bastion-serve`, instead of the tunnel's
forwards. The bastion keeps two workers parked on the Host's lane; the Host
hands one a connection and names a service: `ollama`, `llm` or `pii`. The
bastion reaches the server set on this machine for it (`OLLAMA_ADDR`, else
`127.0.0.1:11434`; `PRIVATAE_LLM_ADDR`; `PRIVATAE_PII_ADDR`), and only when
every address it names is this machine, a container on a network made here
included. A server on another computer is refused, and so is an address sent
in place of a service; nothing is dialled for either. Each connection needs
the `inference` family and goes through the forwards' filter, so a model
scope, the `models` switch and an anchor apply as they do over the tunnel.
One pairing holds at most 16 streams, a stream silent both ways for 10 minutes
is closed, and each one is written to the pairing's audit log. Model
management for such a pairing (the model list, pulls and removals) reaches
`OLLAMA_ADDR` under the same rule, and when an address set here is not this
computer, the bastion prints a line in its output saying which one and what to
set. When the bastion runs as the machine's service, its output is its log:
`journalctl -u privatae-bastion` on Linux, `/var/log/privatae-bastion.log` on
macOS (readable by root only), and
`%SystemRoot%\System32\config\systemprofile\AppData\Roaming\privatae-bastion\bastion.log`
on Windows. When it runs in a terminal (`privatae-bastion run`), the line is
printed there.
The heartbeat lists what the lane serves in `lane_serves`, and advertises
`embed`, `llm-openai` and `pii` only for those. It never advertises the
service slots, or coder runs over TCP.
+42 -2
View File
@@ -27,6 +27,42 @@
directories that are root's and not writable by others unless sticky), or directories that are root's and not writable by others unless sticky), or
no socket is made in it. On an anchored machine its run-up needs the no socket is made in it. On an anchored machine its run-up needs the
standing coder grant like any pairing's. standing coder grant like any pairing's.
- **A Host with no platform reaches your model servers by name, never by
address.** A pairing with a Host on this machine, or in a VM on it, has no
tunnel and so no forwards. Its model servers ride the serve lane instead:
the bastion keeps two workers parked on the Host's lane, and the Host hands
one a connection that names a service, `ollama`, `llm` or `pii`. The
bastion dials the address set on this machine for that service
(`OLLAMA_ADDR`, `PRIVATAE_LLM_ADDR`, `PRIVATAE_PII_ADDR`), and only when
every address it names is this machine: loopback, an address on its
interfaces, or one the kernel delivers here, which includes a container on
a network made on this machine. A server on another computer, an address
or URL sent in place of a service, and a name the lane does not carry are
each refused with a reason before anything is dialled. Each connection
needs the `inference` family and goes through the forwards' own filter:
the model scope, `/api/pull`, `/api/delete` and the other store paths
refused without `models` (on an anchored machine, always), and the model
lists rewritten. One pairing holds at most 16 streams here, and the 17th is
refused as busy. A stream with no byte either way for 10 minutes is closed;
a response's length is not capped. Each job is written to the pairing's
audit log. Such a pairing's heartbeat lists what the lane serves
(`lane_serves`) and advertises `embed`, `llm-openai` and `pii` only for
those, and it never advertises the service slots, or coder runs over TCP,
which do not run there. Before this, it advertised all of them with
nothing serving them. Model management for such a pairing is held to the
same rule: the beat's model-swap probe, `model:list`, `model:pull` with its
probes, and `model:drop` reach `OLLAMA_ADDR` only when it is this machine,
at the addresses checked; otherwise each is refused with the reason before
anything is dialled or an approval is read, and the bastion prints a line in
its output saying which address is not used and what to set. A target set by name is dialled
at each address it was checked for, in turn, as a plain dial of the name
is, so `localhost` reaches a server listening on 127.0.0.1 only when the
name answers `::1` first; no other address is dialled. A tunnel pairing
keeps `OLLAMA_ADDR` wherever it points, as its forwards do. A worker whose
job is refused (malformed, refused or busy) waits a second before it parks
again, so a Host that refuses or misuses every job cannot make this machine
dial in a loop. The egress lane's workers do the same: one that took a job
parks again at once, and one whose job was refused waits a second.
- **Scoped tunnel.** The WireGuard peer's only allowed address is the paired - **Scoped tunnel.** The WireGuard peer's only allowed address is the paired
host's single tunnel address, so the tunnel can reach nothing else. A host's single tunnel address, so the tunnel can reach nothing else. A
pairing that offers anything wider is refused, and a pairing file that pairing that offers anything wider is refused, and a pairing file that
@@ -92,8 +128,12 @@
the rest of `64:ff9b::/32`, the three deprecated IPv6 forms, and any the rest of `64:ff9b::/32`, the three deprecated IPv6 forms, and any
address this machine takes as its own, checked as above and also inside an address this machine takes as its own, checked as above and also inside an
IPv4-mapped, NAT64 or 6to4 address. It IPv4-mapped, NAT64 or 6to4 address. It
keeps your LAN: a private, CGNAT or reserved address that is not this refuses the network around this machine as well, as an anchored machine's
machine's is dialled. A container or VM on a network made on this machine lane does: a private, ULA, CGNAT or reserved address, by address or by any
address a name resolves to. Your own machines are reached through their
approved lanes, never through egress. (Before this, a local machine's lane
kept your LAN: such an address that was not this machine's was dialled.)
A container or VM on a network made on this machine
is this machine, not the LAN; on macOS and Windows, which have no route is this machine, not the LAN; on macOS and Windows, which have no route
lookup here, such an address is dialled. Every address a name resolves to is checked there lookup here, such an address is dialled. Every address a name resolves to is checked there
too, and only those addresses are dialled, in order, the next as soon as too, and only those addresses are dialled, in order, the next as soon as