1 Commits

Author SHA1 Message Date
Privatae 48899b41a9 bastion 0.63.1
ci / personal-data (push) Successful in 4s
ci / test (push) Successful in 4m58s
ci / fuzz (push) Successful in 3m33s
2026-11-03 22:56:18 +00:00
78 changed files with 10366 additions and 307 deletions
+156
View File
@@ -22,6 +22,162 @@ From 0.39.1 on, a heading marks a version *not published* or *held* once
that is settled. The newest entries may not be published yet; the manifest
says which version is. Older entries are not marked.
## 0.63.1
The same program as 0.63.0; its source is now publishable.
## 0.63.0
One Host is in charge of each machine, every pairing comes along when the
machine moves to the service, a Host that waits says who admits it, and a Host
at home governs a server on the internet that can't reach it back.
- **Your Host connects to a server it can reach and that can't reach it
back.** A Host on your own computer, behind your home router, now governs a
server on the internet with no VPS and nothing opened at home. On the
server, `privatae-bastion pair --open` shows a code, good once and for 15
minutes; give your Host the server's address and the code. Your Host then
opens the connections, to TCP 8812 on the server, and every lane runs over
them as before: the bastion is still the TLS client, the Host's CA still
the only one it trusts, its own certificate still presented. The code is
proved by each side to the other over the Host's certificate on that
connection and never sent; a wrong one costs a wait that doubles, and the
fifth closes the window. The port is open only while such a Host is served
or a window is open, keeps nothing but its own Hosts' connections, and
forwards nothing; the audit records the window, each Host connecting and
going away, and refused connections. The wire contract a Host implements is
[docs/host-connects.md](docs/host-connects.md).
- **Nobody touches a firewall for it.** While that port is in use, the
bastion keeps its own rule allowing it inbound, and removes the rule when
the last such Host and window are gone, and on `uninstall`: in Windows
Firewall (inbound TCP 8812, every profile, for the bastion's own program
only), and on Linux in ufw or firewalld when one of them is on. It never
turns a firewall on or off or changes anything else in it, and leaves alone
a rule of the machine's own that already allows the port. A firewall it does
not manage (nftables or iptables rules, ufw for an account that can't change
it, macOS set to block all incoming) is said once, with what to allow, in
`status` and `doctor`. Each rule added or removed is in the audit. A cloud
provider's firewall still needs the port allowed.
- **Your own Host gets back in while strangers flood the port.** Of the 256
connections the port takes in their handshake at once, 64 are kept for
addresses a Host proved itself from in the last 7 days, so your Host gets in
at its first try however many addresses flood it. Any other address is
refused at random once 64 are running, more often as it fills, and always
at 192 (as OpenSSH's `MaxStartups` does), so a new Host still pairs on a
quiet port. The addresses are kept in memory only, never written anywhere;
a bastion that has just restarted knows none until its Hosts connect again.
- **A server you add has your Host in charge.** `pair --open`, run with
machine authority on a server no Host is in charge of, puts the Host that
pairs through it in charge (the anchor), so your Host counts the server as
yours. A local server becomes networked for it, unless it already serves
another Host you said yes to. A Host on the server itself still wins, and an
organisation's policy that names another Host still decides. With a Host in
charge, each terminal and write on the server waits for your approval, as on
any server a Host is in charge of: enrol a second factor there
(`stepup-enroll`), as `pair --open` says.
- **Adding the same server again just works.** The same Host pairing the
same server again through a fresh `pair --open` code replaces its pairing
in place, with no `unpair` first: new credentials, the same state, its grants
and its place in charge kept (also under a new name). It is the same Host
when its connection presents the CA the pairing already holds; another
Host with the same name is still refused.
- **`pair --open` right after `unpair`.** The running bastion lets go of
port 8812 only once it sees the pairing gone, within 15 seconds; `pair
--open` now waits up to 25 seconds for that, and says so, instead of
failing with "address already in use".
- **The tests can't touch a live bastion.** `scripts/e2e-host-connects.sh`
hid `/run/privatae` but not `/run/privatae-bastion`, and on a machine
running a real bastion it replaced that bastion's local sockets. It now puts
an empty mount over all of `/run` and `/tmp` and the bastion's state, and
refuses to start while it can still see the machine's bastion; a test holds
every namespace script in `scripts/` to that. A lane test that raced its
own workers' start no longer fails now and then.
- **The Host on your own computer is the one in charge of it.** A Host that
runs on the same machine as the bastion (over a socket, or on a loopback
lane port, as `npx cseries` sets up) takes charge when it is paired from an
Administrator terminal, also from a remote Host that held it before, and
never "whichever paired first". The Host it replaced is told so on its
last heartbeat, keeps its pairing, and waits for the Host on your computer
to admit it, as any other Host does. The change is recorded in the
machine's pairing audit. `claim-anchor` with no host picks the Host on
this machine, and naming another while one is paired is refused with the
reason.
- **An organisation's policy can name the Host in charge, and only that
overrides it.** On a machine an organisation manages, a policy it deploys
names the Host: on Windows the `AnchorHost` value under
`HKLM\SOFTWARE\Policies\Privatae\Bastion` (Group Policy, Intune), on macOS
the `AnchorHost` key of the managed preference `ai.privatae.bastion` (a
configuration profile), on Linux `anchor_host` in the root-owned
`/etc/privatae-bastion/policy.json`. That Host takes charge when it is
paired, every other Host waits for it, and `claim-anchor` for any other is
refused in words that say an organisation manages the machine. Joining a
domain is not a policy.
- **A Host that waits says who admits it.** `pair` on a machine with a Host
in charge says the code was used, that the new Host waits for that Host to
admit it, and that it asks you to approve it. `status`, `status --json`
(`waiting_for`), `doctor`, the console and the tray say "waiting for
<Host> to admit it", never "not answering". The heartbeat to the Host in
charge lists each waiting pairing with `waiting_for`, `transport`,
`same_machine` and `origin` (paired here, carried from your account, or the
Host that was in charge before), what that Host needs to ask you.
- **Moving to the service keeps every pairing, with no question.** The move
from your account's bastion to the bastion for the whole machine carries
every pairing your account had, also when the service already served a Host
of its own: each comes across with its keys and folders and a control port
of its own. Into a machine with a Host in charge each waits for that Host to
admit it; into one with none it is served as it was. The installer's last
words say what the service serves and what waits, instead of "no Host is
needed".
- **Each pairing runs on its own.** One Host's lanes failing (a port taken,
a certificate that does not read) stopped the whole bastion, and every other
Host with it. Now that pairing alone stops: it is said in the log, the
machine's events and `status`, and it is tried again by itself after a
pause that grows from 15 seconds to 5 minutes.
- **The model server on your machine is found by itself.** With
`PRIVATAE_LLM_ADDR` unset, the bastion looks on this machine's loopback, and
nowhere else, for an OpenAI-compatible server on its usual port: 8080
(llama.cpp), 8000 (vLLM), 1234 (LM Studio), in that order, and serves the
first whose `/v1/models` answers with a list of models; on Linux only one
run by root, the bastion's own account or the machine's owner, never another
account's. It asks again every
minute and as soon as the one it found stops answering, so what the
heartbeat says is served is what the lane reaches. The variable still
overrides it. The log says once what was found, or that nothing was, and
`status` and `doctor` show it.
- **Smaller ones, from a replay of a Windows desktop.** The service's log
names the Windows service manager when it stops the bastion, not the tray.
The owner's `status` and the other reads work over a network sign-in (ssh,
remote PowerShell), which the service's permissions refused with "Access is
denied". `pair` and `claim-anchor` no longer tell you to run or restart the
bastion when one is running: it serves a new pairing within 15 seconds.
## 0.62.1 (not published)
Superseded by 0.63.0, which carries these changes.
Windows, the installer for the whole machine:
- **Installing over a running bastion works.** The tray runs from the
service's own program in every signed-in session, and Windows will not
replace a running program, so a reinstall or upgrade stopped with "Cannot
create a file when that file already exists." The installer now moves the
program in place aside (the bastion removes it at its next start) and puts
the new one in its name, as the bastion's own self-update does.
- **A failed install never leaves the bastion stopped.** The service the
installer stopped is started again before the failure is reported, and the
message says what failed and whether the bastion is running.
- **Your tray is the new version too.** After the install, the tray in your
session is started again from the new program. Run from an Administrator
terminal by hand, the installer says an open tray changes at your next
sign-in. The trays of other people signed in are left running the earlier
program until their next sign-in, never stopped as an earlier install.
- **A reinstall for your account alone runs the new version.** With no
pairing code, it said the bastion was running while the copy already
running was still the earlier program; it now stops that copy before
starting the new one.
## 0.62.0
Windows, with the bastion installed for the whole machine (the
+98 -10
View File
@@ -799,13 +799,46 @@ func lastOK(host string) time.Time {
// pairingEntry is one pairing on this box as told to the ANCHOR host: name,
// handle, state, since. Same privacy line as peers.go — never keys or grants.
//
// For a pairing that waits to be admitted it also says what the anchor Host
// needs to raise the approval prompt for the person (C-Series: the Network
// app's pending pairing, approved with approve_pairing):
//
// waiting_for the Host that admits it: the anchor, which reads its own
// name here (anchor_local.go admitter)
// transport how its lanes would run: "tunnel" (WireGuard through the
// relay, a Cloud Host), "tcp" (a Host's lane port) or "unix"
// (a socket on this machine)
// same_machine its Host runs on this machine (anchor_local.go
// sameMachineHost)
// origin how it came: "paired" (a code redeemed at this machine),
// "carried" (moved in from an account's own bastion when the
// machine moved to the system service, servicecarry.go), or
// "former_anchor" (it was in charge of this machine until a
// Host the rules put first took over, anchor_local.go)
//
// The approval itself is unchanged: pairing_grant_sync with terms the anchor
// signs, {kind: pairing_grant, server: the beat's hostname, anchor: this
// beat's anchor.anchor_host, subject: host_name, subject_handle: handle, caps,
// issued_at, expires}, over the anchor's own mgmt lane (pairing_ops.go).
type pairingEntry struct {
HostName string `json:"host_name"`
Handle string `json:"handle"`
State string `json:"state"`
Since string `json:"since,omitempty"`
HostName string `json:"host_name"`
Handle string `json:"handle"`
State string `json:"state"`
Since string `json:"since,omitempty"`
WaitingFor string `json:"waiting_for,omitempty"`
Transport string `json:"transport,omitempty"`
SameMachine bool `json:"same_machine,omitempty"`
Origin string `json:"origin,omitempty"`
}
// The origins pairingEntry reports.
const (
pairingOriginPaired = "paired"
pairingOriginCarried = "carried"
pairingOriginFormerAnchor = "former_anchor"
)
// pairingsReport lists EVERY pairing on the box, including the one asking.
// "since" is first_seen — when the host was first paired here — kept across
// --replace, so a pending host re-presenting a token keeps its original age
@@ -821,13 +854,41 @@ func pairingsReport() []pairingEntry {
if err != nil {
return nil
}
who := admitter()
replaced := ""
if p, err := anchorReplacedPath(); err == nil {
if raw, err := os.ReadFile(p); err == nil {
var r anchorReplaced
if json.Unmarshal(raw, &r) == nil {
replaced = r.Host
}
}
}
out := make([]pairingEntry, 0, len(cfgs))
for _, c := range cfgs {
if c.HostName == "" || c.invalid != "" {
continue
}
out = append(out, pairingEntry{HostName: c.HostName, Handle: handleOf(c.HostName),
State: c.stateName(), Since: sinceOf(c)})
e := pairingEntry{HostName: c.HostName, Handle: handleOf(c.HostName),
State: c.stateName(), Since: sinceOf(c)}
if c.State == pairingPending {
e.WaitingFor = who
e.Transport = "tunnel"
if c.unixTransport() {
e.Transport = "unix"
} else if c.tcpTransport() {
e.Transport = "tcp"
}
e.SameMachine = sameMachineHost(c)
e.Origin = pairingOriginPaired
if c.Origin == pairingOriginCarried {
e.Origin = pairingOriginCarried
}
if replaced != "" && replaced == c.HostName {
e.Origin = pairingOriginFormerAnchor
}
}
out = append(out, e)
}
return out
}
@@ -855,7 +916,7 @@ func unapprovedAttempts(now time.Time) []peerAnnounce {
if err != nil || now.Sub(since) < unapprovedAfter {
continue
}
out = append(out, peerAnnounce(e))
out = append(out, peerAnnounce{HostName: e.HostName, Handle: e.Handle, State: e.State, Since: e.Since})
}
return out
}
@@ -874,6 +935,16 @@ func pairingEnforcing(self string) map[string]any {
return out
}
out["pairing_state"] = c.stateName()
// A Host the rules put first took the anchor from this one (anchor_local.go):
// {by, at, why, waiting_for}. Its last beat before its lanes stop carries it.
if note := anchorReplacedNote(self); note != nil {
out["anchor_replaced"] = note
}
if c.State == pairingPending {
if who := admitter(); who != "" {
out["waiting_for"] = who
}
}
// The name THIS pairing is known by on this box — what a reach grant must
// name as its subject (service_reach.go). A host can hold more than one name
// (a legacy one and the current one); this says which one the box enforces.
@@ -1001,7 +1072,7 @@ func anchorAlreadyHolds(cfgs []*Config, prev *anchorClaim, host string) bool {
// pickPairing resolves --host against this box's pairings. It accepts the full
// host_name the box reports, or just the handle in front of it, because the
// name the user is looking at depends on which surface they read it from —
// `8abb770bbfde71d0` in Circles, `8abb770bbfde71d0.host.privatae.ai` in the
// `1b2c3d4e5f607182` in Circles, `1b2c3d4e5f607182.host.privatae.ai` in the
// claim-anchor error. Refusing the short form would mean refusing the thing the
// user most obviously typed.
func pickPairing(cfgs []*Config, want string) (*Config, error) {
@@ -1041,6 +1112,12 @@ func cmdClaimAnchor(token string, rotate bool, owner, host string) error {
return err
}
prev := currentAnchor()
// Which Host is in charge (anchor_local.go): the one an organisation's
// policy names, else a Host on this machine. Either decides here, before
// the paths below, which know only "the box's pairings".
if handled, err := claimAnchorByRule(cfgs, prev, token, rotate, owner, host); handled {
return err
}
// Already anchored to the host this asks for (no token, no other host
// named), and that host's pairing is active: done, not an error. `pair`
// anchors a networked box to its first host by itself, and an installer
@@ -1236,11 +1313,22 @@ func cmdClaimAnchor(token string, rotate bool, owner, host string) error {
// singleton on each one as it goes, rather than needing a separate pass.
_ = maybeWriteMarker()
fmt.Printf("anchored to %s (owner %q)\n", cfg.HostName, owner)
fmt.Println(" Every host paired from now on waits, pending, until this anchor host approves it.")
fmt.Println(" " + restartHint())
fmt.Printf(" Every Host paired from now on waits until %s admits it: %s asks you to approve it.\n", cfg.HostName, cfg.HostName)
fmt.Println(" " + anchorFollowNote())
return nil
}
// anchorFollowNote says how the change reaches the bastion: a running one
// reads the anchor on every op and follows the pairing files within
// supervisePeriod, so nothing is restarted.
func anchorFollowNote() string {
if pairDaemonRunning() {
return fmt.Sprintf("The bastion running on this machine follows within %d seconds; nothing needs restarting.",
int(supervisePeriod.Seconds()))
}
return "Start the bastion so it serves it: " + runCommandHint()
}
// writeAnchor signs and installs anchor.json under the box key. A fresh anchor
// (or a rotated one) starts with NO verify key: the new anchor host's key is
// pinned on first contact, never carried over from the previous host.
+537
View File
@@ -0,0 +1,537 @@
package main
// anchor_local.go — who is in charge of this machine: one authoritative Host,
// the anchor (anchor.go), and which Host that is.
//
// A machine needs one authoritative Host, and the code already has that: the
// anchor. The anchor is still the one mechanism, with its signature and
// verification model unchanged (writeAnchor signs it
// under the box key; grants verify under the anchor Host's pinned key). What
// this file decides is WHICH Host holds it, by three rules, strongest first:
//
// 1. An ORGANISATION'S POLICY, explicitly deployed to the machine, names the
// Host in charge (anchorPolicy below). It is the only thing that overrides
// the rule after it, and it is meant for a machine owned by someone other
// than its local admin. Windows: the AnchorHost value under
// HKLM\SOFTWARE\Policies\Privatae\Bastion (Group Policy, Intune). macOS:
// the AnchorHost key of the managed preference domain ai.privatae.bastion
// (a configuration profile). Linux: anchor_host in the root-owned
// /etc/privatae-bastion/policy.json. Joining a domain is not a policy:
// nothing is read from the domain itself. With a policy, claim-anchor for
// any other Host is refused in words that say an organisation manages the
// machine, a new pairing of another Host waits for the named one, and the
// named one takes the anchor when it is paired with machine authority.
// 2. A HOST ON THIS MACHINE is always the one in charge, never "whichever
// paired first". Paired with machine authority it takes the anchor, from a
// remote anchor too; the replaced anchor becomes an ordinary pairing that
// waits for it to admit it (as claim-anchor --rotate leaves the old one),
// the change is recorded in the pairing audit, and the replaced anchor is
// told on one last heartbeat (anchorReplacedNote, run.go superviseOnce).
// A Host on this machine that already holds the anchor keeps it: a second
// one waits, as any other pairing does, and moving the anchor between two
// of them is claim-anchor --rotate as before.
// 3. Otherwise the first Host paired with a networked machine (cmdPair, as
// before this file).
//
// WHAT "ON THIS MACHINE" IS (sameMachineHost). A pairing whose lanes run over
// a unix socket (transport unix): the socket is a file on this machine's own
// disk. Or a pairing whose lane port is a loopback address (transport tcp,
// lane_addr 127.0.0.0/8, ::1 or localhost): loopback never leaves this
// machine's network stack, and it is how a C-Series Host in a VM on this
// computer is reached (npx cseries forwards the VM's lane ports to loopback,
// packages/cseries/src/vm/common.js). Something else forwarded onto loopback
// is something the machine's administrator set up, and taking the anchor needs
// that administrator's authority anyway. Any other address, a LAN one
// included, is another machine. The name the Host gives itself
// (home.cseries.localhost) is not used: it is the Host's word, not the
// machine's.
//
// It adds no power. With machine authority an administrator could already
// claim the anchor on an unanchored box, delete anchor.json, or release an
// anchor (anchor_release.go). What it changes is that the right Host is put in
// charge without anyone having to know the order things were paired in.
import (
"encoding/json"
"fmt"
"net"
"os"
"path/filepath"
"strings"
"time"
)
// anchorPolicy is an organisation's deployed policy naming the Host in charge.
type anchorPolicy struct {
Host string // a host_name, or its handle
Source string // where it was read, as a person would look for it
}
// readAnchorPolicy reads the policy from this platform's managed store
// (anchorpolicy_*.go). A seam: the suite sets a policy without one.
var readAnchorPolicy = readAnchorPolicyOS
// currentAnchorPolicy is the deployed policy, or nil when there is none or it
// names nothing a host name can be.
func currentAnchorPolicy() *anchorPolicy {
p := readAnchorPolicy()
if p == nil {
return nil
}
p.Host = strings.TrimSpace(p.Host)
if !validHostName(p.Host) {
return nil
}
return p
}
// policyNames says whether the policy names host (its full name or handle).
func policyNames(p *anchorPolicy, host string) bool {
if p == nil || host == "" {
return false
}
return strings.EqualFold(p.Host, host) || strings.EqualFold(p.Host, handleOf(host)) ||
strings.EqualFold(handleOf(p.Host), host)
}
// orgManagedRefusal is the words for a Host the policy does not name.
func orgManagedRefusal(p *anchorPolicy, who string) error {
return fmt.Errorf("an organisation manages this machine: its policy (%s) names %s as the Host in charge of it, "+
"so %s cannot take charge here. Whoever manages this machine for the organisation changes that policy", p.Source, p.Host, who)
}
// sameMachineHost: the pairing's Host runs on this machine (see the header).
func sameMachineHost(c *Config) bool {
if c == nil {
return false
}
if c.unixTransport() {
return true
}
if !c.tcpTransport() {
return false
}
return loopbackAddr(c.LaneAddr)
}
// loopbackAddr: host:port whose host is a loopback IP literal or localhost.
func loopbackAddr(hostPort string) bool {
h, _, err := net.SplitHostPort(strings.TrimSpace(hostPort))
if err != nil {
return false
}
if strings.EqualFold(h, "localhost") {
return true
}
ip := net.ParseIP(h)
return ip != nil && ip.IsLoopback()
}
// admitter is the Host a pending pairing waits for: the anchor, or on a
// machine an organisation's policy governs and nobody holds yet, the Host the
// policy names. "" when nothing on this machine admits anyone.
func admitter() string {
if a := currentAnchor(); a != nil {
return a.AnchorHost
}
if boxMode() == modeNetworked {
if p := currentAnchorPolicy(); p != nil {
return p.Host
}
}
return ""
}
// pendingUnderPolicy: on a networked machine with no anchor yet, a policy
// that names another Host makes a new pairing wait for that one.
func pendingUnderPolicy(handle string) bool {
if boxMode() != modeNetworked || currentAnchor() != nil {
return false
}
p := currentAnchorPolicy()
return p != nil && !policyNames(p, handle)
}
// anchorTake is what taking the anchor did, for `pair` and claim-anchor to say.
type anchorTake struct {
Taken bool
Why string // the rule, in words
Replaced string // the previous anchor, if there was one
Waiting []string // pairings that now wait for the new anchor to admit them
}
// The rules by name, in the audit and on the replaced anchor's beat.
const (
anchorWhyPolicy = "policy"
anchorWhyLocal = "same_machine"
anchorWhyFirst = "first_pairing"
)
func anchorWhyWords(why string, c *Config, p *anchorPolicy) string {
switch why {
case anchorWhyPolicy:
return "an organisation's policy (" + p.Source + ") names it as the Host in charge of this machine"
case anchorWhyLocal:
return "it runs on this machine (" + c.laneWhere() + "), and a Host on the machine is always the one in charge of it"
case anchorWhyWindow:
return "you opened the pairing window for it here (`pair --open`), and no Host was in charge of this machine"
}
return "it is the first Host paired with this networked machine"
}
// anchorAfterPair puts the right Host in charge after `pair` wrote cfg (see the
// header). Only on a networked machine and with machine authority, which
// claim-anchor needs anyway. Reports what it did.
func anchorAfterPair(cfg *Config, all []*Config) (anchorTake, error) {
if cfg == nil || boxMode() != modeNetworked || !machineAuthorityHeld() {
return anchorTake{}, nil
}
prev := currentAnchor()
if prev != nil && prev.AnchorHost == cfg.HostName {
return anchorTake{}, nil
}
pol := currentAnchorPolicy()
why := ""
switch {
case pol != nil:
if !policyNames(pol, cfg.HostName) {
return anchorTake{}, nil
}
why = anchorWhyPolicy
case sameMachineHost(cfg):
if prev != nil {
if pc := pairingFor(all, prev.AnchorHost); pc != nil && sameMachineHost(pc) && pc.active() {
// A Host on this machine already holds it: this one waits.
return anchorTake{}, nil
}
}
why = anchorWhyLocal
case prev == nil && len(all) == 1:
why = anchorWhyFirst
default:
return anchorTake{}, nil
}
t, err := takeAnchor(cfg, prev, why, pol, "")
if err != nil {
return anchorTake{}, fmt.Errorf("paired with %s, but putting it in charge of this machine failed: %w\n anchor it with: %s",
cfg.HostName, err, anchorCommandHint())
}
return t, nil
}
// anchorWhyWindow: the person opened a pairing window for this Host at this
// machine (`pair --open`), with machine authority, and nobody was in charge.
const anchorWhyWindow = "pair_open"
// anchorAfterWindow is anchorAfterPair for a Host paired through a window
// opened at this machine (pair_window.go). The window is the person's say-so,
// typed at this machine, for this one Host, so on a machine nobody is in
// charge of yet that Host takes the anchor, as the first Host of a networked
// machine does, whatever else is paired and in either mode. The rules above it
// still come first: an organisation's policy that names another Host leaves
// it waiting, and a Host on this machine wins (a paired one that is not
// revoked keeps the machine for itself, even before it holds the anchor).
//
// A LOCAL machine has no anchor (claim-anchor and set-mode refuse one there),
// so it is made networked first, which set-mode does with the same authority.
// On a local machine already serving another Host the person said yes to, the
// window's Host does not take it: that would leave the other one waiting.
//
// openerAuthority is whether the person who opened the window held machine
// authority; the process committing the pairing (pair --open's own door, or
// the running bastion's) must hold it too.
func anchorAfterWindow(cfg *Config, all []*Config, openerAuthority bool) (anchorTake, error) {
took, err := anchorAfterPair(cfg, all)
if err != nil || took.Taken {
return took, err
}
if cfg == nil || !openerAuthority || !machineAuthorityHeld() || currentAnchor() != nil ||
cfg.invalid != "" || cfg.State == pairingRevoked {
return anchorTake{}, nil
}
pol := currentAnchorPolicy()
why := anchorWhyWindow
if pol != nil {
if !policyNames(pol, cfg.HostName) {
return anchorTake{}, nil
}
why = anchorWhyPolicy
}
// On a networked machine nobody is in charge of, every pairing is refused
// every op (unanchoredNetworkedRefusal), so the others that takeAnchor
// leaves waiting lose nothing they had; they now have a Host to admit them.
// On a local machine the other Hosts are served, so they are not demoted.
local := boxMode() != modeNetworked
for _, o := range all {
if o.HostName == cfg.HostName {
continue
}
if why != anchorWhyPolicy && sameMachineHost(o) && o.invalid == "" && o.State != pairingRevoked {
return anchorTake{}, nil
}
if local && o.active() {
return anchorTake{}, nil
}
}
if local {
if err := setBoxMode(modeNetworked); err != nil {
return anchorTake{}, fmt.Errorf("paired with %s, but making this machine one a Host is in charge of failed: %w", cfg.HostName, err)
}
pairingAudit("set_mode", cfg.HostName, "ok: networked, so the Host paired through the window opened here can be put in charge")
}
t, err := takeAnchor(cfg, nil, why, pol, "")
if err != nil {
return anchorTake{}, fmt.Errorf("paired with %s, but putting it in charge of this machine failed: %w\n anchor it with: %s",
cfg.HostName, err, anchorCommandHint())
}
return t, nil
}
// anchorReplaced is <configDir>/anchor.replaced.json: the last time a Host
// took the anchor from another. The replaced anchor's run reads it on its
// last beat (anchorReplacedNote); the audit keeps the history.
type anchorReplaced struct {
Host string `json:"host"` // the anchor that was replaced
By string `json:"by"`
At string `json:"at"`
Why string `json:"why"`
}
func anchorReplacedPath() (string, error) {
d, err := configDir()
if err != nil {
return "", err
}
return filepath.Join(d, "anchor.replaced.json"), nil
}
// anchorReplacedNote is what the replaced anchor's heartbeat carries, or nil
// when self was not replaced as this machine's anchor.
func anchorReplacedNote(self string) map[string]any {
p, err := anchorReplacedPath()
if err != nil {
return nil
}
raw, err := os.ReadFile(p)
if err != nil {
return nil
}
var r anchorReplaced
if json.Unmarshal(raw, &r) != nil || r.Host == "" || r.Host != self {
return nil
}
a := currentAnchor()
if a == nil || a.AnchorHost != r.By {
return nil
}
return map[string]any{"by": r.By, "at": r.At, "why": r.Why, "waiting_for": r.By}
}
// clearAnchorReplacedFor removes the replaced-anchor record once that Host is
// admitted again.
func clearAnchorReplacedFor(host string) {
p, err := anchorReplacedPath()
if err != nil {
return
}
raw, err := os.ReadFile(p)
if err != nil {
return
}
var r anchorReplaced
if json.Unmarshal(raw, &r) == nil && r.Host == host {
_ = os.Remove(p)
}
}
// takeAnchor makes cfg this machine's anchor, under the box key as
// claim-anchor does (writeAnchor), and every other pairing that was served an
// ordinary one waiting for it. A previous anchor is recorded as replaced.
// owner "" keeps the machine's owner name.
func takeAnchor(cfg *Config, prev *anchorClaim, why string, pol *anchorPolicy, owner string) (anchorTake, error) {
if cfg.invalid != "" {
return anchorTake{}, fmt.Errorf("%s cannot hold the anchor: %s", cfg.HostName, cfg.invalid)
}
if cfg.State == pairingRevoked {
return anchorTake{}, fmt.Errorf("%s is REVOKED on this box — it will not be put in charge; pair that Host again first (`privatae-bastion pair --replace <token>`)", cfg.HostName)
}
owner = strings.TrimSpace(owner)
if owner == "" {
if oc := currentOwnerClaim(); oc != nil && oc.Owner != "" {
owner = oc.Owner
} else if prev != nil {
owner = prev.Owner
} else {
owner = handleOf(cfg.HostName)
}
}
if err := claimNameOK(owner); err != nil {
return anchorTake{}, fmt.Errorf("owner: %w", err)
}
// The anchor never waits on itself.
if !cfg.active() {
if err := cfg.updateOnDisk(func(d *Config) { d.State = pairingActive }); err != nil {
return anchorTake{}, err
}
}
if err := writeAnchor(owner, cfg.HostName); err != nil {
return anchorTake{}, err
}
if _, err := writeOwnerClaim(owner); err != nil {
return anchorTake{}, err
}
// A consent the old anchor left for a rotation is not this.
if p, err := anchorConsentPath(); err == nil {
_ = os.Remove(p)
}
t := anchorTake{Taken: true, Why: anchorWhyWords(why, cfg, pol)}
if prev != nil && prev.AnchorHost != cfg.HostName {
t.Replaced = prev.AnchorHost
}
// Every other pairing that was served now waits for the new anchor: a
// grant from the old one does not verify under the new one's key, so
// "active" would be lanes that refuse every op.
cfgs, err := loadPairings()
if err != nil {
return t, err
}
for _, c := range cfgs {
if c.HostName == cfg.HostName || !c.active() {
continue
}
if err := c.updateOnDisk(func(d *Config) { d.State = pairingPending }); err != nil {
return t, err
}
t.Waiting = append(t.Waiting, c.HostName)
}
at := time.Now().UTC().Format(time.RFC3339)
if t.Replaced != "" {
if p, err := anchorReplacedPath(); err == nil {
if b, err := json.MarshalIndent(anchorReplaced{Host: t.Replaced, By: cfg.HostName, At: at, Why: why}, "", " "); err == nil {
_ = atomicWrite(p, b)
}
}
}
decision := "ok: in charge of this machine because " + t.Why
if t.Replaced != "" {
decision += "; it replaced " + t.Replaced
}
pairingAudit("anchor_take", cfg.HostName, decision)
for _, h := range t.Waiting {
pairingAudit("anchor_take", h, "waits for "+cfg.HostName+" to admit it")
}
_ = maybeWriteMarker()
kickBeats()
return t, nil
}
// claimAnchorByRule is claim-anchor's part of the rules in the header, before
// its own paths run. handled: the command's answer is err. Not handled: the
// rest of claim-anchor decides, as before.
func claimAnchorByRule(cfgs []*Config, prev *anchorClaim, token string, rotate bool, owner, host string) (bool, error) {
named := strings.TrimSpace(host)
if token != "" {
h, _, err := decodeToken(token)
if err != nil {
return true, err
}
named = h
}
if pol := currentAnchorPolicy(); pol != nil {
want := pairingFor(cfgs, pol.Host)
if want == nil {
for _, c := range cfgs {
if policyNames(pol, c.HostName) {
want = c
break
}
}
}
if named != "" && !policyNames(pol, named) {
if c, err := pickPairing(cfgs, named); err != nil || want == nil || c.HostName != want.HostName {
return true, orgManagedRefusal(pol, named)
}
}
if token != "" {
return false, nil // the token names the Host the policy names
}
if want == nil {
return true, fmt.Errorf("an organisation manages this machine: its policy (%s) names %s as the Host in charge "+
"of it, and %s is not paired with this machine yet. Pair it (`privatae-bastion pair <its code>`) and it takes charge",
pol.Source, pol.Host, pol.Host)
}
if prev != nil && prev.AnchorHost == want.HostName && want.active() {
fmt.Printf("already anchored to %s (owner %q)\n", prev.AnchorHost, prev.Owner)
return true, nil
}
t, err := takeAnchor(want, prev, anchorWhyPolicy, pol, owner)
if err != nil {
return true, err
}
printAnchorTake(want.HostName, t)
return true, nil
}
if rotate {
return false, nil
}
var locals []*Config
for _, c := range cfgs {
if sameMachineHost(c) && c.invalid == "" && c.State != pairingRevoked {
locals = append(locals, c)
}
}
if len(locals) == 0 {
return false, nil
}
var want *Config
switch {
case token != "":
if pairingFor(locals, named) == nil {
return true, fmt.Errorf("%s runs on this machine and is the Host in charge of it, so %s does not take the "+
"anchor here; %s admits it instead (it asks you to approve it)", locals[0].HostName, named, locals[0].HostName)
}
return false, nil
case named != "":
c, err := pickPairing(cfgs, named)
if err != nil {
return true, err
}
if !sameMachineHost(c) {
return true, fmt.Errorf("%s runs on this machine and is the Host in charge of it, so %s does not take the "+
"anchor here; %s admits it instead (it asks you to approve it)", locals[0].HostName, c.HostName, locals[0].HostName)
}
want = c
case len(locals) == 1:
want = locals[0]
default:
return false, nil // several Hosts on this machine: --host says which
}
if prev != nil && prev.AnchorHost == want.HostName && want.active() {
fmt.Printf("already anchored to %s (owner %q)\n", prev.AnchorHost, prev.Owner)
return true, nil
}
if prev != nil {
if pc := pairingFor(cfgs, prev.AnchorHost); pc != nil && sameMachineHost(pc) && pc.active() {
return false, nil // moving it between two Hosts on this machine is --rotate, as before
}
}
t, err := takeAnchor(want, prev, anchorWhyLocal, nil, owner)
if err != nil {
return true, err
}
printAnchorTake(want.HostName, t)
return true, nil
}
// printAnchorTake is claim-anchor's report of takeAnchor.
func printAnchorTake(host string, t anchorTake) {
fmt.Printf("anchored to %s: %s.\n", host, t.Why)
if t.Replaced != "" {
fmt.Printf(" %s was in charge before; it is told so, and now waits for %s to admit it.\n", t.Replaced, host)
}
if len(t.Waiting) > 0 {
fmt.Printf(" Waiting for %s to admit them: %s. %s asks you to approve each.\n", host, strings.Join(t.Waiting, ", "), host)
}
fmt.Println(" Every Host paired from now on waits until " + host + " admits it.")
fmt.Println(" " + anchorFollowNote())
}
+530
View File
@@ -0,0 +1,530 @@
//go:build !windows
package main
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
const (
tLocal = "home.cseries.localhost"
tLocal2 = "second.cseries.localhost"
tRemote = "remote.host.privatae.ai"
)
// localBox is a networked box with machine authority in a temp config dir.
func localBox(t *testing.T) {
t.Helper()
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
withMarkerAt(t, filepath.Join(t.TempDir(), "machine.json"))
prev := machineAuthorityHeld
machineAuthorityHeld = func() bool { return true }
t.Cleanup(func() { machineAuthorityHeld = prev })
withPolicy(t, nil)
if err := setBoxMode(modeNetworked); err != nil {
t.Fatal(err)
}
}
func withPolicy(t *testing.T, p *anchorPolicy) {
t.Helper()
prev := readAnchorPolicy
readAnchorPolicy = func() *anchorPolicy {
if p == nil {
return nil
}
c := *p
return &c
}
t.Cleanup(func() { readAnchorPolicy = prev })
}
// writeTCPPairing is a Host reached on a lane port, as pair_tcp.go writes it.
func writeTCPPairing(t *testing.T, host, lane, state string) {
t.Helper()
pd, _ := pairingsDir()
// Its control port is one nothing listens on, whatever runs on the machine
// running the suite (a bastion of its own answers on 7878).
b, _ := json.Marshal(Config{HostName: host, Transport: transportTCP, LaneAddr: lane, WGPrivateKey: "k", State: state,
ListenAddr: "127.0.0.1:1"})
if err := os.WriteFile(filepath.Join(pd, host+".json"), b, 0o600); err != nil {
t.Fatal(err)
}
}
func pairingNamed(t *testing.T, host string) (*Config, []*Config) {
t.Helper()
all, err := loadPairings()
if err != nil {
t.Fatal(err)
}
c := pairingFor(all, host)
if c == nil {
t.Fatalf("%s not paired", host)
}
return c, all
}
func auditLines(t *testing.T) string {
t.Helper()
d, _ := configDir()
b, _ := os.ReadFile(filepath.Join(d, "pairing_audit.jsonl"))
return string(b)
}
// What "on this machine" is: a unix socket, or a lane port on loopback. A LAN
// address, the relay's tunnel, and a Host's own name are not.
//
// MUTATION: return true for every tcp pairing in sameMachineHost. RESULT: red.
func TestSameMachineIsASocketOrALoopbackLane(t *testing.T) {
for _, c := range []struct {
cfg Config
want bool
}{
{Config{Transport: transportUnix, LaneSocket: "/run/h/lane.sock"}, true},
{Config{Transport: transportTCP, LaneAddr: "127.0.0.1:8811"}, true},
{Config{Transport: transportTCP, LaneAddr: "127.3.4.5:8811"}, true},
{Config{Transport: transportTCP, LaneAddr: "[::1]:8811"}, true},
{Config{Transport: transportTCP, LaneAddr: "localhost:8811"}, true},
{Config{Transport: transportTCP, LaneAddr: "10.0.2.2:28811"}, false},
{Config{Transport: transportTCP, LaneAddr: "192.168.1.20:8811"}, false},
{Config{Transport: transportTCP, LaneAddr: "home.cseries.localhost:8811"}, false},
{Config{HostName: "x.cseries.localhost", BastionTunnelIP: "fd12:3456::a:1"}, false},
} {
if got := sameMachineHost(&c.cfg); got != c.want {
t.Errorf("%+v: same machine = %v, want %v", c.cfg, got, c.want)
}
}
}
// The S0 order the other way round: the machine was anchored to a remote Host,
// then a Host on this machine is paired with machine authority. It takes the
// anchor; the remote one becomes an ordinary pairing waiting for it, is told
// on its beat, and the audit says so.
//
// MUTATION: drop the sameMachineHost case from anchorAfterPair. RESULT: red.
// Leave the replaced anchor active in takeAnchor. RESULT: red.
func TestAHostOnThisMachineTakesTheAnchorFromARemoteOne(t *testing.T) {
localBox(t)
writePairing(t, tRemote)
if err := writeAnchor("the owner", tRemote); err != nil {
t.Fatal(err)
}
writeTCPPairing(t, tLocal, "127.0.0.1:8811", pairingPending)
cfg, all := pairingNamed(t, tLocal)
took, err := anchorAfterPair(cfg, all)
if err != nil || !took.Taken || took.Replaced != tRemote {
t.Fatalf("a Host on this machine must take the anchor: %+v %v", took, err)
}
if a := currentAnchor(); a == nil || a.AnchorHost != tLocal || a.Owner != "the owner" {
t.Fatalf("anchor = %+v", a)
}
if got := loadState(t, tLocal); got != pairingActive {
t.Errorf("the new anchor waits on itself: %q", got)
}
if got := loadState(t, tRemote); got != pairingPending {
t.Errorf("the replaced anchor is %q, want pending", got)
}
a := auditLines(t)
if !strings.Contains(a, `"op":"anchor_take"`) || !strings.Contains(a, "it replaced "+tRemote) ||
!strings.Contains(a, "waits for "+tLocal+" to admit it") {
t.Errorf("the audit does not record it:\n%s", a)
}
note := anchorReplacedNote(tRemote)
if note == nil || note["by"] != tLocal || note["why"] != anchorWhyLocal {
t.Fatalf("the replaced anchor's beat is not told: %v", note)
}
if e := pairingEnforcing(tRemote); e["anchor_replaced"] == nil || e["waiting_for"] != tLocal {
t.Errorf("the enforcing report of the replaced anchor: %v", e)
}
if anchorReplacedNote(tLocal) != nil {
t.Error("the new anchor is told it was replaced")
}
var former *pairingEntry
for _, e := range pairingsReport() {
if e.HostName == tRemote {
e := e
former = &e
}
}
if former == nil || former.Origin != pairingOriginFormerAnchor || former.WaitingFor != tLocal || former.Transport != "tunnel" || former.SameMachine {
t.Errorf("the anchor's report of the former anchor: %+v", former)
}
}
// A networked box with no anchor and a remote Host already serving: the Host on
// this machine takes the anchor, and the remote one waits for it.
func TestAHostOnThisMachineAnchorsABoxThatAlreadyServesARemoteHost(t *testing.T) {
localBox(t)
writePairing(t, tRemote)
writeTCPPairing(t, tLocal, "127.0.0.1:8811", "")
cfg, all := pairingNamed(t, tLocal)
took, err := anchorAfterPair(cfg, all)
if err != nil || !took.Taken || took.Replaced != "" || strings.Join(took.Waiting, ",") != tRemote {
t.Fatalf("%+v %v", took, err)
}
if got := loadState(t, tRemote); got != pairingPending {
t.Errorf("the remote Host holds no grant from the new anchor and must wait: %q", got)
}
}
// Never "whichever paired first": a remote Host never takes the anchor from a
// Host on this machine, and a second Host on this machine does not either.
//
// MUTATION: drop the "a Host on this machine already holds it" return.
// RESULT: red.
func TestTheAnchorOnThisMachineIsNotReplaced(t *testing.T) {
for _, newcomer := range []struct{ host, lane string }{{tRemote, "10.0.2.2:28811"}, {tLocal2, "127.0.0.1:9811"}} {
t.Run(newcomer.host, func(t *testing.T) {
localBox(t)
writeTCPPairing(t, tLocal, "127.0.0.1:8811", "")
if err := writeAnchor("the owner", tLocal); err != nil {
t.Fatal(err)
}
writeTCPPairing(t, newcomer.host, newcomer.lane, pairingPending)
cfg, all := pairingNamed(t, newcomer.host)
if took, err := anchorAfterPair(cfg, all); took.Taken || err != nil {
t.Fatalf("%s took the anchor: %+v %v", newcomer.host, took, err)
}
if a := currentAnchor(); a == nil || a.AnchorHost != tLocal {
t.Fatalf("anchor moved: %+v", a)
}
if got := loadState(t, newcomer.host); got != pairingPending {
t.Errorf("state %q", got)
}
})
}
}
// Without machine authority, or on a local machine, nothing is anchored.
func TestTakingTheAnchorNeedsANetworkedMachineAndItsAuthority(t *testing.T) {
localBox(t)
writePairing(t, tRemote)
if err := writeAnchor("the owner", tRemote); err != nil {
t.Fatal(err)
}
writeTCPPairing(t, tLocal, "127.0.0.1:8811", pairingPending)
machineAuthorityHeld = func() bool { return false }
cfg, all := pairingNamed(t, tLocal)
if took, _ := anchorAfterPair(cfg, all); took.Taken {
t.Fatal("anchored without machine authority")
}
machineAuthorityHeld = func() bool { return true }
localBox(t)
if err := setBoxMode(modeLocal); err != nil {
t.Fatal(err)
}
writeTCPPairing(t, tLocal, "127.0.0.1:8811", "")
cfg, all = pairingNamed(t, tLocal)
if took, _ := anchorAfterPair(cfg, all); took.Taken || currentAnchor() != nil {
t.Fatal("anchored a local machine")
}
}
// An organisation's policy wins over the Host on this machine: the Host it
// names takes the anchor, from a Host on this machine too; any other Host
// waits for it; and claim-anchor for any other Host is refused in words that
// say an organisation manages the machine.
//
// MUTATION: check sameMachineHost before the policy in anchorAfterPair.
// RESULT: red. Drop the policy refusal in claimAnchorByRule. RESULT: red.
func TestAnOrganisationsPolicyNamesTheHostInCharge(t *testing.T) {
localBox(t)
withPolicy(t, &anchorPolicy{Host: tRemote, Source: "HKLM\\SOFTWARE\\Policies\\Privatae\\Bastion AnchorHost"})
// Unanchored: the Host on this machine waits for the one the policy names.
if _, state, err := pairPrecheckState(false, handleOf(tLocal)); err != nil || state != pairingPending {
t.Fatalf("a Host the policy does not name must wait: %q %v", state, err)
}
if _, state, _ := pairPrecheckState(false, handleOf(tRemote)); state != "" {
t.Fatalf("the Host the policy names waits for nobody: %q", state)
}
writeTCPPairing(t, tLocal, "127.0.0.1:8811", pairingPending)
cfg, all := pairingNamed(t, tLocal)
if took, _ := anchorAfterPair(cfg, all); took.Taken {
t.Fatal("the Host on this machine took the anchor against the organisation's policy")
}
if got := admitter(); got != tRemote {
t.Errorf("a pending pairing waits for %q, want the policy's Host", got)
}
err := cmdClaimAnchor("", false, "", tLocal)
if err == nil || !strings.Contains(err.Error(), "an organisation manages this machine") ||
!strings.Contains(err.Error(), "names "+tRemote) {
t.Fatalf("claim-anchor for another Host: %v", err)
}
if err := cmdClaimAnchor("", false, "", ""); err == nil || !strings.Contains(err.Error(), "not paired with this machine yet") {
t.Fatalf("claim-anchor before the policy's Host is paired: %v", err)
}
// The policy's Host is paired: it takes the anchor, the other one waits.
writePairingState(t, tRemote, "")
cfg, all = pairingNamed(t, tRemote)
took, err := anchorAfterPair(cfg, all)
if err != nil || !took.Taken {
t.Fatalf("the policy's Host must take the anchor: %+v %v", took, err)
}
if a := currentAnchor(); a == nil || a.AnchorHost != tRemote {
t.Fatalf("anchor = %+v", a)
}
if !strings.Contains(auditLines(t), "an organisation's policy") {
t.Errorf("the audit does not name the policy: %s", auditLines(t))
}
if err := cmdClaimAnchor("", true, "", tLocal); err == nil || !strings.Contains(err.Error(), "an organisation manages this machine") {
t.Fatalf("claim-anchor --rotate to another Host: %v", err)
}
}
// A policy deployed after a Host on this machine took the anchor: claim-anchor
// with no argument hands it to the Host the policy names, without --rotate.
func TestClaimAnchorFollowsAPolicyDeployedLater(t *testing.T) {
localBox(t)
writeTCPPairing(t, tLocal, "127.0.0.1:8811", "")
if err := writeAnchor("the owner", tLocal); err != nil {
t.Fatal(err)
}
writePairingState(t, tRemote, pairingPending)
withPolicy(t, &anchorPolicy{Host: handleOf(tRemote), Source: "/etc/privatae-bastion/policy.json anchor_host"})
if err := cmdClaimAnchor("", false, "", ""); err != nil {
t.Fatal(err)
}
if a := currentAnchor(); a == nil || a.AnchorHost != tRemote {
t.Fatalf("anchor = %+v", a)
}
if got := loadState(t, tLocal); got != pairingPending {
t.Errorf("the Host on this machine is %q, want waiting for the policy's Host", got)
}
}
// claim-anchor with no argument on a box that serves a remote Host and a Host
// on this machine picks the one on this machine; naming the remote one is
// refused while a Host on this machine is paired.
//
// MUTATION: return false at the top of claimAnchorByRule's same-machine part.
// RESULT: red (the old "say which one" refusal).
func TestClaimAnchorPicksTheHostOnThisMachine(t *testing.T) {
localBox(t)
writePairing(t, tRemote)
writeTCPPairing(t, tLocal, "127.0.0.1:8811", "")
if err := cmdClaimAnchor("", false, "", tRemote); err == nil || !strings.Contains(err.Error(), "runs on this machine and is the Host in charge") {
t.Fatalf("naming the remote Host: %v", err)
}
if err := cmdClaimAnchor("", false, "", ""); err != nil {
t.Fatal(err)
}
if a := currentAnchor(); a == nil || a.AnchorHost != tLocal {
t.Fatalf("anchor = %+v", a)
}
if got := loadState(t, tRemote); got != pairingPending {
t.Errorf("the remote Host is %q", got)
}
}
// The replaced anchor's running beat is beaten once more before its lanes stop.
//
// MUTATION: drop the tellReplacedAnchor call in superviseOnce, or the
// tellReplacedAnchorsBeforeStop body. RESULT: red.
func TestTheReplacedAnchorIsToldOnALastBeat(t *testing.T) {
localBox(t)
writePairing(t, tRemote)
if err := writeAnchor("the owner", tRemote); err != nil {
t.Fatal(err)
}
writeTCPPairing(t, tLocal, "127.0.0.1:8811", pairingPending)
remote, _ := pairingNamed(t, tRemote)
cfg, all := pairingNamed(t, tLocal)
if took, err := anchorAfterPair(cfg, all); err != nil || !took.Taken {
t.Fatal(took, err)
}
var told any
beat := func() { told = pairingEnforcing(tRemote)["anchor_replaced"] }
_, unregister := registerRunBeat(tRemote, beat)
defer unregister()
done := make(chan struct{})
close(done)
supervisor.Lock()
supervisor.runs = map[string]*pairingRun{tRemote: {cancel: func() {}, done: done, cfg: remote}}
supervisor.Unlock()
t.Cleanup(func() { supervisor.Lock(); supervisor.runs = nil; supervisor.Unlock() })
// The daemon stopping first (an installer's Restart-Service right after
// `pair`) tells it as well.
tellReplacedAnchorsBeforeStop()
if told == nil {
t.Fatal("the daemon stopped without a last beat to the replaced anchor")
}
told = nil
superviseOnce()
if told == nil {
t.Fatal("the replaced anchor was stopped without a last beat that says so")
}
// Admitted again, it is no longer told.
clearAnchorReplacedFor(tRemote)
if anchorReplacedNote(tRemote) != nil {
t.Error("an admitted former anchor still hears it was replaced")
}
}
// One pairing's fatal error stops that pairing alone: nothing reaches the
// daemon's fatal channel, the failure is recorded for status, and it is tried
// again after a pause.
//
// MUTATION: send the error to supervisor.errc again. RESULT: red.
func TestOnePairingsFatalErrorStopsOnlyThatPairing(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
freshLocalEvents(t)
// A Host over TCP whose CA does not read: runUnixPairing fails at once.
bad := &Config{HostName: "broken.example.test", Transport: transportTCP, LaneAddr: "127.0.0.1:1", Bearer: "b",
CACert: "not a certificate", ListenAddr: "127.0.0.1:0"}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
errc := make(chan error, 4)
supervisor.Lock()
prevCtx, prevErrc, prevRuns := supervisor.ctx, supervisor.errc, supervisor.runs
supervisor.ctx, supervisor.errc, supervisor.runs = ctx, errc, nil
supervisor.Unlock()
t.Cleanup(func() {
supervisor.Lock()
supervisor.ctx, supervisor.errc, supervisor.runs = prevCtx, prevErrc, prevRuns
supervisor.Unlock()
pairingRecovered(bad.HostName)
})
if !startPairing(bad) {
t.Fatal("not started")
}
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if _, ok := loadPairingFailures()[bad.HostName]; ok {
break
}
time.Sleep(20 * time.Millisecond)
}
select {
case err := <-errc:
t.Fatalf("one pairing's error went to the whole daemon: %v", err)
case <-time.After(100 * time.Millisecond):
}
f, ok := loadPairingFailures()[bad.HostName]
if !ok || f.Error == "" || f.Tries != 1 {
t.Fatalf("the failure is not recorded: %+v", f)
}
if pairingRetryDue(bad.HostName) {
t.Error("tried again with no pause")
}
if retryPause(1) != supervisePeriod || retryPause(3) != 4*supervisePeriod || retryPause(40) != retryMaxPause {
t.Errorf("pauses %v %v %v", retryPause(1), retryPause(3), retryPause(40))
}
found := false
for _, e := range localEventsNewestFirst() {
found = found || strings.Contains(e.Text, "The other Hosts go on being served")
}
if !found {
t.Error("the machine's events do not say so")
}
}
// The Windows service manager's stop is logged as its own, not the tray's.
// The service handler cannot run here, so its source is read.
//
// MUTATION: call requestShutdown() in sysservice_windows.go. RESULT: red.
func TestTheServiceManagersStopIsNotLoggedAsTheTrays(t *testing.T) {
src, err := os.ReadFile("sysservice_windows.go")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(src), "requestShutdownFor(shutdownFromService)") || strings.Contains(string(src), "requestShutdown()") {
t.Error("the service's stop does not name the service manager")
}
run, _ := os.ReadFile("run.go")
if !strings.Contains(string(run), `"\nstopping — " + shutdownWhy`) {
t.Error("cmdRun does not log who asked it to stop")
}
if strings.Contains(shutdownFromService, "tray") {
t.Error(shutdownFromService)
}
}
// Over a network logon (ssh) the service cannot be queried for its process id;
// the ask pipe is then checked by its owner, SYSTEM, instead of refused.
//
// MUTATION: drop the ERROR_ACCESS_DENIED branch. RESULT: red.
func TestTheAskPipeIsCheckedByItsOwnerWhenTheServiceCannotBeQueried(t *testing.T) {
src, err := os.ReadFile("approvalpipe_windows.go")
if err != nil {
t.Fatal(err)
}
s := string(src)
i := strings.Index(s, "func pipeServerIsService(")
if i < 0 {
t.Fatal("no pipeServerIsService")
}
body := s[i : i+strings.Index(s[i:], "\n}\n")]
if !strings.Contains(body, "errors.Is(err, windows.ERROR_ACCESS_DENIED)") || !strings.Contains(body, "return pipeOwnedBySystem(h)") {
t.Error("a network logon is still refused")
}
if !strings.Contains(s, "!owner.IsWellKnown(windows.WinLocalSystemSid) && !owner.IsWellKnown(windows.WinBuiltinAdministratorsSid)") {
t.Error("the owner check does not require LocalSystem or the Administrators group")
}
}
// The last beat to a replaced anchor (lastBeat, from superviseOnce's
// goroutine) never runs beside the run's own beat or path failover: both write
// cfg.Endpoints and the WG device, and before lastBeat a run's beats all came
// from its one loop.
func TestTheLastBeatNeverRunsBesideTheRunsOwnBeat(t *testing.T) {
const host = "serial-beat.host.privatae.ai"
var mu sync.Mutex
in, most := 0, 0
entered := make(chan struct{}, 4)
release := make(chan struct{})
beat := func() {
mu.Lock()
in++
if in > most {
most = in
}
mu.Unlock()
entered <- struct{}{}
<-release
mu.Lock()
in--
mu.Unlock()
}
rb, unregister := registerRunBeat(host, beat)
defer unregister()
old := lastBeatWait
lastBeatWait = 200 * time.Millisecond
t.Cleanup(func() { lastBeatWait = old })
loopDone := make(chan struct{})
go func() { defer close(loopDone); rb.run() }() // the run's own beat, in flight
<-entered
failoverRan := make(chan struct{})
go func() { rb.do(func() {}); close(failoverRan) }() // the tick's failover waits too
if !lastBeat(host) {
t.Fatal("lastBeat found no beat for a running pairing")
}
select {
case <-entered:
t.Fatal("the last beat ran beside the run's own beat")
default:
}
select {
case <-failoverRan:
t.Fatal("path failover ran beside a beat")
default:
}
close(release)
<-loopDone
<-failoverRan
mu.Lock()
defer mu.Unlock()
if most != 1 {
t.Fatalf("%d beats ran at once, want 1", most)
}
}
+35
View File
@@ -0,0 +1,35 @@
//go:build darwin
package main
import (
"os"
"os/exec"
"strings"
"syscall"
)
// anchorPolicyPlist is the machine-wide managed preference domain a
// configuration profile (MDM) installs for the bastion. The folder is root's
// and the system writes it; a file there not owned by root is not a policy.
const anchorPolicyPlist = "/Library/Managed Preferences/ai.privatae.bastion.plist"
func readAnchorPolicyOS() *anchorPolicy {
fi, err := os.Lstat(anchorPolicyPlist)
if err != nil || !fi.Mode().IsRegular() || fi.Mode().Perm()&0o022 != 0 {
return nil
}
if st, ok := fi.Sys().(*syscall.Stat_t); !ok || st.Uid != 0 {
return nil
}
// defaults reads the binary and the XML form alike.
out, err := exec.Command("/usr/bin/defaults", "read", strings.TrimSuffix(anchorPolicyPlist, ".plist"), "AnchorHost").Output()
if err != nil {
return nil
}
v := strings.TrimSpace(string(out))
if v == "" {
return nil
}
return &anchorPolicy{Host: v, Source: "the managed preference ai.privatae.bastion AnchorHost"}
}
+37
View File
@@ -0,0 +1,37 @@
//go:build linux
package main
import (
"os"
"path/filepath"
"testing"
)
// The policy readers accept only what the machine's administrator or policy
// engine could have written: on Linux a file root owns that nobody else can
// write. Run as an ordinary user, the suite's own file is not root's, so it is
// no policy.
func TestALinuxPolicyFileMustBeRoots(t *testing.T) {
dir := t.TempDir()
prev := anchorPolicyFile
anchorPolicyFile = filepath.Join(dir, "policy.json")
t.Cleanup(func() { anchorPolicyFile = prev })
if err := os.WriteFile(anchorPolicyFile, []byte(`{"anchor_host":"`+"managed.host.privatae.ai"+`"}`), 0o644); err != nil {
t.Fatal(err)
}
p := readAnchorPolicyOS()
if os.Geteuid() == 0 {
if p == nil || p.Host != "managed.host.privatae.ai" {
t.Fatalf("root's own policy file was not read: %+v", p)
}
_ = os.Chmod(anchorPolicyFile, 0o666)
if readAnchorPolicyOS() != nil {
t.Error("a policy file anyone can write was read")
}
return
}
if p != nil {
t.Errorf("a policy file root does not own was read: %+v", p)
}
}
+37
View File
@@ -0,0 +1,37 @@
//go:build !windows && !darwin
package main
import (
"encoding/json"
"os"
"strings"
"syscall"
)
// anchorPolicyFile is the managed configuration an organisation deploys to a
// Linux machine (by its configuration management) for the bastion: root's,
// and writable by nobody else. A file anyone else could have written is not a
// policy.
var anchorPolicyFile = "/etc/privatae-bastion/policy.json"
func readAnchorPolicyOS() *anchorPolicy {
fi, err := os.Lstat(anchorPolicyFile)
if err != nil || !fi.Mode().IsRegular() || fi.Mode().Perm()&0o022 != 0 {
return nil
}
if st, ok := fi.Sys().(*syscall.Stat_t); !ok || st.Uid != 0 {
return nil
}
raw, err := os.ReadFile(anchorPolicyFile)
if err != nil {
return nil
}
var p struct {
AnchorHost string `json:"anchor_host"`
}
if json.Unmarshal(raw, &p) != nil || strings.TrimSpace(p.AnchorHost) == "" {
return nil
}
return &anchorPolicy{Host: p.AnchorHost, Source: anchorPolicyFile + " anchor_host"}
}
+27
View File
@@ -0,0 +1,27 @@
//go:build windows
package main
import (
"strings"
"golang.org/x/sys/windows/registry"
)
// anchorPolicyKey is where Group Policy and Intune deploy a machine policy for
// the bastion: under HKLM\SOFTWARE\Policies, which only an administrator or
// the policy engine writes. Domain membership is not read.
const anchorPolicyKey = `SOFTWARE\Policies\Privatae\Bastion`
func readAnchorPolicyOS() *anchorPolicy {
k, err := registry.OpenKey(registry.LOCAL_MACHINE, anchorPolicyKey, registry.QUERY_VALUE)
if err != nil {
return nil
}
defer k.Close()
v, _, err := k.GetStringValue("AnchorHost")
if err != nil || strings.TrimSpace(v) == "" {
return nil
}
return &anchorPolicy{Host: v, Source: `HKLM\` + anchorPolicyKey + ` AnchorHost`}
}
+29
View File
@@ -441,6 +441,18 @@ func pipeServerIsService(h windows.Handle) error {
return fmt.Errorf("cannot tell which process serves the pipe: %w", err)
}
svcPID, err := bastionServicePID()
if errors.Is(err, windows.ERROR_ACCESS_DENIED) {
// The service's default permissions let an INTERACTIVE logon query it,
// not a network one (ssh, a remote PowerShell): the owner's own
// `status` over ssh was refused as "Access is denied". The pipe's
// owner says what the process id would: servePipe sets no owner, so
// the pipe's is the creating token's default owner, which for
// LocalSystem is BUILTIN\Administrators (seen on Windows Server 2022)
// or SYSTEM, and no program of an ordinary account can make an object
// either owns. An administrator's own program could; an administrator
// already holds the service itself.
return pipeOwnedBySystem(h)
}
if err != nil {
return err
}
@@ -450,6 +462,23 @@ func pipeServerIsService(h windows.Handle) error {
return nil
}
// pipeOwnedBySystem refuses a pipe whose owner is not LocalSystem or the
// Administrators group, the two owners LocalSystem's objects get.
func pipeOwnedBySystem(h windows.Handle) error {
sd, err := windows.GetSecurityInfo(h, windows.SE_FILE_OBJECT, windows.OWNER_SECURITY_INFORMATION)
if err != nil {
return fmt.Errorf("cannot tell who made the pipe: %w", err)
}
owner, _, err := sd.Owner()
if err != nil || owner == nil {
return fmt.Errorf("cannot tell who made the pipe: %v", err)
}
if !owner.IsWellKnown(windows.WinLocalSystemSid) && !owner.IsWellKnown(windows.WinBuiltinAdministratorsSid) {
return fmt.Errorf("the pipe was made by %s, not by the %s service; nothing was sent", owner.String(), winServiceName)
}
return nil
}
// bastionServicePID is the service's process id from the Service Control
// Manager, asked with the query right every signed-in user holds. 0 when it is
// not running.
+6
View File
@@ -377,6 +377,12 @@ func TestEveryListenerThisPackageOpensIsInTheTable(t *testing.T) {
"ops_action_linux.go:opActionProbeUnit:net.Listen": {"tcp", 1, "the action-op probe, 127.0.0.1:0, open while one probe unit dials it"},
"coders_linux.go:coderListenUnix:net.Listen": {"unix", 1, "a coder run's egress.sock, a unix socket file in its run dir, not a network port"},
"coders_other.go:coderListenUnix:net.Listen": {"unix", 1, "the same, off Linux (coder mode refuses there)"},
// The one listener on every address, by design: the door a Host that
// this server can't reach connects to (hostdoor.go). Bound only while
// such a pairing is served or a pairing window is open; it keeps only
// connections whose certificate verifies under a served pairing's
// pinned CA, and forwards nothing (TestTheDoorRefusesAnythingButItsPinnedHost).
"hostdoor.go:listenDoor:lc.Listen": {"tcp", 1, "the door your Host connects to, every address, TCP 8812 (docs/host-connects.md)"},
// Beside a Windows system service: the tray's own approvals window,
// and the service's approvals pipe (approvalpipe.go), which takes no
// remote client (PIPE_REJECT_REMOTE_CLIENTS) and, by its ACL, only
+19 -1
View File
@@ -52,6 +52,11 @@ type Config struct {
// pairing's age and not the time its credentials were last rotated (the
// file's mtime, which is what pairings from before this field fall back to).
FirstSeen string `json:"first_seen,omitempty"`
// Origin: how the pairing came to this bastion, for the Host that admits
// it (pairingsReport). "" was paired here; "carried" was moved in from an
// account's own bastion when the machine moved to the system service
// (servicecarry.go).
Origin string `json:"origin,omitempty"`
// How this pairing reaches its host (lane_socket.go). "" is WireGuard
// through the relay to the host's tunnel address on :8443, which is every
@@ -66,6 +71,12 @@ type Config struct {
LaneSocket string `json:"lane_socket,omitempty"`
RunsDir string `json:"runs_dir,omitempty"`
LaneAddr string `json:"lane_addr,omitempty"`
// HostDials marks a "tcp" pairing whose Host opens the connections
// (hostdoor.go): a Host this server can't reach, behind a home router,
// connects to this server's door at DoorAddr, and the lanes run over the
// connections it opens. LaneAddr is empty: nothing here ever dials it.
HostDials bool `json:"host_dials,omitempty"`
DoorAddr string `json:"door_addr,omitempty"`
// retired is set, under saveMu, once the run holding this copy has been
// stopped or has ended (retireRun, run.go), and is never cleared; from then
@@ -101,9 +112,13 @@ const (
// unixTransport says whether this pairing's lanes go over a local socket.
func (c *Config) unixTransport() bool { return c.Transport == transportUnix }
// tcpTransport says whether this pairing's lanes go to a Host's lane port.
// tcpTransport says whether this pairing's lanes go to a Host's lane port,
// or come from a Host that connects to this server (hostDials).
func (c *Config) tcpTransport() bool { return c.Transport == transportTCP }
// hostDials: a tcp pairing whose Host connects to this server (hostdoor.go).
func (c *Config) hostDials() bool { return c.tcpTransport() && c.HostDials }
// localTransport: a Host with no platform, over either local transport. Its
// lanes run without a WireGuard device (run.go runUnixPairing).
func (c *Config) localTransport() bool { return c.unixTransport() || c.tcpTransport() }
@@ -419,6 +434,9 @@ func cmdUnpair(key string) error {
if err != nil {
return err
}
// With the daemon running, its door's close does this too; stopped, it
// is done here, so the rule never outlives the last pairing that used it.
doorFirewallReleaseUnused()
fmt.Printf("unpaired %s — its tunnel and lanes are down; the context store and DCN state stay on disk\n", host)
return nil
}
+20 -20
View File
@@ -35,7 +35,7 @@ func TestLegacyConfigMigration(t *testing.T) {
t.Fatal(err)
}
legacy := map[string]string{
"host_name": "tenant-3.host.privatae.ai",
"host_name": "tenant-9003.host.privatae.ai",
"bearer": "mcp_x",
"listen_addr": "127.0.0.1:7878",
}
@@ -48,7 +48,7 @@ func TestLegacyConfigMigration(t *testing.T) {
if err != nil {
t.Fatalf("loadPairings: %v", err)
}
if len(cfgs) != 1 || cfgs[0].HostName != "tenant-3.host.privatae.ai" {
if len(cfgs) != 1 || cfgs[0].HostName != "tenant-9003.host.privatae.ai" {
t.Fatalf("migrated pairings = %+v, want the one legacy host", cfgs)
}
if got, want := cfgs[0].ContextRoot, filepath.Join(cd, "context"); got != want {
@@ -57,7 +57,7 @@ func TestLegacyConfigMigration(t *testing.T) {
if got, want := cfgs[0].DCNDir, filepath.Join(cd, "dcn"); got != want {
t.Errorf("DCNDir = %q, want the explicit legacy default %q", got, want)
}
if _, err := os.Stat(filepath.Join(cd, "pairings", "tenant-3.host.privatae.ai.json")); err != nil {
if _, err := os.Stat(filepath.Join(cd, "pairings", "tenant-9003.host.privatae.ai.json")); err != nil {
t.Errorf("pairings/<host>.json missing after migration: %v", err)
}
if _, err := os.Stat(filepath.Join(cd, "config.json")); !os.IsNotExist(err) {
@@ -76,27 +76,27 @@ func TestLegacyConfigMigration(t *testing.T) {
// existing pairing — both checked pre-redeem so a refusal can't burn the token.
func TestCheckPairConflict(t *testing.T) {
cfgs := []*Config{
{HostName: "tenant-3.host.privatae.ai"},
{HostName: "tenant-122.host.privatae.ai"},
{HostName: "tenant-9003.host.privatae.ai"},
{HostName: "tenant-9122.host.privatae.ai"},
}
if _, err := checkPairConflict(cfgs, "tenant-122", false); err == nil {
if _, err := checkPairConflict(cfgs, "tenant-9122", false); err == nil {
t.Fatal("pairing an already-paired host must be refused without --replace")
}
got, err := checkPairConflict(cfgs, "tenant-122", true)
if err != nil || got == nil || got.HostName != "tenant-122.host.privatae.ai" {
got, err := checkPairConflict(cfgs, "tenant-9122", true)
if err != nil || got == nil || got.HostName != "tenant-9122.host.privatae.ai" {
t.Fatalf("--replace on a paired host = (%v, %v), want its existing pairing", got, err)
}
if got, err := checkPairConflict(cfgs, "tenant-9", false); err != nil || got != nil {
if got, err := checkPairConflict(cfgs, "tenant-9500", false); err != nil || got != nil {
t.Fatalf("a NEW host must pair cleanly, got (%v, %v)", got, err)
}
if _, err := checkPairConflict(cfgs, "tenant-9", true); err == nil {
if _, err := checkPairConflict(cfgs, "tenant-9500", true); err == nil {
t.Fatal("--replace on an unpaired host must be refused (nothing to replace)")
}
// "tenant-1" is a prefix of tenant-122's host name but a different handle —
// "tenant-91" is a prefix of tenant-9122's host name but a different handle —
// it must NOT collide.
if got, err := checkPairConflict(cfgs, "tenant-1", false); err != nil || got != nil {
t.Fatalf("handle 'tenant-1' must not match tenant-122, got (%v, %v)", got, err)
if got, err := checkPairConflict(cfgs, "tenant-91", false); err != nil || got != nil {
t.Fatalf("handle 'tenant-91' must not match tenant-9122, got (%v, %v)", got, err)
}
}
@@ -124,13 +124,13 @@ func TestNextListenAddr(t *testing.T) {
// prefix; ambiguity is an error, not a guess.
func TestFindPairingHostFlag(t *testing.T) {
cfgs := []*Config{
{HostName: "tenant-3.host.privatae.ai"},
{HostName: "tenant-122.host.privatae.ai"},
{HostName: "tenant-9003.host.privatae.ai"},
{HostName: "tenant-9122.host.privatae.ai"},
}
for _, key := range []string{"tenant-3.host.privatae.ai", "tenant-3", "tenant-3."} {
for _, key := range []string{"tenant-9003.host.privatae.ai", "tenant-9003", "tenant-9003."} {
c, err := findPairing(cfgs, key)
if err != nil || c == nil || c.HostName != "tenant-3.host.privatae.ai" {
t.Errorf("findPairing(%q) = (%v, %v), want tenant-3", key, c, err)
if err != nil || c == nil || c.HostName != "tenant-9003.host.privatae.ai" {
t.Errorf("findPairing(%q) = (%v, %v), want tenant-9003", key, c, err)
}
}
if c, err := findPairing(cfgs, "nope"); err != nil || c != nil {
@@ -145,8 +145,8 @@ func TestFindPairingHostFlag(t *testing.T) {
// a pin synced for one host may never verify (or even be visible to) another's ops.
func TestPerHostDcnDirs(t *testing.T) {
cd := isolateConfigDir(t)
a := &Config{HostName: "tenant-3.host.privatae.ai"}
b := &Config{HostName: "tenant-122.host.privatae.ai"}
a := &Config{HostName: "tenant-9003.host.privatae.ai"}
b := &Config{HostName: "tenant-9122.host.privatae.ai"}
dirA, dirB := a.dcnDir(), b.dcnDir()
if dirA == "" || dirB == "" || dirA == dirB {
t.Fatalf("per-host dcn dirs must differ: %q vs %q", dirA, dirB)
+9
View File
@@ -267,11 +267,20 @@ func readDoctor(readOpts) (doctorRead, error) {
Fix: asAdmin("quarantine clear --asker " + account)})
}
for _, c := range doorFirewallChecks(s.Firewall) {
add(c)
}
if s.LLM != nil {
add(doctorCheck{Name: "model server", OK: true, Detail: sentenceCase(s.LLM.words()) + "."})
}
for _, p := range s.Pairings {
switch {
case p.Problem != "":
add(doctorCheck{Name: "host " + p.Host, Problem: "The pairing with " + p.Host + " is not usable: " +
p.Problem + ".", Fix: "The bastion's log says more: " + logHint()})
case p.WaitingFor != "":
add(doctorCheck{Name: "host " + p.Host, OK: true, Detail: p.Host + " waits for " + p.WaitingFor +
" to admit it; " + p.WaitingFor + " asks you to approve it."})
case !p.Answering && s.Running != "no":
add(doctorCheck{Name: "host " + p.Host, Problem: "The bastion is not serving " + p.Host +
" (nothing answers on " + p.Local + ").", Fix: "The bastion's log says why: " + logHint()})
+1059
View File
File diff suppressed because it is too large Load Diff
+50
View File
@@ -0,0 +1,50 @@
package main
import "testing"
// --listapps as macOS prints it before 15 (the state on the entry's line, or
// on the next) and from 15 (names alone, "(state: 1)" allowed, 4 blocked),
// as shown in developer.apple.com/forums/thread/757909.
func TestTheMacListOfAppsIsReadEntryByEntry(t *testing.T) {
const before15 = "ALF: total number of apps = 4 \n\n" +
"1 : /Applications/Google Chrome.app ( Block incoming connections )\n" +
"2 : /usr/local/bin/privatae-bastion.old ( Allow incoming connections )\n" +
"3 : /usr/local/bin/privatae-bastion \n \t ( Allow incoming connections ) \n\n" +
"4 : /opt/example/privatae-bastion ( Block incoming connections )\n"
apps := parseMacListApps(before15)
if len(apps) != 4 {
t.Fatalf("entries: %+v", apps)
}
for _, c := range []struct {
prog string
listed, allowed bool
}{
{"/usr/local/bin/privatae-bastion", true, true},
{"/opt/example/privatae-bastion", true, false},
{"/usr/local/bin/privatae", false, false},
{"/Applications/Google Chrome.app", true, false},
{"/usr/local/bin/privatae-bastion.old", true, true},
} {
if l, a := macAppState(apps, c.prog); l != c.listed || a != c.allowed {
t.Fatalf("%s: listed %v allowed %v", c.prog, l, a)
}
}
const from15 = "2024-06-24 16:21:15.599 socketfilterfw[2988:52866] ApplicationFirewall::AFGetAllApplications() processing response dictionary\n\n" +
"Total number of apps = 3 \n\n" +
"privatae-bastion.old (state: 1)\n" +
"privatae-bastion (state: 4)\n" +
"cupsd (state: 1)\n"
apps = parseMacListApps(from15)
if len(apps) != 3 {
t.Fatalf("entries: %+v", apps)
}
if l, a := macAppState(apps, "/usr/local/bin/privatae-bastion"); !l || a {
t.Fatalf("blocked by name: listed %v allowed %v", l, a)
}
if l, a := macAppState(apps, "/usr/sbin/cupsd"); !l || !a {
t.Fatalf("allowed by name: listed %v allowed %v", l, a)
}
// MUTATION: the next line's state ignored. RESULT: red, /usr/local/bin/privatae-bastion not allowed.
// MUTATION: any state counts as allowed. RESULT: red, blocked read as allowed.
}
+7
View File
@@ -0,0 +1,7 @@
//go:build !windows
package main
// systemDir is System32 on Windows; elsewhere the Windows firewall is never
// used, and the tests read the path the rule commands are built with.
func systemDir() string { return `C:\Windows\System32` }
+13
View File
@@ -0,0 +1,13 @@
//go:build windows
package main
import "golang.org/x/sys/windows"
// systemDir is System32, from Windows itself rather than the environment.
func systemDir() string {
if d, err := windows.GetSystemDirectory(); err == nil && d != "" {
return d
}
return `C:\Windows\System32`
}
+811
View File
@@ -0,0 +1,811 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"path"
"path/filepath"
"sort"
"strings"
"sync"
"testing"
"time"
)
// fwStandin plays this machine's firewall commands: netsh, ufw, firewall-cmd,
// socketfilterfw, nft and iptables, each with the state it would keep. Every
// command run is recorded, so a test can say what was never run.
type fwStandin struct {
mu sync.Mutex
ran []string
fail map[string]bool // a verb that fails ("add")
// Windows: rule name -> its add arguments.
win map[string]string
// ufw: on or off, and its rules as `ufw status` lists them.
ufwOn bool
ufwRules []string
// firewalld: running, ports now and permanently.
fwdOn bool
fwdNow, fwdPerm map[string]bool
macOn, macAllIn bool
macApps map[string]bool
nftRuleset, ipts string
}
func newFWStandin() *fwStandin {
return &fwStandin{fail: map[string]bool{}, win: map[string]string{}, fwdNow: map[string]bool{},
fwdPerm: map[string]bool{}, macApps: map[string]bool{}}
}
var errExit1 = errors.New("exit status 1")
func (f *fwStandin) exec(name string, args ...string) (string, error) {
f.mu.Lock()
defer f.mu.Unlock()
line := path.Base(strings.ReplaceAll(name, `\`, "/")) + " " + strings.Join(args, " ")
f.ran = append(f.ran, line)
a := strings.Join(args, " ")
switch path.Base(strings.ReplaceAll(name, `\`, "/")) {
case "netsh.exe":
if !strings.HasPrefix(a, "advfirewall firewall ") || len(args) < 5 {
return "", fmt.Errorf("netsh: %s", a)
}
rule := strings.TrimPrefix(args[4], "name=")
switch args[2] {
case "show":
if _, ok := f.win[rule]; ok {
return "Rule Name: " + rule + "\n", nil
}
return "No rules match the specified criteria.\n", errExit1
case "add":
if f.fail["add"] {
return "The requested operation requires elevation (Run as administrator).", errExit1
}
f.win[rule] = strings.Join(args[5:], " ")
return "Ok.\n", nil
case "delete":
if _, ok := f.win[rule]; !ok {
return "No rules match the specified criteria.\n", errExit1
}
delete(f.win, rule)
return "\nDeleted 1 rule(s).\nOk.\n", nil
}
case "ufw":
switch {
case a == "status":
if !f.ufwOn {
return "Status: inactive\n", nil
}
return "Status: active\n\nTo Action From\n-- ------ ----\n" +
strings.Join(f.ufwRules, "\n") + "\n", nil
case strings.HasPrefix(a, "allow "):
port := args[1]
c := ""
if len(args) == 4 && args[2] == "comment" {
c = " # " + args[3]
}
f.ufwRules = append(f.ufwRules, port+" ALLOW Anywhere"+c,
port+" (v6) ALLOW Anywhere (v6)"+c)
return "Rule added\nRule added (v6)\n", nil
case strings.HasPrefix(a, "delete allow "):
port := args[2]
var keep []string
for _, r := range f.ufwRules {
if strings.Fields(r)[0] != port {
keep = append(keep, r)
}
}
if len(keep) == len(f.ufwRules) {
return "Could not delete non-existent rule\n", errExit1
}
f.ufwRules = keep
return "Rule deleted\nRule deleted (v6)\n", nil
}
case "firewall-cmd":
perm := len(args) > 0 && args[0] == "--permanent"
set := f.fwdNow
if perm {
set, args = f.fwdPerm, args[1:]
}
if len(args) == 0 {
break
}
switch {
case args[0] == "--state":
if f.fwdOn {
return "running\n", nil
}
return "not running\n", errors.New("exit status 252")
case strings.HasPrefix(args[0], "--query-port="):
if set[strings.TrimPrefix(args[0], "--query-port=")] {
return "yes\n", nil
}
return "no\n", errExit1
case strings.HasPrefix(args[0], "--add-port="):
if perm && f.fail["permanent"] {
return "Error: COMMAND_FAILED\n", errExit1
}
set[strings.TrimPrefix(args[0], "--add-port=")] = true
return "success\n", nil
case strings.HasPrefix(args[0], "--remove-port="):
p := strings.TrimPrefix(args[0], "--remove-port=")
if !set[p] {
return "Warning: NOT_ENABLED: " + p + "\nsuccess\n", nil
}
delete(set, p)
return "success\n", nil
}
case "socketfilterfw":
switch args[0] {
case "--getglobalstate":
if f.macOn {
return "Firewall is enabled. (State = 1)\n", nil
}
return "Firewall is disabled. (State = 0)\n", nil
case "--getblockall":
if f.macAllIn {
return "Firewall is set to block all non-essential incoming connections\nBlock all ENABLED!\n", nil
}
return "Block all DISABLED!\n", nil
case "--listapps":
// As macOS before 15 prints it (developer.apple.com/forums/thread/757909).
var b strings.Builder
fmt.Fprintf(&b, "ALF: total number of apps = %d \n\n", len(f.macApps))
var ps []string
for p := range f.macApps {
ps = append(ps, p)
}
sort.Strings(ps)
for i, p := range ps {
st := "Block"
if f.macApps[p] {
st = "Allow"
}
fmt.Fprintf(&b, "%d : %s ( %s incoming connections )\n", i+1, p, st)
}
return b.String(), nil
case "--add":
f.macApps[args[1]] = false
return "The file " + args[1] + " added to firewall\n", nil
case "--unblockapp":
if f.fail["unblock"] {
return "Error\n", errExit1
}
f.macApps[args[1]] = true
return "Incoming connection to the application is permitted\n", nil
case "--remove":
delete(f.macApps, args[1])
return "The file " + args[1] + " removed from firewall\n", nil
}
case "nft":
return f.nftRuleset, nil
case "iptables":
return f.ipts, nil
}
return "", fmt.Errorf("the stand-in does not know %s", line)
}
func (f *fwStandin) commands() []string {
f.mu.Lock()
defer f.mu.Unlock()
return append([]string(nil), f.ran...)
}
// fwWorld stands the firewall in for goos, with the door wanted while *want.
func fwWorld(t *testing.T, goos string, root bool) (*fwStandin, *bool) {
t.Helper()
uiTestConfigDir(t)
f := newFWStandin()
want := true
prevExec, prevFor, prevBy, prevHave, prevRoot, prevProg, prevWant, prevRead :=
doorFWExec, doorFirewallFor, doorFirewallByID, doorFWHave, doorFWRoot, doorFWProgram, doorPortWanted, doorFWReadFile
t.Cleanup(func() {
doorFWExec, doorFirewallFor, doorFirewallByID, doorFWHave, doorFWRoot, doorFWProgram, doorPortWanted, doorFWReadFile =
prevExec, prevFor, prevBy, prevHave, prevRoot, prevProg, prevWant, prevRead
})
doorFWExec = f.exec
doorFirewallFor = func(port string) (doorFirewall, string) { return detectDoorFirewall(goos, port) }
doorFirewallByID = func(id string) doorFirewall { return doorFirewallOf(goos, id) }
doorFWHave = func(paths ...string) string { return paths[0] }
doorFWRoot = func() bool { return root }
doorFWProgram = func() (string, error) { return `C:\Program Files\Privatae\privatae-bastion.exe`, nil }
doorPortWanted = func(string) bool { return want }
doorFWReadFile = func(string) ([]byte, error) { return nil, errors.New("no such file") }
return f, &want
}
func fwAudit(t *testing.T) string {
t.Helper()
d, _ := configDir()
b, _ := os.ReadFile(filepath.Join(d, "pairing_audit.jsonl"))
return string(b)
}
func countOp(t *testing.T, op string) int {
return strings.Count(fwAudit(t), `"op":"`+op+`"`)
}
// Windows: the rule is the bastion's own by name, inbound, TCP on the door's
// port only, every profile, for this program only, made by netsh from
// System32. It is made once, kept while the door is in use, and removed when
// the last pairing and window are gone, each step in the audit and in status.
func TestTheWindowsRuleIsTheBastionsOwnAndGoesWithTheDoor(t *testing.T) {
f, want := fwWorld(t, "windows", false)
said := doorFirewallEnsure(":8812")
rule := doorFWRuleName("8812")
got, ok := f.win[rule]
if !ok {
t.Fatalf("no rule %q: %v", rule, f.commands())
}
for _, part := range []string{"dir=in", "action=allow", "protocol=TCP", "localport=8812",
`program=C:\Program Files\Privatae\privatae-bastion.exe`, "profile=any", "enable=yes"} {
if !strings.Contains(got, part) {
t.Fatalf("the rule lacks %s: %s", part, got)
}
}
for _, c := range f.commands() {
if !strings.HasPrefix(c, "netsh.exe advfirewall firewall ") {
t.Fatalf("ran %q", c)
}
}
if !strings.HasPrefix(netshPath(), `C:\Windows\System32`) {
t.Fatalf("netsh is not run by its absolute path: %s", netshPath())
}
if !strings.Contains(said, "added its own rule in Windows Firewall") {
t.Fatalf("said %q", said)
}
// Again (the daemon's door after the window's): kept, not doubled.
doorFirewallEnsure(":8812")
if n := strings.Count(strings.Join(f.commands(), "\n"), " add rule "); n != 1 {
t.Fatalf("%d adds", n)
}
s := readDoorFirewallStatus()
if s == nil || len(s.Rules) != 1 || !strings.Contains(s.Rules[0], "Windows Firewall allows TCP 8812") {
t.Fatalf("status: %+v", s)
}
// Still in use: the rule stays.
doorFirewallRelease(":8812")
if _, ok := f.win[rule]; !ok {
t.Fatal("removed while the door is still in use")
}
// The last pairing and window gone: removed.
*want = false
doorFirewallRelease(":8812")
if _, ok := f.win[rule]; ok {
t.Fatal("the rule outlived the door")
}
if s := readDoorFirewallStatus(); s != nil {
t.Fatalf("status after: %+v", s)
}
if countOp(t, "door_firewall_opened") != 1 || countOp(t, "door_firewall_closed") != 1 {
t.Fatalf("the audit:\n%s", fwAudit(t))
}
// MUTATION: release removes nothing. RESULT: red, "the rule outlived the door".
// MUTATION: the add drops program=. RESULT: red, "the rule lacks program=".
// MUTATION: ensure adds without asking has(). RESULT: red, "2 adds".
}
// Windows: the bastion moved (another path): its rule is replaced by one for
// the new path; and an add refused (no administrator) is said once, with the
// command that does it by hand, in status and doctor.
func TestTheWindowsRuleFollowsTheProgramAndAFailureIsSaidOnce(t *testing.T) {
f, _ := fwWorld(t, "windows", false)
doorFirewallEnsure(":8812")
doorFWProgram = func() (string, error) { return `D:\Privatae\privatae-bastion.exe`, nil }
doorFirewallEnsure(":8812")
if got := f.win[doorFWRuleName("8812")]; !strings.Contains(got, `program=D:\Privatae\privatae-bastion.exe`) {
t.Fatalf("the rule names %s", got)
}
if st := loadDoorFW(); len(st.Rules) != 1 || st.Rules[0].Program != `D:\Privatae\privatae-bastion.exe` {
t.Fatalf("the record: %+v", st.Rules)
}
g, _ := fwWorld(t, "windows", false)
g.fail["add"] = true
doorFirewallEnsure(":8812")
doorFirewallEnsure(":8812")
if n := countOp(t, "door_firewall_failed"); n != 1 {
t.Fatalf("the failure audited %d times", n)
}
s := readDoorFirewallStatus()
if s == nil || len(s.Needs) != 1 || !strings.Contains(s.Needs[0], "New-NetFirewallRule") {
t.Fatalf("status: %+v", s)
}
cs := doorFirewallChecks(s)
if len(cs) != 1 || cs[0].OK || !strings.Contains(cs[0].Problem, "TCP 8812") {
t.Fatalf("doctor: %+v", cs)
}
// MUTATION: the program is not compared. RESULT: red, the rule names the old path.
// MUTATION: the note is audited on every ensure. RESULT: red, audited 2 times.
}
// Linux ufw: a rule only while ufw is on, with the bastion's comment; ufw is
// never turned on, off or reset; a rule of the machine's own for the port is
// neither doubled nor removed.
func TestTheUfwRuleIsAddedOnlyWhenUfwIsOnAndNeverTheMachinesOwn(t *testing.T) {
f, want := fwWorld(t, "linux", true)
// Off: nothing at all.
if said := doorFirewallEnsure(":8812"); said != "" || len(loadDoorFW().Rules) != 0 {
t.Fatalf("ufw off: %q %v", said, f.commands())
}
f.ufwOn = true
doorFirewallEnsure(":8812")
st := strings.Join(f.ufwRules, "\n")
if !strings.Contains(st, "8812/tcp") || !strings.Contains(st, "# "+doorFWComment) {
t.Fatalf("ufw rules: %s", st)
}
if s := readDoorFirewallStatus(); s == nil || len(s.Rules) != 1 || !strings.Contains(s.Rules[0], "ufw allows TCP 8812") {
t.Fatalf("status: %+v", s)
}
*want = false
doorFirewallRelease(":8812")
if len(f.ufwRules) != 0 {
t.Fatalf("left: %v", f.ufwRules)
}
for _, c := range f.commands() {
for _, never := range []string{"ufw enable", "ufw disable", "ufw reset", "ufw default", "ufw reload"} {
if strings.HasPrefix(c, never) {
t.Fatalf("ran %q", c)
}
}
}
// The machine's own rule for the port.
g, want2 := fwWorld(t, "linux", true)
g.ufwOn = true
g.ufwRules = []string{"8812/tcp ALLOW Anywhere", "8812/tcp (v6) ALLOW Anywhere (v6)"}
said := doorFirewallEnsure(":8812")
if !strings.Contains(said, "already allowed") || len(g.ufwRules) != 2 {
t.Fatalf("said %q, rules %v", said, g.ufwRules)
}
*want2 = false
doorFirewallRelease(":8812")
if len(g.ufwRules) != 2 {
t.Fatalf("the machine's own rule was removed: %v", g.commands())
}
// MUTATION: ufwRuleFor counts any ALLOW as ours. RESULT: red, the machine's own rule was removed.
// MUTATION: detection ignores "Status: inactive". RESULT: red, ufw off.
}
// Linux firewalld: the port added now and permanently, and removed the same
// way; a port already open in it is the machine's own.
func TestTheFirewalldPortIsAddedAndRemovedNowAndPermanently(t *testing.T) {
f, want := fwWorld(t, "linux", true)
f.fwdOn = true
doorFirewallEnsure(":8812")
if !f.fwdNow["8812/tcp"] || !f.fwdPerm["8812/tcp"] {
t.Fatalf("firewalld: %v %v", f.fwdNow, f.fwdPerm)
}
doorFirewallEnsure(":8812")
if n := strings.Count(strings.Join(f.commands(), "\n"), "--add-port"); n != 2 {
t.Fatalf("%d adds: %v", n, f.commands())
}
*want = false
doorFirewallRelease(":8812")
if f.fwdNow["8812/tcp"] || f.fwdPerm["8812/tcp"] {
t.Fatalf("left: %v %v", f.fwdNow, f.fwdPerm)
}
g, want2 := fwWorld(t, "linux", true)
g.fwdOn = true
g.fwdNow["8812/tcp"], g.fwdPerm["8812/tcp"] = true, true
doorFirewallEnsure(":8812")
*want2 = false
doorFirewallRelease(":8812")
if !g.fwdNow["8812/tcp"] || !g.fwdPerm["8812/tcp"] {
t.Fatal("the machine's own open port was closed")
}
// MUTATION: remove only the runtime port. RESULT: red, left permanently.
// MUTATION: has() reports a port it did not add as ours. RESULT: red, the machine's own port closed.
}
// Linux: a firewall the bastion does not manage (nftables, iptables), or ufw
// it may not change, is said once, with what to allow, in status and doctor,
// and nothing in it is changed.
func TestAFirewallTheBastionDoesNotManageIsSaidOnceAndLeftAlone(t *testing.T) {
f, want := fwWorld(t, "linux", true)
f.nftRuleset = "table inet filter {\n\tchain input {\n\t\ttype filter hook input priority filter; policy drop;\n\t}\n}\n"
said := doorFirewallEnsure(":8812")
doorFirewallEnsure(":8812")
if !strings.Contains(said, "nftables") || !strings.Contains(said, "tcp dport 8812 accept") {
t.Fatalf("said %q", said)
}
if n := countOp(t, "door_firewall_unmanaged"); n != 1 {
t.Fatalf("said %d times", n)
}
for _, c := range f.commands() {
if strings.HasPrefix(c, "nft ") && c != "nft list ruleset" {
t.Fatalf("ran %q", c)
}
}
s := readDoorFirewallStatus()
if s == nil || len(s.Needs) != 1 || !strings.Contains(s.Needs[0], "nftables") {
t.Fatalf("status: %+v", s)
}
if cs := doorFirewallChecks(s); len(cs) != 1 || cs[0].OK {
t.Fatalf("doctor: %+v", cs)
}
*want = false
doorFirewallRelease(":8812")
if s := readDoorFirewallStatus(); s != nil {
t.Fatalf("still said with no door: %+v", s)
}
// iptables, and an account that may not change ufw.
g, _ := fwWorld(t, "linux", true)
g.ipts = "-P INPUT DROP\n"
if said := doorFirewallEnsure(":8812"); !strings.Contains(said, "iptables -I INPUT -p tcp --dport 8812 -j ACCEPT") {
t.Fatalf("iptables: %q", said)
}
h, _ := fwWorld(t, "linux", false)
doorFWReadFile = func(string) ([]byte, error) { return []byte("ENABLED=yes\n"), nil }
if said := doorFirewallEnsure(":8812"); !strings.Contains(said, "sudo ufw allow 8812/tcp") {
t.Fatalf("not root: %q", said)
}
for _, c := range h.commands() {
if strings.HasPrefix(c, "ufw ") {
t.Fatalf("an account that may not change ufw ran %q", c)
}
}
// MUTATION: nftFiltersInput always false. RESULT: red, said "".
// MUTATION: the note is not cleared at release. RESULT: red, still said with no door.
}
// macOS: with the application firewall on, this program is allowed; with it
// off nothing is done; "block all incoming" is only said.
func TestTheMacFirewallAllowsTheProgramOnlyWhenItIsOn(t *testing.T) {
f, want := fwWorld(t, "darwin", true)
doorFirewallEnsure(":8812")
if len(f.macApps) != 0 {
t.Fatal("added with the firewall off")
}
f.macOn = true
doorFirewallEnsure(":8812")
prog, _ := doorFWProgram()
if allowed, ok := f.macApps[prog]; !ok || !allowed {
t.Fatalf("apps: %v", f.macApps)
}
*want = false
doorFirewallRelease(":8812")
if len(f.macApps) != 0 {
t.Fatalf("left: %v", f.macApps)
}
g, _ := fwWorld(t, "darwin", true)
g.macOn, g.macAllIn = true, true
if said := doorFirewallEnsure(":8812"); !strings.Contains(said, "blocks all incoming") || len(g.macApps) != 0 {
t.Fatalf("block all: %q %v", said, g.macApps)
}
// MUTATION: --unblockapp not run. RESULT: red, apps not allowed.
}
// uninstall removes every rule the bastion added, whatever uses the door.
func TestUninstallRemovesTheDoorsOwnRules(t *testing.T) {
f, _ := fwWorld(t, "windows", false)
doorFirewallEnsure(":8812")
doorFirewallEnsure(":9443")
said, err := doorFirewallRemoveAll()
if err != nil || len(said) != 2 || len(f.win) != 0 {
t.Fatalf("uninstall: %v %v, left %v", said, err, f.win)
}
if len(loadDoorFW().Rules) != 0 {
t.Fatal("the record outlived the rules")
}
// MUTATION: RemoveAll skips the removal. RESULT: red, rules left.
}
// A rule left behind (a `pair --open` killed before its release, a pairing
// that went while nothing ran) is removed by the daemon's sweep once nothing
// uses its port, and kept while something does.
func TestTheDaemonsSweepRemovesARuleNothingUses(t *testing.T) {
f, want := fwWorld(t, "windows", false)
doorFirewallEnsure(":8812")
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { doorFirewallSweep(ctx); close(done) }()
cancel()
<-done
if _, ok := f.win[doorFWRuleName("8812")]; !ok {
t.Fatal("the sweep removed a rule in use")
}
*want = false
doorFirewallReleaseUnused()
if _, ok := f.win[doorFWRuleName("8812")]; ok {
t.Fatal("the sweep left a rule nothing uses")
}
// MUTATION: doorFirewallReleaseUnused skips recorded rules. RESULT: red, the sweep left a rule.
}
// ufw deletes by rule, whatever its comment: a rule the bastion recorded and
// the person later replaced with their own, or a record that names a port the
// bastion never opened, is the machine's, and is never removed.
func TestUfwNeverRemovesARuleThatIsNotItsOwnWhateverTheRecordSays(t *testing.T) {
f, want := fwWorld(t, "linux", true)
f.ufwOn = true
doorFirewallEnsure(":8812")
// The person deletes the bastion's rule and puts their own in its place.
f.ufwRules = []string{"8812/tcp ALLOW Anywhere", "8812/tcp (v6) ALLOW Anywhere (v6)"}
doorFirewallEnsure(":8812")
*want = false
doorFirewallRelease(":8812")
if len(f.ufwRules) != 2 {
t.Fatalf("the person's own rule was removed: %v", f.commands())
}
// A record naming a port the bastion never opened (written by hand, or
// left from another time): the sweep leaves the machine's rule there.
g, want2 := fwWorld(t, "linux", true)
g.ufwOn = true
g.ufwRules = []string{"22/tcp ALLOW Anywhere"}
if err := saveDoorFW(doorFWState{Rules: []doorFWRule{{Firewall: "ufw", Port: "22", Added: "2026-01-01T00:00:00Z"}}}); err != nil {
t.Fatal(err)
}
*want2 = false
doorFirewallReleaseUnused()
for _, c := range g.commands() {
if strings.HasPrefix(c, "ufw delete") {
t.Fatalf("ran %q on a rule that is not the bastion's", c)
}
}
if len(g.ufwRules) != 1 || len(loadDoorFW().Rules) != 0 {
t.Fatalf("rules %v, record %+v", g.ufwRules, loadDoorFW().Rules)
}
// MUTATION: ufw remove deletes without reading the comment. RESULT: red, the person's own rule was removed.
}
// Only a rule that lets a Host in from anywhere counts as the machine's own
// allow: an outbound rule, or one for one source or interface, does not, so
// the bastion adds its own rather than saying the port is already allowed.
func TestUfwAnOutboundOrOneSourceRuleIsNotTheWayIn(t *testing.T) {
for _, r := range []string{
"8812/tcp ALLOW OUT Anywhere",
"8812/tcp ALLOW 10.0.0.0/8",
"8812/tcp on eth1 ALLOW Anywhere",
"8812/tcp DENY Anywhere",
} {
f, _ := fwWorld(t, "linux", true)
f.ufwOn = true
f.ufwRules = []string{r}
said := doorFirewallEnsure(":8812")
if !strings.Contains(said, "added its own rule in ufw") {
t.Fatalf("with %q: said %q", r, said)
}
}
for _, r := range []string{
"8812/tcp ALLOW Anywhere",
"8812/tcp ALLOW IN Anywhere",
"8812 (v6) ALLOW Anywhere (v6)",
} {
if _, other := ufwRuleFor(r, "8812"); !other {
t.Fatalf("%q lets a Host in from anywhere", r)
}
}
// MUTATION: any line with ALLOW counts. RESULT: red, said "already allowed".
}
// firewalld: a permanent add that fails takes the runtime port back out, so
// no open port is left that no record says is the bastion's.
func TestFirewalldAFailedPermanentAddLeavesNoPortOpen(t *testing.T) {
f, _ := fwWorld(t, "linux", true)
f.fwdOn = true
f.fail["permanent"] = true
said := doorFirewallEnsure(":8812")
if !strings.Contains(said, "could not add") {
t.Fatalf("said %q", said)
}
if f.fwdNow["8812/tcp"] {
t.Fatalf("the runtime port was left open with no record: %v", f.commands())
}
// Next time it is tried again, not taken for the machine's own.
f.fail["permanent"] = false
if said := doorFirewallEnsure(":8812"); !strings.Contains(said, "added its own rule") {
t.Fatalf("said %q", said)
}
// MUTATION: no undo of the runtime add. RESULT: red, the runtime port was left open.
}
// macOS: an --unblockapp that fails takes the --add back out, so no entry is
// left that would read as the machine's own and never go.
func TestTheMacAFailedUnblockLeavesNoEntry(t *testing.T) {
f, _ := fwWorld(t, "darwin", true)
f.macOn = true
f.fail["unblock"] = true
if said := doorFirewallEnsure(":8812"); !strings.Contains(said, "could not add") {
t.Fatalf("said %q", said)
}
if len(f.macApps) != 0 {
t.Fatalf("left an entry with no record: %v", f.macApps)
}
// MUTATION: no --remove after a failed --unblockapp. RESULT: red, left an entry.
}
// realDoorPortWanted is the door-in-use check fwWorld stands in.
var realDoorPortWanted = doorPortWanted
// sweepOnce runs the daemon's sweep's first pass.
func sweepOnce() {
ctx, cancel := context.WithCancel(context.Background())
cancel()
doorFirewallSweep(ctx)
}
// A door claims the window (its file is moved aside) before the pairing is
// saved: between the two, neither is there to read. The window is still in
// use then, so a sweep in that gap keeps the rule, and puts it back if it had
// gone; once the window has its result and no pairing uses the port, the
// rule goes.
func TestAWindowClaimedForAPairingStillBeingSavedKeepsTheRule(t *testing.T) {
f, _ := fwWorld(t, "windows", false)
doorPortWanted = realDoorPortWanted
code, err := newWindowCode()
if err != nil {
t.Fatal(err)
}
now := time.Now().UTC()
w := &pairWindow{ID: "w-claimed", Code: code, Door: ":8812", Opened: now.Format(time.RFC3339),
Expires: now.Add(windowLifetime).Format(time.RFC3339)}
if err := saveWindowFile(w); err != nil {
t.Fatal(err)
}
doorFirewallEnsure(":8812")
rule := doorFWRuleName("8812")
if _, ok := f.win[rule]; !ok {
t.Fatalf("no rule for the open window: %v", f.commands())
}
if !claimWindow(w.ID) {
t.Fatal("the window could not be claimed")
}
sweepOnce()
if _, ok := f.win[rule]; !ok {
t.Fatal("the sweep removed the rule while a pairing on the window was being saved")
}
// The rule gone in that gap (by hand, or a release that ran then): back.
delete(f.win, rule)
sweepOnce()
if _, ok := f.win[rule]; !ok {
t.Fatal("the sweep did not put back the rule of a window still in use")
}
endWindow(w.ID, windowResult{Error: "the Host refused"})
sweepOnce()
if _, ok := f.win[rule]; ok {
t.Fatal("the rule outlived the window")
}
// MUTATION: doorPortsInUse ignores a claimed window. RESULT: red, the sweep removed the rule while a pairing ... was being saved.
// MUTATION: claimedWindowUnderWay counts a window with its result. RESULT: red, the rule outlived the window.
}
// A rule that goes while its door is in use (removed by hand, the firewall
// reset) is put back by the daemon's sweep; a port a rule of the machine's
// own allows is still never given a second one; nothing in use, nothing added.
func TestTheSweepPutsBackARuleThatWentWhileTheDoorIsInUse(t *testing.T) {
f, want := fwWorld(t, "windows", false)
doorFirewallEnsure(":8812")
rule := doorFWRuleName("8812")
delete(f.win, rule) // removed by hand
sweepOnce()
if got, ok := f.win[rule]; !ok || !strings.Contains(got, "localport=8812") {
t.Fatalf("not put back: %v", f.commands())
}
if n := countOp(t, "door_firewall_opened"); n != 2 {
t.Fatalf("opened %d times:\n%s", n, fwAudit(t))
}
*want = false
sweepOnce()
if _, ok := f.win[rule]; ok {
t.Fatal("a rule nothing uses was kept")
}
sweepOnce()
if _, ok := f.win[rule]; ok {
t.Fatal("the sweep added a rule nothing uses")
}
// ufw: the bastion's rule deleted while the door is in use, back with its comment.
g, _ := fwWorld(t, "linux", true)
g.ufwOn = true
doorFirewallEnsure(":8812")
g.ufwRules = nil
sweepOnce()
if st := strings.Join(g.ufwRules, "\n"); !strings.Contains(st, "8812/tcp") || !strings.Contains(st, "# "+doorFWComment) {
t.Fatalf("ufw not put back: %v", g.commands())
}
// ufw: the machine's own rule allows the port: never a second one.
h, _ := fwWorld(t, "linux", true)
h.ufwOn = true
h.ufwRules = []string{"8812/tcp ALLOW Anywhere"}
doorFirewallEnsure(":8812")
sweepOnce()
sweepOnce()
for _, c := range h.commands() {
if strings.HasPrefix(c, "ufw allow") {
t.Fatalf("ran %q with the machine's own rule there", c)
}
}
// MUTATION: the sweep only releases. RESULT: red, not put back (and in
// TestAWindowClaimedForAPairingStillBeingSavedKeepsTheRule).
}
// macOS: an entry is for one program path; the bastion that moved removes the
// entry by the path it recorded, both when its door is released and when it
// makes the entry for its new path.
func TestTheMacEntryIsRemovedByThePathItWasAddedFor(t *testing.T) {
const a, b = "/usr/local/bin/privatae-bastion", "/opt/example/privatae-bastion"
f, want := fwWorld(t, "darwin", true)
f.macOn = true
doorFWProgram = func() (string, error) { return a, nil }
doorFirewallEnsure(":8812")
if !f.macApps[a] {
t.Fatalf("apps: %v", f.macApps)
}
doorFWProgram = func() (string, error) { return b, nil }
*want = false
doorFirewallRelease(":8812")
if len(f.macApps) != 0 {
t.Fatalf("the moved bastion left its old path allowed: %v", f.macApps)
}
// Moved while the door is in use: the old path's entry goes, the new one's is made.
g, _ := fwWorld(t, "darwin", true)
g.macOn = true
doorFWProgram = func() (string, error) { return a, nil }
doorFirewallEnsure(":8812")
doorFWProgram = func() (string, error) { return b, nil }
doorFirewallEnsure(":8812")
if _, ok := g.macApps[a]; ok || !g.macApps[b] {
t.Fatalf("apps: %v", g.macApps)
}
if st := loadDoorFW(); len(st.Rules) != 1 || st.Rules[0].Program != b {
t.Fatalf("the record: %+v", st.Rules)
}
// MUTATION: remove uses this program's path, not the record's. RESULT: red, the moved bastion left its old path allowed.
// MUTATION: the moved case removes this program's path. RESULT: red, apps: both paths.
}
// macOS: --listapps is read entry by entry, by the exact path, and only an
// "allow incoming" entry counts: a blocked entry is the person's setting and
// is said, not changed; a longer path (…/privatae-bastion.old) is not this
// program's.
func TestTheMacReadsEachEntryAndCountsOnlyAnAllow(t *testing.T) {
const prog = "/usr/local/bin/privatae-bastion"
f, _ := fwWorld(t, "darwin", true)
f.macOn = true
doorFWProgram = func() (string, error) { return prog, nil }
f.macApps[prog] = false // the person blocked it
said := doorFirewallEnsure(":8812")
if !strings.Contains(said, "block incoming connections for this program") {
t.Fatalf("said %q", said)
}
for _, c := range f.commands() {
if strings.Contains(c, "--unblockapp") || strings.Contains(c, "--add") || strings.Contains(c, "--remove") {
t.Fatalf("changed the person's setting: %q", c)
}
}
if s := readDoorFirewallStatus(); s == nil || len(s.Needs) != 1 {
t.Fatalf("status: %+v", s)
}
g, _ := fwWorld(t, "darwin", true)
g.macOn = true
doorFWProgram = func() (string, error) { return prog, nil }
g.macApps[prog+".old"] = true
if said := doorFirewallEnsure(":8812"); !strings.Contains(said, "added its own rule") || !g.macApps[prog] {
t.Fatalf("a longer path counted as this program: said %q, apps %v", said, g.macApps)
}
// Its own entry, later set to block by the person: left as it is at release.
h, want := fwWorld(t, "darwin", true)
h.macOn = true
doorFWProgram = func() (string, error) { return prog, nil }
doorFirewallEnsure(":8812")
h.macApps[prog] = false
*want = false
doorFirewallRelease(":8812")
if allowed, ok := h.macApps[prog]; !ok || allowed {
t.Fatalf("the person's block was removed: %v", h.macApps)
}
// MUTATION: has() counts a listed program whatever its state. RESULT: red, said "already allowed".
// MUTATION: the path is matched as a prefix. RESULT: red, a longer path counted as this program.
// MUTATION: remove runs --remove on a blocked entry. RESULT: red, the person's block was removed.
}
+206
View File
@@ -0,0 +1,206 @@
//go:build linux
package main
import (
"crypto/tls"
"errors"
"net"
"strings"
"sync"
"testing"
"time"
)
// fwFake is a firewall the bastion manages, kept in memory.
type fwFake struct {
mu sync.Mutex
rules map[string]bool
adds int
}
func (f *fwFake) id() string { return "ufw" }
func (f *fwFake) name() string { return "ufw" }
func (f *fwFake) has(port string) (bool, bool, error) {
f.mu.Lock()
defer f.mu.Unlock()
return f.rules[port], false, nil
}
func (f *fwFake) add(port string) error {
f.mu.Lock()
defer f.mu.Unlock()
f.rules[port] = true
f.adds++
return nil
}
func (f *fwFake) remove(port, _ string) error {
f.mu.Lock()
defer f.mu.Unlock()
if !f.rules[port] {
return errors.New("no such rule")
}
delete(f.rules, port)
return nil
}
func (f *fwFake) manual(port string) string { return "allow " + port }
func (f *fwFake) open(port string) bool { ok, _, _ := f.has(port); return ok }
func fwFakeWorld(t *testing.T) *fwFake {
t.Helper()
f := &fwFake{rules: map[string]bool{}}
prevFor, prevBy := doorFirewallFor, doorFirewallByID
t.Cleanup(func() { doorFirewallFor, doorFirewallByID = prevFor, prevBy })
doorFirewallFor = func(string) (doorFirewall, string) { return f, "" }
doorFirewallByID = func(string) doorFirewall { return f }
return f
}
// The window's own door opens the way in while the window is open, and a
// window that ends with nothing paired takes it away again.
func TestAPairingWindowOpensTheFirewallAndClosingItRemovesTheRule(t *testing.T) {
pairingBox(t)
f := fwFakeWorld(t)
addr := freeDoorAddr(t)
port := doorPort(addr)
_, out, res := openWindowFor(t, addr)
if !f.open(port) || !strings.Contains(out.String(), "added its own rule in ufw") {
t.Fatalf("the window is open and the firewall is not: %v\n%s", f.rules, out.String())
}
w := loadOpenWindow()
removeWindow(w.ID) // closed at the terminal, as Ctrl-C does
select {
case <-res:
case <-time.After(5 * time.Second):
t.Fatal("pair --open did not end")
}
if f.open(port) {
t.Fatal("the rule outlived a window that paired nothing")
}
a := pairingAuditText(t)
if !strings.Contains(a, `"door_firewall_opened"`) || !strings.Contains(a, `"door_firewall_closed"`) {
t.Fatalf("the audit:\n%s", a)
}
// MUTATION: pair --open does not release. RESULT: red, the rule outlived the window.
}
// A pairing made through the window keeps the rule; the daemon's door keeps
// it while it serves the pairing and when the bastion stops (the pairing has
// not gone); unpairing removes it.
func TestThePairingsDoorKeepsTheRuleUntilThePairingGoes(t *testing.T) {
pairingBox(t)
laneTestProcess(t)
// Opened without machine authority, so the Host is not put in charge
// (anchorAfterWindow) and `unpair` below may remove it: which Host is in
// charge is TestAServerPairedThroughTheWindowHasThatHostInCharge's.
prevAuth := machineAuthorityHeld
machineAuthorityHeld = func() bool { return false }
t.Cleanup(func() { machineAuthorityHeld = prevAuth })
f := fwFakeWorld(t)
addr := freeDoorAddr(t)
port := doorPort(addr)
code, _, res := openWindowFor(t, addr)
ca := newTestCA(t, "home host CA")
h := newStandinHost(t, ca, "home", "home.cseries.localhost", addr)
if done, err := h.pair(formatWindowCode(code)); err != nil || done.State != pairingActive {
t.Fatalf("pair: %+v %v", done, err)
}
if err := <-res; err != nil {
t.Fatal(err)
}
if !f.open(port) {
t.Fatal("the window's end removed the rule its active pairing needs")
}
cfgs, _ := loadPairings()
cfg := pairingFor(cfgs, "home")
// Its local control on a free port: the machine running the suite may
// have its own bastion on the default one.
cfg.ListenAddr = freeDoorAddr(t)
stop := runPairingBG(t, cfg)
doorWaitFor(t, 5*time.Second, "the door to bind", func() bool {
c, err := net.DialTimeout("tcp", addr, time.Second)
if err == nil {
_ = c.Close()
}
return err == nil
})
if err := stop(); err != nil && !errors.Is(err, errPairingNotActive) {
t.Fatal(err)
}
if !f.open(port) {
t.Fatal("the bastion stopping removed the rule of a pairing that is still there")
}
if f.adds != 1 {
t.Fatalf("%d adds", f.adds)
}
if err := cmdUnpair("home"); err != nil {
t.Fatal(err)
}
if f.open(port) {
t.Fatal("the rule outlived the last pairing")
}
// MUTATION: cmdUnpair does not release. RESULT: red, the rule outlived the last pairing.
// MUTATION: doorPortWanted ignores pairings. RESULT: red, the window's end removed the rule.
}
// The daemon's door alone: bound, it opens the way in; its last pairing
// stopped and gone, the rule goes with it.
func TestTheDaemonsDoorRemovesTheRuleWhenItsLastPairingGoes(t *testing.T) {
pairingBox(t)
laneTestProcess(t)
f := fwFakeWorld(t)
ca := newTestCA(t, "home host CA")
addr := freeDoorAddr(t)
cfg := hostDialsPairingFor(t, ca, "home.cseries.localhost", addr) // never saved: gone once it stops
stop := runPairingBG(t, cfg)
doorWaitFor(t, 5*time.Second, "the rule", func() bool { return f.open(doorPort(addr)) })
if err := stop(); err != nil {
t.Fatal(err)
}
doorWaitFor(t, 5*time.Second, "the rule to go", func() bool { return !f.open(doorPort(addr)) })
// MUTATION: the door's release does not call doorFirewallRelease. RESULT: red, timed out waiting for the rule to go.
// MUTATION: bindLoop does not ensure. RESULT: red, timed out waiting for the rule.
}
// fwHung is a firewall whose commands hang until let go.
type fwHung struct {
*fwFake
gate chan struct{}
}
func (f *fwHung) has(port string) (bool, bool, error) { <-f.gate; return f.fwFake.has(port) }
// A firewall command that hangs never holds the daemon's door shut: the door
// serves while the bastion waits on the firewall.
func TestTheDoorServesWhileTheFirewallHangs(t *testing.T) {
pairingBox(t)
laneTestProcess(t)
f := &fwHung{fwFake: &fwFake{rules: map[string]bool{}}, gate: make(chan struct{})}
prevFor, prevBy := doorFirewallFor, doorFirewallByID
t.Cleanup(func() { doorFirewallFor, doorFirewallByID = prevFor, prevBy })
doorFirewallFor = func(string) (doorFirewall, string) { return f, "" }
doorFirewallByID = func(string) doorFirewall { return f }
var once sync.Once
release := func() { once.Do(func() { close(f.gate) }) }
t.Cleanup(release)
ca := newTestCA(t, "home host CA")
addr := freeDoorAddr(t)
cfg := hostDialsPairingFor(t, ca, "home.cseries.localhost", addr)
stop := runPairingBG(t, cfg)
doorWaitFor(t, 5*time.Second, "the door to answer a handshake", func() bool {
c, err := net.DialTimeout("tcp", addr, time.Second)
if err != nil {
return false
}
defer c.Close()
_ = c.SetDeadline(time.Now().Add(time.Second))
err = tls.Client(c, &tls.Config{InsecureSkipVerify: true, ServerName: tcpServerName}).Handshake()
var ne net.Error
return err == nil || !(errors.As(err, &ne) && ne.Timeout())
})
release()
doorWaitFor(t, 5*time.Second, "the rule", func() bool { return f.open(doorPort(addr)) })
if err := stop(); err != nil {
t.Fatal(err)
}
// MUTATION: bindLoop ensures before it serves. RESULT: red, timed out waiting for the door to answer a handshake.
}
+1 -5
View File
@@ -114,11 +114,7 @@ func (c *wsConn) SetWriteDeadline(t time.Time) error { return c.ws.SetWriteDeadl
func runEgressWorkers(pctx context.Context, self string, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer string,
dialOut egressDialer, n int) (wait func()) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
dialer := laneWSDialer(tlsCfg, dialHost)
var wg sync.WaitGroup
for i := 0; i < n; i++ {
wg.Add(1)
+8 -1
View File
@@ -31,4 +31,11 @@ var agentTokenEnvNames = []string{"PRIVATAE_AGENT_TOKEN", "SKEMA_AGENT_TOKEN"}
// written.
func piiTarget() string { return os.Getenv("PRIVATAE_PII_ADDR") }
func llmTarget() string { return os.Getenv("PRIVATAE_LLM_ADDR") }
// llmTarget is PRIVATAE_LLM_ADDR, or, unset, the OpenAI-compatible server
// found on this machine's loopback (llmlocal.go), "" when there is none.
func llmTarget() string {
if v := os.Getenv("PRIVATAE_LLM_ADDR"); v != "" {
return v
}
return discoveredLLM().Addr
}
+1 -5
View File
@@ -41,11 +41,7 @@ const gitForgeHost = "git.privatae.ai"
// runGitLane serves the loopback git proxy: one host WebSocket per local connection.
func runGitLane(ln net.Listener, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer string) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
dialer := laneWSDialer(tlsCfg, dialHost)
for {
local, err := ln.Accept()
if err != nil {
+1 -5
View File
@@ -107,11 +107,7 @@ type harnessSpec struct {
// instead of re-dialing a closed tunnel for the life of the process.
func runHarnessWorkers(pctx context.Context, self, dcnDir string, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer string, n int) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
dialer := laneWSDialer(tlsCfg, dialHost)
for i := 0; i < n; i++ {
go func() {
hold := laneHoldFor(self)
+778
View File
@@ -0,0 +1,778 @@
package main
// hostdoor.go: the door a Host connects to when this server can't reach it.
//
// A Host on a person's own computer sits behind their home router: a server
// on the internet can't open a connection to it, but it can open one to the
// server. So for such a pairing (Config.HostDials) the Host opens the TCP
// connections, to this server's door, and the lanes run over them exactly as
// they run over a connection this bastion dialled to a Host's lane port: this
// bastion is the TLS client, the Host's CA is the only root, the Host's
// certificate must name host.cseries.internal, and this bastion presents its
// operator certificate. Only the TCP direction is the other way round.
//
// One port serves every such pairing, and the Host's certificate says which
// pairing a connection is for: the door does the TLS handshake itself, as the
// client, and a connection whose Host certificate does not verify under the
// pinned CA of exactly one pairing it serves is closed before a byte of a lane
// crosses it. What passes is held, idle, in that pairing's pool, and a lane
// that would have dialled takes one from there instead (hostDoorDial). The TLS
// is already done, so the lane runs on it as it is (lanedial.go).
//
// The door is bound only while a pairing of this kind is served here (or the
// person has a pairing window open on it, pair_window.go). It never dials,
// never forwards a byte anywhere (a connection it keeps is a lane of a pairing
// it already serves, and nothing else is ever read from it), and the only
// thing it answers before a Host has proved itself is the TLS handshake. The
// Host keeps a few idle connections open and opens another each time one is
// taken; when it goes away, its connections close and it opens new ones when
// it is back. This server never reconnects to it: it can't.
//
// The wire contract the Host implements is docs/host-connects.md.
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"math/rand/v2"
"net"
"strings"
"sync"
"sync/atomic"
"time"
)
const (
// doorDefaultAddr is where the door listens: every address of the
// machine, IPv4 and IPv6, port 8812 (beside the Host's own 8810/8811).
doorDefaultAddr = ":8812"
// The two protocols a Host can name in the handshake (ALPN): a lane
// connection of a pairing this server serves, or a pairing through an
// open window. The Host chooses; the door offers the second only while a
// window is open.
laneALPN = "privatae-lane/1"
pairALPN = "privatae-pair/1"
// A connection has this long to finish its TLS handshake.
doorHandshakeTimeout = 10 * time.Second
// At most this many handshakes are in progress at once, and this many
// from one address (an IPv6 address counts as its /64, which one machine
// is commonly given whole): a scan or a flood costs the door a bounded
// number of goroutines.
doorMaxHandshakes = 256
doorMaxHandshakesPerSource = 8
// Of those, this many are kept for addresses a Host proved itself from
// within doorKnownHostFor (doorRememberHost): a stranger is admitted only
// while fewer than doorMaxHandshakes-doorReservedHandshakes are in
// progress, so strangers from any number of addresses never hold the last
// slots, and the person's own Host reconnects at its first try while the
// port is flooded. The reserve holds every handshake of 8 such addresses
// at once; one Host connects from one.
doorReservedHandshakes = 64
// Random early drop for strangers, as OpenSSH's MaxStartups (default
// 10:30:100): with fewer than doorEarlyDropStart handshakes in progress a
// stranger is always admitted (a new Host pairing, Hosts redialling after
// this bastion restarted); from there a stranger is refused with a chance
// that starts at doorEarlyDropRate and rises in a line to certain at the
// strangers' limit. A flood then fills the strangers' share slowly and
// never the reserve.
doorEarlyDropStart = 64
doorEarlyDropRate = 0.3
// An address a Host proved itself from is remembered for this long after
// its last verified handshake, in memory only (never written anywhere,
// gone when the bastion stops), at most doorKnownHostsMax of them. A Host
// connected keeps it fresh; this covers one away for a week (a laptop
// shut over a holiday) whose home address did not change.
doorKnownHostFor = 7 * 24 * time.Hour
doorKnownHostsMax = 64
// A pairing holds at most this many idle connections; one more is closed.
// The Host keeps hostSpareConns idle (docs/host-connects.md), so this is
// far above anything a well-behaved Host opens.
doorMaxIdle = 32
// TCP keepalive on every connection the door keeps, under the 30 s a
// home router's NAT may forget an idle connection after.
doorKeepAlive = 25 * time.Second
// A refusal from one address is written to the audit once per this, and
// at most doorRefusalAuditSources addresses are written per period: a
// scan from many addresses must not fill the audit or this map either.
doorRefusalAuditEvery = 10 * time.Minute
doorRefusalAuditSources = 256
)
// doorTakeWait is how long a lane waits for its Host's next connection. A
// lane's own context can cut it shorter.
const doorTakeWait = egressDialTimeout
// hostOpenedConn is a connection a Host opened to the door, its TLS done. Idle
// in a pool, a watcher reads it, so a connection the Host or the network closed
// leaves the pool at once; a lane that takes one stops the watcher first.
type hostOpenedConn struct {
*tls.Conn
p *doorPairing
state atomic.Int32 // connIdle, connTaken, connDead
done chan struct{}
werr error // the watcher's read result, set before done closes
closed atomic.Bool
}
const (
connIdle int32 = iota
connTaken
connDead
)
// Close closes the connection and counts it gone, once. It audits nothing
// and calls nothing but the close: every Close in this program can reach it
// as far as the static checks can tell (claims_static_test.go), so the
// "disconnected" audit is written by the pairing's own loop (auditLoop).
func (c *hostOpenedConn) Close() error {
err := c.Conn.Close()
if c.closed.CompareAndSwap(false, true) {
c.p.connClosed()
}
return err
}
// watch reads the idle connection. A Host sends nothing on a connection until
// a lane has spoken on it, so any read that returns is the end of it, unless
// the read was the take's own stop.
func (c *hostOpenedConn) watch() {
var b [1]byte
n, err := c.Conn.Read(b[:])
if n > 0 {
err = errors.New("the Host sent data on a connection no lane had used")
}
c.werr = err
close(c.done)
if c.state.CompareAndSwap(connIdle, connDead) {
_ = c.Close()
}
}
// claim takes an idle connection for a lane: the watcher stopped, its read
// deadline cleared. False when the connection is dead (and it is closed).
func (c *hostOpenedConn) claim() bool {
if !c.state.CompareAndSwap(connIdle, connTaken) {
return false
}
_ = c.Conn.SetReadDeadline(time.Unix(1, 0))
<-c.done
var ne net.Error
if c.werr != nil && !(errors.As(c.werr, &ne) && ne.Timeout()) {
_ = c.Close()
return false
}
_ = c.Conn.SetReadDeadline(time.Time{})
return true
}
// doorPairing is one pairing the door serves: its pinned CA, the operator
// certificate it presents, and its idle connections.
type doorPairing struct {
host string
roots *x509.CertPool
cert tls.Certificate
idle chan *hostOpenedConn
gone chan struct{} // the last connection closed (auditLoop)
live atomic.Int32 // connections open now, idle or taken
from atomic.Value // string: the address the last connection came from
// mu orders put against drain: once stopped, no connection is held.
mu sync.Mutex
stopped bool
}
func (p *doorPairing) connClosed() {
if p.live.Add(-1) == 0 {
select {
case p.gone <- struct{}{}:
default:
}
}
}
// auditLoop writes "disconnected" each time the pairing's last connection
// closes, until quit. Off Close's path on purpose (hostOpenedConn.Close).
func (p *doorPairing) auditLoop(quit <-chan struct{}) {
for {
select {
case <-p.gone:
if p.live.Load() == 0 {
from, _ := p.from.Load().(string)
pairingAudit("host_disconnected", p.host, "its last connection to this server closed (from "+from+")")
}
case <-quit:
return
}
}
}
// put holds a verified connection for the pairing's lanes.
func (p *doorPairing) put(tc *tls.Conn, src string) {
p.mu.Lock()
defer p.mu.Unlock()
if p.stopped {
// The pairing stopped while this connection was in its handshake.
_ = tc.Close()
return
}
c := &hostOpenedConn{Conn: tc, p: p, done: make(chan struct{})}
p.from.Store(src)
if p.live.Add(1) == 1 {
pairingAudit("host_connected", p.host, "connected to this server from "+src)
// Back from away: it hears from this server now, not on the minute.
defer kickBeats()
}
select {
case p.idle <- c:
go c.watch()
default:
// More idle connections than any Host keeps: this one is closed.
c.state.Store(connDead)
_ = c.Close()
}
}
// take is a lane's dial: the next idle connection the Host opened, waiting at
// most doorTakeWait for one.
func (p *doorPairing) take(ctx context.Context) (net.Conn, error) {
ctx, cancel := context.WithTimeout(ctx, doorTakeWait)
defer cancel()
for {
select {
case c := <-p.idle:
if c.claim() {
return c, nil
}
case <-ctx.Done():
return nil, fmt.Errorf("%s has not connected to this server within %s (your Host connects to this server; "+
"it does when it is running and can reach it)", p.host, doorTakeWait)
}
}
}
// drain closes every idle connection: the pairing stopped.
func (p *doorPairing) drain() {
p.mu.Lock()
p.stopped = true
p.mu.Unlock()
for {
select {
case c := <-p.idle:
if c.state.CompareAndSwap(connIdle, connDead) {
_ = c.Close()
}
default:
return
}
}
}
// hostDoor is one listening door.
type hostDoor struct {
addr string
// lanes: this door serves pairings' lanes (the running bastion's door).
// A pairing window opened with no bastion holding the port is served by
// a door of its own that serves the window alone.
lanes bool
mu sync.Mutex
ln net.Listener
pairings map[string]*doorPairing
refs int
inflight int
perSource map[string]int
refused map[string]time.Time
win windowState
closed bool
// draw is the early drop's random number in [0, 1) (doorDraw).
draw func() float64
}
// doorDraw is where a new door's early drop draws from. A variable so a test
// can fix it; nothing else assigns it.
var doorDraw = rand.Float64
func newHostDoor(addr string, lanes bool) *hostDoor {
return &hostDoor{addr: addr, lanes: lanes, pairings: map[string]*doorPairing{},
perSource: map[string]int{}, refused: map[string]time.Time{}, draw: doorDraw}
}
// listenDoor binds addr with keepalive on every accepted connection.
func listenDoor(addr string) (net.Listener, error) {
lc := net.ListenConfig{KeepAlive: doorKeepAlive}
return lc.Listen(context.Background(), "tcp", addr)
}
// serve accepts until the listener closes.
func (d *hostDoor) serve(ln net.Listener) {
for {
raw, err := ln.Accept()
if err != nil {
if errors.Is(err, net.ErrClosed) {
return
}
time.Sleep(50 * time.Millisecond)
continue
}
src := remoteHost(raw.RemoteAddr())
if !d.admitHandshake(sourceKey(src)) {
_ = raw.Close()
continue
}
go d.handle(raw, src)
}
}
func remoteHost(a net.Addr) string {
if a == nil {
return ""
}
h, _, err := net.SplitHostPort(a.String())
if err != nil {
return a.String()
}
return h
}
// sourceKey is the address the per-source limits count: an IPv4 address as
// it is, an IPv6 address as its /64.
func sourceKey(h string) string {
ip := net.ParseIP(h)
if ip == nil || ip.To4() != nil {
return h
}
return (&net.IPNet{IP: ip.Mask(net.CIDRMask(64, 128)), Mask: net.CIDRMask(64, 128)}).String()
}
// admitHandshake takes a handshake slot for src (a sourceKey), or refuses:
// the global and per-source limits for everyone, and for a source no Host
// proved itself from lately, the strangers' share and its early drop.
func (d *hostDoor) admitHandshake(src string) bool {
known := doorKnownHost(src, time.Now())
d.mu.Lock()
defer d.mu.Unlock()
if d.inflight >= doorMaxHandshakes || d.perSource[src] >= doorMaxHandshakesPerSource {
return false
}
if !known && !d.admitStranger(d.inflight) {
return false
}
d.inflight++
d.perSource[src]++
return true
}
// admitStranger is the early drop for a stranger's handshake with n in
// progress. Called with d.mu held.
func (d *hostDoor) admitStranger(n int) bool {
full := doorMaxHandshakes - doorReservedHandshakes
switch {
case n < doorEarlyDropStart:
return true
case n >= full:
return false
}
drop := doorEarlyDropRate + (1-doorEarlyDropRate)*float64(n-doorEarlyDropStart)/float64(full-doorEarlyDropStart)
return d.draw() >= drop
}
// doorKnown is the addresses (sourceKeys) a Host proved itself from, each
// with when it is forgotten. In memory only: an address is never written to
// disk, and a bastion that restarts knows none until its Hosts connect again.
var doorKnown struct {
sync.Mutex
until map[string]time.Time
}
// doorRememberHost notes that a Host proved itself from src (a sourceKey):
// its certificate verified under a pairing's pinned CA, or it paired through
// the window with the code.
func doorRememberHost(src string, now time.Time) {
doorKnown.Lock()
defer doorKnown.Unlock()
if doorKnown.until == nil {
doorKnown.until = map[string]time.Time{}
}
if _, ok := doorKnown.until[src]; !ok && len(doorKnown.until) >= doorKnownHostsMax {
oldest := ""
for k, t := range doorKnown.until {
if !now.Before(t) {
delete(doorKnown.until, k)
continue
}
if oldest == "" || t.Before(doorKnown.until[oldest]) {
oldest = k
}
}
if len(doorKnown.until) >= doorKnownHostsMax {
delete(doorKnown.until, oldest)
}
}
doorKnown.until[src] = now.Add(doorKnownHostFor)
}
// doorKnownHost is whether a Host proved itself from src within
// doorKnownHostFor.
func doorKnownHost(src string, now time.Time) bool {
doorKnown.Lock()
defer doorKnown.Unlock()
t, ok := doorKnown.until[src]
if ok && !now.Before(t) {
delete(doorKnown.until, src)
return false
}
return ok
}
func (d *hostDoor) handshakeDone(src string) {
d.mu.Lock()
defer d.mu.Unlock()
d.inflight--
if d.perSource[src]--; d.perSource[src] <= 0 {
delete(d.perSource, src)
}
}
// match is the pairing whose pinned CA the Host's certificate verifies under,
// for host.cseries.internal: exactly one, or a refusal.
func (d *hostDoor) match(chain []*x509.Certificate) (*doorPairing, error) {
if len(chain) == 0 {
return nil, errors.New("the connection presented no certificate")
}
inter := x509.NewCertPool()
for _, c := range chain[1:] {
inter.AddCert(c)
}
d.mu.Lock()
defer d.mu.Unlock()
var hit *doorPairing
for _, p := range d.pairings {
if _, err := chain[0].Verify(x509.VerifyOptions{Roots: p.roots, Intermediates: inter, DNSName: tcpServerName}); err != nil {
continue
}
if hit != nil {
return nil, errors.New("the certificate verifies for more than one pairing here — refusing")
}
hit = p
}
if hit == nil {
return nil, errors.New("the certificate is not that of a Host this server serves")
}
return hit, nil
}
// handle does the TLS handshake as the client and keeps the connection only
// for what the Host proved: a lane of the one pairing its certificate names,
// or a pairing through the open window.
func (d *hostDoor) handle(raw net.Conn, src string) {
released := false
release := func() {
if !released {
released = true
d.handshakeDone(sourceKey(src))
}
}
defer release()
_ = raw.SetDeadline(time.Now().Add(doorHandshakeTimeout))
win := d.openWindow()
var protos []string
if d.lanes {
protos = append(protos, laneALPN)
}
if win != nil {
protos = append(protos, pairALPN)
}
if len(protos) == 0 {
_ = raw.Close()
return
}
var matched *doorPairing
var pairCA *x509.Certificate
cfg := &tls.Config{
ServerName: tcpServerName, MinVersion: tls.VersionTLS13, NextProtos: protos,
// The chain is checked in VerifyConnection against the pinned CAs
// alone (or, for a pairing, against the CA it carries, which the code
// then vouches for): the system roots play no part.
InsecureSkipVerify: true,
VerifyConnection: func(cs tls.ConnectionState) error {
var err error
switch {
case cs.NegotiatedProtocol == laneALPN && d.lanes:
matched, err = d.match(cs.PeerCertificates)
case cs.NegotiatedProtocol == pairALPN && win != nil:
pairCA, err = chainCA(cs.PeerCertificates)
default:
err = fmt.Errorf("the connection named no protocol this server speaks (%q)", cs.NegotiatedProtocol)
}
return err
},
GetClientCertificate: func(*tls.CertificateRequestInfo) (*tls.Certificate, error) {
if matched != nil {
return &matched.cert, nil
}
// A pairing: this bastion holds no certificate from the Host yet.
return &tls.Certificate{}, nil
},
}
tc := tls.Client(raw, cfg)
if err := tc.Handshake(); err != nil {
d.refuse(src, err)
_ = raw.Close()
return
}
release()
switch tc.ConnectionState().NegotiatedProtocol {
case laneALPN:
_ = raw.SetDeadline(time.Time{})
d.mu.Lock()
live := !d.closed && d.pairings[matched.host] == matched
d.mu.Unlock()
if !live {
_ = tc.Close()
return
}
doorRememberHost(sourceKey(src), time.Now())
matched.put(tc, src)
case pairALPN:
d.serveWindow(tc, win, pairCA, src)
}
}
// refuse audits a refused connection, once per address per
// doorRefusalAuditEvery: a scan of the port must not fill the audit.
func (d *hostDoor) refuse(src string, err error) {
key := sourceKey(src)
d.mu.Lock()
last, seen := d.refused[key]
now := time.Now()
if seen && now.Sub(last) < doorRefusalAuditEvery {
d.mu.Unlock()
return
}
if len(d.refused) >= doorRefusalAuditSources {
for k, t := range d.refused {
if now.Sub(t) >= doorRefusalAuditEvery {
delete(d.refused, k)
}
}
if len(d.refused) >= doorRefusalAuditSources {
// This period's share of the audit is spent.
d.mu.Unlock()
return
}
}
d.refused[key] = now
d.mu.Unlock()
pairingAudit("door_refused", src, "refused: "+err.Error())
}
// chainCA is the CA a pairing Host's chain carries, checked to sign its leaf
// for host.cseries.internal. The code proves the Host holds it
// (pair_window.go); nothing else does.
func chainCA(chain []*x509.Certificate) (*x509.Certificate, error) {
if len(chain) < 2 {
return nil, errors.New("the Host presented no certificate authority with its certificate")
}
for _, c := range chain[1:] {
if !c.IsCA {
continue
}
roots := x509.NewCertPool()
roots.AddCert(c)
if _, err := chain[0].Verify(x509.VerifyOptions{Roots: roots, DNSName: tcpServerName}); err == nil {
return c, nil
}
}
return nil, errors.New("the Host's certificate is not signed by a certificate authority it presented, for " + tcpServerName)
}
// ── the running bastion's doors ─────────────────────────────────────────────
var doors struct {
sync.Mutex
m map[string]*hostDoor
}
// doorBindRetry is how often a door that could not bind tries again (the
// port is held by a pairing window's own door until that window closes).
var doorBindRetry = 2 * time.Second
// newDoorPairing is cfg as the door serves it: its pinned CA, its operator
// certificate, an empty pool.
func newDoorPairing(cfg *Config) (*doorPairing, error) {
cert, err := tls.X509KeyPair([]byte(cfg.OperatorCert), []byte(cfg.OperatorKey))
if err != nil {
return nil, fmt.Errorf("operator cert/key: %w", err)
}
roots := x509.NewCertPool()
if !roots.AppendCertsFromPEM([]byte(cfg.CACert)) {
return nil, errors.New("bad CA cert in config")
}
return &doorPairing{host: cfg.HostName, roots: roots, cert: cert, idle: make(chan *hostOpenedConn, doorMaxIdle),
gone: make(chan struct{}, 1)}, nil
}
// doorOpen registers an active host-dials pairing with the door on its
// address, binding the door if this is its first pairing. The returned
// release unregisters it and closes the door after its last pairing.
func doorOpen(cfg *Config) (func(), error) {
p, err := newDoorPairing(cfg)
if err != nil {
return nil, err
}
addr := cfg.doorAddr()
quit := make(chan struct{})
go p.auditLoop(quit)
doors.Lock()
if doors.m == nil {
doors.m = map[string]*hostDoor{}
}
d := doors.m[addr]
first := d == nil
if first {
d = newHostDoor(addr, true)
doors.m[addr] = d
}
d.mu.Lock()
if _, dup := d.pairings[cfg.HostName]; dup {
d.mu.Unlock()
doors.Unlock()
close(quit)
return nil, fmt.Errorf("%s is already served on %s", cfg.HostName, addr)
}
d.pairings[cfg.HostName] = p
d.refs++
d.mu.Unlock()
doors.Unlock()
if first {
go d.bindLoop()
}
return func() {
defer close(quit)
doors.Lock()
d.mu.Lock()
if d.pairings[cfg.HostName] == p {
delete(d.pairings, cfg.HostName)
}
d.refs--
last := d.refs == 0
if last {
d.closed = true
delete(doors.m, addr)
}
ln := d.ln
d.mu.Unlock()
doors.Unlock()
p.drain()
if last && ln != nil {
_ = ln.Close()
}
if last {
// The rule stays while a pairing on this port is still active
// (the bastion stopping is not the pairing going) or a window
// is open on it; otherwise it goes now.
doorFirewallRelease(addr)
}
}, nil
}
// bindLoop binds the door and serves it, trying again while the port is held
// and the door still has a pairing.
func (d *hostDoor) bindLoop() {
logged := false
for {
d.mu.Lock()
if d.closed {
d.mu.Unlock()
return
}
d.mu.Unlock()
ln, err := listenDoor(d.addr)
if err == nil {
d.mu.Lock()
if d.closed {
d.mu.Unlock()
_ = ln.Close()
return
}
d.ln = ln
d.mu.Unlock()
fmt.Printf("door: your Host connects to this server on %s (TCP)\n", d.addr)
// The way in through this machine's own firewall, while the door
// is in use (doorfw.go); removed when its last pairing goes. Beside
// serving, never before it: a firewall command that hangs (each
// may take doorFWExecWait) never holds the door shut.
go func() {
still := func() bool { d.mu.Lock(); defer d.mu.Unlock(); return !d.closed }
if said := doorFirewallEnsureWhile(d.addr, still); said != "" {
fmt.Println("door: " + said)
}
}()
d.serve(ln)
return
}
if !logged {
fmt.Printf("door: cannot listen on %s yet (%v); trying again every %s\n", d.addr, err, doorBindRetry)
logged = true
}
time.Sleep(doorBindRetry)
}
}
// hostDoorDial is a host-dials pairing's lane dial: the next connection its
// Host opened to the door.
func hostDoorDial(cfg *Config) func(ctx context.Context) (net.Conn, error) {
addr, host := cfg.doorAddr(), cfg.HostName
return func(ctx context.Context) (net.Conn, error) {
doors.Lock()
d := doors.m[addr]
doors.Unlock()
if d == nil {
return nil, fmt.Errorf("no door is open on %s for %s", addr, host)
}
d.mu.Lock()
p := d.pairings[host]
d.mu.Unlock()
if p == nil {
return nil, fmt.Errorf("%s is not served on %s", host, addr)
}
return p.take(ctx)
}
}
// hostDialsLaneDialer is localLaneDialer for a pairing whose Host connects
// to this server. The TLS configuration is the one the door uses, kept for
// the lanes' own use; every connection the dial returns has it done.
func hostDialsLaneDialer(cfg *Config) (func(ctx context.Context) (net.Conn, error), *tls.Config, error) {
if cfg.Bearer == "" {
return nil, nil, errors.New("this pairing holds no bearer, and its lane never opens without one")
}
cert, err := tls.X509KeyPair([]byte(cfg.OperatorCert), []byte(cfg.OperatorKey))
if err != nil {
return nil, nil, fmt.Errorf("operator cert/key: %w", err)
}
tc, err := hostCATLS(cfg.CACert, &cert)
if err != nil {
return nil, nil, fmt.Errorf("bad CA cert in config: %w", err)
}
return hostDoorDial(cfg), tc, nil
}
// doorAddr is where this pairing's Host connects: the address it was paired
// on, or the default.
func (c *Config) doorAddr() string {
if strings.TrimSpace(c.DoorAddr) != "" {
return c.DoorAddr
}
return doorDefaultAddr
}
// doorPort is the door's port, for the person's words.
func doorPort(addr string) string {
if _, p, err := net.SplitHostPort(addr); err == nil {
return p
}
return addr
}
+309
View File
@@ -0,0 +1,309 @@
//go:build linux
package main
import (
"crypto/ed25519"
"encoding/base64"
"net"
"os"
"strings"
"testing"
"time"
)
// windowBox is a pairing box whose `pair --open` runs with machine authority
// (held) and no organisation's policy, in the given mode.
func windowBox(t *testing.T, networked, held bool) {
t.Helper()
pairingBox(t)
prev := machineAuthorityHeld
machineAuthorityHeld = func() bool { return true }
t.Cleanup(func() { machineAuthorityHeld = prev })
withPolicy(t, nil)
if networked {
if err := setBoxMode(modeNetworked); err != nil {
t.Fatal(err)
}
}
machineAuthorityHeld = func() bool { return held }
}
// pairThroughWindow opens a window on a free port, pairs h's Host through it,
// and waits for `pair --open` to finish. h is made for that port.
func pairThroughWindow(t *testing.T, ca *testCA, handle, name string) (windowDoneReq, string) {
t.Helper()
addr := freeDoorAddr(t)
code, out, res := openWindowFor(t, addr)
h := newStandinHost(t, ca, handle, name, addr)
done, err := h.pair(formatWindowCode(code))
if err != nil {
t.Fatalf("pair: %v\n%s", err, pairingAuditText(t))
}
select {
case err := <-res:
if err != nil {
t.Fatalf("pair --open: %v\n%s", err, out.String())
}
case <-time.After(15 * time.Second):
t.Fatal("pair --open never ended")
}
return done, out.String()
}
// 1. A server paired through `pair --open`, by someone with machine
// authority, on a machine nobody is in charge of, has that Host in charge:
// the anchor, the existing mechanism. A local machine is made networked for
// it (an anchor is a networked machine's). The rules before it still win: a
// Host on this machine, an organisation's policy naming another Host. Nobody
// without machine authority puts anyone in charge, and a local machine that
// already serves a Host the person said yes to keeps serving it.
//
// MUTATION: windowCommit calls anchorAfterPair instead of anchorAfterWindow.
// RESULT: red (local, and networked beside another pairing: no anchor).
// MUTATION: drop the sameMachineHost check in anchorAfterWindow. RESULT: red.
// MUTATION: drop the local-mode active check. RESULT: red.
// MUTATION: anchorAfterWindow ignores openerAuthority. RESULT: red.
func TestAServerPairedThroughTheWindowHasThatHostInCharge(t *testing.T) {
const name = "home201721e4.cseries.localhost"
t.Run("local machine, nobody in charge: networked, anchored, active", func(t *testing.T) {
windowBox(t, false, true)
done, out := pairThroughWindow(t, newTestCA(t, "home CA"), "home", name)
if done.State != pairingActive {
t.Fatalf("done: %+v", done)
}
if a := currentAnchor(); a == nil || a.AnchorHost != name {
t.Fatalf("anchor = %+v, want %s", a, name)
}
if boxMode() != modeNetworked {
t.Fatalf("mode %s: a machine with an anchor is networked", boxMode())
}
if !strings.Contains(out, name+" is now the Host in charge of this machine") || !strings.Contains(out, "pair --open") {
t.Fatalf("the person is not told who is in charge:\n%s", out)
}
if a := pairingAuditText(t); !strings.Contains(a, `"op":"anchor_take"`) {
t.Fatalf("no anchor_take in the audit:\n%s", a)
}
})
t.Run("networked, unanchored, another pairing already there: anchored, the other waits", func(t *testing.T) {
windowBox(t, true, true)
writePairing(t, tRemote)
_, _ = pairThroughWindow(t, newTestCA(t, "home CA"), "home", name)
if a := currentAnchor(); a == nil || a.AnchorHost != name {
t.Fatalf("anchor = %+v, want %s", a, name)
}
if got := loadState(t, name); got != pairingActive {
t.Fatalf("the anchor never waits on itself: %q", got)
}
if got := loadState(t, tRemote); got != pairingPending {
t.Fatalf("the other Host is %q, want pending (waiting for %s)", got, name)
}
})
t.Run("a Host on this machine is paired: it stays the one in charge", func(t *testing.T) {
windowBox(t, true, true)
writeTCPPairing(t, tLocal2, "127.0.0.1:8811", pairingPending)
_, _ = pairThroughWindow(t, newTestCA(t, "home CA"), "home", name)
if a := currentAnchor(); a != nil {
t.Fatalf("a Host on this machine lost the machine to the window's Host: anchor %+v", a)
}
})
t.Run("an organisation's policy names another Host: it waits for that one", func(t *testing.T) {
windowBox(t, true, true)
withPolicy(t, &anchorPolicy{Host: "corp.host.privatae.ai", Source: "test policy"})
done, _ := pairThroughWindow(t, newTestCA(t, "home CA"), "home", name)
if a := currentAnchor(); a != nil {
t.Fatalf("anchor %+v against the policy", a)
}
if done.State != pairingPending || done.AdmittedBy != "corp.host.privatae.ai" {
t.Fatalf("done: %+v, want pending for the Host the policy names", done)
}
})
t.Run("an organisation's policy names this Host: in charge, on a local machine too", func(t *testing.T) {
windowBox(t, false, true)
withPolicy(t, &anchorPolicy{Host: name, Source: "test policy"})
writeTCPPairing(t, tLocal2, "127.0.0.1:8811", pairingRevoked)
_, _ = pairThroughWindow(t, newTestCA(t, "home CA"), "home", name)
if a := currentAnchor(); a == nil || a.AnchorHost != name {
t.Fatalf("anchor = %+v, want %s (the policy names it)", a, name)
}
})
t.Run("no machine authority: nobody is put in charge, the mode is left", func(t *testing.T) {
windowBox(t, false, false)
_, _ = pairThroughWindow(t, newTestCA(t, "home CA"), "home", name)
if a := currentAnchor(); a != nil || boxMode() != modeLocal {
t.Fatalf("without authority: anchor %+v, mode %s", a, boxMode())
}
})
t.Run("opened without machine authority, committed by a door that has it: nobody in charge", func(t *testing.T) {
windowBox(t, false, true)
cfg := Config{HostName: name, State: ""}
if took, err := anchorAfterWindow(&cfg, []*Config{&cfg}, false); took.Taken || err != nil || currentAnchor() != nil {
t.Fatalf("a window opened without authority put %s in charge: %+v %v", name, took, err)
}
})
t.Run("a local machine already serving another Host: it keeps serving it", func(t *testing.T) {
windowBox(t, false, true)
writePairing(t, tRemote)
_, _ = pairThroughWindow(t, newTestCA(t, "home CA"), "home", name)
if a := currentAnchor(); a != nil || boxMode() != modeLocal {
t.Fatalf("anchor %+v, mode %s: the Host already served here would wait", a, boxMode())
}
if got := loadState(t, tRemote); got == pairingPending {
t.Fatal("the Host already served here was left waiting")
}
})
}
// 2. The same Host pairing the same server again, through a fresh window,
// replaces its pairing: no `unpair` first. The same CA is the same Host,
// whatever handle it gives (a C-Series Host's handle is "home" while its name
// carries an id). Its state, the anchor and its grants are kept, and the old
// credentials are gone.
//
// MUTATION: drop the doorPairingWithCA lookup in windowCommit. RESULT: red,
// "this Host is already paired with this server as ...".
func TestTheSameHostPairingAgainReplacesItsPairing(t *testing.T) {
const name = "home201721e4.cseries.localhost"
windowBox(t, true, true)
ca := newTestCA(t, "home CA")
_, _ = pairThroughWindow(t, ca, "home", name)
if a := currentAnchor(); a == nil || a.AnchorHost != name {
t.Fatalf("first pairing: anchor %+v", a)
}
before, _ := pairingNamed(t, name)
gp, err := pairingGrantPath(name)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(gp, []byte(`{"kept":true}`), 0o600); err != nil {
t.Fatal(err)
}
done, out := pairThroughWindow(t, ca, "home", name)
if done.State != pairingActive || done.HostName != name {
t.Fatalf("pairing again: %+v\n%s", done, out)
}
after, all := pairingNamed(t, name)
if len(all) != 1 {
t.Fatalf("%d pairings after pairing again, want the one", len(all))
}
if after.OperatorCert == before.OperatorCert || after.OperatorKey == before.OperatorKey || after.WGPrivateKey == before.WGPrivateKey {
t.Fatal("the old credentials were kept")
}
if after.ListenAddr != before.ListenAddr || after.FirstSeen != before.FirstSeen || after.State != before.State {
t.Fatalf("the pairing was not kept in place: %+v -> %+v", before, after)
}
if a := currentAnchor(); a == nil || a.AnchorHost != name {
t.Fatalf("the anchor was not kept: %+v", a)
}
if b, err := os.ReadFile(gp); err != nil || string(b) != `{"kept":true}` {
t.Fatalf("the grant was not kept: %q %v", b, err)
}
// Another Host (another CA) with the same handle is never this one.
done, err = func() (windowDoneReq, error) {
addr := freeDoorAddr(t)
code, _, res := openWindowFor(t, addr)
h := newStandinHost(t, newTestCA(t, "another CA"), "home", name, addr)
d, err := h.pair(formatWindowCode(code))
<-res
return d, err
}()
if err != nil || done.State != "refused" || !strings.Contains(done.Error, "different Host") {
t.Fatalf("another Host of the same name: %+v %v", done, err)
}
if c, _ := pairingNamed(t, name); c.Bearer != after.Bearer {
t.Fatal("another Host's pairing replaced this one's")
}
}
// The same Host paired again under a new name (same CA) stays in charge under
// that name: the anchor follows it, its pinned key kept.
//
// MUTATION: drop carryAnchorToNewName from windowCommit. RESULT: red.
func TestTheAnchorFollowsItsHostToANewName(t *testing.T) {
windowBox(t, true, true)
ca := newTestCA(t, "home CA")
_, _ = pairThroughWindow(t, ca, "home", "home1.cseries.localhost")
pub, _, _ := ed25519.GenerateKey(nil)
if err := pinAnchorVerifyKey(base64.StdEncoding.EncodeToString(pub)); err != nil {
t.Fatal(err)
}
f, _ := loadAnchorFile()
pin := f.VerifyPubkey
_, _ = pairThroughWindow(t, ca, "home", "home2.cseries.localhost")
all, _ := loadPairings()
if len(all) != 1 || all[0].HostName != "home2.cseries.localhost" {
t.Fatalf("pairings: %v", hostNames(all))
}
f, _ = loadAnchorFile()
if a := currentAnchor(); a == nil || a.AnchorHost != "home2.cseries.localhost" || f.VerifyPubkey != pin || pin == "" {
t.Fatalf("anchor %+v (pin kept: %v)", a, f != nil && f.VerifyPubkey == pin)
}
}
// The anchor moves only with machine authority, as every other anchor write
// (claim-anchor --repin's wall): a process without it leaves anchor.json as
// it is.
//
// MUTATION: drop the machineAuthorityHeld check in carryAnchorToNewName.
// RESULT: red, the anchor names the new name.
func TestTheAnchorMovesToANewNameOnlyWithMachineAuthority(t *testing.T) {
windowBox(t, true, true)
ca := newTestCA(t, "home CA")
_, _ = pairThroughWindow(t, ca, "home", "home1.cseries.localhost")
if a := currentAnchor(); a == nil || a.AnchorHost != "home1.cseries.localhost" {
t.Fatalf("first pairing: anchor %+v", a)
}
machineAuthorityHeld = func() bool { return false }
_, _ = pairThroughWindow(t, ca, "home", "home2.cseries.localhost")
if a := currentAnchor(); a == nil || a.AnchorHost != "home1.cseries.localhost" {
t.Fatalf("without machine authority the anchor was rewritten: %+v", a)
}
}
// 3. Right after `unpair` the running bastion still holds the door's port
// until it sees the pairing gone (a supervision period). `pair --open` waits
// for it to let go instead of failing "address already in use".
//
// MUTATION: drop the awaitDoorReleased call in cmdPairOpen. RESULT: red.
func TestPairOpenWaitsForTheRunningBastionToLetGoOfThePort(t *testing.T) {
pairingBox(t)
prevRunning, prevWait := pairDaemonRunning, doorReleaseWait
pairDaemonRunning = func() bool { return true }
doorReleaseWait = 10 * time.Second
t.Cleanup(func() { pairDaemonRunning, doorReleaseWait = prevRunning, prevWait })
addr := freeDoorAddr(t)
held, err := listenDoor(addr) // the running bastion's door, its pairing just removed
if err != nil {
t.Fatal(err)
}
time.AfterFunc(1500*time.Millisecond, func() { _ = held.Close() })
code, out, res := openWindowFor(t, addr)
if !strings.Contains(out.String(), "waiting for it to let go") {
t.Fatalf("the person is not told why it waits:\n%s", out.String())
}
h := newStandinHost(t, newTestCA(t, "home CA"), "home", "home.cseries.localhost", addr)
if done, err := h.pair(formatWindowCode(code)); err != nil || done.State != pairingActive {
t.Fatalf("pair after the port was let go: %+v %v", done, err)
}
if err := <-res; err != nil {
t.Fatalf("pair --open: %v\n%s", err, out.String())
}
// Another program that never lets go: it still fails, after the wait, and
// says what to do.
doorReleaseWait = 300 * time.Millisecond
addr2 := freeDoorAddr(t)
other, err := net.Listen("tcp", addr2)
if err != nil {
t.Fatal(err)
}
defer other.Close()
err = cmdPairOpen(addr2, &syncBuf{})
if err == nil || !strings.Contains(err.Error(), "choose another with --port") {
t.Fatalf("a port another program holds: %v", err)
}
if !strings.Contains(err.Error(), "address already in use") {
t.Fatalf("not refused for the address in use: %v", err)
}
}
+479
View File
@@ -0,0 +1,479 @@
//go:build linux
package main
import (
"bytes"
"context"
"crypto/tls"
"net"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
// freeDoorAddr is a loopback port nothing listens on.
func freeDoorAddr(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
a := ln.Addr().String()
_ = ln.Close()
return a
}
// syncBuf is a bytes.Buffer two goroutines can share.
type syncBuf struct {
mu sync.Mutex
b bytes.Buffer
}
func (s *syncBuf) Write(p []byte) (int, error) { s.mu.Lock(); defer s.mu.Unlock(); return s.b.Write(p) }
func (s *syncBuf) String() string { s.mu.Lock(); defer s.mu.Unlock(); return s.b.String() }
// openWindowFor runs `pair --open` on addr in the background and returns the
// code it shows, its output, and its result.
func openWindowFor(t *testing.T, addr string) (code string, out *syncBuf, result <-chan error) {
t.Helper()
prevPoll := windowPollEvery
windowPollEvery = 20 * time.Millisecond
t.Cleanup(func() { windowPollEvery = prevPoll })
out = &syncBuf{}
res := make(chan error, 1)
go func() { res <- cmdPairOpen(addr, out) }()
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if w := loadOpenWindow(); w != nil && strings.Contains(out.String(), formatWindowCode(w.Code)) {
return w.Code, out, res
}
time.Sleep(10 * time.Millisecond)
}
t.Fatalf("pair --open showed no code: %q", out.String())
return "", nil, nil
}
func pairingAuditText(t *testing.T) string {
t.Helper()
d, _ := configDir()
b, _ := os.ReadFile(filepath.Join(d, "pairing_audit.jsonl"))
return string(b)
}
// The code the person reads is the code both sides key the proofs with,
// however it is typed back.
func TestTheWindowCodeIsReadBackHoweverItIsTyped(t *testing.T) {
c, err := newWindowCode()
if err != nil || normalizeWindowCode(c) != c || len(c) != windowCodeLen {
t.Fatalf("a fresh code is not normalised: %q %v", c, err)
}
f := formatWindowCode(c)
if strings.Count(f, "-") != 3 || normalizeWindowCode(f) != c || normalizeWindowCode(strings.ToLower(f)) != c {
t.Fatalf("the shown code %q does not read back as %q", f, c)
}
if normalizeWindowCode("O1IL0-00000-00000-00000") != "01110000000000000000" {
t.Fatal("O, I and L read as the digits they look like")
}
for _, bad := range []string{"", "ABCDE", c + "0", strings.Replace(c, c[:1], "U", 1)} {
if normalizeWindowCode(bad) != "" {
t.Fatalf("%q is not a code", bad)
}
}
for args, want := range map[string]string{"--open": doorDefaultAddr, "--open --port 9443": ":9443", "--port=9443 --open": ":9443"} {
got, ok, err := parsePairOpenArgs(strings.Fields(args))
if !ok || err != nil || got != want {
t.Fatalf("%s: %q %v %v", args, got, ok, err)
}
}
for _, args := range []string{"--open --port 0", "--open --port x", "--open skb_x"} {
if _, ok, err := parsePairOpenArgs(strings.Fields(args)); !ok || err == nil {
t.Fatalf("%s must be refused", args)
}
}
if _, ok, _ := parsePairOpenArgs([]string{"skb_x"}); ok {
t.Fatal("a token pair is not --open")
}
}
// A Host that connects pairs through the window: the code proved both ways,
// the pairing saved with its CA pinned and no address to dial, the window
// closed after one use, and each step in the audit.
func TestAHostThatConnectsPairsThroughTheWindow(t *testing.T) {
pairingBox(t)
addr := freeDoorAddr(t)
code, out, res := openWindowFor(t, addr)
ca := newTestCA(t, "home host CA")
h := newStandinHost(t, ca, "home", "home.cseries.localhost", addr)
done, err := h.pair(formatWindowCode(code))
if err != nil {
t.Fatalf("%v\n%s", err, pairingAuditText(t))
}
if done.State != pairingActive || done.HostName != "home.cseries.localhost" {
t.Fatalf("done: %+v", done)
}
select {
case err := <-res:
if err != nil {
t.Fatalf("pair --open: %v\n%s", err, out.String())
}
case <-time.After(10 * time.Second):
t.Fatal("pair --open never saw the pairing end its window")
}
if !strings.Contains(out.String(), "Your Host connects to this server") || !strings.Contains(out.String(), "Paired with home.cseries.localhost") {
t.Fatalf("what the person reads:\n%s", out.String())
}
cfgs, _ := loadPairings()
c := pairingFor(cfgs, "home")
if c == nil || !c.hostDials() || c.DoorAddr != addr || c.LaneAddr != "" || !sameCAs(c.CACert, ca.pem) ||
!strings.HasPrefix(c.Bearer, "mcp_dials_home_") || operatorCertRefusal(c.OperatorCert, ca.pem, c.OperatorKey) != nil {
t.Fatalf("the pairing: %+v", c)
}
if w := readWindowFile(); w != nil {
t.Fatalf("the window outlived its pairing: %+v", w)
}
// One use: the same code again finds no window.
if _, err := h.pair(code); err == nil {
t.Fatal("a used code paired again")
}
a := pairingAuditText(t)
for _, op := range []string{`"window_opened"`, `"window_paired"`} {
if !strings.Contains(a, op) {
t.Fatalf("the audit has no %s:\n%s", op, a)
}
}
}
// SECURITY: a wrong code gets nothing, costs a wait that doubles, and the
// fifth closes the window. A proof made over another certificate (one
// relayed from another connection) is a wrong code. A Host never signs for a
// server that did not prove the code.
func TestAWrongCodeIsThrottledAndClosesTheWindow(t *testing.T) {
pairingBox(t)
prev := windowFailWait
windowFailWait = 100 * time.Millisecond
t.Cleanup(func() { windowFailWait = prev })
addr := freeDoorAddr(t)
code, _, res := openWindowFor(t, addr)
ca := newTestCA(t, "home host CA")
h := newStandinHost(t, ca, "home", "home.cseries.localhost", addr)
relayed := newStandinHost(t, ca, "home", "home.cseries.localhost", addr)
relayed.lieLeaf = true
if _, err := relayed.pair(code); err == nil {
t.Fatal("a proof over another certificate paired")
}
// The wait after it: the next try, at once, is not even offered a pairing.
if _, err := h.pair(code); err == nil || !strings.Contains(err.Error(), "handshake") {
t.Fatalf("a try inside the wait must be refused at the handshake, got %v", err)
}
wrong := "ZZZZZ-ZZZZZ-ZZZZZ-ZZZZZ"
for i := 2; i <= windowMaxFailures; i++ {
time.Sleep(time.Duration(1<<(i-2))*windowFailWait + 50*time.Millisecond)
if _, err := h.pair(wrong); err == nil {
t.Fatalf("wrong code %d paired", i)
}
}
select {
case err := <-res:
if err == nil || !strings.Contains(err.Error(), "wrong codes") {
t.Fatalf("pair --open after %d wrong codes: %v", windowMaxFailures, err)
}
case <-time.After(5 * time.Second):
t.Fatal("the window stayed open after five wrong codes")
}
if _, err := h.pair(code); err == nil {
t.Fatal("the right code paired after the window closed")
}
cfgs, _ := loadPairings()
if len(cfgs) != 0 || h.csrs != 0 || relayed.csrs != 0 {
t.Fatalf("a wrong code got something: %d pairings, %d CSRs signed", len(cfgs), h.csrs+relayed.csrs)
}
if n := strings.Count(pairingAuditText(t), `"window_wrong_code"`); n != windowMaxFailures {
t.Fatalf("%d wrong codes audited, want %d", n, windowMaxFailures)
}
// The Host's side: a server whose proof does not check gets no signature.
addr2 := freeDoorAddr(t)
code2, _, res2 := openWindowFor(t, addr2)
strict := newStandinHost(t, ca, "home", "home.cseries.localhost", addr2)
strict.badProof = true
if d, err := strict.pair(code2); err != nil || d.State != "refused" {
t.Fatalf("a Host refusing the server's proof: %+v %v", d, err)
}
if err := <-res2; err == nil {
t.Fatal("the window reported a pairing the Host refused")
}
if cfgs, _ := loadPairings(); len(cfgs) != 0 || strict.csrs != 0 {
t.Fatal("a pairing was kept that the Host never signed for")
}
}
// hostDialsPairingFor is a host-dials pairing with an operator certificate
// from ca and its door on addr.
func hostDialsPairingFor(t *testing.T, ca *testCA, host, addr string) *Config {
t.Helper()
c := tcpPairingFor(t, ca, host, "")
c.HostDials, c.DoorAddr = true, addr
c.Bearer = "mcp_dials_lane_bearer"
return c
}
// runPairingBG runs the pairing until the returned stop.
func runPairingBG(t *testing.T, cfg *Config) (stop func() error) {
ctx, cancel := context.WithCancel(context.Background())
errc := make(chan error, 1)
go func() { errc <- runPairing(ctx, cfg) }()
return func() error {
cancel()
select {
case err := <-errc:
return err
case <-time.After(10 * time.Second):
t.Fatal("the pairing did not stop")
}
return nil
}
}
func doorWaitFor(t *testing.T, d time.Duration, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(d)
for time.Now().Before(deadline) {
if ok() {
return
}
time.Sleep(20 * time.Millisecond)
}
t.Fatalf("timed out waiting for %s", what)
}
// The lanes run over the connections the Host opens, exactly as over a
// dialled lane port: the heartbeat with the bearer, every worker pool, a
// harness job on a parked worker, every handshake under host.cseries.internal
// and with the operator certificate (the stand-in requires and verifies it).
// The door is bound only while the pairing is served, and the bastion never
// dials anything.
func TestTheLanesRunOverTheConnectionsTheHostOpens(t *testing.T) {
pairingBox(t)
laneTestProcess(t)
ca := newTestCA(t, "home host CA")
addr := freeDoorAddr(t)
cfg := hostDialsPairingFor(t, ca, "home.cseries.localhost", addr)
h := newStandinHost(t, ca, "home", cfg.HostName, addr)
h.lane.jobs <- []byte(`{"harness_id":"h1","mode":"echo","prompt":"hello over a Host-opened connection"}`)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go h.serveLanes(ctx)
stop := runPairingBG(t, cfg)
doorWaitFor(t, 10*time.Second, "the heartbeat and every worker", func() bool {
hellos, beats := h.lane.snapshot()
return len(beats) > 0 && len(hellos) >= harnessWorkerCount+sessionWorkerCount+mgmtWorkerCount+egressWorkerCount+serveWorkerCount
})
select {
case r := <-h.lane.replies:
if !strings.HasSuffix(r, "exit=0") {
t.Fatalf("the harness job: %s", r)
}
case <-time.After(10 * time.Second):
t.Fatal("a parked harness worker served no job")
}
hellos, beats := h.lane.snapshot()
if beats[0] != "Bearer "+cfg.Bearer {
t.Fatalf("the heartbeat carries the bearer: %q", beats)
}
count := map[string]int{}
for _, hl := range hellos {
path, msg, _ := strings.Cut(hl, " ")
if !strings.HasPrefix(msg, "BEARER "+cfg.Bearer) {
t.Fatalf("a worker opened without the bearer: %q", hl)
}
if strings.Contains(msg, "lane=session") {
path += " session"
}
count[path]++
}
if count["/bastion-harness"] < harnessWorkerCount || count["/bastion-harness session"] != sessionWorkerCount ||
count["/bastion-mgmt"] != mgmtWorkerCount || count["/bastion-egress"] != egressWorkerCount ||
count["/bastion-serve"] != serveWorkerCount {
t.Fatalf("the worker pools over Host-opened connections: %v", count)
}
h.lane.mu.Lock()
sni := append([]string(nil), h.lane.sni...)
h.lane.mu.Unlock()
for _, n := range sni {
if n != tcpServerName {
t.Fatalf("a handshake asked for %q", n)
}
}
if !strings.Contains(pairingAuditText(t), `"host_connected"`) {
t.Fatal("the Host's connection is not in the audit")
}
if err := stop(); err != nil {
t.Fatal(err)
}
// Closed with its last pairing.
doorWaitFor(t, 5*time.Second, "the door to close", func() bool {
ln, err := net.Listen("tcp", addr)
if err == nil {
_ = ln.Close()
}
return err == nil
})
}
// SECURITY: the door keeps a connection only from the Host whose pinned CA
// its certificate verifies under, for host.cseries.internal, naming the lane
// protocol. Another CA, the right CA under another name, no protocol, and a
// pairing protocol with no window open all end at the handshake: nothing is
// pooled and no lane ever runs on them.
func TestTheDoorRefusesAnythingButItsPinnedHost(t *testing.T) {
pairingBox(t)
laneTestProcess(t)
ca := newTestCA(t, "home host CA")
addr := freeDoorAddr(t)
cfg := hostDialsPairingFor(t, ca, "home.cseries.localhost", addr)
stop := runPairingBG(t, cfg)
defer stop()
doorWaitFor(t, 5*time.Second, "the door", func() bool {
c, err := net.Dial("tcp", addr)
if err == nil {
_ = c.Close()
}
return err == nil
})
impostor := newTestCA(t, "impostor")
cases := map[string]struct {
cert tls.Certificate
protos []string
}{
"another CA": {impostor.serverCert(t, tcpServerName), []string{laneALPN}},
"the Host's own name": {ca.serverCert(t, "home.cseries.localhost"), []string{laneALPN}},
"no protocol": {ca.serverCert(t, tcpServerName), nil},
"pairing, no window": {ca.serverCert(t, tcpServerName), []string{pairALPN}},
}
for name, tc := range cases {
raw, err := net.Dial("tcp", addr)
if err != nil {
t.Fatal(err)
}
s := tls.Server(raw, &tls.Config{Certificates: []tls.Certificate{tc.cert}, NextProtos: tc.protos,
ClientAuth: tls.RequestClientCert})
_ = s.SetDeadline(time.Now().Add(5 * time.Second))
herr := s.Handshake()
if herr == nil {
// TLS 1.3: the server's side can finish before the client's
// verdict arrives; the verdict is the connection closing.
_, herr = s.Read(make([]byte, 1))
}
if herr == nil {
t.Fatalf("%s: the door kept the connection", name)
}
if got := len(s.ConnectionState().PeerCertificates); got != 0 {
t.Fatalf("%s: the door presented this bastion's certificate to a stranger", name)
}
_ = raw.Close()
}
// Nothing was pooled: a lane's dial finds no connection. Its wait is
// cut short by the context: doorTakeWait is a constant, read by every
// lane, an earlier test's still running ones too.
short, cancelShort := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancelShort()
if c, err := hostDoorDial(cfg)(short); err == nil {
_ = c.Close()
t.Fatal("a refused connection reached the pool")
}
if !strings.Contains(pairingAuditText(t), `"door_refused"`) {
t.Fatal("the refusals are not in the audit")
}
}
// The Host redials, the bastion never does: a Host that stops and comes back
// gets its lanes again, and so does a bastion that restarts, all on
// connections the Host opens. A connection the Host closes while idle leaves
// the pool at once.
func TestTheHostReconnectsAfterEitherSideRestarts(t *testing.T) {
pairingBox(t)
laneTestProcess(t)
ca := newTestCA(t, "home host CA")
addr := freeDoorAddr(t)
cfg := hostDialsPairingFor(t, ca, "home.cseries.localhost", addr)
h := newStandinHost(t, ca, "home", cfg.HostName, addr)
beats := func() int { _, b := h.lane.snapshot(); return len(b) }
stop := runPairingBG(t, cfg)
ctx, cancel := context.WithCancel(context.Background())
go h.serveLanes(ctx)
doorWaitFor(t, 10*time.Second, "the first heartbeat", func() bool { return beats() >= 1 })
// The Host stops: every connection closes, and the pool empties.
cancel()
doorWaitFor(t, 5*time.Second, "the pool to empty", func() bool {
doors.Lock()
d := doors.m[addr]
doors.Unlock()
d.mu.Lock()
p := d.pairings[cfg.HostName]
d.mu.Unlock()
return p.live.Load() == 0
})
doorWaitFor(t, 5*time.Second, "the Host going away in the audit", func() bool {
return strings.Contains(pairingAuditText(t), `"host_disconnected"`)
})
// It comes back: a beat at once (the reconnect brings the lanes up; the
// beat loop's next tick or a new run sends it).
n := beats()
ctx2, cancel2 := context.WithCancel(context.Background())
defer cancel2()
go h.serveLanes(ctx2)
if err := stop(); err != nil {
t.Fatal(err)
}
// The bastion restarts while the Host keeps redialling.
stop = runPairingBG(t, cfg)
defer stop()
doorWaitFor(t, 15*time.Second, "a heartbeat after both restarts", func() bool { return beats() > n })
if h.dials.Load() < 2*hostSpareConns {
t.Fatalf("the Host opened %d connections over two runs", h.dials.Load())
}
}
// The terminal closing the window and a door committing a pairing on it are
// one rename each, across processes: whichever comes first wins. A window
// closed first is never claimed; a window claimed first is reported as the
// pairing it made, never "nothing was paired".
func TestClosingAWindowAndClaimingItCannotBothWin(t *testing.T) {
pairingBox(t)
open := func(id string) {
w := &pairWindow{ID: id, Code: "0123456789ABCDEFGHJK", Door: ":8812", Opened: time.Now().UTC().Format(time.RFC3339),
Expires: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)}
if err := saveWindowFile(w); err != nil {
t.Fatal(err)
}
}
open("a")
if removeWindow("a") {
t.Fatal("an unclaimed window closed as claimed")
}
if claimWindow("a") {
t.Fatal("a window closed at the terminal was claimed")
}
open("b")
if !claimWindow("b") {
t.Fatal("an open window could not be claimed")
}
if !removeWindow("b") {
t.Fatal("closing a claimed window did not say it was claimed")
}
endWindow("b", windowResult{Host: "home.cseries.localhost", State: pairingActive})
if res := awaitClaimedResult("b"); res == nil || res.Host != "home.cseries.localhost" {
t.Fatalf("the claimed window's result: %+v", res)
}
if claimedWindow("b") != nil {
t.Fatal("the claimed file outlived its result")
}
}
+123
View File
@@ -0,0 +1,123 @@
//go:build linux
package main
import (
"crypto/tls"
"errors"
"fmt"
"io"
"net"
"net/http"
"strings"
"sync"
"testing"
"time"
)
// A connection that finishes its handshake after its pairing stopped is
// closed, never held idle with a watcher nobody will ever stop.
func TestAConnectionPutAfterThePairingStoppedIsClosed(t *testing.T) {
pairingBox(t)
p := &doorPairing{host: "home", idle: make(chan *hostOpenedConn, doorMaxIdle)}
p.drain()
a, b := net.Pipe()
t.Cleanup(func() { _ = a.Close(); _ = b.Close() })
p.put(tls.Client(a, &tls.Config{InsecureSkipVerify: true}), "192.0.2.1")
if n, live := len(p.idle), p.live.Load(); n != 0 || live != 0 {
t.Fatalf("a stopped pairing holds %d idle connections, %d live", n, live)
}
_ = b.SetReadDeadline(time.Now().Add(2 * time.Second))
if n, err := b.Read(make([]byte, 1)); err != io.EOF {
t.Fatalf("the connection was left open (read %d, %v)", n, err)
}
}
// Every connection that was offered the window while it was open runs into
// the same rules when its exchange starts: a window closed by the fifth wrong
// code, inside a wrong code's wait, or closed at the terminal, starts nothing.
func TestAWindowClosedDuringAHandshakeStartsNothing(t *testing.T) {
pairingBox(t)
addr := freeDoorAddr(t)
w := &pairWindow{ID: "w1", Code: strings.Repeat("A", windowCodeLen), Door: addr,
Expires: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)}
if err := saveWindowFile(w); err != nil {
t.Fatal(err)
}
d := newHostDoor(addr, false)
if d.openWindow() == nil {
t.Fatal("the window is not offered")
}
d.win.failures = windowMaxFailures
if d.windowBegin(w) {
t.Fatal("a pairing started after the fifth wrong code")
}
d.win.failures, d.win.next = 1, time.Now().Add(time.Hour)
if d.windowBegin(w) {
t.Fatal("a pairing started inside a wrong code's wait")
}
d.win.failures, d.win.next = 0, time.Time{}
removeWindow(w.ID)
if d.windowBegin(w) {
t.Fatal("a pairing started on a window closed at the terminal")
}
}
// A window closed at the terminal while the exchange runs: the Host is not
// asked to sign anything.
func TestAWindowClosedDuringTheExchangeIsNeverRedeemed(t *testing.T) {
pairingBox(t)
ca := newTestCA(t, "home host CA")
var mu sync.Mutex
var paths []string
c := &http.Client{Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) {
mu.Lock()
paths = append(paths, r.URL.Path)
mu.Unlock()
return nil, fmt.Errorf("no Host here")
})}
w := &pairWindow{ID: "gone", Code: strings.Repeat("A", windowCodeLen), Door: ":8812"}
_, res := windowCommit(c, w, windowHelloResp{HostHandle: "home", HostName: "home.cseries.localhost"}, ca.pem,
func() string { return "proof" })
mu.Lock()
defer mu.Unlock()
if len(paths) != 0 || !strings.Contains(res.Error, "window closed") {
t.Fatalf("a closed window was redeemed: requests %v, result %+v", paths, res)
}
}
type roundTripFunc func(*http.Request) (*http.Response, error)
func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
// One IPv6 machine is one source: its /64 counts as one address.
func TestTheHandshakeLimitCountsAnIPv6NetworkAsOneSource(t *testing.T) {
d := newHostDoor(":0", true)
for i := 1; i <= doorMaxHandshakesPerSource; i++ {
if !d.admitHandshake(sourceKey(fmt.Sprintf("2001:db8:1:2::%x", i))) {
t.Fatalf("handshake %d refused", i)
}
}
if d.admitHandshake(sourceKey("2001:db8:1:2::ffff")) {
t.Fatal("a ninth handshake from one /64 was admitted")
}
if !d.admitHandshake(sourceKey("2001:db8:1:3::1")) || !d.admitHandshake(sourceKey("192.0.2.7")) {
t.Fatal("another source was refused")
}
}
// A scan from many addresses writes a bounded number of refusals.
func TestAScanFromManyAddressesDoesNotFillTheAudit(t *testing.T) {
pairingBox(t)
d := newHostDoor(":0", true)
for i := 0; i < 4*doorRefusalAuditSources; i++ {
d.refuse(fmt.Sprintf("2001:db8:%x::1", i), errors.New("scan"))
}
if n := strings.Count(pairingAuditText(t), `"door_refused"`); n > doorRefusalAuditSources {
t.Fatalf("%d refusals audited from a scan, at most %d", n, doorRefusalAuditSources)
}
if len(d.refused) > doorRefusalAuditSources {
t.Fatalf("the refusal map grew to %d", len(d.refused))
}
}
+402
View File
@@ -0,0 +1,402 @@
//go:build linux
package main
// The door's handshake slots under a flood: the global and per-source caps,
// the share kept for addresses a Host proved itself from, and the random
// early drop for strangers (hostdoor.go, admitHandshake).
import (
"crypto/tls"
"fmt"
"io"
"math"
"net"
"sync"
"sync/atomic"
"testing"
"time"
)
// freshDoorMemory forgets every Host address, before and after the test.
func freshDoorMemory(t *testing.T) {
t.Helper()
reset := func() {
doorKnown.Lock()
doorKnown.until = nil
doorKnown.Unlock()
}
reset()
t.Cleanup(reset)
}
// neverDrops is a draw the early drop never refuses on (it refuses when the
// draw is below the drop chance, which is under 1 short of the limit).
func neverDrops() float64 { return math.Nextafter(1, 0) }
// alwaysDrops is a draw the early drop refuses on wherever it applies.
func alwaysDrops() float64 { return 0 }
func strangersShare() int { return doorMaxHandshakes - doorReservedHandshakes }
// The global and per-source caps hold for everyone, a known Host's address
// included, and strangers never take the reserved share, however lucky.
func TestTheHandshakeCapsStillHold(t *testing.T) {
freshDoorMemory(t)
now := time.Now()
d := newHostDoor(":0", true)
d.draw = neverDrops
// Strangers: admitted up to their share, from enough addresses, and not
// one more.
for i := 0; i < strangersShare(); i++ {
if !d.admitHandshake(fmt.Sprintf("203.0.113.%d", i/doorMaxHandshakesPerSource)) {
t.Fatalf("stranger handshake %d refused below the strangers' share", i)
}
}
if d.admitHandshake("198.51.100.200") {
t.Fatalf("a stranger took a slot of the reserve (%d in progress)", d.inflight)
}
// Known addresses: the rest, 8 from each, and then nothing for anyone.
for i := 0; d.inflight < doorMaxHandshakes; i++ {
src := fmt.Sprintf("198.51.100.%d", i)
doorRememberHost(src, now)
for j := 0; j < doorMaxHandshakesPerSource; j++ {
if !d.admitHandshake(src) {
t.Fatalf("handshake %d of known %s refused with %d in progress", j, src, d.inflight)
}
}
if d.inflight < doorMaxHandshakes && d.admitHandshake(src) {
t.Fatalf("known %s got a ninth handshake", src)
}
}
doorRememberHost("192.0.2.1", now)
if d.inflight != doorMaxHandshakes || d.admitHandshake("192.0.2.1") {
t.Fatalf("a handshake past the global cap of %d (%d in progress)", doorMaxHandshakes, d.inflight)
}
// A known address is held to 8 at once on an empty door too.
e := newHostDoor(":0", true)
for j := 0; j < doorMaxHandshakesPerSource; j++ {
if !e.admitHandshake("192.0.2.1") {
t.Fatal("a known address was refused on an empty door")
}
}
if e.admitHandshake("192.0.2.1") {
t.Fatal("a known address got a ninth handshake at once")
}
}
// Strangers are dropped early as the door fills: never below the start,
// with the chance rising from the rate to certain at the strangers' share.
// A known Host's address is never early-dropped.
func TestStrangersAreDroppedEarlyAsTheDoorFills(t *testing.T) {
freshDoorMemory(t)
d := newHostDoor(":0", true)
full := strangersShare()
for _, c := range []struct {
n int
draw float64
admits bool
}{
{0, 0, true},
{doorEarlyDropStart - 1, 0, true},
{doorEarlyDropStart, doorEarlyDropRate - 0.01, false},
{doorEarlyDropStart, doorEarlyDropRate + 0.01, true},
{(doorEarlyDropStart + full) / 2, (1+doorEarlyDropRate)/2 - 0.01, false},
{(doorEarlyDropStart + full) / 2, (1+doorEarlyDropRate)/2 + 0.01, true},
{full - 1, 0.99, false},
{full - 1, neverDrops(), true},
{full, neverDrops(), false},
} {
draw := c.draw
d.draw = func() float64 { return draw }
if got := d.admitStranger(c.n); got != c.admits {
t.Fatalf("with %d in progress and a draw of %v: admitted %v, want %v", c.n, c.draw, got, c.admits)
}
}
// Through admitHandshake: an unlucky stranger is refused at the start,
// a known address with the same draw is not, up to the global cap.
d = newHostDoor(":0", true)
d.draw = alwaysDrops
for i := 0; i < doorEarlyDropStart; i++ {
if !d.admitHandshake(fmt.Sprintf("203.0.113.%d", i)) {
t.Fatalf("stranger %d refused below the start", i)
}
}
if d.admitHandshake("203.0.113.250") {
t.Fatal("an unlucky stranger was admitted past the start")
}
doorRememberHost("198.51.100.7", time.Now())
d.inflight = doorMaxHandshakes - 1
if !d.admitHandshake("198.51.100.7") {
t.Fatal("a known Host's address was early-dropped")
}
}
// An address is remembered for doorKnownHostFor after its last verified
// handshake, in a bounded memory that forgets the oldest first.
func TestAHostAddressIsRememberedForAWhileAndBounded(t *testing.T) {
freshDoorMemory(t)
now := time.Now()
doorRememberHost("198.51.100.7", now)
if !doorKnownHost("198.51.100.7", now.Add(doorKnownHostFor-time.Second)) {
t.Fatal("forgotten before its time")
}
if doorKnownHost("198.51.100.7", now.Add(doorKnownHostFor)) {
t.Fatal("remembered past its time")
}
if doorKnownHost("198.51.100.8", now) {
t.Fatal("an address no Host proved itself from is known")
}
// A fresh handshake renews it.
doorRememberHost("198.51.100.7", now)
doorRememberHost("198.51.100.7", now.Add(doorKnownHostFor-time.Second))
if !doorKnownHost("198.51.100.7", now.Add(doorKnownHostFor+time.Hour)) {
t.Fatal("a renewed address was forgotten")
}
// Bounded: the oldest goes first, the newest stays.
for i := 0; i < 2*doorKnownHostsMax; i++ {
doorRememberHost(fmt.Sprintf("2001:db8:%x::/64", i), now.Add(time.Duration(i)*time.Second))
}
doorKnown.Lock()
n := len(doorKnown.until)
doorKnown.Unlock()
if n > doorKnownHostsMax {
t.Fatalf("the memory holds %d addresses, at most %d", n, doorKnownHostsMax)
}
last := now.Add(time.Duration(2*doorKnownHostsMax) * time.Second)
if !doorKnownHost(fmt.Sprintf("2001:db8:%x::/64", 2*doorKnownHostsMax-1), last) {
t.Fatal("the newest address was forgotten")
}
if doorKnownHost("2001:db8:0::/64", last) {
t.Fatal("the oldest address outlived a full memory")
}
}
// pipeDoor is a door served over in-memory connections from any address the
// test names, with one pairing for ca's Host.
type pipeDoor struct {
d *hostDoor
p *doorPairing
ca *testCA
conns chan net.Conn
quit chan struct{}
accepts atomic.Int64
pushed int64
mu sync.Mutex
ends []net.Conn
}
func (l *pipeDoor) Accept() (net.Conn, error) {
l.accepts.Add(1)
select {
case c := <-l.conns:
return c, nil
case <-l.quit:
return nil, net.ErrClosed
}
}
func (l *pipeDoor) Close() error { return nil }
func (l *pipeDoor) Addr() net.Addr { return &net.TCPAddr{Port: 8812} }
type fromConn struct {
net.Conn
from net.Addr
}
func (c fromConn) RemoteAddr() net.Addr { return c.from }
func newPipeDoor(t *testing.T) *pipeDoor {
t.Helper()
ca := newTestCA(t, "home host CA")
cfg := hostDialsPairingFor(t, ca, "home.cseries.localhost", ":8812")
p, err := newDoorPairing(cfg)
if err != nil {
t.Fatal(err)
}
l := &pipeDoor{d: newHostDoor(":8812", true), p: p, ca: ca, conns: make(chan net.Conn), quit: make(chan struct{})}
l.d.pairings[cfg.HostName] = p
go l.d.serve(l)
t.Cleanup(func() {
close(l.quit)
l.mu.Lock()
for _, c := range l.ends {
_ = c.Close()
}
l.mu.Unlock()
p.drain()
doorWaitFor(t, 15*time.Second, "every handshake to end", func() bool { return l.inflight() == 0 })
})
return l
}
// connect hands the door a connection from ip and waits until the door has
// admitted or closed it. It returns the far end.
func (l *pipeDoor) connect(t *testing.T, ip string) net.Conn {
t.Helper()
near, far := net.Pipe()
l.mu.Lock()
l.ends = append(l.ends, far)
l.mu.Unlock()
l.conns <- fromConn{Conn: near, from: &net.TCPAddr{IP: net.ParseIP(ip), Port: 40000}}
l.pushed++
for l.accepts.Load() <= l.pushed {
time.Sleep(50 * time.Microsecond)
}
return far
}
// stranger connects from ip and never answers the handshake: a slot held
// until the handshake's time runs out.
func (l *pipeDoor) stranger(t *testing.T, ip string) {
far := l.connect(t, ip)
go func() { _, _ = io.Copy(io.Discard, far) }()
}
// host connects from ip as the pairing's Host and returns how long its
// handshake took, or the error that ended it.
func (l *pipeDoor) host(t *testing.T, ip string, within time.Duration) (time.Duration, error) {
t.Helper()
start := time.Now()
far := l.connect(t, ip)
s := tls.Server(far, &tls.Config{Certificates: []tls.Certificate{l.ca.serverCert(t, tcpServerName)},
NextProtos: []string{laneALPN}, MinVersion: tls.VersionTLS13, ClientAuth: tls.RequireAnyClientCert})
_ = s.SetDeadline(start.Add(within))
err := s.Handshake()
_ = s.SetDeadline(time.Time{})
return time.Since(start), err
}
func (l *pipeDoor) inflight() int {
l.d.mu.Lock()
defer l.d.mu.Unlock()
return l.d.inflight
}
// hostReconnectBound is how long the person's own Host may take to get a
// connection through a flooded door: its first try is admitted, so this is
// one handshake over an in-memory connection, with room for -race and a
// loaded machine. A refused try instead costs the Host's backoff (1 s, then
// doubling to 30 s, docs/host-connects.md).
const hostReconnectBound = 2 * time.Second
// SECURITY: 64 addresses flooding the door hold every slot strangers may
// take, and the person's own Host, connecting from the address it proved
// itself from, gets in at its first try, every time, within
// hostReconnectBound. The same Host from an address it never proved itself
// from waits like any stranger; what the reserve keeps is the address.
func TestTheOwnHostGetsInWhileStrangersFloodTheDoor(t *testing.T) {
pairingBox(t)
freshDoorMemory(t)
l := newPipeDoor(t)
const home = "198.51.100.7"
// Before the flood: a Host that connected once is known by its address.
if _, err := l.host(t, home, hostReconnectBound); err != nil {
t.Fatalf("the Host's first connection: %v", err)
}
doorWaitFor(t, 5*time.Second, "the connection in the pool", func() bool { return l.p.live.Load() == 1 })
if !doorKnownHost(home, time.Now()) {
t.Fatal("a Host's verified connection did not make its address known")
}
// The flood: 64 addresses, round and round, until every slot strangers
// may hold is held, with the real random early drop.
attempts := 0
for l.inflight() < strangersShare() {
if attempts++; attempts > 50000 {
t.Fatalf("the flood holds %d slots after %d connections", l.inflight(), attempts)
}
l.stranger(t, fmt.Sprintf("203.0.113.%d", attempts%64))
}
// Full for strangers: from any address, new or flooding, none gets in.
for i := 0; i < 64; i++ {
l.stranger(t, fmt.Sprintf("192.0.2.%d", i))
}
if n := l.inflight(); n != strangersShare() {
t.Fatalf("strangers hold %d slots, at most %d", n, strangersShare())
}
// The Host reconnects: each of its spare connections at the first try.
for i := 0; i < hostSpareConns; i++ {
took, err := l.host(t, home, hostReconnectBound)
if err != nil {
t.Fatalf("the Host's connection %d during the flood (after %d flood connections): %v", i, attempts, err)
}
if took > hostReconnectBound {
t.Fatalf("the Host's connection %d took %s, bound %s", i, took, hostReconnectBound)
}
t.Logf("the Host's connection %d through the flood (%d flood connections): %s", i, attempts, took)
}
doorWaitFor(t, 5*time.Second, "the Host's connections in the pool", func() bool {
return l.p.live.Load() == 1+hostSpareConns
})
// The same Host from an address it never proved itself from: refused.
if _, err := l.host(t, "198.51.100.99", hostReconnectBound); err == nil {
t.Fatal("an unknown address got past a door full for strangers")
}
}
// A Host never seen before pairs through the window when nobody floods the
// door, even with a draw the early drop would refuse on, and its address is
// known from then on, so its lane connections count as its own.
func TestANewHostPairsWhenNobodyFloods(t *testing.T) {
pairingBox(t)
freshDoorMemory(t)
prev := doorDraw
doorDraw = alwaysDrops
t.Cleanup(func() { doorDraw = prev })
addr := freeDoorAddr(t)
code, out, res := openWindowFor(t, addr)
ca := newTestCA(t, "home host CA")
h := newStandinHost(t, ca, "home", "home.cseries.localhost", addr)
done, err := h.pair(formatWindowCode(code))
if err != nil {
t.Fatalf("%v\n%s", err, pairingAuditText(t))
}
if done.State != pairingActive {
t.Fatalf("done: %+v", done)
}
select {
case err := <-res:
if err != nil {
t.Fatalf("pair --open: %v\n%s", err, out.String())
}
case <-time.After(10 * time.Second):
t.Fatal("pair --open never saw the pairing end its window")
}
host, _, _ := net.SplitHostPort(addr)
if !doorKnownHost(sourceKey(host), time.Now()) {
t.Fatal("a Host that paired through the window is not known by its address")
}
}
// SECURITY: only a verified handshake for a pairing the door still serves
// makes an address known: a certificate under another CA does not, nor the
// Host's own certificate once the door has stopped serving its pairing.
func TestOnlyAServedHostsVerifiedHandshakeMakesItsAddressKnown(t *testing.T) {
pairingBox(t)
freshDoorMemory(t)
l := newPipeDoor(t)
// handshake connects from ip with cert and waits for the door's verdict,
// the connection closing, so whatever the door does for it is done.
handshake := func(ip string, cert tls.Certificate) {
far := l.connect(t, ip)
s := tls.Server(far, &tls.Config{Certificates: []tls.Certificate{cert}, NextProtos: []string{laneALPN},
MinVersion: tls.VersionTLS13, ClientAuth: tls.RequireAnyClientCert})
_ = s.SetDeadline(time.Now().Add(5 * time.Second))
if err := s.Handshake(); err == nil {
_, _ = s.Read(make([]byte, 1))
}
_ = s.Close()
}
const impostorIP, stoppedIP = "198.51.100.60", "198.51.100.61"
impostor := newTestCA(t, "impostor")
handshake(impostorIP, impostor.serverCert(t, tcpServerName))
if doorKnownHost(impostorIP, time.Now()) {
t.Fatal("a certificate under another CA made its address known")
}
l.d.mu.Lock()
l.d.closed = true
l.d.mu.Unlock()
handshake(stoppedIP, l.ca.serverCert(t, tcpServerName))
if doorKnownHost(stoppedIP, time.Now()) {
t.Fatal("a handshake for a pairing the door no longer serves made its address known")
}
}
+371
View File
@@ -0,0 +1,371 @@
//go:build linux
package main
// A stand-in for a Host that connects to a server (docs/host-connects.md):
// it pairs through a window with the code, then keeps hostSpareConns idle
// lane connections open to the server's door and serves the lane routes of
// newLaneHost over them. The end-to-end run (hostdoor_e2e_linux_test.go)
// runs it in a network namespace behind a NAT.
import (
"context"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"io"
"log"
"net"
"net/http"
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
"time"
)
// hostSpareConns is how many idle connections the stand-in keeps open, the
// number docs/host-connects.md asks a Host to keep.
const hostSpareConns = 4
type standinHost struct {
ca *testCA
handle string
name string
server string // the door, host:port
// pairing
pairLeaf tls.Certificate // leaf + CA, as a pairing presents them
lieLeaf bool // prove over another certificate (a relayed proof)
badProof bool // check the bastion's proof against the wrong code
bearerN atomic.Int32
mu sync.Mutex
done []windowDoneReq
csrs int
// lanes
lane *laneHost
handler http.Handler
laneTLS *tls.Config
dials atomic.Int32
idle atomic.Int32
open atomic.Int32
}
func newStandinHost(t *testing.T, ca *testCA, handle, name, server string) *standinHost {
t.Helper()
pl := ca.serverCert(t, tcpServerName)
pl.Certificate = append(pl.Certificate, ca.cert.Raw)
h := &standinHost{ca: ca, handle: handle, name: name, server: server, pairLeaf: pl}
h.lane, h.handler, h.laneTLS = newLaneHost(ca.serverCert(t, tcpServerName), ca)
return h
}
// pair connects to the door and pairs with code. It returns what the
// bastion's done said, or the error that ended the exchange.
func (h *standinHost) pair(code string) (windowDoneReq, error) {
norm := normalizeWindowCode(code)
raw, err := net.DialTimeout("tcp", h.server, 5*time.Second)
if err != nil {
return windowDoneReq{}, err
}
tc := tls.Server(raw, &tls.Config{Certificates: []tls.Certificate{h.pairLeaf}, NextProtos: []string{pairALPN},
MinVersion: tls.VersionTLS13})
_ = tc.SetDeadline(time.Now().Add(20 * time.Second))
if err := tc.Handshake(); err != nil {
_ = raw.Close()
return windowDoneReq{}, fmt.Errorf("handshake: %w", err)
}
leafSum := sha256.Sum256(h.pairLeaf.Certificate[0])
if h.lieLeaf {
leafSum = sha256.Sum256([]byte("another connection's certificate"))
}
caSum := sha256.Sum256(h.ca.cert.Raw)
var nb, nh []byte
gotDone := make(chan windowDoneReq, 1)
mux := http.NewServeMux()
mux.HandleFunc("/v1/pair/hello", func(w http.ResponseWriter, r *http.Request) {
var req windowHelloReq
_ = json.NewDecoder(r.Body).Decode(&req)
nb, _ = hex.DecodeString(req.Nonce)
nh = make([]byte, 32)
_, _ = rand.Read(nh)
_ = json.NewEncoder(w).Encode(windowHelloResp{V: 1, HostHandle: h.handle, HostName: h.name,
Nonce: hex.EncodeToString(nh), Proof: windowProof(norm, "host", leafSum[:], caSum[:], nb, nh)})
})
mux.HandleFunc("/v1/pair/redeem", func(w http.ResponseWriter, r *http.Request) {
var req windowRedeemReq
_ = json.NewDecoder(r.Body).Decode(&req)
key := norm
if h.badProof {
key = "0000000000000000000Z"
}
real := sha256.Sum256(h.pairLeaf.Certificate[0])
if req.Proof != windowProof(key, "bastion", real[:], caSum[:], nb, nh) {
w.WriteHeader(http.StatusForbidden)
_ = json.NewEncoder(w).Encode(map[string]string{"error": "this server did not prove the code"})
return
}
h.mu.Lock()
h.csrs++
h.mu.Unlock()
_ = json.NewEncoder(w).Encode(map[string]any{"operator_cert": h.ca.signCSR(req.OperatorCSR), "ca_cert": h.ca.pem,
"bearer": fmt.Sprintf("mcp_dials_%s_%d", h.handle, h.bearerN.Add(1)), "transport": windowTransport,
"host_handle": h.handle, "host_name": h.name})
})
mux.HandleFunc("/v1/pair/done", func(w http.ResponseWriter, r *http.Request) {
var req windowDoneReq
_ = json.NewDecoder(r.Body).Decode(&req)
h.mu.Lock()
h.done = append(h.done, req)
h.mu.Unlock()
_, _ = w.Write([]byte("{}"))
gotDone <- req
})
ln := &oneConnListener{c: tc, closed: make(chan struct{})}
srv := &http.Server{Handler: mux, ErrorLog: log.New(io.Discard, "", 0)}
go func() { _ = srv.Serve(ln) }()
defer srv.Close()
select {
case d := <-gotDone:
return d, nil
case <-ln.gone():
return windowDoneReq{}, errors.New("the server closed the pairing connection")
case <-time.After(20 * time.Second):
return windowDoneReq{}, errors.New("no done within 20 s")
}
}
// oneConnListener hands out one connection, then blocks until closed.
type oneConnListener struct {
c net.Conn
once sync.Once
closed chan struct{}
cmu sync.Mutex
shut bool
}
func (l *oneConnListener) Accept() (net.Conn, error) {
var c net.Conn
l.once.Do(func() { c = &watchClose{Conn: l.c, l: l} })
if c != nil {
return c, nil
}
<-l.closed
return nil, net.ErrClosed
}
func (l *oneConnListener) gone() <-chan struct{} { return l.closed }
func (l *oneConnListener) Close() error {
l.cmu.Lock()
defer l.cmu.Unlock()
if !l.shut {
l.shut = true
close(l.closed)
}
return nil
}
func (l *oneConnListener) Addr() net.Addr { return l.c.LocalAddr() }
type watchClose struct {
net.Conn
l *oneConnListener
}
func (w *watchClose) Read(b []byte) (int, error) {
n, err := w.Conn.Read(b)
var ne net.Error
if err != nil && !(errors.As(err, &ne) && ne.Timeout()) {
_ = w.l.Close() // the http server's own read aborts are timeouts
}
return n, err
}
// serveLanes keeps hostSpareConns idle connections open to the door until
// ctx ends, serving the lane routes over each, and closes them all at the end
// (a Host that stops). Redials back off from 100 ms to 2 s.
func (h *standinHost) serveLanes(ctx context.Context) {
acc := make(chan net.Conn)
ln := &chanListener{c: acc, done: ctx.Done(), addr: &net.TCPAddr{}}
srv := &http.Server{Handler: h.handler, ErrorLog: log.New(io.Discard, "", 0)}
go func() { _ = srv.Serve(ln) }()
var all sync.Map
kick := make(chan struct{}, 64)
wait := 100 * time.Millisecond
for ctx.Err() == nil {
if h.idle.Load() >= hostSpareConns || h.open.Load() >= 64 {
select {
case <-kick:
case <-time.After(time.Second):
case <-ctx.Done():
}
continue
}
h.dials.Add(1)
d := net.Dialer{Timeout: 5 * time.Second, KeepAlive: 25 * time.Second}
raw, err := d.DialContext(ctx, "tcp", h.server)
if err == nil {
tc := tls.Server(raw, h.laneTLS)
_ = tc.SetDeadline(time.Now().Add(10 * time.Second))
if err = tc.HandshakeContext(ctx); err == nil {
_ = tc.SetDeadline(time.Time{})
h.idle.Add(1)
h.open.Add(1)
c := &usedConn{Conn: tc, h: h, kick: kick}
all.Store(c, true)
select {
case acc <- c:
case <-ctx.Done():
_ = c.Close()
}
wait = 100 * time.Millisecond
continue
}
_ = raw.Close()
}
select {
case <-time.After(wait):
case <-ctx.Done():
}
wait = min(2*wait, 2*time.Second)
}
_ = srv.Close()
all.Range(func(k, _ any) bool { _ = k.(*usedConn).Close(); return true })
}
// usedConn tells the stand-in when a lane first speaks on a connection (it
// is no longer idle) and when it closes.
type usedConn struct {
net.Conn
h *standinHost
kick chan struct{}
used atomic.Bool
closed atomic.Bool
}
func (c *usedConn) Read(b []byte) (int, error) {
n, err := c.Conn.Read(b)
if n > 0 && c.used.CompareAndSwap(false, true) {
c.h.idle.Add(-1)
c.nudge()
}
return n, err
}
func (c *usedConn) Close() error {
if c.closed.CompareAndSwap(false, true) {
if c.used.CompareAndSwap(false, true) {
c.h.idle.Add(-1)
}
c.h.open.Add(-1)
c.nudge()
}
return c.Conn.Close()
}
func (c *usedConn) nudge() {
select {
case c.kick <- struct{}{}:
default:
}
}
type chanListener struct {
c chan net.Conn
done <-chan struct{}
addr net.Addr
}
func (l *chanListener) Accept() (net.Conn, error) {
select {
case c := <-l.c:
return c, nil
case <-l.done:
return nil, net.ErrClosed
}
}
func (l *chanListener) Close() error { return nil }
func (l *chanListener) Addr() net.Addr { return l.addr }
// TestStandinHostProcess is the stand-in as a process of its own, for the
// end-to-end run behind a NAT (scripts/e2e-host-connects.sh) and on another
// machine's bastion. Skipped unless STANDIN_MODE is set:
//
// STANDIN_MODE=pair STANDIN_STATE=<dir> STANDIN_SERVER=<host:port> STANDIN_CODE=<code>
// STANDIN_MODE=serve STANDIN_STATE=<dir> STANDIN_SERVER=<host:port>
//
// The CA is kept in STANDIN_STATE, so a restarted stand-in is the same Host.
func TestStandinHostProcess(t *testing.T) {
mode := os.Getenv("STANDIN_MODE")
if mode == "" {
t.Skip("the stand-in Host process (scripts/e2e-host-connects.sh)")
}
dir, server := os.Getenv("STANDIN_STATE"), os.Getenv("STANDIN_SERVER")
handle := firstNonEmpty(os.Getenv("STANDIN_HANDLE"), "home")
name := firstNonEmpty(os.Getenv("STANDIN_NAME"), handle+".cseries.localhost")
ca := standinCA(t, dir, name)
h := newStandinHost(t, ca, handle, name, server)
logf := func(f string, a ...any) {
fmt.Printf(time.Now().UTC().Format("15:04:05.000Z ")+"standin "+f+"\n", a...)
}
switch mode {
case "pair":
d, err := h.pair(os.Getenv("STANDIN_CODE"))
if err != nil {
logf("PAIR FAILED: %v", err)
t.Fatal(err)
}
logf("PAIRED: done=%+v csrs=%d", d, h.csrs)
case "serve":
h.lane.jobs <- []byte(`{"harness_id":"e2e","mode":"echo","prompt":"a job over a connection the Host opened"}`)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go h.serveLanes(ctx)
var lb, lh int
for {
time.Sleep(time.Second)
hellos, beats := h.lane.snapshot()
if len(beats) != lb || len(hellos) != lh {
logf("lanes: dials=%d open=%d idle=%d beats=%d worker-hellos=%d", h.dials.Load(), h.open.Load(), h.idle.Load(), len(beats), len(hellos))
lb, lh = len(beats), len(hellos)
}
select {
case r := <-h.lane.replies:
logf("HARNESS JOB served over a Host-opened connection: %s", r)
default:
}
}
}
}
// standinCA is the stand-in's CA, made once and kept in dir.
func standinCA(t *testing.T, dir, cn string) *testCA {
_ = os.MkdirAll(dir, 0o700)
cp, kp := filepath.Join(dir, "ca.pem"), filepath.Join(dir, "ca.key")
if c, err := os.ReadFile(cp); err == nil {
k, _ := os.ReadFile(kp)
cb, _ := pem.Decode(c)
kb, _ := pem.Decode(k)
if cb != nil && kb != nil {
cert, err1 := x509.ParseCertificate(cb.Bytes)
key, err2 := x509.ParseECPrivateKey(kb.Bytes)
if err1 == nil && err2 == nil {
return &testCA{cert: cert, key: key, pem: string(c)}
}
}
}
ca := newTestCA(t, cn+" stand-in CA")
kd, _ := x509.MarshalECPrivateKey(ca.key)
_ = os.WriteFile(cp, []byte(ca.pem), 0o600)
_ = os.WriteFile(kp, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: kd}), 0o600)
return ca
}
+80
View File
@@ -74,3 +74,83 @@ func TestInstallPs1ElevatesForTheWholeMachineUnderPwsh(t *testing.T) {
t.Fatalf("%v\n%s", err, out)
}
}
// A whole-machine install replaces a program the tray holds open. The tray
// runs from the service's own file in every session, Windows will not replace
// a running program, and the install stopped with "Cannot create a file when
// that file already exists." with the service it had stopped left stopped.
// The program is moved aside instead, a failure starts the service again, and
// this person's tray is started from the new program.
//
// MUTATIONS (run with testdata/install_ps1_swap.ps1 under pwsh): drop the move
// aside from Install-Program; drop Start-Service from Stop-InstallFailed; drop
// the session filter from Restart-Tray; let Unregister-ScheduledTask's failure
// escape; drop the .old/.bak exclusion from the earlier-install stop. RESULT:
// each red.
func TestInstallPs1ReplacesAProgramTheTrayHoldsOpen(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "deploy", "install.ps1"))
if err != nil {
t.Fatal(err)
}
s := string(raw)
for _, want := range []string{
"Install-Program $tmp $exe",
"Move-Item -LiteralPath $target -Destination $aside",
`foreach ($name in @("$target.old", "$target.bak"))`,
"$stoppedService = $true",
`Stop-InstallFailed $stoppedService "The Windows service could not be registered`,
"Start-Service -Name 'PrivataeBastion' -ErrorAction Stop",
"if (Restart-Tray $installedExe)",
// The logon task is removed with the service stopped: a failure starts it again.
`Stop-InstallFailed $stoppedService ("The logon task from an earlier install could not be removed`,
// The trays running from the program moved aside are not an earlier install.
"($asideNames -notcontains $_.Path)",
} {
if !strings.Contains(s, want) {
t.Errorf("install.ps1 lacks %q", want)
}
}
if strings.Contains(s, "Move-Item -Force -Path $tmp -Destination $exe") {
t.Error("install.ps1 moves the download over the program in place, which fails while a tray runs it")
}
// A per-user reinstall with no -Pair stops the copy already running (the
// old program, moved aside) before it starts the task again, or it would
// say "running now" over the old version.
//
// MUTATION: drop the Stop-Process loop from that branch. RESULT: red.
if i := strings.Index(s, "# Started now, not only at the next sign-in"); i < 0 {
t.Error("install.ps1 no longer starts a per-user bastion at install")
} else if seg := s[i:]; !strings.Contains(seg[:strings.Index(seg, "Start-ScheduledTask")], "Stop-Process -Id $_.Id -Force") {
t.Error("a per-user reinstall starts the task without stopping the copy already running the old program")
}
if strings.Contains(s, `throw "could not register the Windows service`) {
t.Error("a failed service registration throws with the service it stopped left stopped")
}
// The aside names are ones the bastion clears at its next start.
src, err := os.ReadFile("update.go")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(src), `[]string{".old", ".new", ".bak"}`) {
t.Error("cleanupStaleBinaries no longer clears .old and .bak, which the installer leaves beside the program")
}
// The tray is started from the ordinary account, after the elevated step.
if r, e := strings.Index(s, "if (Restart-Tray $installedExe)"), strings.Index(s, "-Verb RunAs"); r < 0 || e < 0 || r < e {
t.Error("the tray is not restarted after the elevated step")
}
}
// Install-Program, Stop-InstallFailed and Restart-Tray, run by pwsh with
// Windows stood in (testdata/install_ps1_swap.ps1). Skipped where pwsh is not
// installed.
func TestInstallPs1SwapRunsUnderPwsh(t *testing.T) {
pwsh, err := exec.LookPath("pwsh")
if err != nil {
t.Skip("pwsh is not installed: install.ps1's swap is checked only by TestInstallPs1ReplacesAProgramTheTrayHoldsOpen here")
}
out, err := exec.Command(pwsh, "-NoProfile", "-NonInteractive", "-File",
filepath.Join("testdata", "install_ps1_swap.ps1"), filepath.Join("..", "deploy", "install.ps1")).CombinedOutput()
if err != nil || !strings.HasPrefix(strings.TrimSpace(string(out)), "ok:") {
t.Fatalf("%v\n%s", err, out)
}
}
+81
View File
@@ -0,0 +1,81 @@
package main
// lanedial.go: the one place a lane turns the connection its pairing's
// transport hands it into the mutual TLS the lane runs inside.
//
// Every transport but one hands a lane a plain connection (the tunnel, the
// lane socket, a Host's lane port), and the TLS is done here exactly as the
// WebSocket library did it before: the pairing's lane configuration, the
// server name it names, the handshake under the dial's context, and the name
// checked once more. A Host that connects to this server (hostdoor.go) hands
// the lane a connection whose TLS the door already did, under the same
// configuration (the Host's CA pinned, host.cseries.internal, this bastion's
// operator certificate): that one is passed on as it is. So the lane code is
// the same for every pairing; only the dial differs.
import (
"context"
"crypto/tls"
"net"
"net/http"
"time"
"github.com/gorilla/websocket"
)
// laneTLSDial is the TLS dial every lane uses: dialHost's connection with the
// lane's TLS done on it, or a Host-opened connection as the door left it.
func laneTLSDial(dialHost func(ctx context.Context) (net.Conn, error), tlsCfg *tls.Config) func(ctx context.Context, network, addr string) (net.Conn, error) {
return func(ctx context.Context, _, addr string) (net.Conn, error) {
c, err := dialHost(ctx)
if err != nil {
return nil, err
}
if hc, ok := c.(*hostOpenedConn); ok {
return hc, nil
}
cfg := tlsCfg.Clone()
if cfg == nil {
cfg = &tls.Config{}
}
if cfg.ServerName == "" {
if h, _, err := net.SplitHostPort(addr); err == nil {
cfg.ServerName = h
} else {
cfg.ServerName = addr
}
}
tc := tls.Client(c, cfg)
if err := tc.HandshakeContext(ctx); err != nil {
_ = c.Close()
return nil, err
}
if !cfg.InsecureSkipVerify {
if err := tc.VerifyHostname(cfg.ServerName); err != nil {
_ = tc.Close()
return nil, err
}
}
return tc, nil
}
}
// laneWSDialer is the WebSocket dialer every lane worker pool uses.
func laneWSDialer(tlsCfg *tls.Config, dialHost func(ctx context.Context) (net.Conn, error)) *websocket.Dialer {
return &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
NetDialTLSContext: laneTLSDial(dialHost, tlsCfg),
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
}
// laneHTTPTransport is the HTTP transport of a local pairing's requests (the
// heartbeat, git credentials, store proposals, context pulls).
func laneHTTPTransport(dialHost func(ctx context.Context) (net.Conn, error), tlsCfg *tls.Config) *http.Transport {
return &http.Transport{
DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { return dialHost(ctx) },
DialTLSContext: laneTLSDial(dialHost, tlsCfg),
TLSClientConfig: tlsCfg.Clone(),
}
}
+32 -4
View File
@@ -21,9 +21,13 @@ import (
type heldHost struct {
hellos atomic.Int32
hold atomic.Bool
tls *tls.Config
dial func(ctx context.Context) (net.Conn, error)
url string
// together: a held socket is closed only once this many hellos came in
// (or after barrierWait), so every worker of a pool has dialled before
// the first refusal starts the hold. 0 closes each at once.
together atomic.Int32
tls *tls.Config
dial func(ctx context.Context) (net.Conn, error)
url string
}
func startHeldHost(t *testing.T) *heldHost {
@@ -42,6 +46,9 @@ func startHeldHost(t *testing.T) *heldHost {
}
h.hellos.Add(1)
if h.hold.Load() {
for end := time.Now().Add(barrierWait); h.hellos.Load() < h.together.Load() && time.Now().Before(end); {
time.Sleep(5 * time.Millisecond)
}
_ = c.WriteControl(websocket.CloseMessage,
websocket.FormatCloseMessage(laneHeldCloseCode, "this bastion is quarantined"), time.Now().Add(time.Second))
return
@@ -66,6 +73,9 @@ func startHeldHost(t *testing.T) *heldHost {
return h
}
// barrierWait bounds how long a held socket waits for the rest of its pool.
const barrierWait = 15 * time.Second
// shortHolds makes the holds short enough for a test. Set before any worker
// runs, and never from a parallel subtest.
func shortHolds(t *testing.T, first, max time.Duration) {
@@ -109,6 +119,17 @@ func lanePools(dir string) map[string]func(ctx context.Context, host string, h *
// worker of any lane dials it again before the hold is over: one hello per
// worker, where a pool used to dial again at once or a second later, 30 to 80
// times a second from one held machine.
//
// The Host closes the sockets only once every worker has said hello
// (together). Closing the first at once made the test race the pool's own
// start: a worker not yet past its first waitClear when another was refused
// rightly waited out the hold before dialling, and the count read one of two
// (a full gate, 2026-11-03). The hold is timed from the refusals, which come
// after the last hello, so the 2 s look is inside the 3 s hold however slowly
// the workers start.
//
// MUTATION: drop hold.refused from a lane's worker (mgmt.go). RESULT: red,
// that lane dials again within the hold.
func TestEveryLanePoolWaitsOutAHostThatHoldsThisMachine(t *testing.T) {
shortHolds(t, 3*time.Second, time.Minute)
for lane, run := range lanePools(t.TempDir()) {
@@ -117,10 +138,17 @@ func TestEveryLanePoolWaitsOutAHostThatHoldsThisMachine(t *testing.T) {
host := "held-" + lane + ".example.test"
ownHold(t, host)
h := startHeldHost(t)
const n = 2
h.together.Store(n)
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
const n = 2
go run(ctx, host, h, n)
for end := time.Now().Add(barrierWait); h.hellos.Load() < n && time.Now().Before(end); {
time.Sleep(5 * time.Millisecond)
}
if got := h.hellos.Load(); got < n {
t.Fatalf("only %d of %d workers dialled the Host in %s", got, n, barrierWait)
}
time.Sleep(2 * time.Second)
if got := h.hellos.Load(); got != n {
t.Fatalf("%d dials within the hold from %d workers; want one each", got, n)
+207
View File
@@ -0,0 +1,207 @@
package main
// llmlocal.go — finding the OpenAI-compatible model server on this machine
// when PRIVATAE_LLM_ADDR names none.
//
// Before 0.63.0 that lane existed only when someone set the variable in the
// bastion's environment, which no installer does, so a machine running
// llama.cpp on its usual port served no model to its Host. Now, with the
// variable unset, the bastion looks on this machine's loopback, and nowhere
// else, at the ports the common servers
// listen on by default: 127.0.0.1:8080 (llama.cpp's llama-server), :8000
// (vLLM), :1234 (LM Studio), in that order. Each is asked GET /v1/models with a
// short timeout, and the first whose answer is OpenAI-shaped (a "data" list of
// models, each with an "id") is the server. Something else answering on one of
// those ports (a web app on 8080, say) is not taken for a model server, nor,
// on Linux, a server another account runs (llmListenerTrusted).
//
// The answer is kept and asked again every llmReprobeEvery, and at once after
// it failed (llmTargetFailed: the beat's probe of it, a dial of it), so the
// heartbeat that advertises the lane and the forward that serves it read the
// same address: both call llmTarget (env.go), the one function for the lane.
// The variable, when set, overrides all of this, as before. What was found, or
// that nothing was, is logged once per change and shown by `status` and
// `doctor`.
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strings"
"sync"
"time"
)
// llmCandidate is one place a local OpenAI-compatible server listens by
// default. Loopback only, by construction: the addresses are fixed here.
type llmCandidate struct {
Addr string
Server string // whose default port it is, for the words
}
var llmCandidates = []llmCandidate{
{"127.0.0.1:8080", "llama.cpp"},
{"127.0.0.1:8000", "vLLM"},
{"127.0.0.1:1234", "LM Studio"},
}
const (
llmReprobeEvery = 60 * time.Second
llmProbeTimeout = 800 * time.Millisecond
)
// llmProbe asks addr for its models; ok only for an OpenAI-shaped answer. A
// seam for the suite.
var llmProbe = probeOpenAIShaped
// llmTrusted: the program on that port may be handed the Host's prompts
// (llmlocal_linux.go: root's, this bastion's user's or the owner's). A seam.
var llmTrusted = llmListenerTrusted
// probeOpenAIShaped is GET http://addr/v1/models: a 200 whose body is
// {"data": [{"id": "..."}, ...]} with at least one model and an id on each.
func probeOpenAIShaped(addr string) ([]string, bool) {
cl := &http.Client{Timeout: llmProbeTimeout}
resp, err := cl.Get("http://" + addr + "/v1/models")
if err != nil {
return nil, false
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, false
}
return openAIShapedModels(io.LimitReader(resp.Body, 1<<20))
}
// openAIShapedModels reads an OpenAI /v1/models answer: the model ids, and
// whether it is one.
func openAIShapedModels(r io.Reader) ([]string, bool) {
var list struct {
Data []map[string]any `json:"data"`
}
if err := json.NewDecoder(r).Decode(&list); err != nil || len(list.Data) == 0 {
return nil, false
}
var ids []string
for _, m := range list.Data {
id, _ := m["id"].(string)
if strings.TrimSpace(id) == "" {
return nil, false
}
ids = append(ids, id)
}
return ids, true
}
// llmFound is what the bastion found on this machine.
type llmFound struct {
Addr string `json:"addr,omitempty"`
Server string `json:"server,omitempty"` // whose default port it answered on
Models []string `json:"models,omitempty"`
}
var llmLocal = struct {
sync.Mutex
found llmFound
at time.Time
probed bool
said string // what was last logged
}{}
// discoveredLLM is the server found on this machine, asked again when the
// last answer is older than llmReprobeEvery or was marked failed.
func discoveredLLM() llmFound {
llmLocal.Lock()
defer llmLocal.Unlock()
if llmLocal.probed && time.Since(llmLocal.at) < llmReprobeEvery {
return llmLocal.found
}
f := llmFound{}
for _, c := range llmCandidates {
if models, ok := llmProbe(c.Addr); ok && llmTrusted(c.Addr) {
f = llmFound{Addr: c.Addr, Server: c.Server, Models: models}
break
}
}
llmLocal.found, llmLocal.at, llmLocal.probed = f, time.Now(), true
// Said in the running bastion's log, once per change; a command such as
// `status` shows it in its own words instead.
if line := llmFoundWords(f); line != llmLocal.said && daemonRunningHere() {
llmLocal.said = line
fmt.Printf("[llm] %s\n", line)
}
return f
}
// daemonRunningHere: this process is the running bastion (cmdRun set up its
// supervisor), not a command beside it.
func daemonRunningHere() bool {
supervisor.Lock()
defer supervisor.Unlock()
return supervisor.ctx != nil
}
// llmTargetFailed marks the found server as failed when addr is it, so the
// next llmTarget asks again rather than serve a server that went away.
func llmTargetFailed(addr string) {
if addr == "" || os.Getenv("PRIVATAE_LLM_ADDR") != "" {
return
}
llmLocal.Lock()
defer llmLocal.Unlock()
if llmLocal.found.Addr == addr {
llmLocal.probed = false
}
}
// llmCandidateList is the places looked, for the words.
func llmCandidateList() string {
var out []string
for _, c := range llmCandidates {
out = append(out, c.Addr+" ("+c.Server+")")
}
return strings.Join(out, ", ")
}
func llmFoundWords(f llmFound) string {
if f.Addr == "" {
return "no OpenAI-compatible model server found on this machine (looked on " + llmCandidateList() +
"); set PRIVATAE_LLM_ADDR to name another on this machine"
}
s := fmt.Sprintf("found an OpenAI-compatible model server on %s, %s's port", f.Addr, f.Server)
if len(f.Models) > 0 {
s += ": " + strings.Join(f.Models, ", ")
}
return s
}
// statusLLM is the model-server line `status` and `doctor` show.
type statusLLM struct {
Addr string `json:"addr,omitempty"`
Source string `json:"source"` // "PRIVATAE_LLM_ADDR", "found" or "none"
Server string `json:"server,omitempty"`
Models []string `json:"models,omitempty"`
}
func llmStatus() *statusLLM {
if v := os.Getenv("PRIVATAE_LLM_ADDR"); v != "" {
return &statusLLM{Addr: v, Source: "PRIVATAE_LLM_ADDR"}
}
f := discoveredLLM()
if f.Addr == "" {
return &statusLLM{Source: "none"}
}
return &statusLLM{Addr: f.Addr, Source: "found", Server: f.Server, Models: f.Models}
}
func (l *statusLLM) words() string {
switch l.Source {
case "PRIVATAE_LLM_ADDR":
return l.Addr + " (set by PRIVATAE_LLM_ADDR)"
case "found":
return llmFoundWords(llmFound{Addr: l.Addr, Server: l.Server, Models: l.Models})
}
return llmFoundWords(llmFound{})
}
+75
View File
@@ -0,0 +1,75 @@
//go:build linux
package main
import (
"os"
"strconv"
"strings"
)
// llmListenerTrusted: the program listening on the loopback port addr names
// runs as root, as this bastion's own user, or as the machine's recorded
// owner. Found without anyone setting PRIVATAE_LLM_ADDR, a model server is
// handed the Host's prompts, and any account on the machine can listen on
// 127.0.0.1:8080 while llama.cpp is not running; a server of another account
// is not taken (the variable still names one on purpose). The kernel's own
// table says who owns each listening socket (/proc/net/tcp, tcp6).
func llmListenerTrusted(addr string) bool {
port := portOfAddr(addr)
if port <= 0 {
return false
}
var uids []uint32
for _, f := range []string{"/proc/net/tcp", "/proc/net/tcp6"} {
if raw, err := os.ReadFile(f); err == nil {
uids = append(uids, loopbackListenerUIDs(string(raw), port)...)
}
}
owner, _, ownerKnown, _ := ownerAccountLoad()
return listenersTrusted(uids, uint32(os.Getuid()), owner, ownerKnown)
}
// listenersTrusted: there is a listener, and every one is root's, self's or
// the owner's.
func listenersTrusted(uids []uint32, self, owner uint32, ownerKnown bool) bool {
if len(uids) == 0 {
return false
}
for _, u := range uids {
if u != 0 && u != self && (!ownerKnown || u != owner) {
return false
}
}
return true
}
// loopbackListenerUIDs reads a /proc/net/tcp table: the uid of every socket
// LISTENing on port at a loopback or any-address local address.
func loopbackListenerUIDs(table string, port int) []uint32 {
var out []uint32
for _, line := range strings.Split(table, "\n")[1:] {
f := strings.Fields(line)
if len(f) < 8 || f[3] != "0A" {
continue
}
host, p, ok := strings.Cut(f[1], ":")
if !ok {
continue
}
if n, err := strconv.ParseUint(p, 16, 16); err != nil || int(n) != port {
continue
}
switch host {
case "0100007F", "00000000", "00000000000000000000000001000000", "00000000000000000000000000000000":
default:
if !strings.HasSuffix(host, "7F") || len(host) != 8 { // 127.x.y.z, little-endian
continue
}
}
if u, err := strconv.ParseUint(f[7], 10, 32); err == nil {
out = append(out, uint32(u))
}
}
return out
}
+64
View File
@@ -0,0 +1,64 @@
//go:build linux
package main
import (
"net"
"os"
"strconv"
"testing"
)
// Found without PRIVATAE_LLM_ADDR, a model server another account runs on the
// port is not handed the Host's prompts; root's, this bastion's and its own
// listener are.
//
// MUTATION: accept any uid in listenersTrusted. RESULT: red.
func TestAModelServerOfAnotherAccountIsNotTaken(t *testing.T) {
table := " sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n" +
" 0: 0100007F:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000 1001 0 1 1 0 100 0 0 10 0\n" +
" 1: 0100007F:1F40 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 2 1 0 100 0 0 10 0\n" +
" 2: 0100007F:1F40 0100007F:9999 01 00000000:00000000 00:00000000 00000000 1002 0 3 1 0 100 0 0 10 0\n"
if got := loopbackListenerUIDs(table, 8080); len(got) != 1 || got[0] != 1001 {
t.Errorf("8080: %v", got)
}
if got := loopbackListenerUIDs(table, 8000); len(got) != 1 || got[0] != 0 {
t.Errorf("8000 (an established connection is not a listener): %v", got)
}
for _, c := range []struct {
uids []uint32
owner bool
want bool
}{
{[]uint32{0}, false, true}, {[]uint32{1000}, false, true}, {[]uint32{1001}, true, true},
{[]uint32{1001}, false, false}, {[]uint32{1002}, true, false}, {[]uint32{0, 1002}, true, false}, {nil, true, false},
} {
if got := listenersTrusted(c.uids, 1000, 1001, c.owner); got != c.want {
t.Errorf("listeners %v (owner recorded %v): trusted %v, want %v", c.uids, c.owner, got, c.want)
}
}
// A listener of this process's own is trusted, a port nobody listens on is not.
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer ln.Close()
if !llmListenerTrusted(ln.Addr().String()) {
t.Errorf("this process's own listener (uid %d) is not trusted", os.Getuid())
}
_, p, _ := net.SplitHostPort(ln.Addr().String())
n, _ := strconv.Atoi(p)
_ = ln.Close()
if llmListenerTrusted("127.0.0.1:" + strconv.Itoa(n)) {
t.Error("a port nobody listens on is trusted")
}
// Discovery skips an untrusted server for the next one.
bad, good := openAIServer(t, oneModel), openAIServer(t, oneModel)
withLLMCandidates(t, bad, good)
prev := llmTrusted
llmTrusted = func(a string) bool { return a != bad }
t.Cleanup(func() { llmTrusted = prev })
if got := llmTarget(); got != good {
t.Errorf("llmTarget = %q, want the trusted one", got)
}
}
+8
View File
@@ -0,0 +1,8 @@
//go:build !linux
package main
// llmListenerTrusted is Linux's check of who listens (llmlocal_linux.go). On
// Windows and macOS the found server is taken as it answers; who owns the
// port is not checked there yet.
func llmListenerTrusted(string) bool { return true }
+189
View File
@@ -0,0 +1,189 @@
package main
import (
"encoding/json"
"net"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
// withLLMCandidates points the discovery at addrs (in order) with the real
// probe, and starts from no answer.
func withLLMCandidates(t *testing.T, addrs ...string) {
t.Helper()
prevC, prevP := llmCandidates, llmProbe
llmCandidates = nil
for i, a := range addrs {
llmCandidates = append(llmCandidates, llmCandidate{Addr: a, Server: []string{"llama.cpp", "vLLM", "LM Studio"}[i%3]})
}
llmProbe = probeOpenAIShaped
resetLLMDiscovery()
t.Cleanup(func() { llmCandidates, llmProbe = prevC, prevP; resetLLMDiscovery() })
t.Setenv("PRIVATAE_LLM_ADDR", "")
}
func resetLLMDiscovery() {
llmLocal.Lock()
llmLocal.found, llmLocal.probed, llmLocal.said = llmFound{}, false, ""
llmLocal.Unlock()
}
// openAIServer answers /v1/models with body.
func openAIServer(t *testing.T, body string) string {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/models" {
http.NotFound(w, r)
return
}
_, _ = w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
return strings.TrimPrefix(srv.URL, "http://")
}
// deadAddr is a loopback port nothing listens on.
func deadAddr(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
a := ln.Addr().String()
_ = ln.Close()
return a
}
const oneModel = `{"object":"list","data":[{"id":"qwen3-4b-q4_k_m.gguf","object":"model"}]}`
// None answering: no lane, and status says where it looked.
//
// MUTATION: return the first candidate whatever it answers. RESULT: red.
func TestNoLocalModelServerMeansNoLane(t *testing.T) {
withLLMCandidates(t, deadAddr(t), deadAddr(t))
if got := llmTarget(); got != "" {
t.Fatalf("llmTarget = %q with nothing answering", got)
}
if ok, _ := openAICompatModels(); ok {
t.Error("the beat would advertise a model server that is not there")
}
if l := llmStatus(); l.Source != "none" || !strings.Contains(l.words(), "no OpenAI-compatible model server found") {
t.Errorf("status: %+v %q", l, l.words())
}
}
// One answering: the lane names it, the beat's probe and the forward agree,
// and status says which server it is.
//
// MUTATION: have llmTarget return only the variable. RESULT: red.
func TestALocalModelServerIsFound(t *testing.T) {
a := openAIServer(t, oneModel)
withLLMCandidates(t, deadAddr(t), a)
if got := llmTarget(); got != a {
t.Fatalf("llmTarget = %q, want %q", got, a)
}
ok, models := openAICompatModels()
if !ok || strings.Join(models, ",") != "qwen3-4b-q4_k_m.gguf" {
t.Errorf("the beat's probe: %v %v", ok, models)
}
if s, _ := serveServiceNamed("llm"); s.target() != a {
t.Error("the serve lane reads another address than the beat")
}
l := llmStatus()
if l.Source != "found" || l.Addr != a || !strings.Contains(l.words(), "found an OpenAI-compatible model server on "+a) {
t.Errorf("status: %+v %q", l, l.words())
}
raw, _ := json.Marshal(statusRead{LLM: l})
if !strings.Contains(string(raw), `"source":"found"`) {
t.Errorf("status --json: %s", raw)
}
}
// Several answering: the first in the order wins (llama.cpp, vLLM, LM Studio).
//
// MUTATION: keep looking after the first answer and take the last. RESULT: red.
func TestTheFirstLocalModelServerInOrderWins(t *testing.T) {
first, second := openAIServer(t, oneModel), openAIServer(t, `{"data":[{"id":"other"}]}`)
withLLMCandidates(t, first, second)
if got := llmTarget(); got != first {
t.Fatalf("llmTarget = %q, want the first, %q", got, first)
}
}
// The variable overrides what is found.
//
// MUTATION: look before reading the variable. RESULT: red.
func TestTheVariableOverridesTheLocalModelServer(t *testing.T) {
a := openAIServer(t, oneModel)
withLLMCandidates(t, a)
t.Setenv("PRIVATAE_LLM_ADDR", "127.0.0.1:18080")
if got := llmTarget(); got != "127.0.0.1:18080" {
t.Fatalf("llmTarget = %q", got)
}
if l := llmStatus(); l.Source != "PRIVATAE_LLM_ADDR" {
t.Errorf("status: %+v", l)
}
}
// Something else on the port (a web app, a server whose list is not OpenAI's)
// is not taken for a model server.
//
// MUTATION: accept any 200 in probeOpenAIShaped. RESULT: red.
func TestAnAnswerThatIsNotOpenAIShapedIsIgnored(t *testing.T) {
for _, body := range []string{`<html>hello</html>`, `{"models":[{"name":"x"}]}`, `{"data":[]}`, `{"data":[{"name":"x"}]}`, `{"data":"x"}`} {
bad := openAIServer(t, body)
good := openAIServer(t, oneModel)
withLLMCandidates(t, bad, good)
if got := llmTarget(); got != good {
t.Errorf("%s: llmTarget = %q, want the OpenAI-shaped one", body, got)
}
}
}
// Only loopback is ever asked.
//
// MUTATION: add a LAN address to llmCandidates. RESULT: red.
func TestOnlyLoopbackIsAskedForAModelServer(t *testing.T) {
want := []string{"127.0.0.1:8080", "127.0.0.1:8000", "127.0.0.1:1234"}
if len(llmCandidates) != len(want) {
t.Fatalf("%+v", llmCandidates)
}
for i, c := range llmCandidates {
if c.Addr != want[i] || !loopbackAddr(c.Addr) {
t.Errorf("candidate %d is %q", i, c.Addr)
}
}
}
// A found server that goes away is asked for again at once after a failure,
// not served for the rest of the interval; one that starts later is found at
// the next interval.
//
// MUTATION: drop llmTargetFailed from openAICompatModels. RESULT: red.
func TestAModelServerThatWentAwayIsAskedForAgain(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte(oneModel)) }))
a := strings.TrimPrefix(srv.URL, "http://")
withLLMCandidates(t, a)
if llmTarget() != a {
t.Fatal("not found")
}
srv.Close()
if ok, _ := openAICompatModels(); ok {
t.Fatal("a closed server answered")
}
if got := llmTarget(); got != "" {
t.Errorf("after a failed probe llmTarget still names %q", got)
}
// Later: the cached "none" is asked again once the interval is over.
b := openAIServer(t, oneModel)
llmCandidates = []llmCandidate{{Addr: b, Server: "llama.cpp"}}
llmLocal.Lock()
llmLocal.at = time.Now().Add(-llmReprobeEvery - time.Second)
llmLocal.Unlock()
if got := llmTarget(); got != b {
t.Errorf("a server that started later is not found: %q", got)
}
}
+16 -6
View File
@@ -423,17 +423,30 @@ func startLaneHost(t *testing.T, sock string, cert tls.Certificate, clientCA *te
// socket, or a Host's lane port (local_host_tcp_linux_test.go).
func startLaneHostOn(t *testing.T, raw net.Listener, cert tls.Certificate, clientCA *testCA) *laneHost {
t.Helper()
h, mux, tc := newLaneHost(cert, clientCA)
ln := tls.NewListener(countingListener{raw, &h.accepted}, tc)
srv := &http.Server{Handler: mux, ErrorLog: log.New(io.Discard, "", 0)}
go func() { _ = srv.Serve(ln) }()
t.Cleanup(func() { _ = srv.Close() })
return h
}
// newLaneHost is a lane Host's routes and its TLS server configuration,
// served on a listener (startLaneHostOn) or over connections it opens itself
// (hostdoor_linux_test.go). It speaks the lane protocol name a Host that
// connects to a server chooses (laneALPN); a bastion that dials offers none.
func newLaneHost(cert tls.Certificate, clientCA *testCA) (*laneHost, http.Handler, *tls.Config) {
h := &laneHost{jobs: make(chan []byte, 4), replies: make(chan string, 4)}
pool := x509.NewCertPool()
pool.AddCert(clientCA.cert)
ln := tls.NewListener(countingListener{raw, &h.accepted}, &tls.Config{Certificates: []tls.Certificate{cert},
tc := &tls.Config{Certificates: []tls.Certificate{cert}, NextProtos: []string{laneALPN},
ClientCAs: pool, ClientAuth: tls.RequireAndVerifyClientCert,
VerifyConnection: func(cs tls.ConnectionState) error {
h.mu.Lock()
h.sni = append(h.sni, cs.ServerName)
h.mu.Unlock()
return nil
}})
}}
up := websocket.Upgrader{}
mux := http.NewServeMux()
mux.HandleFunc("/api/network/bastions/heartbeat", func(w http.ResponseWriter, r *http.Request) {
@@ -496,10 +509,7 @@ func startLaneHostOn(t *testing.T, raw net.Listener, cert tls.Certificate, clien
for _, p := range []string{"/bastion-harness", "/bastion-mgmt", "/bastion-egress", "/bastion-serve"} {
mux.HandleFunc(p, ws)
}
srv := &http.Server{Handler: mux, ErrorLog: log.New(io.Discard, "", 0)}
go func() { _ = srv.Serve(ln) }()
t.Cleanup(func() { _ = srv.Close() })
return h
return h, mux, tc
}
func (h *laneHost) snapshot() (hellos, beats []string) {
+11
View File
@@ -253,6 +253,7 @@ func takeHostFlag(args []string) (string, []string) {
const pairUsage = "usage: privatae-bastion pair [--replace] [--redeem-url unix:///<socket> | --redeem-url https://<host>:<port>/<path> --host-ca <pem file>] <token> | -\n" +
" privatae-bastion pair <pairing code skp_…> | -\n" +
" privatae-bastion pair --open [--port 8812] (on a server: your Host connects to it with the code this shows)\n" +
" - reads the token or pairing code from standard input, so it stays out of the process list and shell history\n" +
" exit status: 0 paired, 3 already paired with this Host (nothing changed), anything else a failure"
@@ -315,6 +316,16 @@ func main() {
case "service":
err = cmdService(os.Args[2:])
case "pair":
// pair --open: a pairing window for a Host that connects to this
// server (pair_window.go).
if addr, open, oerr := parsePairOpenArgs(os.Args[2:]); open {
if oerr != nil {
fmt.Fprintln(os.Stderr, oerr)
os.Exit(2)
}
err = cmdPairOpen(addr, os.Stdout)
break
}
token, redeemURL, hostCA, replace, perr := parsePairArgs(os.Args[2:])
if perr == nil {
token, perr = pairToken(token, os.Stdin, stdinIsConsole(), os.Stderr)
+12
View File
@@ -59,6 +59,10 @@ func TestMain(m *testing.M) {
if len(os.Args) >= 3 && os.Args[1] == opActionChildSubcommand {
os.Exit(opActionChildMain(os.Args[2]))
}
// The suite never asks the build machine's own loopback for a model
// server (llmlocal.go): a llama.cpp left running on 8080 there would make
// every lane test depend on it. llmlocal_test.go sets its own.
llmProbe = func(string) ([]string, bool) { return nil, false }
if len(os.Args) >= 3 && os.Args[1] == opActionProbeSubcommand {
os.Exit(opActionProbeMain(os.Args[2]))
}
@@ -138,6 +142,14 @@ func TestMain(m *testing.M) {
// stops every op unit its names match: the suite's names are its own, so
// it never stops an op the machine's own bastion is running.
opIsolateOpsUnits()
// THE FIREWALL IS THE MACHINE'S. A door the suite binds asks for its own
// inbound rule (doorfw.go): never in the firewall of the machine running
// the suite. The firewall tests stand their own firewalls in.
doorFirewallFor = func(string) (doorFirewall, string) { return nil, "" }
doorFirewallByID = func(string) doorFirewall { return nil }
doorFWExec = func(name string, _ ...string) (string, error) {
return "", fmt.Errorf("the suite never runs the machine's firewall (%s)", name)
}
code := m.Run()
opIsolatedOpsUnitsDone()
+1 -5
View File
@@ -340,11 +340,7 @@ func refuseMgmt(ws *websocket.Conn, why string) {
// the identity the pairing_* ops are gated on.
func runMgmtWorkers(pctx context.Context, self string, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer, ctxRoot, dcnDir, backupRoot string, n int) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
dialer := laneWSDialer(tlsCfg, dialHost)
for i := 0; i < n; i++ {
go func() {
hold := laneHoldFor(self)
+1 -5
View File
@@ -502,11 +502,7 @@ const mirrorForwardMaxConns = 8
// inventing a framing layer for no gain.
func serveMirrorForward(ctx context.Context, gw, subnet, wsURL string,
tlsCfg *tls.Config, dialHost func(ctx context.Context) (net.Conn, error), bearer string) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
dialer := laneWSDialer(tlsCfg, dialHost)
var allowed *net.IPNet
if subnet != "" {
if _, n, err := net.ParseCIDR(subnet); err == nil {
+3 -2
View File
@@ -129,8 +129,9 @@ func cmdSetMode(mode string) error {
if mode == modeNetworked {
// Platform-correct, for the same reason the refusal text is: telling a
// Windows user to run `sudo` is how a real instruction becomes noise.
fmt.Printf(" Next: anchor this machine to your host — %s\n", anchorCommandHint())
fmt.Println(" Every host paired after the anchor waits, pending, until the anchor host approves it.")
fmt.Println(" Next: pair it with the Host that should be in charge of it. That Host takes charge as it is")
fmt.Println(" paired (a Host on this machine always does; so does one an organisation's policy names), and every")
fmt.Println(" other Host waits until it admits it. To put a Host already paired in charge: " + anchorCommandHint())
}
return nil
}
+101 -52
View File
@@ -195,16 +195,18 @@ func pairPrecheck(cfgs []*Config, handle string, replace, forceActive bool) (exi
if boxMode() == modeLocal && len(cfgs) > 0 {
return nil, "", &localSecondHostError{serving: cfgs[0].HostName}
}
return pairPrecheckState(forceActive)
return pairPrecheckState(forceActive, handle)
}
// pairPrecheckState is the state a new pairing gets once pairPrecheck's walls
// are passed: active for claim-anchor, pending behind an anchor, else "".
func pairPrecheckState(forceActive bool) (*Config, string, error) {
// are passed: active for claim-anchor, pending behind an anchor (or, on a
// machine an organisation's policy governs and no Host holds yet, behind the
// Host the policy names: anchor_local.go), else "".
func pairPrecheckState(forceActive bool, handle string) (*Config, string, error) {
if forceActive {
return nil, pairingActive, nil
}
if currentAnchor() != nil {
if currentAnchor() != nil || pendingUnderPolicy(handle) {
return nil, pairingPending, nil
}
return nil, "", nil
@@ -280,29 +282,29 @@ func cmdPair(token, redeemURL, hostCAFile string, replace bool) error {
if err != nil {
return err
}
// A networked machine is a server. Left unanchored it refuses every op to
// every host (capsAllowIn: unanchoredNetworkedRefusal) until someone runs
// claim-anchor, a step people missed. So the first pairing on a networked
// box with no anchor anchors it, as `claim-anchor` with no token would, and
// from then on every terminal and write waits for the person's approval and
// every later pairing for this host's. Only with machine authority, which
// claim-anchor needs anyway.
// A networked machine is a server, with one Host in charge of it: the
// anchor (anchor.go). Left unanchored it refuses every op to every host
// (capsAllowIn: unanchoredNetworkedRefusal) until someone runs
// claim-anchor, a step people missed. So with machine authority, which
// claim-anchor needs anyway, `pair` puts the right Host in charge by
// itself (anchor_local.go anchorAfterPair): the Host an organisation's
// policy names, else a Host on this machine, else the first Host paired.
// The pairing itself succeeded, so a failed anchor is reported, not
// returned: `pair` failing here would read as "could not pair" to a caller
// (npx cseries exits on it) when the pairing exists. claim-anchor, which
// such a caller runs next, reports the anchor's own failure.
anchored, err := anchorFirstNetworkedPairing(cfg, all)
took, err := anchorAfterPair(cfg, all)
if err != nil {
fmt.Fprintf(os.Stderr, "warning: %v\n", err)
}
if anchored {
if took.Taken {
if fresh, lerr := loadPairings(); lerr == nil {
if c := pairingFor(fresh, cfg.HostName); c != nil {
cfg, all = c, fresh
}
}
}
pairReport(os.Stdout, cfg, all, anchored)
pairReport(os.Stdout, cfg, all, took)
return nil
}
@@ -327,45 +329,77 @@ func pairOptsFor(token, redeemURL, hostCAFile string, replace bool) (string, pai
}
// pairReport is what `pair` prints once the pairing stands. Whatever the form
// (anchored just now, over TCP, over a unix socket, pending, or over the
// tunnel), it ends by saying to enrol a second factor when this pairing has
// none and no approval device is pinned: without one, a local machine installs
// a folder grant on the host's signed grant alone, and an anchored one has no
// way to approve anything the host asks for.
func pairReport(w io.Writer, cfg *Config, all []*Config, anchored bool) {
// (put in charge just now, over TCP, over a unix socket, waiting to be
// admitted, or over the tunnel), it ends by saying to enrol a second factor
// when this pairing has none and no approval device is pinned: without one, a
// local machine installs a folder grant on the host's signed grant alone, and
// an anchored one has no way to approve anything the host asks for.
//
// It never tells the person to start a bastion that is already running: a
// running bastion serves a new pairing by itself within supervisePeriod
// (run.go superviseLoop), and on a machine whose bastion is a system service
// the command that starts it is the service manager's (pairServeNote).
func pairReport(w io.Writer, cfg *Config, all []*Config, took anchorTake) {
enrol := ""
if !secondFactorAvailable(cfg.dcnDir()) {
enrol = stepupEnrollHint(cfg.HostName, len(all))
}
defer func() { pairEnrolNote(w, cfg.HostName, enrol) }()
if anchored {
fmt.Fprintf(w, "\nThis machine is networked and anchored to %s: every terminal and every write\n", cfg.HostName)
fmt.Fprintln(w, "waits for your approval, and any later host waits for this one to approve it.")
if took.Taken {
fmt.Fprintf(w, "\n%s is now the Host in charge of this machine: %s.\n", cfg.HostName, took.Why)
if took.Replaced != "" {
fmt.Fprintf(w, "%s was in charge before. It is told so, and waits for %s to admit it, as any other Host does.\n",
took.Replaced, cfg.HostName)
}
if others := without(took.Waiting, took.Replaced); len(others) > 0 {
fmt.Fprintf(w, "Waiting for %s to admit them: %s.\n", cfg.HostName, strings.Join(others, ", "))
}
fmt.Fprintln(w, "Every terminal and every write waits for your approval, and any other Host waits")
fmt.Fprintf(w, "until %s admits it: %s asks you to approve it.\n", cfg.HostName, cfg.HostName)
if enrol != "" {
fmt.Fprintf(w, "Enrol your second factor now, so you can approve here:\n %s\n\n", enrol)
enrol = ""
}
}
if cfg.tcpTransport() && cfg.active() {
if !cfg.active() {
who := admitter()
if who == "" {
fmt.Fprintf(w, "Paired with %s — PENDING. Nothing runs for it until it is admitted.\n", cfg.HostName)
return
}
why := "the Host in charge of this computer"
if p := currentAnchorPolicy(); p != nil && boxMode() == modeNetworked && policyNames(p, who) {
why += ", named by an organisation's policy (" + p.Source + ")"
}
fmt.Fprintf(w, "Paired with %s — WAITING for %s to admit it.\n\n", cfg.HostName, who)
fmt.Fprintf(w, "The code was used: %s now lists this computer, but nothing runs for it yet (no lanes).\n", cfg.HostName)
fmt.Fprintf(w, "%s is %s, and it admits every other Host:\n", who, why)
fmt.Fprintf(w, "%s asks you to approve %s. Once you do, %s is served at once, with no restart.\n", who, cfg.HostName, cfg.HostName)
fmt.Fprintf(w, "Until then `privatae-bastion status` shows it as waiting for %s.\n", who)
return
}
if cfg.hostDials() {
fmt.Fprintf(w, "Paired with %s. Your Host connects to this server; this server never connects to it.\n\n", cfg.HostName)
fmt.Fprintf(w, " lanes: over the connections your Host opens to port %s (mutual TLS, the Host's CA pinned, name %s)\n coder runs: none over TCP\n",
doorPort(cfg.doorAddr()), tcpServerName)
fmt.Fprintf(w, " firewall: this server needs TCP port %s reachable inbound; the bastion keeps its own rule for it in a firewall it manages "+
"here (status shows it), and nothing on your Host's network needs opening\n", doorPort(cfg.doorAddr()))
fmt.Fprintf(w, " local: http://%s\n\n%s\n", cfg.ListenAddr, pairServeNote())
return
}
if cfg.tcpTransport() {
fmt.Fprintf(w, "Paired with %s over TCP.\n\n", cfg.HostName)
fmt.Fprintf(w, " lanes: %s (mutual TLS, the Host's CA pinned, name %s)\n coder runs: none over TCP\n", cfg.LaneAddr, tcpServerName)
fmt.Fprintf(w, " local: http://%s\n\nStart (or restart) the bastion:\n privatae-bastion run\n", cfg.ListenAddr)
fmt.Fprintf(w, " local: http://%s\n\n%s\n", cfg.ListenAddr, pairServeNote())
return
}
if cfg.unixTransport() && cfg.active() {
if cfg.unixTransport() {
fmt.Fprintf(w, "Paired with %s, a Host on this machine.\n\n", cfg.HostName)
fmt.Fprintf(w, " lanes: %s (mutual TLS, the Host's CA pinned)\n coder runs: a socket per run in %s\n", cfg.LaneSocket, cfg.RunsDir)
fmt.Fprintf(w, " local: http://%s\n\nStart (or restart) the bastion:\n privatae-bastion run\n", cfg.ListenAddr)
fmt.Fprintf(w, " local: http://%s\n\n%s\n", cfg.ListenAddr, pairServeNote())
return
}
if !cfg.active() {
fmt.Fprintf(w, "Paired with %s — PENDING.\n\n", cfg.HostName)
fmt.Fprintln(w, "This machine is anchored, so a new host waits until the anchor host approves it.")
fmt.Fprintln(w, "Nothing is started for it yet: no tunnel, no lanes. It shows on the anchor host's")
fmt.Fprintln(w, "Network app as a pending pairing; once approved the daemon opens its lanes by itself.")
return
}
fmt.Fprintf(w, "Paired with %s.\n\nOne `privatae-bastion run` serves every pairing:\n", cfg.HostName)
fmt.Fprintf(w, "Paired with %s.\n\nOne bastion serves every pairing:\n", cfg.HostName)
for _, c := range all {
// The listen address is the CLI's loopback control port for that host
// (add-folder, git-credential). The local secret that guards it is read
@@ -373,10 +407,37 @@ func pairReport(w io.Writer, cfg *Config, all []*Config, anchored bool) {
// operator client to hand it to since the MCP relay was removed.
fmt.Fprintf(w, "\n %s\n local: http://%s\n", c.HostName, c.ListenAddr)
}
fmt.Fprintf(w, "\nStart (or restart) the tunnels:\n privatae-bastion run\n")
fmt.Fprintf(w, "\n%s\n", pairServeNote())
fmt.Fprintf(w, "\ngit to your forge goes through this bastion — one-time setup:\n privatae-bastion git-setup\n")
}
// pairDaemonRunning: a bastion is serving on this machine (its console
// answers, or the system service runs, waiting for its first pairing). A seam
// for the suite.
var pairDaemonRunning = func() bool { return uiDaemonListening() || systemServiceRunning() }
// pairServeNote is `pair`'s last line about serving the new pairing: a running
// bastion picks it up by itself; otherwise the command that starts this
// machine's bastion (the service manager's on a service machine).
func pairServeNote() string {
if pairDaemonRunning() {
return fmt.Sprintf("The bastion running on this machine serves it within %d seconds; nothing needs restarting.",
int(supervisePeriod.Seconds()))
}
return "Start the bastion so it serves it:\n " + runCommandHint()
}
// without is list less drop.
func without(list []string, drop string) []string {
var out []string
for _, s := range list {
if s != drop {
out = append(out, s)
}
}
return out
}
// pairEnrolNote says what a pairing with no second factor means on this kind
// of machine, and how to enrol one. hint "" says nothing.
func pairEnrolNote(w io.Writer, host, hint string) {
@@ -393,19 +454,6 @@ func pairEnrolNote(w io.Writer, host, hint string) {
"sessions are refused. Enrol it now, so each needs a code from this machine:\n %s\n", host, hint)
}
// anchorFirstNetworkedPairing anchors a networked machine that has no anchor
// to the pairing just made, when it is the machine's only one and the caller
// holds machine authority (see cmdPair). Reports whether it anchored.
func anchorFirstNetworkedPairing(cfg *Config, all []*Config) (bool, error) {
if boxMode() != modeNetworked || currentAnchor() != nil || len(all) != 1 || !machineAuthorityHeld() {
return false, nil
}
if err := cmdClaimAnchor("", false, "", cfg.HostName); err != nil {
return false, fmt.Errorf("paired with %s, but anchoring this networked machine to it failed: %w\n anchor it with: %s", cfg.HostName, err, anchorCommandHint())
}
return true, nil
}
// redeemURLBase is the platform's enrollment ingress. A variable so a test can
// serve a redeem response on loopback; nothing outside tests assigns it.
var redeemURLBase = "https://api.privatae.ai"
@@ -509,7 +557,7 @@ func pairWith(token string, opts pairOpts) (*Config, []*Config, error) {
if !yes {
return nil, nil, fmt.Errorf("nothing was paired: %s stays the only Host this computer serves", second.serving)
}
existing, state, err = pairPrecheckState(forceActive)
existing, state, err = pairPrecheckState(forceActive, handle)
}
// The CA this pair trusts, as a fingerprint: the pairing code's, or that
// of the --host-ca file. "" for a platform or unix-socket pairing, whose
@@ -738,8 +786,9 @@ func commitPairing(cfg, existing *Config, replace bool, state string) (*Config,
// host renamed when its address moved). unpair() is deliberately not
// reused: it refuses to remove the anchor's own pairing, which is exactly
// the case here — the anchor is re-paired like every other host. The anchor
// RECORD is untouched by design; anchor.json still names the old host and
// only `claim-anchor --rotate` may re-sign it.
// RECORD is untouched here; anchor.json still names the old host until
// `claim-anchor --repin` re-signs it, or, for a Host paired again through a
// window under its pinned CA, carryAnchorToNewName (pair_window.go) does.
if replace && existing != nil && existing.HostName != cfg.HostName {
if err := removeStalePairingFile(existing.HostName); err != nil {
return nil, nil, fmt.Errorf("paired as %s, but the superseded pairing file for %s "+
+11 -5
View File
@@ -43,7 +43,7 @@ func TestPairAnchorsANetworkedMachineToItsFirstHost(t *testing.T) {
box(t, true)
writePairingState(t, tMember, pairingPending)
cfg, all := only(t, tMember)
anchored, err := anchorFirstNetworkedPairing(cfg, all)
anchored, err := anchoredAfterPair(cfg, all)
if err != nil || !anchored {
t.Fatalf("the first pairing on a networked box must anchor it: anchored=%v err=%v", anchored, err)
}
@@ -64,7 +64,7 @@ func TestPairAnchorsANetworkedMachineToItsFirstHost(t *testing.T) {
box(t, false)
writePairing(t, tMember)
cfg, all := only(t, tMember)
if anchored, err := anchorFirstNetworkedPairing(cfg, all); anchored || err != nil {
if anchored, err := anchoredAfterPair(cfg, all); anchored || err != nil {
t.Fatalf("a local machine must not be anchored: %v %v", anchored, err)
}
if currentAnchor() != nil {
@@ -81,7 +81,7 @@ func TestPairAnchorsANetworkedMachineToItsFirstHost(t *testing.T) {
}
writePairingState(t, tMember, pairingPending)
cfg, all := only(t, tMember)
if anchored, err := anchorFirstNetworkedPairing(cfg, all); anchored || err != nil {
if anchored, err := anchoredAfterPair(cfg, all); anchored || err != nil {
t.Fatalf("an anchored machine must not be re-anchored: %v %v", anchored, err)
}
if a := currentAnchor(); a == nil || a.AnchorHost != tAnchor {
@@ -98,7 +98,7 @@ func TestPairAnchorsANetworkedMachineToItsFirstHost(t *testing.T) {
writePairing(t, tAnchor)
writePairingState(t, tMember, pairingPending)
cfg, all := only(t, tMember)
if anchored, err := anchorFirstNetworkedPairing(cfg, all); anchored || err != nil {
if anchored, err := anchoredAfterPair(cfg, all); anchored || err != nil {
t.Fatalf("with two pairings the person picks the anchor: %v %v", anchored, err)
}
})
@@ -109,8 +109,14 @@ func TestPairAnchorsANetworkedMachineToItsFirstHost(t *testing.T) {
writePairingState(t, tMember, pairingPending)
held = false
cfg, all := only(t, tMember)
if anchored, err := anchorFirstNetworkedPairing(cfg, all); anchored || err != nil {
if anchored, err := anchoredAfterPair(cfg, all); anchored || err != nil {
t.Fatalf("without machine authority nothing is anchored: %v %v", anchored, err)
}
})
}
// anchoredAfterPair is anchorAfterPair's yes or no.
func anchoredAfterPair(cfg *Config, all []*Config) (bool, error) {
t, err := anchorAfterPair(cfg, all)
return t.Taken, err
}
+1 -1
View File
@@ -19,7 +19,7 @@ import (
func pairReportOf(cfg *Config, all []*Config, anchored bool) string {
var b strings.Builder
pairReport(&b, cfg, all, anchored)
pairReport(&b, cfg, all, anchorTake{Taken: anchored, Why: "it is the first Host paired with this networked machine"})
return b.String()
}
+29 -14
View File
@@ -219,20 +219,8 @@ func tcpAnswerRefusal(rr tcpRedeemResp, redeem *url.URL, hostCA, opKey string, e
if !sameCAs(rr.caCert(), hostCA) {
return errors.New("the Host answered with a CA other than the --host-ca it was paired with — refusing")
}
pair, err := tls.X509KeyPair([]byte(rr.operatorCert()), []byte(opKey))
if err != nil {
return fmt.Errorf("the Host's operator_cert is not for this bastion's key: %v", err)
}
leaf, err := x509.ParseCertificate(pair.Certificate[0])
if err != nil {
return fmt.Errorf("the Host's operator_cert does not parse: %v", err)
}
pool := x509.NewCertPool()
for _, c := range pemCerts(hostCA) {
pool.AddCert(c)
}
if _, err := leaf.Verify(x509.VerifyOptions{Roots: pool, KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}}); err != nil {
return fmt.Errorf("the Host's operator_cert is not signed by its CA for client use: %v", err)
if err := operatorCertRefusal(rr.operatorCert(), hostCA, opKey); err != nil {
return err
}
// Again here: the pairing the answer's name lands on may not be the one
// the token's handle found before the redeem.
@@ -250,6 +238,27 @@ func tcpAnswerRefusal(rr tcpRedeemResp, redeem *url.URL, hostCA, opKey string, e
return nil
}
// operatorCertRefusal: the operator certificate a Host answered with is for
// this bastion's key, and its CA signed it for client use.
func operatorCertRefusal(certPEM, hostCA, opKey string) error {
pair, err := tls.X509KeyPair([]byte(certPEM), []byte(opKey))
if err != nil {
return fmt.Errorf("the Host's operator_cert is not for this bastion's key: %v", err)
}
leaf, err := x509.ParseCertificate(pair.Certificate[0])
if err != nil {
return fmt.Errorf("the Host's operator_cert does not parse: %v", err)
}
pool := x509.NewCertPool()
for _, c := range pemCerts(hostCA) {
pool.AddCert(c)
}
if _, err := leaf.Verify(x509.VerifyOptions{Roots: pool, KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}}); err != nil {
return fmt.Errorf("the Host's operator_cert is not signed by its CA for client use: %v", err)
}
return nil
}
// tcpPinRefusal is THE PIN, as for a unix pairing: a --replace keeps the CA
// the pairing was made with. pairWithTokenVia asks it before the redeem
// dials, because --host-ca is the only root that redeem trusts: under
@@ -328,6 +337,9 @@ func tcpLaneDialer(cfg *Config) (func(ctx context.Context) (net.Conn, error), *t
// localLaneDialer is the lane transport of a pairing with a Host that has no
// platform: the lane socket or the lane port.
func localLaneDialer(cfg *Config) (func(ctx context.Context) (net.Conn, error), *tls.Config, error) {
if cfg.hostDials() {
return hostDialsLaneDialer(cfg)
}
if cfg.tcpTransport() {
return tcpLaneDialer(cfg)
}
@@ -336,6 +348,9 @@ func localLaneDialer(cfg *Config) (func(ctx context.Context) (net.Conn, error),
// laneWhere is where a local pairing's lanes go, for the log and status.
func (c *Config) laneWhere() string {
if c.hostDials() {
return "connections your Host opens to this server's port " + doorPort(c.doorAddr())
}
if c.tcpTransport() {
return c.LaneAddr
}
File diff suppressed because it is too large Load Diff
+3
View File
@@ -146,6 +146,9 @@ func handlePairingGrantSync(ws *websocket.Conn, _ string, args []string) {
return
}
anchorMu.Unlock()
// A former anchor admitted again has heard it was replaced: its beats
// stop saying so (anchor_local.go anchorReplacedNote).
clearAnchorReplacedFor(c.HostName)
// Open the lanes now: the user approved on the anchor host and expects the
// member host to come up, not to wait for someone to restart the service.
started := false
+136
View File
@@ -0,0 +1,136 @@
package main
// pairing_retry.go — each pairing runs on its own.
//
// Until 0.63.0 one pairing's fatal error (a port it could not bind, a lane
// that would not come up, a bad certificate in its file) went to cmdRun and
// stopped the whole daemon, so every other Host on the machine went dark with
// it, and on Windows the service manager restarted all of them into the same
// failure every five seconds. Now that error stops that pairing alone. It is
// reported (the log, the machine's events, `status`), and the supervisor
// (run.go superviseOnce) starts it again by itself after a pause that doubles
// from supervisePeriod up to retryMaxPause, the other pairings serving all
// along. A run that stays up for retryResetAfter clears the record.
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"time"
)
const (
retryMaxPause = 5 * time.Minute
retryResetAfter = 2 * time.Minute
)
// pairingFailure is one pairing's last fatal error, as status reads it.
type pairingFailure struct {
Error string `json:"error"`
At string `json:"at"`
NextTry string `json:"next_try"`
Tries int `json:"tries"`
}
var pairingRetry = struct {
sync.Mutex
next map[string]time.Time
tries map[string]int
}{next: map[string]time.Time{}, tries: map[string]int{}}
func pairingFailuresPath() (string, error) {
d, err := configDir()
if err != nil {
return "", err
}
return filepath.Join(d, "pairing_failures.json"), nil
}
// loadPairingFailures is the record status reads; empty when there is none.
func loadPairingFailures() map[string]pairingFailure {
out := map[string]pairingFailure{}
p, err := pairingFailuresPath()
if err != nil {
return out
}
if raw, err := os.ReadFile(p); err == nil {
_ = json.Unmarshal(raw, &out)
}
return out
}
func savePairingFailuresLocked(m map[string]pairingFailure) {
p, err := pairingFailuresPath()
if err != nil {
return
}
if len(m) == 0 {
_ = os.Remove(p)
return
}
if b, err := json.MarshalIndent(m, "", " "); err == nil {
_ = atomicWrite(p, b)
}
}
// retryPause is the pause before try n+1 after n failures in a row.
func retryPause(n int) time.Duration {
d := supervisePeriod
for i := 1; i < n && d < retryMaxPause; i++ {
d *= 2
}
return min(d, retryMaxPause)
}
// pairingFailed records host's fatal error and when it is tried again, and
// says so. ranFor is how long the run had been up: one that served for
// retryResetAfter starts the count again.
func pairingFailed(host string, err error, ranFor time.Duration) {
pairingRetry.Lock()
defer pairingRetry.Unlock()
if ranFor >= retryResetAfter {
pairingRetry.tries[host] = 0
}
pairingRetry.tries[host]++
n := pairingRetry.tries[host]
now := time.Now()
next := now.Add(retryPause(n))
pairingRetry.next[host] = next
m := loadPairingFailures()
m[host] = pairingFailure{Error: err.Error(), At: now.UTC().Format(time.RFC3339),
NextTry: next.UTC().Format(time.RFC3339), Tries: n}
savePairingFailuresLocked(m)
why := strings.TrimRight(strings.TrimSpace(err.Error()), ".")
line := fmt.Sprintf("stopped: %s. The other Hosts go on being served; it is tried again by itself at %s.",
why, next.Local().Format("15:04:05"))
fmt.Printf("[%s] %s\n", host, line)
recordLocalEvent("pairing", host+" "+line)
}
// pairingRetryDue: host has no failure pending a pause, or its pause is over.
func pairingRetryDue(host string) bool {
pairingRetry.Lock()
defer pairingRetry.Unlock()
next, ok := pairingRetry.next[host]
return !ok || !time.Now().Before(next)
}
// pairingRecovered clears host's record: its run has stayed up, or it ended
// cleanly.
func pairingRecovered(host string) {
pairingRetry.Lock()
defer pairingRetry.Unlock()
if _, ok := pairingRetry.next[host]; !ok {
return
}
delete(pairingRetry.next, host)
delete(pairingRetry.tries, host)
m := loadPairingFailures()
if _, ok := m[host]; ok {
delete(m, host)
savePairingFailuresLocked(m)
}
}
+205
View File
@@ -0,0 +1,205 @@
//go:build !windows
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// anchoredWithPending is the S0 machine: anchored to a Host on this machine,
// with a remote Host that redeemed its code and waits.
func anchoredWithPending(t *testing.T) {
t.Helper()
localBox(t)
writeTCPPairing(t, tLocal, "127.0.0.1:8811", "")
if err := writeAnchor("the owner", tLocal); err != nil {
t.Fatal(err)
}
pd, _ := pairingsDir()
b, _ := json.Marshal(Config{HostName: tRemote, Transport: transportTCP, LaneAddr: "10.0.2.2:28811", WGPrivateKey: "k",
State: pairingPending, ListenAddr: "127.0.0.1:7879", Origin: pairingOriginCarried})
if err := os.WriteFile(filepath.Join(pd, tRemote+".json"), b, 0o600); err != nil {
t.Fatal(err)
}
}
// `pair` on an anchored machine says the code was used, that the Host waits
// for the anchor to admit it, and that the anchor asks the person.
//
// MUTATION: restore the 0.62 text ("a new host waits until the anchor host
// approves it"). RESULT: red.
func TestPairOnAnAnchoredMachineSaysWhoAdmitsTheHost(t *testing.T) {
anchoredWithPending(t)
cfg, all := pairingNamed(t, tRemote)
out := pairReportOf(cfg, all, false)
for _, want := range []string{
"Paired with " + tRemote + " — WAITING for " + tLocal + " to admit it.",
"The code was used",
tLocal + " asks you to approve " + tRemote,
"served at once, with no restart",
} {
if !strings.Contains(out, want) {
t.Errorf("pair does not say %q:\n%s", want, out)
}
}
if strings.Contains(out, "not answering") || strings.Contains(out, "privatae-bastion run") {
t.Errorf("pair names a dead end:\n%s", out)
}
}
// `pair` never tells the person to start a bastion that is running: the
// running one serves the new pairing by itself. With none running it names
// this machine's own start command.
//
// MUTATION: restore "Start (or restart) the bastion:\n privatae-bastion run".
// RESULT: red.
func TestPairNeverSaysRunWhenABastionServesTheMachine(t *testing.T) {
localBox(t)
writeTCPPairing(t, tLocal, "127.0.0.1:8811", "")
cfg, all := pairingNamed(t, tLocal)
prev := pairDaemonRunning
t.Cleanup(func() { pairDaemonRunning = prev })
pairDaemonRunning = func() bool { return true }
out := pairReportOf(cfg, all, false)
if strings.Contains(out, "privatae-bastion run") || !strings.Contains(out, "serves it within 15 seconds; nothing needs restarting") {
t.Errorf("with a bastion running:\n%s", out)
}
pairDaemonRunning = func() bool { return false }
out = pairReportOf(cfg, all, false)
if !strings.Contains(out, "Start the bastion so it serves it:\n "+runCommandHint()) {
t.Errorf("with none running:\n%s", out)
}
}
// `status` and `status --json` say a pending Host waits for the anchor to
// admit it, never that it is not answering, and the machine is RUNNING when
// every Host it may serve is served.
//
// MUTATION: count pending pairings as served in readStatus. RESULT: red
// (PARTLY RUNNING). Drop WaitingFor. RESULT: red.
func TestStatusSaysAPendingHostWaitsForTheAnchor(t *testing.T) {
anchoredWithPending(t)
s, err := readStatus(readOpts{})
if err != nil {
t.Fatal(err)
}
var pend *statusPairing
for i := range s.Pairings {
if s.Pairings[i].Host == tRemote {
pend = &s.Pairings[i]
}
}
if pend == nil || pend.WaitingFor != tLocal {
t.Fatalf("status --json does not say who admits it: %+v", s.Pairings)
}
raw, _ := json.Marshal(s)
if !strings.Contains(string(raw), `"waiting_for":"`+tLocal+`"`) {
t.Errorf("the JSON lacks waiting_for: %s", raw)
}
// As the daemon reads it, with the anchor's own lane answering.
s.Running = "yes"
for i := range s.Pairings {
if s.Pairings[i].Host == tLocal {
s.Pairings[i].Answering = true
}
}
var b strings.Builder
printStatus(&b, s)
out := b.String()
if !strings.HasPrefix(out, "RUNNING — serving 1 host(s)") {
t.Errorf("headline:\n%s", out)
}
if !strings.Contains(out, "Waiting for "+tLocal+" to admit it: "+tRemote) ||
!strings.Contains(out, "(waiting for "+tLocal+" to admit it)") {
t.Errorf("status does not say who admits it:\n%s", out)
}
if strings.Contains(out, "not answering") {
t.Errorf("a pending Host reads as not answering:\n%s", out)
}
}
// The console says the same, and the page shows it in place of "not answering".
//
// MUTATION: drop the WaitingFor branch in status.html. RESULT: red.
func TestTheConsoleSaysAPendingHostWaitsForTheAnchor(t *testing.T) {
anchoredWithPending(t)
s := collectUIStatus()
if strings.Contains(s.StateNote, "not answering") || !strings.Contains(s.StateNote, "wait(s) for "+tLocal+" to admit them") {
t.Errorf("state %q: %q", s.State, s.StateNote)
}
found := false
for _, p := range s.Pairings {
found = found || (p.Host == tRemote && p.WaitingFor == tLocal)
}
if !found {
t.Errorf("pairings %+v", s.Pairings)
}
page, err := os.ReadFile(filepath.Join("uiassets", "status.html"))
if err != nil {
t.Fatal(err)
}
i, j := strings.Index(string(page), "{{if .WaitingFor}}<span class=\"tag warn\">not served until"), strings.Index(string(page), "not answering</span>")
if i < 0 || j < 0 || i > j {
t.Error("the page shows a pending Host as not answering")
}
if line := trayStateLine(s); !strings.Contains(line, "waiting for "+tLocal) {
t.Errorf("tray: %q", line)
}
}
// The anchor's heartbeat carries each pending pairing with what the anchor
// Host needs for the approval prompt: who admits it, how it would run, where
// its Host is, and how it came.
//
// MUTATION: drop the pending fields in pairingsReport. RESULT: red.
func TestTheAnchorsBeatCarriesWhatItNeedsToAskThePerson(t *testing.T) {
anchoredWithPending(t)
var e *pairingEntry
for _, p := range pairingsReport() {
if p.HostName == tRemote {
p := p
e = &p
}
}
if e == nil || e.State != pairingPending || e.WaitingFor != tLocal || e.Transport != "tcp" || e.SameMachine ||
e.Origin != pairingOriginCarried || e.Handle != handleOf(tRemote) {
t.Fatalf("%+v", e)
}
for _, p := range pairingsReport() {
if p.HostName == tLocal && (p.WaitingFor != "" || p.Origin != "") {
t.Errorf("an active pairing carries the pending fields: %+v", p)
}
}
// The unapproved list keeps its four fields.
raw, _ := json.Marshal(unapprovedAttempts(time.Now().Add(time.Hour)))
if strings.Contains(string(raw), "waiting_for") {
t.Errorf("unapproved attempts grew fields: %s", raw)
}
}
// The installer's last words come from what the service holds: what it serves
// and what waits, before the "no Host is needed" line it said whatever the
// service served.
//
// MUTATION: drop the status --json read. RESULT: red.
func TestTheInstallerSaysWhatTheServiceServesAndWhatWaits(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "deploy", "install.ps1"))
if err != nil {
t.Fatal(err)
}
s := string(raw)
read := strings.Index(s, "$st = (& $exe status --json 2>$null | Out-String) | ConvertFrom-Json")
waits := strings.Index(s, "waits for $($p.waiting_for) to admit it")
none := strings.Index(s, "no Host is needed.")
if read < 0 || waits < 0 || none < 0 || !(read < waits && waits < none) {
t.Error("the installer's end text does not come from what the service serves")
}
if !strings.Contains(s, "if ($served -or $waits) {") {
t.Error("the 'no Host is needed' line is not kept for a service that serves none")
}
}
+4 -10
View File
@@ -106,8 +106,9 @@ func runCommandHintFor(goos string) string {
return "systemctl start privatae-bastion"
}
// restartHint is how to bring the daemon back on THIS machine after a change
// that only takes at startup.
// restartHintFor is how to bring the daemon back after a change that only
// takes at startup, on goos (claim-anchor and pair no longer need it: a
// running bastion follows the anchor and the pairing files by itself).
//
// It said `systemctl restart privatae-bastion` unconditionally — the third
// Linux instruction printed at a Windows user inside one command's output,
@@ -116,7 +117,7 @@ func runCommandHintFor(goos string) string {
// service: deploy/install.ps1 installs the exe and registers a logon task. So
// the honest instruction there is to start it again. A networked install on
// macOS or Windows runs as a system service (sysservice.go), and for that
// machine restartHint names the service manager's own restart instead.
// machine serviceRestartHintFor names the service manager's own restart instead.
//
// TAKES THE PLATFORM AS AN ARGUMENT, and that is the point. The first version
// branched on runtime.GOOS directly, so the only machine that could test the
@@ -124,13 +125,6 @@ func runCommandHintFor(goos string) string {
// passed clean on the Linux builder. That is the same hole that let 'darwin'
// match 'win' in the dashboard: a platform branch nothing off-platform can
// reach is a platform branch nothing checks.
func restartHint() string {
if systemService.installed && (runtime.GOOS == "darwin" || runtime.GOOS == "windows") {
return serviceRestartHintFor(runtime.GOOS)
}
return restartHintFor(runtime.GOOS)
}
func restartHintFor(goos string) string {
if goos == "windows" {
return "Start the bastion again so the change takes: privatae-bastion run\n" +
+122
View File
@@ -0,0 +1,122 @@
package main
import (
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"testing"
)
// The member repository publishes this tree to strangers byte for byte, and
// its release step refuses outright a tree that carries any of the classes
// below. Each is checked here as well, so a change that brings one in fails
// before it merges, not after a release has been built and signed from it.
//
// A class is described, never named: a comment says what happened and why, not
// on which of our machines, by whom, under which ticket, or on which day it was
// decided. Test data is invented (example.com, .example, the 9000 range).
//
// Not checked here: names held outside this repository (they cannot be listed
// in a published file; the release step and the personal-data check read them
// from the machine they are kept on), and the two classes the release step
// compares against the previous release instead of refusing outright: paths
// under root's home directory, which the bastion's own state and its tests of
// credential-store refusals legitimately use, and the bastion's own tunnel
// addresses in the private range.
//
// Every pattern is written so that this file does not match it: a bracketed
// letter keeps the text from being the word it finds.
var publishedInternalClasses = []struct {
name string
re *regexp.Regexp
}{
{"a machine of the platform's own fleet",
regexp.MustCompile(`mgmt-[12]|platform-meta[l]|entity-poo[l]|edge-vi[n]|edge-hi[l]|dmz-[a-z]+-[0-9]`)},
{"a credential store of the platform",
regexp.MustCompile(`gitea-[s]kema-credentials|/roo[t]/\.config/git|INTERNAL_API_SECRE[T]`)},
{"a ticket id", regexp.MustCompile(`T-[0-9]{4}\b|R-[0-9]{3}\b`)},
{"the name of a person or a review agent who worked on the code",
regexp.MustCompile(`\b(Fabl[e]|Maddi[e]|Opu[s])\b|[Cc]ase[y]`)},
{"a dated decision", regexp.MustCompile(`(Decided|Reported|Measured|measured) 20[0-9]{2}-[0-9]{2}`)},
{"a design document reference", regexp.MustCompile(`DESIGN-[a-z]|plan [§]`)},
{"the old module path", regexp.MustCompile(`github\.com/privata[e]`)},
{"the platform's install directory", regexp.MustCompile(`/opt/privata[e]`)},
{"an address in the platform's fleet mesh", regexp.MustCompile(`fd0[0]:aa:`)},
}
// publishedInternalHits returns "class: line N" for every line of src that a
// class matches.
func publishedInternalHits(src string) []string {
var hits []string
for i, line := range strings.Split(src, "\n") {
for _, c := range publishedInternalClasses {
if c.re.MatchString(line) {
hits = append(hits, strconv.Itoa(i+1)+": "+c.name)
}
}
}
return hits
}
func TestPublishedSourceNamesNothingInternal(t *testing.T) {
root, err := filepath.Abs("..")
if err != nil {
t.Fatal(err)
}
files := publishedFiles(t, root)
if len(files) < 50 {
t.Fatalf("found only %d files under %s; the scan is not looking at the repository", len(files), root)
}
var hits []string
for _, rel := range files {
for _, h := range publishedInternalHits(rel) {
hits = append(hits, rel+" (in the file name) "+h[strings.Index(h, " ")+1:])
}
data, err := os.ReadFile(filepath.Join(root, rel))
if err != nil {
continue // a tracked file deleted in the working tree
}
for _, h := range publishedInternalHits(string(data)) {
hits = append(hits, rel+":"+h)
}
}
// The lines themselves are not printed: a hit of the first two classes can
// be the very value that must not be published, and a CI log is published too.
if len(hits) > 0 {
t.Errorf("%d published lines name something internal; describe it instead, with invented values:\n %s",
len(hits), strings.Join(hits, "\n "))
}
}
// The check itself: each class fires on a planted line and stays quiet on its
// look-alikes. Planted lines carry a "|" that is removed before they are read.
func TestPublishedInternalCheckCatchesEachClass(t *testing.T) {
live := func(s string) string { return strings.ReplaceAll(s, "|", "") }
for _, line := range []string{
"seen on mgmt|-2 at noon", "ssh platform-met|al", "the entity-po|ol nodes", "via edge-v|in",
"edge-h|il took over", "dmz-ro|uter-1 is up",
"read /ro|ot/.config/g|it/creds", "gitea-sk|ema-credentials", "INTERNAL_API_SEC|RET=x",
"fixed in T-|1234", "see R-|101.",
"Fab|le found it", "per Mad|die", "Op|us said so", "Ca|sey asked", `owner "ca|sey"`,
"Decid|ed 2026-10-01: keep it", "measu|red 2026-11-03 on a box",
"DESIGN|-bastion.md", "plan| § 4",
"github.com/priv|atae/bastion", "/opt/priv|atae/bin", "addr fd0|0:aa::1",
} {
if len(publishedInternalHits(live(line))) == 0 {
t.Errorf("%q: not caught", live(line))
}
}
for _, line := range []string{
"mgmt-3 is not ours", "platform metal", "edge-v",
"dmz-router", "~/.config/git", "T-123 and T-12345x", "R-1000",
"Fables and Opuses", "Maddies", "a case, yes", "decided in 2026",
"design doc", "github.com/example/thing", "/opt/example/privatae-bastion",
"fd12:3456::a:1", "fdaa::1",
} {
if hits := publishedInternalHits(live(line)); len(hits) != 0 {
t.Errorf("%q: false hit %v", line, hits)
}
}
}
+19 -10
View File
@@ -7,6 +7,7 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
@@ -19,16 +20,10 @@ func laneSock(t *testing.T) {
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = ln.Close() })
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go askSockServe(c)
}
}()
// The globals are set before the server goroutine starts and restored only
// after it and every connection it served have returned: a client's
// connect is no happens-before edge the race detector sees, so a server
// that read them unordered with these writes is a data race under -race.
prevPath, prevLane, prevID := readSockPath, readsGoThroughLane, askPeerIdentityFn
t.Cleanup(func() { readSockPath, readsGoThroughLane, askPeerIdentityFn = prevPath, prevLane, prevID })
readSockPath = func() string { return sock }
@@ -42,6 +37,20 @@ func laneSock(t *testing.T) {
}
return uid, who, err
}
var served sync.WaitGroup
served.Add(1)
t.Cleanup(func() { _ = ln.Close(); served.Wait() })
go func() {
defer served.Done()
for {
c, err := ln.Accept()
if err != nil {
return
}
served.Add(1)
go func() { defer served.Done(); askSockServe(c) }()
}
}()
}
const laneTestUID = 1000
+247 -43
View File
@@ -153,6 +153,10 @@ func cmdRun() error {
go superviseLoop(ctx)
// The machine owner's approved stops (stop_local.go), Host or no Host.
go localStopLoop(ctx)
// The door's own firewall rules (doorfw.go): one left by a `pair --open`
// that was killed, or a pairing that went while nothing ran, is removed
// here once nothing uses its port.
go doorFirewallSweep(ctx)
// The local UI (ui.go): ONE per machine, not one per pairing, because it
// shows the whole box. Started last so it renders a daemon that is already
// up, and non-fatal by construction — a machine that cannot bind the UI port
@@ -168,13 +172,15 @@ func cmdRun() error {
select {
case <-sig:
fmt.Println("\nshutting down")
tellReplacedAnchorsBeforeStop()
cancel() // closes every pairing's server + WG device via runPairing's exits
return nil
case <-shutdownCh:
// "Stop the bastion" in the tray menu. Deliberately the SAME path as a
// signal rather than an os.Exit: a machine that stops serving must tear
// its tunnels down, not vanish and leave the host waiting for a beat.
fmt.Println("\nstopping — asked from the tray")
fmt.Println("\nstopping — " + shutdownWhy)
tellReplacedAnchorsBeforeStop()
cancel()
return nil
case err := <-errc:
@@ -245,10 +251,13 @@ func startPairing(cfg *Config) bool {
pctx, pcancel := context.WithCancel(supervisor.ctx)
run := &pairingRun{cancel: pcancel, done: make(chan struct{}), cfg: cfg}
supervisor.runs[cfg.HostName] = run
errc := supervisor.errc
go func() {
defer close(run.done)
started := time.Now()
// A run that stays up has recovered from any failure before it.
up := time.AfterFunc(retryResetAfter, func() { pairingRecovered(cfg.HostName) })
err := runPairing(pctx, cfg)
up.Stop()
// Retired BEFORE it leaves the registry: leaving is what lets the
// pairing start again, and this run's heartbeat may still be running.
retireRun(cfg)
@@ -257,12 +266,15 @@ func startPairing(cfg *Config) bool {
delete(supervisor.runs, cfg.HostName)
}
supervisor.Unlock()
// nil is a clean exit (ctx cancelled: shutdown or a revoke) and must not
// take the process down. Anything else is fatal for the whole daemon, as
// before: the supervisor restarts all lanes together, never half-homed.
if err != nil && errc != nil {
errc <- fmt.Errorf("[%s] %w", cfg.HostName, err)
// nil is a clean exit (ctx cancelled: shutdown or a revoke). Anything
// else stops THIS pairing only: it is reported, and superviseOnce
// starts it again after a pause (pairing_retry.go). It used to stop the
// whole daemon, and every other Host with it.
if err != nil && pctx.Err() == nil {
pairingFailed(cfg.HostName, err, time.Since(started))
return
}
pairingRecovered(cfg.HostName)
}()
return true
}
@@ -311,6 +323,9 @@ func superviseOnce() {
c, ok := onDisk[h]
switch {
case !ok || !c.active():
if ok {
tellReplacedAnchor(h)
}
fmt.Printf("[%s] no longer active on disk — stopping its tunnel and lanes\n", h)
stopPairing(h)
case r.cfg != nil && c.WGPrivateKey != r.cfg.WGPrivateKey:
@@ -324,12 +339,108 @@ func superviseOnce() {
}
}
for _, c := range cfgs {
if c.active() {
if c.active() && pairingRetryDue(c.HostName) {
startPairing(c) // no-op when already running
}
}
}
// tellReplacedAnchor: the anchor a Host the rules put first replaced
// (anchor_local.go) is told so on one last beat, its enforcing report saying
// who is in charge now and that it waits to be admitted, before its lanes stop.
func tellReplacedAnchor(h string) {
if anchorReplacedNote(h) != nil && lastBeat(h) {
fmt.Printf("[%s] no longer in charge of this machine — told it on a last heartbeat\n", h)
}
}
// tellReplacedAnchorsBeforeStop is tellReplacedAnchor for every running
// pairing the files on disk no longer serve, when the daemon stops: an
// installer that pairs a Host on this machine and restarts the service at
// once stops the replaced anchor before superviseOnce would have seen it.
func tellReplacedAnchorsBeforeStop() {
cfgs, err := loadPairings()
if err != nil {
return
}
supervisor.Lock()
var running []string
for h := range supervisor.runs {
running = append(running, h)
}
supervisor.Unlock()
for _, h := range running {
if c := pairingFor(cfgs, h); c != nil && c.HostName == h && !c.active() {
tellReplacedAnchor(h)
}
}
}
// runBeats are the running pairings' beats, for one last beat before a
// pairing stops (superviseOnce).
var runBeats = struct {
sync.Mutex
m map[string]*runBeat
}{m: map[string]*runBeat{}}
// runBeat is one run's beat, serialised. lastBeat beats from superviseOnce's
// goroutine while the run's own loop (runBeatLoop) may be beating or failing
// over, and neither is safe to run beside the other: maybeRotateEndpoint and
// maybeFailover both write cfg.Endpoints and the WG device. Before lastBeat
// every beat of a run came from its one loop, so nothing locked them.
type runBeat struct {
mu sync.Mutex
beat func()
}
// do runs f with no beat of this run beside it.
func (r *runBeat) do(f func()) {
r.mu.Lock()
defer r.mu.Unlock()
f()
}
// run is one beat.
func (r *runBeat) run() { r.do(r.beat) }
// registerRunBeat records host's beat while its run lasts. The run beats
// through the returned runBeat from then on; the func takes it out.
func registerRunBeat(host string, beat func()) (*runBeat, func()) {
mine := &runBeat{beat: beat}
runBeats.Lock()
runBeats.m[host] = mine
runBeats.Unlock()
return mine, func() {
runBeats.Lock()
if runBeats.m[host] == mine { // a later run of host keeps its own
delete(runBeats.m, host)
}
runBeats.Unlock()
}
}
// lastBeatWait bounds the last beat: a Host that does not answer does not hold
// the stop.
var lastBeatWait = 10 * time.Second
// lastBeat beats host once more, now, and waits for it (bounded). False when
// host has no running beat.
func lastBeat(host string) bool {
runBeats.Lock()
b := runBeats.m[host]
runBeats.Unlock()
if b == nil {
return false
}
done := make(chan struct{})
go func() { defer close(done); b.run() }()
select {
case <-done:
case <-time.After(lastBeatWait):
}
return true
}
// stopPairing cancels one running pairing and waits (bounded) for its tunnel,
// listener and lanes to exit. Returns whether anything was running.
func stopPairing(host string) bool {
@@ -687,13 +798,15 @@ func runPairing(ctx context.Context, cfg *Config) error {
// box costs 2.9 s because it hashes every executable once, and no beat
// may ever wait on that.
go runDeltaRefresh(ctx)
beat() // first beat: flip online + bind fp now, don't wait a full interval
rb, unregisterBeat := registerRunBeat(cfg.HostName, beat)
defer unregisterBeat()
rb.run() // first beat: flip online + bind fp now, don't wait a full interval
// Every 60 s, and at once after a pairing on this box changes state
// (beatnow.go), so this Host learns of it without waiting for the tick.
// Path failover BEFORE a ticked beat: if the current relay entry has gone
// dead (no handshake for RejectAfterTime) the beat could not reach the
// host anyway; move to the next candidate first, no platform round trip.
runBeatLoop(ctx, 60*time.Second, func() { maybeFailover(cfg, dev, startedAt) }, beat)
runBeatLoop(ctx, 60*time.Second, func() { rb.do(func() { maybeFailover(cfg, dev, startedAt) }) }, rb.run)
}()
srv := &http.Server{Addr: cfg.ListenAddr, Handler: mux}
@@ -769,11 +882,17 @@ func runUnixPairing(ctx context.Context, cfg *Config) error {
if err != nil {
return err
}
// A Host that connects to this server: its door is open while it is
// served (hostdoor.go), and closed after the last such pairing stops.
if cfg.hostDials() {
closeDoor, err := doorOpen(cfg)
if err != nil {
return err
}
defer closeDoor()
}
lane := cfg.laneWhere()
upstream := &http.Client{Transport: &http.Transport{
DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: laneTLS.Clone(),
}}
upstream := &http.Client{Transport: laneHTTPTransport(dialHost, laneTLS)}
// Coder runs: the box-wide loop, then a unix pairing's run sockets, which
// are registered before the harness workers come up so a run-up that
@@ -908,10 +1027,12 @@ func runUnixPairing(ctx context.Context, cfg *Config) error {
go runAlertsRefresh(ctx)
go runOtherBastionRefresh(ctx)
go runDeltaRefresh(ctx)
beat()
rb, unregisterBeat := registerRunBeat(cfg.HostName, beat)
defer unregisterBeat()
rb.run()
// Every 60 s, and at once after a pairing on this box changes state
// (beatnow.go).
runBeatLoop(ctx, 60*time.Second, nil, beat)
runBeatLoop(ctx, 60*time.Second, nil, rb.run)
}()
srv := &http.Server{Addr: cfg.ListenAddr, Handler: mux}
@@ -1105,6 +1226,12 @@ type statusRead struct {
Pairings []statusPairing `json:"pairings"`
Mode string `json:"mode,omitempty"`
Anchor *statusAnchor `json:"anchor,omitempty"`
// LLM: the OpenAI-compatible model server the llm lane serves
// (llmlocal.go): set, found on this machine, or none.
LLM *statusLLM `json:"llm,omitempty"`
// Firewall: the door's way in through this machine's firewall
// (doorfw.go): the bastion's own rules, and what it could not do.
Firewall *statusFirewall `json:"firewall,omitempty"`
}
type statusPairing struct {
@@ -1121,6 +1248,14 @@ type statusPairing struct {
Endpoint string `json:"endpoint,omitempty"`
Local string `json:"local"`
Answering bool `json:"answering"`
// WaitingFor: a pending pairing waits for this Host, the one in charge of
// the machine, to admit it (anchor_local.go admitter). Nothing runs for
// it until then, so it is never "not answering".
WaitingFor string `json:"waiting_for,omitempty"`
// Failing: the run last stopped on this error and is tried again by
// itself (pairing_retry.go); the other pairings are served meanwhile.
// Cleared once a run has stayed up for retryResetAfter.
Failing *pairingFailure `json:"failing,omitempty"`
}
type statusAnchor struct {
@@ -1134,27 +1269,45 @@ func readStatus(readOpts) (statusRead, error) {
if err != nil {
return statusRead{}, fmt.Errorf("this machine's pairings cannot be read: %w", err)
}
s := statusRead{Version: clientVersion, Running: "no", Pairings: []statusPairing{}}
s := statusRead{Version: clientVersion, Running: "no", Pairings: []statusPairing{}, Firewall: readDoorFirewallStatus()}
if len(cfgs) == 0 {
// With no Host the only thing listening is the console, so that is
// what says whether the daemon is up.
if uiDaemonListening() {
s.Running = "yes"
}
s.LLM = llmStatus()
return s, nil
}
running := 0
running, served := 0, 0
who := admitter()
failures := loadPairingFailures()
for _, cfg := range cfgs {
p := statusPairing{Host: cfg.HostName, State: cfg.stateName(), Local: cfg.ListenAddr,
Answering: bastionListening(cfg.ListenAddr)}
if p.Answering {
running++
if cfg.invalid == "" && cfg.State == pairingPending && who != "" {
p.WaitingFor = who
}
// Until a run has stayed up for retryResetAfter, its last failure is
// shown, whether or not the port answers now.
if f, ok := failures[cfg.HostName]; ok && cfg.active() {
p.Failing = &f
}
// Only a pairing that is served counts: one waiting to be admitted
// has nothing to answer with.
if cfg.active() {
served++
if p.Answering {
running++
}
}
switch {
case cfg.invalid != "":
p.Problem = cfg.invalid
case cfg.unixTransport():
p.Transport, p.Lanes, p.Runs = "local", cfg.LaneSocket, cfg.RunsDir
case cfg.hostDials():
p.Transport, p.Lanes = "tcp", cfg.laneWhere()
case cfg.tcpTransport():
p.Transport, p.Lanes = "tcp", cfg.LaneAddr
default:
@@ -1163,12 +1316,18 @@ func readStatus(readOpts) (statusRead, error) {
s.Pairings = append(s.Pairings, p)
}
switch {
case running == len(cfgs):
case served == 0:
// Nothing to serve yet: the console says whether the daemon is up.
if uiDaemonListening() {
s.Running = "yes"
}
case running == served:
s.Running = "yes"
case running > 0:
s.Running = "partly"
}
s.Mode = boxMode()
s.LLM = llmStatus()
if a := currentAnchor(); a != nil {
s.Anchor = &statusAnchor{Host: a.AnchorHost, Owner: a.Owner, Since: a.ClaimedAt}
}
@@ -1214,6 +1373,10 @@ func printStatus(w io.Writer, s statusRead) {
} else {
fmt.Fprintf(w, "NOT RUNNING, and no Host is connected. Start it with `%s`.\n", runCommandHint())
}
if s.LLM != nil {
fmt.Fprintf(w, "model server: %s\n", s.LLM.words())
}
printDoorFirewallStatus(w, s.Firewall)
return
}
// IS IT RUNNING, AND WHERE — the first thing status should answer and the
@@ -1225,22 +1388,41 @@ func printStatus(w io.Writer, s statusRead) {
// a self-update the bastion has NO console at all — it is serving, and there
// is nothing on screen that says so. status printed pairings and mode and
// never once said whether a daemon was alive.
served := 0
served, servable := 0, 0
waiting := map[string][]string{}
var admitters []string
for _, p := range s.Pairings {
if p.Answering {
served++
if p.WaitingFor != "" {
if len(waiting[p.WaitingFor]) == 0 {
admitters = append(admitters, p.WaitingFor)
}
waiting[p.WaitingFor] = append(waiting[p.WaitingFor], p.Host)
continue
}
if p.State == pairingActive {
servable++
if p.Answering {
served++
}
}
}
switch s.Running {
case "yes":
fmt.Fprintf(w, "RUNNING — serving %d host(s) on this machine.\n", len(s.Pairings))
case "partly":
fmt.Fprintf(w, "PARTLY RUNNING — %d of %d hosts are being served; the rest are not answering.\n",
served, len(s.Pairings))
default:
switch {
case s.Running == "no":
fmt.Fprintf(w, "NOT RUNNING — nothing is serving these pairings. Start it with `%s`.\n", runCommandHint())
case servable == 0:
fmt.Fprintln(w, "RUNNING — no Host is served yet.")
case s.Running == "partly":
fmt.Fprintf(w, "PARTLY RUNNING — %d of %d hosts are being served; the rest are not answering.\n",
served, servable)
default:
fmt.Fprintf(w, "RUNNING — serving %d host(s) on this machine.\n", servable)
}
if served > 0 && runtime.GOOS == "windows" {
for _, who := range admitters {
hosts := waiting[who]
fmt.Fprintf(w, " Waiting for %s to admit it: %s. %s asks you to approve it; it is served from then on.\n",
who, strings.Join(hosts, ", "), who)
}
if served > 0 && runtime.GOOS == "windows" && !systemService.installed {
fmt.Fprintln(w, " (No window? After a self-update it restarts itself detached, so it")
fmt.Fprintln(w, " has no console. Task Manager → privatae-bastion.exe is where it lives.)")
}
@@ -1251,7 +1433,10 @@ func printStatus(w io.Writer, s statusRead) {
continue
}
live := "not answering"
if p.Answering {
switch {
case p.WaitingFor != "":
live = "waiting for " + p.WaitingFor + " to admit it"
case p.Answering:
live = "answering"
}
switch p.Transport {
@@ -1265,11 +1450,19 @@ func printStatus(w io.Writer, s statusRead) {
fmt.Fprintf(w, "paired with: %s [%s]\n tunnel: %s -> %s via %s\n local: http://%s (%s)\n",
p.Host, p.State, p.BastionIP, p.HostIP, p.Endpoint, p.Local, live)
}
if f := p.Failing; f != nil {
fmt.Fprintf(w, " last stopped: %s (at %s, try %d); tried again by itself from %s, the other Hosts served meanwhile\n",
strings.TrimRight(f.Error, "."), uiWhen(f.At), f.Tries, uiWhen(f.NextTry))
}
}
fmt.Fprintf(w, "mode: %s\n", s.Mode)
if a := s.Anchor; a != nil {
fmt.Fprintf(w, "anchor: %s (owner %q, since %s)\n", a.Host, a.Owner, a.Since)
}
if s.LLM != nil {
fmt.Fprintf(w, "model server: %s\n", s.LLM.words())
}
printDoorFirewallStatus(w, s.Firewall)
}
// ollamaModels reports whether a local ollama with the embeddings API answers
@@ -1340,12 +1533,18 @@ func ollamaModelsAt(addrs []string) (bool, []string) {
const llmForwardPort = 18080
// openAICompatModels detects a non-ollama LLM server (llama.cpp llama-server,
// vLLM, anything OpenAI-compatible) via GET /v1/models. Only PRIVATAE_LLM_ADDR is
// probed — unlike ollama there is no conventional default port worth guessing,
// and the operator setting the env IS the opt-in. Same 200-is-alive rule as
// /api/tags: a live server with an unparseable list still counts.
// vLLM, anything OpenAI-compatible) via GET /v1/models at llmTarget:
// PRIVATAE_LLM_ADDR, or the server found on this machine's loopback
// (llmlocal.go). Same 200-is-alive rule as /api/tags: a live server with an
// unparseable list still counts. A found server that no longer answers is
// asked for again (llmTargetFailed).
func openAICompatModels() (bool, []string) {
return openAICompatModelsAt(llmTarget())
t := llmTarget()
ok, models := openAICompatModelsAt(t)
if !ok {
llmTargetFailed(t)
}
return ok, models
}
// openAICompatModelsAt is openAICompatModels' probe of one address.
@@ -1393,14 +1592,14 @@ func localOllamaAddr() string {
// serveLLMForward is serveOllamaForward's twin for the OpenAI-compatible
// server: host dials <bastion tunnel IP>:18080, we proxy raw TCP to
// PRIVATAE_LLM_ADDR. Only runs when the env is set (the operator's opt-in).
// llmTarget (PRIVATAE_LLM_ADDR, or the server found on this machine, llmlocal.go).
// The "llm-openai" heartbeat capability is the host's signal that this
// listener exists and speaks the /v1 dialect.
func serveLLMForward(ctx context.Context, self string, tnet *netstack.Net, gwIP netip.Addr) {
target := llmTarget()
if target == "" {
return
}
// Listening whether or not a server is there now: the one found on this
// machine (llmlocal.go) may start after the bastion, and the beat
// advertises the lane only while llmTarget names one. Each connection
// goes to the server llmTarget names at that moment.
ln, err := tnet.ListenTCP(&net.TCPAddr{IP: gwIP.AsSlice(), Port: llmForwardPort})
if err != nil {
return
@@ -1415,6 +1614,11 @@ func serveLLMForward(ctx context.Context, self string, tnet *netstack.Net, gwIP
if refused(in) {
continue
}
target := llmTarget()
if target == "" {
_ = in.Close()
continue
}
// A pairing whose grant NAMES models gets an HTTP-aware proxy
// that checks the model on each request; one with a free hand gets the
// raw splice this lane has always been (inferencefilter.go).
+81
View File
@@ -0,0 +1,81 @@
package main
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// Every script in scripts/ that starts a bastion in namespaces of its own
// hides ALL of /run and /tmp, never a chosen subdirectory of them, and refuses
// to go on while it can still see the machine's own bastion. A run that put a
// tmpfs over /run/privatae but not /run/privatae-bastion replaced a live
// bastion's ask.sock and context.sock on the machine running the tests.
//
// MUTATION: the e2e script's /run mount back to /run/privatae alone. RESULT: red.
// MUTATION: drop its guard for /run/privatae-bastion. RESULT: red.
// MUTATION: move the guard after the first start of the bastion. RESULT: red.
// MUTATION: drop the PID 1 check before the first mount. RESULT: red.
func TestNamespaceScriptsHideAllOfRunAndTmpAndRefuseToSeeTheLiveBastion(t *testing.T) {
paths, err := filepath.Glob(filepath.Join("..", "scripts", "*.sh"))
if err != nil {
t.Fatal(err)
}
n := 0
for _, p := range paths {
raw, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
s := string(raw)
if !strings.Contains(s, "unshare") {
continue
}
n++
name := filepath.Base(p)
lines := strings.Split(s, "\n")
at := func(re string) int {
r := regexp.MustCompile(re)
for i, l := range lines {
if strings.HasPrefix(strings.TrimSpace(l), "#") {
continue
}
if r.MatchString(l) {
return i
}
}
return -1
}
runAt := at(`mount -t tmpfs tmpfs /run\s*$`)
tmpAt := at(`mount -t tmpfs tmpfs /tmp\s*$`)
if runAt < 0 || tmpAt < 0 {
t.Errorf("%s: no tmpfs over all of /run (%d) and /tmp (%d)", name, runAt, tmpAt)
}
if i := at(`mount -t tmpfs tmpfs /(run|tmp)/`); i >= 0 && (runAt < 0 || i < runAt) {
t.Errorf("%s:%d hides a subdirectory of /run or /tmp instead of all of it: %s", name, i+1, lines[i])
}
guard := at(`/run/privatae-bastion`)
if guard < 0 || guard < runAt || guard < tmpAt {
t.Errorf("%s: no guard, after hiding /run and /tmp, that refuses to run while /run/privatae-bastion is visible", name)
}
if i := at(`"\$B"|privatae-bastion" (run|pair)|\$BAST`); i >= 0 && guard > i {
t.Errorf("%s:%d starts a bastion before the guard", name, i+1)
}
if at(`unshare .*--pid`) < 0 || at(`unshare .*--mount`) < 0 || at(`unshare .*--net`) < 0 {
t.Errorf("%s: a bastion runs in private PID, mount and network namespaces", name)
}
// The re-exec into the namespaces is chosen by an environment variable
// (E2E_INNER): one set by the caller skips the unshare, and every mount
// below then lands on this machine's own /run. So before any mount the
// script proves it is PID 1 of a namespace of its own.
pid1 := at(`\[ "?\$\$"? (=|-eq) "?1"? \]`)
if m := at(`^\s*(mount|mkdir -p [^;&]*&& mount) `); pid1 < 0 || (m >= 0 && pid1 > m) {
t.Errorf("%s: no check, before its first mount (line %d), that it runs as PID 1 of its own namespace (line %d)", name, m+1, pid1+1)
}
}
if n == 0 {
t.Fatal("no namespace script found: the glob is wrong")
}
}
+1 -5
View File
@@ -607,11 +607,7 @@ func serveTargetNoticeClear(self string, s serveService) {
// pairing's workers exit.
func runServeWorkers(pctx context.Context, self, dcnDir, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer string, n int) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
dialer := laneWSDialer(tlsCfg, dialHost)
var spawn func()
spawn = func() {
go func() {
+462 -20
View File
@@ -15,7 +15,15 @@ package main
// into the service's before the service starts, when the service has no
// pairing of its own yet:
//
// - only then: a service that already serves a Host is never merged into;
// - a service that already serves a Host takes the account's pairings too,
// with no question (a move to the service never leaves a pairing
// behind). Only the pairings for Hosts the service does not serve
// come in, each with its own folders and a loopback port of its own
// (carryAddedHosts); nothing of the account's that the service keeps
// machine-wide (keys, owner, code, anchor, mode, audit) ever becomes the
// service's. On an anchored machine each added pairing waits, pending,
// for the anchor Host to admit it, as any new pairing there does; on an
// unanchored one it is served as it was;
// - not while a bastion still runs from that folder (its run lock is held,
// waited for a moment), and the lock is held through the copy, the check
// and the emptying, so none starts in the middle;
@@ -76,6 +84,16 @@ type carryResult struct {
// differed; that version is beside each, named with carryAsideSuffix.
Aside []string
Note string // why nothing was carried, when nothing was
// Beside: the Hosts the service already served, when Hosts were added
// beside them (carryAddedHosts); Hosts are then the ones added.
Beside []string
// Pending: the added pairings wait for the machine's anchor to admit them;
// Anchor names it.
Pending bool
Anchor string
// Stayed: the account's pairings for Hosts the service already served,
// left in the account's folder.
Stayed []string
// Left: the pairings are in the service, but the account's folder could
// not be removed, and still holds copies of their keys.
Left string
@@ -114,6 +132,29 @@ func carryLock(f *os.File) error {
}
}
// carryTakeLock takes the account folder's run lock for a carry and returns
// what lets it go. It is held until only the lock itself is left to remove, so
// no bastion starts from the folder while it is copied, checked and emptied;
// letting it go twice is harmless.
func carryTakeLock(from string) (func(), error) {
lock, err := os.OpenFile(filepath.Join(from, "run.lock"), os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
return nil, fmt.Errorf("cannot open the run lock under %s: %w", from, err)
}
if err := carryLock(lock); err != nil {
_ = lock.Close()
return nil, fmt.Errorf("a bastion is still running from %s (it holds that folder's run lock), so its pairings "+
"were not moved into the service. Stop it, then run `privatae-bastion service install` again", from)
}
locked := true
return func() {
if locked {
_ = lock.Close()
locked = false
}
}, nil
}
// carryStateInto moves the bastion state at from into to (see the header).
func carryStateInto(from, to string) (carryResult, error) {
var res carryResult
@@ -136,27 +177,11 @@ func carryStateInto(from, to string) (carryResult, error) {
return res, fmt.Errorf("cannot read the service's pairings under %s: %w", to, err)
}
if len(have) > 0 {
res.Note = fmt.Sprintf("the service already serves %s, so the pairings under %s (%s) were left where they are",
strings.Join(have, ", "), from, strings.Join(hosts, ", "))
return res, nil
return carryAddedHosts(from, to, hosts, have)
}
lock, err := os.OpenFile(filepath.Join(from, "run.lock"), os.O_CREATE|os.O_RDWR, 0o600)
unlock, err := carryTakeLock(from)
if err != nil {
return res, fmt.Errorf("cannot open the run lock under %s: %w", from, err)
}
if err := carryLock(lock); err != nil {
_ = lock.Close()
return res, fmt.Errorf("a bastion is still running from %s (it holds that folder's run lock), so its pairings "+
"were not moved into the service. Stop it, then run `privatae-bastion service install` again", from)
}
// Held until only the lock itself is left to remove, so no bastion starts
// from the folder while it is copied, checked and emptied.
locked := true
unlock := func() {
if locked {
_ = lock.Close()
locked = false
}
return res, err
}
defer unlock()
// A copy that fails takes back every file it wrote: a service left with
@@ -373,6 +398,25 @@ func carryReport(r carryResult, from string) string {
}
s := " Moved this machine's pairings into the service from " + from + ": " + strings.Join(r.Hosts, ", ") +
". They go on serving as they were; nothing needs pairing again."
if len(r.Beside) > 0 {
s = " Added " + strings.Join(r.Hosts, ", ") + " to the service, beside " + strings.Join(r.Beside, ", ") +
", from " + from + "; nothing needs pairing again."
if !r.Pending {
s += " Each goes on serving as it was."
}
if r.Pending {
if r.Anchor != "" {
s += " Each waits until " + r.Anchor + ", the Host in charge of this machine, admits it: " + r.Anchor +
" asks you to approve it, and it is served from then on."
} else {
s += " Each waits until the Host in charge of this machine admits it."
}
}
if len(r.Stayed) > 0 {
s += " The service already served " + strings.Join(r.Stayed, ", ") + ", so your account's pairing for it " +
"was left in " + from + "."
}
}
if r.Kept > 0 {
s += fmt.Sprintf(" %d file(s) the service already had were kept as the service's.", r.Kept)
}
@@ -385,3 +429,401 @@ func carryReport(r carryResult, from string) string {
}
return s
}
// carryHostRoots are the folders under from that belong to host's pairing
// alone (config.go contextRoot, dcnDir; backupfetch.go backupRootPath;
// mirror.go mirrorDir), and any other folder or file its pairing names under
// from, relative to from. A pairing that keeps its state in a folder every
// pairing shares (the layout before 0.10.0, migrateLegacyConfig) is refused:
// that folder is not its alone to move.
func carryHostRoots(raw []byte, from, host string) ([]string, error) {
roots := map[string]bool{}
for _, top := range []string{"context", "dcn", "backups", "mirror"} {
roots[filepath.Join(top, host)] = true
}
var v any
dec := json.NewDecoder(bytes.NewReader(raw))
dec.UseNumber()
if err := dec.Decode(&v); err != nil {
return nil, fmt.Errorf("the pairing for %s does not read: %w", host, err)
}
var walkErr error
var walk func(any)
walk = func(x any) {
switch t := x.(type) {
case string:
if _, ok := rebasePath(t, from, from); !ok {
return
}
rel, err := filepath.Rel(from, t)
if err != nil {
return
}
first := strings.SplitN(filepath.ToSlash(rel), "/", 2)[0]
switch {
case first == "pairings":
return
case rel == "." || !strings.Contains(filepath.ToSlash(rel), "/"):
if walkErr == nil {
walkErr = fmt.Errorf("the pairing for %s keeps its state in %s, which every pairing under %s "+
"shares, so it was not added", host, t, from)
}
return
}
roots[rel] = true
case []any:
for _, e := range t {
walk(e)
}
case map[string]any:
for _, e := range t {
walk(e)
}
}
}
walk(v)
if walkErr != nil {
return nil, walkErr
}
var out []string
for r := range roots {
out = append(out, r)
}
sort.Strings(out)
// A root inside another is the other's already.
var kept []string
for _, r := range out {
inside := false
for _, k := range kept {
if strings.HasPrefix(filepath.ToSlash(r)+"/", filepath.ToSlash(k)+"/") {
inside = true
break
}
}
if !inside {
kept = append(kept, r)
}
}
return kept, nil
}
// carryUsedAddrs are the loopback addresses the service's own pairings hold:
// their control ports and git proxies (config.go nextListenAddr,
// nextGitProxyAddr). An added pairing that held one of them in the account
// would fail to bind beside the pairing that has it here.
func carryUsedAddrs(to string) (map[string]bool, error) {
used := map[string]bool{}
hosts, err := pairingHosts(to)
if err != nil {
return nil, err
}
for _, h := range hosts {
raw, err := os.ReadFile(filepath.Join(to, "pairings", h+".json"))
if err != nil {
return nil, err
}
var c Config
if err := json.Unmarshal(raw, &c); err != nil {
return nil, fmt.Errorf("the service's pairing for %s does not read: %w", h, err)
}
used[c.ListenAddr], used[c.GitProxyAddr] = true, true
}
delete(used, "")
return used, nil
}
// carryFreeAddr is the first loopback port from start up that used does not
// hold, marked used.
func carryFreeAddr(start int, used map[string]bool) string {
for port := start; ; port++ {
addr := fmt.Sprintf("127.0.0.1:%d", port)
if !used[addr] {
used[addr] = true
return addr
}
}
}
// carriedPairing is an account's pairing as the service keeps it: its paths
// moved under to, its loopback ports its own among the service's, and on an
// anchored machine pending, as `pair` writes any new pairing there
// (pairPrecheckState). Every other field, its keys and bearer included, is
// as it was.
func carriedPairing(raw []byte, from, to string, used map[string]bool, anchored bool) ([]byte, error) {
var m map[string]any
dec := json.NewDecoder(bytes.NewReader(rebaseJSONPaths(raw, from, to)))
dec.UseNumber()
if err := dec.Decode(&m); err != nil {
return nil, err
}
la, _ := m["listen_addr"].(string)
if la == "" || used[la] {
la = carryFreeAddr(7878, used)
}
used[la] = true
m["listen_addr"] = la
if ga, _ := m["git_proxy_addr"].(string); ga != "" {
if used[ga] {
ga = carryFreeAddr(7978, used)
}
used[ga] = true
m["git_proxy_addr"] = ga
}
// On a machine an organisation's policy governs, a Host it does not name
// waits for the one it does, anchored yet or not, as `pair` writes it
// (pairPrecheckState).
host, _ := m["host_name"].(string)
if p := currentAnchorPolicy(); anchored || (p != nil && !policyNames(p, host)) {
m["state"] = pairingPending
}
m["origin"] = pairingOriginCarried
return json.MarshalIndent(m, "", " ")
}
// carryAddedHosts adds the account's pairings for Hosts the service does not
// serve to a service that serves others (see the
// header). The account's pairings for Hosts the service already serves stay
// where they are. When none stays, the rest of the account's folder (its
// keys, owner, code, anchor, mode, audit) is kept beside the service's own
// files with carryAsideSuffix, never in their place, and the folder is
// removed, as a whole carry removes it.
func carryAddedHosts(from, to string, hosts, have []string) (carryResult, error) {
var res carryResult
serving := map[string]bool{}
for _, h := range have {
serving[h] = true
}
var add, stay []string
for _, h := range hosts {
if serving[h] {
stay = append(stay, h)
} else {
add = append(add, h)
}
}
if len(add) == 0 {
res.Note = fmt.Sprintf("the service already serves every Host the pairings under %s are for (%s), so nothing "+
"was added", from, strings.Join(hosts, ", "))
return res, nil
}
unlock, err := carryTakeLock(from)
if err != nil {
return res, err
}
defer unlock()
_, aerr := os.Lstat(filepath.Join(to, "anchor.json"))
anchored := aerr == nil
anchorName := ""
if raw, err := os.ReadFile(filepath.Join(to, "anchor.json")); err == nil {
var a struct {
AnchorHost string `json:"anchor_host"`
}
if json.Unmarshal(raw, &a) == nil && validHostName(a.AnchorHost) {
anchorName = a.AnchorHost
}
}
used, err := carryUsedAddrs(to)
if err != nil {
return res, fmt.Errorf("cannot read the service's pairings under %s: %w", to, err)
}
type item struct {
rel string
data []byte // set: written as it is; nil: read from from and rebased
aside bool // kept beside the service's file of that name, never in its place
}
var items, last []item
carried := map[string]bool{} // rel paths (and roots) this carry moves
for _, h := range add {
praw, err := os.ReadFile(filepath.Join(from, "pairings", h+".json"))
if err != nil {
return res, fmt.Errorf("cannot read the pairing for %s under %s: %w", h, from, err)
}
roots, err := carryHostRoots(praw, from, h)
if err != nil {
return res, err
}
for _, r := range roots {
carried[r] = true
err := filepath.WalkDir(filepath.Join(from, r), func(p string, d fs.DirEntry, err error) error {
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
return err
}
if !d.Type().IsRegular() {
return nil
}
rel, err := filepath.Rel(from, p)
if err != nil {
return err
}
items = append(items, item{rel: rel})
return nil
})
if err != nil {
return res, fmt.Errorf("cannot read %s: %w", filepath.Join(from, r), err)
}
}
data, err := carriedPairing(praw, from, to, used, anchored)
if err != nil {
return res, fmt.Errorf("the pairing for %s does not read: %w", h, err)
}
prel := filepath.Join("pairings", h+".json")
grel := filepath.Join("pairings", h+pairingGrantSuffix)
carried[prel] = true
last = append(last, item{rel: prel, data: data})
// A grant is the anchor's word for this machine. On an anchored
// machine the account's grant was another anchor's, and the pairing
// waits for this one's instead; it is kept beside, like any file of
// the account's the service does not take.
if _, err := os.Lstat(filepath.Join(from, grel)); err == nil && !anchored {
carried[grel] = true
last = append(last, item{rel: grel})
}
}
if len(stay) == 0 {
// Nothing stays: the rest of the folder goes beside the service's.
err := filepath.WalkDir(from, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, err := filepath.Rel(from, p)
if err != nil {
return err
}
if carried[rel] {
if d.IsDir() {
return filepath.SkipDir
}
return nil
}
if !d.Type().IsRegular() || (rel == d.Name() && carrySkip[d.Name()]) {
return nil
}
items = append(items, item{rel: rel, aside: true})
return nil
})
if err != nil {
return res, fmt.Errorf("cannot read %s: %w", from, err)
}
}
written := map[string][]byte{}
undo := func() {
for p := range written {
_ = os.Remove(p)
}
}
for _, it := range append(items, last...) {
data := it.data
if data == nil {
if data, err = os.ReadFile(filepath.Join(from, it.rel)); err != nil {
undo()
return res, fmt.Errorf("cannot read %s: %w. Nothing was removed from %s", filepath.Join(from, it.rel), err, from)
}
if strings.HasSuffix(it.rel, ".json") {
data = rebaseJSONPaths(data, from, to)
}
}
dst := filepath.Join(to, it.rel)
if it.aside {
dst += carryAsideSuffix
}
if have, err := os.ReadFile(dst); err == nil {
if bytes.Equal(have, data) {
continue
}
if !it.aside {
res.Kept++
dst += carryAsideSuffix
if _, err := os.Lstat(dst); err == nil {
undo()
return res, fmt.Errorf("%s is already there; the account's %s was not put beside the service's. "+
"Nothing was removed from %s", dst, it.rel, from)
}
res.Aside = append(res.Aside, dst)
} else {
undo()
return res, fmt.Errorf("%s is already there; the account's %s was not put beside it. Nothing was "+
"removed from %s", dst, it.rel, from)
}
} else if !errors.Is(err, fs.ErrNotExist) {
undo()
return res, err
}
if err := os.MkdirAll(filepath.Dir(dst), 0o700); err != nil {
undo()
return res, err
}
if err := carryWriteFile(dst, data, 0o600); err != nil {
_ = os.Remove(dst)
undo()
return res, fmt.Errorf("could not write %s: %w. Nothing was removed from %s", dst, err, from)
}
written[dst] = data
}
for p, want := range written {
got, err := os.ReadFile(p)
if err != nil || !bytes.Equal(got, want) {
undo()
return res, fmt.Errorf("%s did not read back as written; nothing was removed from %s", p, from)
}
}
got, err := pairingHosts(to)
if err != nil {
undo()
return res, err
}
in := map[string]bool{}
for _, h := range got {
in[h] = true
}
for _, h := range append(append([]string{}, have...), add...) {
if !in[h] {
undo()
return res, fmt.Errorf("the service's folder %s holds the pairings %v after the copy, not %s's; nothing "+
"was removed from %s", to, got, h, from)
}
}
res.Hosts, res.Beside, res.Pending, res.Stayed, res.Anchor = add, have, anchored, stay, anchorName
// From here the service has every added pairing, so what will not go is
// said, not a failure.
var leftErr error
if len(stay) > 0 {
var rels []string
for r := range carried {
rels = append(rels, r)
}
sort.Strings(rels)
for _, r := range rels {
if err := carryRemoveAll(filepath.Join(from, r)); err != nil && leftErr == nil {
leftErr = err
}
}
} else {
ents, err := os.ReadDir(from)
if err != nil {
leftErr = err
}
for _, e := range ents {
if e.Name() == "run.lock" {
continue
}
if err := carryRemoveAll(filepath.Join(from, e.Name())); err != nil && leftErr == nil {
leftErr = err
}
}
unlock()
if err := carryRemoveAll(from); err != nil && leftErr == nil {
leftErr = err
}
}
if leftErr != nil {
res.Left = fmt.Sprintf("part of %s could not be removed (%v). It may still hold copies of the added pairings' "+
"keys: delete those files", from, leftErr)
}
return res, nil
}
+192 -26
View File
@@ -94,32 +94,6 @@ func TestAServiceInstallCarriesTheAccountsPairingsIntoTheService(t *testing.T) {
}
}
// A service that already serves a Host is never merged into, and the
// account's state stays where it was.
func TestACarryNeverMergesIntoAServiceThatServesAHost(t *testing.T) {
from, to := carryTree(t)
p := filepath.Join(to, "pairings", "box-two.example.test.json")
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(`{"host_name":"box-two.example.test"}`), 0o600); err != nil {
t.Fatal(err)
}
res, err := carryStateInto(from, to)
if err != nil {
t.Fatal(err)
}
if len(res.Hosts) != 0 || !strings.Contains(res.Note, "box-two.example.test") {
t.Fatalf("carried into a serving service: %+v", res)
}
if _, err := os.Stat(filepath.Join(from, "pairings", "box-one.example.test.json")); err != nil {
t.Errorf("the account's pairing was touched: %v", err)
}
if _, err := os.Stat(filepath.Join(to, "pairings", "box-one.example.test.json")); err == nil {
t.Error("the account's pairing was merged in")
}
}
// Not while a bastion still runs from the folder: its next write would be
// lost, and two bastions would hold the same keys.
func TestACarryWaitsForTheAccountsBastionToStop(t *testing.T) {
@@ -351,3 +325,195 @@ func TestACarryWhoseSourceCannotBeRemovedStillSucceeds(t *testing.T) {
t.Errorf("the install does not say the account's copy is still there: %q", line)
}
}
// writeCarryFile writes body at rel under dir.
func writeCarryFile(t *testing.T, dir, rel, body string) {
t.Helper()
p := filepath.Join(dir, filepath.FromSlash(rel))
if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
}
// addHostsTree is carryTree with the account's pairing on the first control
// port, and a service that already serves box-two on that same port.
func addHostsTree(t *testing.T) (from, to string) {
t.Helper()
from, to = carryTree(t)
writeCarryFile(t, from, "pairings/box-one.example.test.json", `{"host_name":"box-one.example.test","bearer":"mcp_test",`+
`"listen_addr":"127.0.0.1:7878","git_proxy_addr":"127.0.0.1:7978",`+
`"context_root":"`+filepath.ToSlash(filepath.Join(from, "context", "box-one.example.test"))+`",`+
`"dcn_dir":"`+filepath.ToSlash(filepath.Join(from, "dcn", "box-one.example.test"))+`"}`)
writeCarryFile(t, to, "pairings/box-two.example.test.json", `{"host_name":"box-two.example.test",`+
`"listen_addr":"127.0.0.1:7878","git_proxy_addr":"127.0.0.1:7978"}`)
writeCarryFile(t, to, "ui.secret", "service secret")
return from, to
}
// With no question asked, the account's pairing for a Host the service does
// not serve is added beside the service's: its keys as they were, its
// folders the service's, a control port of its own. Nothing machine-wide of
// the account's takes the service's place; it is kept beside it.
//
// MUTATION: leave the account's pairings where they are when the service
// serves a Host (the 0.62 rule). RESULT: red. Keep the account's listen_addr
// in carriedPairing. RESULT: red (two
// pairings on 127.0.0.1:7878). Write the account's other files without
// carryAsideSuffix. RESULT: red (the account's ui.secret becomes the service's).
func TestAMoveAddsTheAccountsOtherHostBesideTheServices(t *testing.T) {
from, to := addHostsTree(t)
writeCarryFile(t, from, "mode.json", `{"mode":"networked"}`)
res, err := carryStateInto(from, to)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(to, "mode.json")); err == nil {
t.Error("the account's mode became the machine's, which had none of its own")
}
if b, _ := os.ReadFile(filepath.Join(to, "mode.json"+carryAsideSuffix)); !strings.Contains(string(b), "networked") {
t.Errorf("the account's mode was not kept beside: %q", b)
}
if strings.Join(res.Hosts, ",") != "box-one.example.test" || strings.Join(res.Beside, ",") != "box-two.example.test" {
t.Fatalf("added %v beside %v (%s)", res.Hosts, res.Beside, res.Note)
}
raw, err := os.ReadFile(filepath.Join(to, "pairings", "box-one.example.test.json"))
if err != nil {
t.Fatalf("the added pairing is not in the service: %v", err)
}
var c Config
if err := json.Unmarshal(raw, &c); err != nil {
t.Fatal(err)
}
if c.Bearer != "mcp_test" || c.State != "" {
t.Errorf("the pairing changed on the way: %+v", c)
}
if c.ListenAddr == "127.0.0.1:7878" || c.ListenAddr == "" || c.GitProxyAddr == "127.0.0.1:7978" || c.GitProxyAddr == c.ListenAddr {
t.Errorf("the added pairing shares a port with the service's: listen %q, git %q", c.ListenAddr, c.GitProxyAddr)
}
if !samePath(c.ContextRoot, filepath.Join(to, "context", "box-one.example.test")) {
t.Errorf("the added pairing's context is still the account's: %q", c.ContextRoot)
}
if b, _ := os.ReadFile(filepath.Join(to, "dcn", "box-one.example.test", "grants.json")); !strings.Contains(string(b), "notes") {
t.Errorf("the added pairing's grants did not come along: %q", b)
}
if b, _ := os.ReadFile(filepath.Join(to, "pairings", "box-two.example.test.json")); !strings.Contains(string(b), "7878") {
t.Errorf("the service's own pairing changed: %s", b)
}
if b, _ := os.ReadFile(filepath.Join(to, "ui.secret")); string(b) != "service secret" {
t.Errorf("the service's own secret was replaced: %q", b)
}
if b, _ := os.ReadFile(filepath.Join(to, "ui.secret"+carryAsideSuffix)); string(b) != "account secret" {
t.Errorf("the account's secret was not kept beside the service's: %q", b)
}
if _, err := os.Stat(filepath.Join(to, "run.holder.json")); err == nil {
t.Error("a running bastion's file came along")
}
if _, err := os.Stat(from); !errors.Is(err, os.ErrNotExist) {
t.Errorf("the account's folder, whose pairings all moved, is still there: %v", err)
}
if line := carryReport(res, from); !strings.Contains(line, "Added box-one.example.test to the service, beside box-two.example.test") {
t.Errorf("the report: %q", line)
}
}
// On an anchored machine an added pairing waits for the anchor, as any new
// pairing there does; the account's grant was another anchor's and is kept
// beside, never as this machine's.
//
// MUTATION: drop the pending state in carriedPairing. RESULT: red.
func TestAnAddedHostWaitsForTheAnchor(t *testing.T) {
from, to := addHostsTree(t)
writeCarryFile(t, to, "anchor.json", `{"owner":"box-two","anchor_host":"box-two.example.test"}`)
writeCarryFile(t, from, "pairings/box-one.example.test"+pairingGrantSuffix, `{"grant":"another anchor's"}`)
res, err := carryStateInto(from, to)
if err != nil {
t.Fatal(err)
}
raw, _ := os.ReadFile(filepath.Join(to, "pairings", "box-one.example.test.json"))
var c Config
_ = json.Unmarshal(raw, &c)
if c.State != pairingPending || !res.Pending || c.Origin != pairingOriginCarried {
t.Errorf("the added pairing does not wait for the anchor as a carried one: state %q origin %q, %+v", c.State, c.Origin, res)
}
if line := carryReport(res, from); !strings.Contains(line, "waits until box-two.example.test, the Host in charge of this machine, admits it") {
t.Errorf("the report does not say who admits it: %q", line)
}
if _, err := os.Stat(filepath.Join(to, "pairings", "box-one.example.test"+pairingGrantSuffix)); err == nil {
t.Error("another anchor's grant became this machine's")
}
if b, _ := os.ReadFile(filepath.Join(to, "anchor.json")); !strings.Contains(string(b), "box-two") {
t.Errorf("the machine's anchor changed: %s", b)
}
}
// The account's pairing for a Host the service already serves stays, and so
// does everything that is not an added Host's: only what was added leaves.
func TestAnAddLeavesThePairingForAHostTheServiceServes(t *testing.T) {
from, to := addHostsTree(t)
writeCarryFile(t, from, "pairings/box-two.example.test.json", `{"host_name":"box-two.example.test","bearer":"mcp_account"}`)
res, err := carryStateInto(from, to)
if err != nil {
t.Fatal(err)
}
if strings.Join(res.Hosts, ",") != "box-one.example.test" || strings.Join(res.Stayed, ",") != "box-two.example.test" {
t.Fatalf("%+v", res)
}
if b, _ := os.ReadFile(filepath.Join(to, "pairings", "box-two.example.test.json")); strings.Contains(string(b), "mcp_account") {
t.Error("the account's pairing replaced the service's for the same Host")
}
for _, rel := range []string{"pairings/box-two.example.test.json", "ui.secret"} {
if _, err := os.Stat(filepath.Join(from, filepath.FromSlash(rel))); err != nil {
t.Errorf("%s left the account's folder: %v", rel, err)
}
}
for _, rel := range []string{"pairings/box-one.example.test.json", "dcn/box-one.example.test", "context/box-one.example.test"} {
if _, err := os.Stat(filepath.Join(from, filepath.FromSlash(rel))); err == nil {
t.Errorf("%s, which moved, is still in the account's folder", rel)
}
}
if _, err := os.Stat(filepath.Join(to, "ui.secret"+carryAsideSuffix)); err == nil {
t.Error("the account's other files moved while a pairing of its stays")
}
}
// A pairing that keeps its state in a folder every pairing shares (before
// 0.10.0) is not moved: that folder is not its alone.
func TestAnAddRefusesAPairingInTheSharedFolders(t *testing.T) {
from := filepath.Join(t.TempDir(), "privatae-bastion")
raw := []byte(`{"host_name":"old.example.test","context_root":"` + filepath.ToSlash(filepath.Join(from, "context")) + `"}`)
if _, err := carryHostRoots(raw, from, "old.example.test"); err == nil {
t.Error("a pairing in the shared context folder would be moved")
}
raw = []byte(`{"host_name":"new.example.test","backup_root":"/mnt/nas","runs_dir":"` +
filepath.ToSlash(filepath.Join(from, "runs", "new")) + `"}`)
roots, err := carryHostRoots(raw, from, "new.example.test")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(roots, ","), filepath.Join("runs", "new")) || strings.Contains(strings.Join(roots, ","), "nas") {
t.Errorf("roots %v", roots)
}
}
// Under an organisation's policy, a carried Host it does not name waits for the
// one it names, even on a machine nobody holds yet.
//
// MUTATION: drop the policy clause in carriedPairing. RESULT: red.
func TestACarriedHostWaitsForTheHostAPolicyNames(t *testing.T) {
prev := readAnchorPolicy
readAnchorPolicy = func() *anchorPolicy { return &anchorPolicy{Host: "corp.example.test", Source: "test"} }
t.Cleanup(func() { readAnchorPolicy = prev })
from, to := addHostsTree(t)
if _, err := carryStateInto(from, to); err != nil {
t.Fatal(err)
}
raw, _ := os.ReadFile(filepath.Join(to, "pairings", "box-one.example.test.json"))
var c Config
_ = json.Unmarshal(raw, &c)
if c.State != pairingPending {
t.Errorf("a Host the policy does not name came in %q", c.State)
}
}
+1 -5
View File
@@ -76,11 +76,7 @@ const sessionWorkerCount = 4
// self is the pairing this pool belongs to, for the caps gate on an anchored box.
func runSessionWorkers(pctx context.Context, self string, wsURL string, tlsCfg *tls.Config,
dialHost func(ctx context.Context) (net.Conn, error), bearer string, n int, senv sessionEnv) {
dialer := &websocket.Dialer{
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { return dialHost(ctx) },
TLSClientConfig: tlsCfg,
HandshakeTimeout: 20 * time.Second,
}
dialer := laneWSDialer(tlsCfg, dialHost)
for i := 0; i < n; i++ {
go func() {
hold := laneHoldFor(self)
+23 -9
View File
@@ -143,9 +143,10 @@ func TestARunThatEndedWritesNothing(t *testing.T) {
if !startPairing(cfg) {
t.Fatal("setup: the pairing was not started")
}
select {
case <-errc: // runPairing refused the address and returned
case <-time.After(5 * time.Second):
// runPairing refused the address and returned; that pairing alone stopped
// (pairing_retry.go).
_ = errc
if waitPairingFailure(t, host) == "" {
t.Fatal("setup: the run did not end")
}
cfg.Endpoint = "203.0.113.99:1"
@@ -181,12 +182,25 @@ func TestSuperviseRestartsAPairingMadeAgainWithItsNewFile(t *testing.T) {
if old.Err() == nil {
t.Fatal("the run holding the replaced key is still up")
}
select {
case err := <-errc:
if !strings.Contains(err.Error(), `"from-the-new-file"`) {
t.Fatalf("a run started, but not from the new file: %v", err)
}
case <-time.After(5 * time.Second):
_ = errc
switch got := waitPairingFailure(t, host); {
case got == "":
t.Fatal("no run was started with the new file")
case !strings.Contains(got, `"from-the-new-file"`):
t.Fatalf("a run started, but not from the new file: %v", got)
}
}
// waitPairingFailure waits (bounded) for host's run to stop on an error, and
// returns it; "" when none came. A run's error stops that pairing alone and is
// recorded (pairing_retry.go), no longer sent to the daemon's fatal channel.
func waitPairingFailure(t *testing.T, host string) string {
t.Helper()
t.Cleanup(func() { pairingRecovered(host) })
for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline); time.Sleep(20 * time.Millisecond) {
if f, ok := loadPairingFailures()[host]; ok {
return f.Error
}
}
return ""
}
+4
View File
@@ -167,3 +167,7 @@ func writeRootFile(p string, data []byte, mode os.FileMode) error {
}
return os.Rename(tmp, p)
}
// systemServiceRunning: the launchd daemon runs `run` at once, whose console
// answers for it (pair.go pairDaemonRunning).
func systemServiceRunning() bool { return false }
+4
View File
@@ -18,3 +18,7 @@ func installSystemService() error { return errLinuxServiceIsTheUnit }
func uninstallSystemService() error { return errLinuxServiceIsTheUnit }
func printSystemServiceStatus() error { return errLinuxServiceIsTheUnit }
// systemServiceRunning: no service manager of ours here; the console answers
// for a running bastion (pair.go pairDaemonRunning).
func systemServiceRunning() bool { return false }
+10 -1
View File
@@ -158,7 +158,7 @@ func (bastionService) Execute(_ []string, r <-chan svc.ChangeRequest, s chan<- s
s <- c.CurrentStatus
case svc.Stop, svc.Shutdown:
s <- svc.Status{State: svc.StopPending}
requestShutdown()
requestShutdownFor(shutdownFromService)
select {
case <-done:
case <-time.After(20 * time.Second):
@@ -440,3 +440,12 @@ func printSystemServiceStatus() error {
}
return nil
}
// systemServiceRunning: the service runs now. A service waiting for its first
// pairing has no console yet, and still serves a pairing written beside it
// within moments (runSystemService waitForPairing), so `pair` asks this as
// well as the console.
func systemServiceRunning() bool {
pid, err := bastionServicePID()
return err == nil && pid != 0
}
+8 -1
View File
@@ -19,10 +19,15 @@ $dir = Join-Path ([IO.Path]::GetTempPath()) ("o'brien-o$([char]0x2019)neil two-
New-Item -ItemType Directory -Path $dir | Out-Null
$env:LOCALAPPDATA = $dir
$env:TMPDIR = $dir
$env:ProgramFiles = $dir
function Write-Host { param($Object) $script:out += "$Object`n" }
function Run-Case([string]$answer, [bool]$installed, [string]$pair) {
$script:out = ''; $script:cmd = ''; $script:pairSeen = ''
$script:tray = ''
# The tray is started again from the new program after a yes (Restart-Tray,
# run by testdata/install_ps1_swap.ps1).
function Restart-Tray { param($exe) $script:tray = $exe; return $true }
$Service = $true; $isAdmin = $false; $serviceInstalled = $installed; $Shared = $false
$elevatedDone = $false; $fellBack = $false; $Pair = $pair
$installScript = 'https://privatae.ai/bastion/install.ps1'
@@ -58,7 +63,7 @@ function Run-Case([string]$answer, [bool]$installed, [string]$pair) {
$threw = $null
try { . $block } catch { $threw = $_.Exception.Message }
return @{ threw = $threw; out = $script:out; cmd = $script:cmd; pairSeen = $script:pairSeen;
service = $Service; done = $elevatedDone; fellBack = $fellBack }
service = $Service; done = $elevatedDone; fellBack = $fellBack; tray = $script:tray }
}
function Fail([string]$why) { Write-Output "FAIL: $why"; exit 1 }
@@ -69,9 +74,11 @@ if ($r.pairSeen -ne 'skp_INVENTEDCODE') { Fail "the elevated step could not read
if ($r.cmd -notmatch "-Personal -OwnerSid 'S-1-5-21-1-2-3-1001'") { Fail "kind or owner: $($r.cmd)" }
if ($r.cmd -notmatch "o''brien-o$([char]0x2019)$([char]0x2019)neil") { Fail "the temp folder's quotes are not doubled: $($r.cmd)" }
if ($r.out -notmatch 'Installed: C:\\Program Files') { Fail "the elevated step's output is not shown: $($r.out)" }
if ($r.tray -ne (Join-Path $dir 'Privatae\privatae-bastion.exe')) { Fail "the tray was not started again from the installed program: '$($r.tray)'" }
if (Get-ChildItem -LiteralPath $dir -Filter 'privatae-bastion-pair-*' -ErrorAction SilentlyContinue) { Fail 'the code file was left behind' }
$r = Run-Case 'no' $false ''
if ($r.tray) { Fail 'a declined install restarted the tray' }
if ($r.threw -or $r.done -or $r.service -or -not $r.fellBack -or $r.out -notmatch 'installed for your account only') { Fail "a no must fall back to this account: $($r.threw) $($r.out)" }
$r = Run-Case 'no' $true ''
+100
View File
@@ -0,0 +1,100 @@
# Runs install.ps1's Install-Program, Stop-InstallFailed and Restart-Tray with
# Windows' services and processes stood in. pwsh on Linux:
# pwsh -NoProfile -File testdata/install_ps1_swap.ps1 ../deploy/install.ps1
# Exits 0 when every case holds, and prints the first that does not.
param([string]$Script)
$ErrorActionPreference = 'Stop'
$src = Get-Content -Raw $Script
foreach ($name in 'Install-Program', 'Stop-InstallFailed', 'Restart-Tray') {
$m = [regex]::Match($src, "(?ms)^function $name\(.*?^\}\r?\n")
if (-not $m.Success) { Write-Output "FAIL: install.ps1 has no $name"; exit 1 }
. ([scriptblock]::Create($m.Value))
}
function Fail([string]$why) { Write-Output "FAIL: $why"; exit 1 }
$dir = Join-Path ([IO.Path]::GetTempPath()) ("install-swap-" + [guid]::NewGuid().ToString())
New-Item -ItemType Directory -Path $dir | Out-Null
$exe = Join-Path $dir 'privatae-bastion.exe'
$new = Join-Path $dir 'download.exe'
# The program in place moves aside to .old and the new one takes its name.
Set-Content -LiteralPath $exe -Value 'old'
Set-Content -LiteralPath "$exe.old" -Value 'older'
Set-Content -LiteralPath $new -Value 'new'
Install-Program $new $exe
if ((Get-Content -LiteralPath $exe) -ne 'new') { Fail 'the new program is not in place' }
if ((Get-Content -LiteralPath "$exe.old") -ne 'old') { Fail 'the earlier program was not moved aside to .old' }
if (Test-Path -LiteralPath $new) { Fail 'the download was left behind' }
# A new file that cannot be moved in puts the earlier program back.
$threw = $false
try { Install-Program (Join-Path $dir 'missing.exe') $exe } catch { $threw = $true }
if (-not $threw) { Fail 'a failed move did not fail the install' }
if ((Get-Content -LiteralPath $exe) -ne 'new') { Fail 'a failed move left no program in place' }
# A first install has nothing to move aside.
$fresh = Join-Path $dir 'fresh.exe'
Set-Content -LiteralPath $new -Value 'new'
Install-Program $new $fresh
if ((Get-Content -LiteralPath $fresh) -ne 'new' -or (Test-Path -LiteralPath "$fresh.old")) { Fail 'a first install' }
# A stopped service is started again before the failure is reported.
$script:started = 0
function Start-Service { param($Name, $ErrorAction) $script:started++ }
try { Stop-InstallFailed $true 'It failed.' } catch { $msg = $_.Exception.Message }
if ($script:started -ne 1 -or $msg -notmatch 'It failed\. The bastion''s service is running again\.') { Fail "restart after a failure: $script:started, $msg" }
function Start-Service { param($Name, $ErrorAction) throw 'access denied' }
try { Stop-InstallFailed $true 'It failed.' } catch { $msg = $_.Exception.Message }
if ($msg -notmatch 'could not be started again \(access denied\)' -or $msg -notmatch 'Start-Service PrivataeBastion') { Fail "a restart that fails: $msg" }
$script:started = 0
function Start-Service { param($Name, $ErrorAction) $script:started++ }
try { Stop-InstallFailed $false 'It failed.' } catch { $msg = $_.Exception.Message }
if ($script:started -ne 0 -or $msg -ne 'It failed.') { Fail "a service this install did not stop was started: $msg" }
# Only this session's trays from the installed program are stopped, and the
# tray is started from the new one.
$me = (Get-Process -Id $PID).SessionId
$script:stopped = @(); $script:launched = $null
function Get-CimInstance { param($ClassName, $Filter, $ErrorAction)
@([pscustomobject]@{ ProcessId = 11; SessionId = $me; ExecutablePath = "$exe.old"; CommandLine = "`"$exe`" tray" },
[pscustomobject]@{ ProcessId = 12; SessionId = $me + 1; ExecutablePath = $exe; CommandLine = "`"$exe`" tray" },
[pscustomobject]@{ ProcessId = 13; SessionId = $me; ExecutablePath = $exe; CommandLine = "`"$exe`" run" },
[pscustomobject]@{ ProcessId = 14; SessionId = $me; ExecutablePath = 'C:\elsewhere\privatae-bastion.exe'; CommandLine = 'x tray' }) }
function Stop-Process { param($Id, [switch]$Force, $ErrorAction) $script:stopped += $Id }
function Wait-Process { param($Id, $Timeout, $ErrorAction) }
function Start-Process { param($FilePath, $ArgumentList) $script:launched = "$FilePath $ArgumentList" }
if (-not (Restart-Tray $exe)) { Fail 'the tray was not started' }
if (($script:stopped -join ',') -ne '11') { Fail "stopped: $($script:stopped -join ',')" }
if ($script:launched -ne "$exe tray") { Fail "started: $script:launched" }
# The service block that follows the swap: the service is stopped by then, so
# a logon task that cannot be removed starts it again, and the trays running
# from the program moved aside are not stopped as an earlier install.
$m = [regex]::Match($src, '(?ms)^ if \(\$Service\) \{\r?\n \$task = .*?(?=^ \} else \{)')
if (-not $m.Success) { Write-Output 'FAIL: install.ps1 has no service block'; exit 1 }
$serviceBlock = [scriptblock]::Create($m.Value + "}`n")
function Write-Host { param($Object) }
$Service = $true; $OwnerSid = 'S-1-5-21-1-2-3-1001'; $stoppedService = $true
$exe = Join-Path $dir 'privatae-bastion.exe'
function Get-ScheduledTask { param($TaskName, $ErrorAction) [pscustomobject]@{ TaskName = $TaskName } }
function Unregister-ScheduledTask { param($TaskName, $Confirm) throw 'access denied' }
$script:started = 0; $msg = $null
function Start-Service { param($Name, $ErrorAction) $script:started++ }
try { . $serviceBlock } catch { $msg = $_.Exception.Message }
if ($script:started -ne 1 -or $msg -notmatch 'could not be removed \(access denied\)' -or $msg -notmatch 'service is running again') {
Fail "a logon task that cannot be removed left the service stopped: $script:started, $msg"
}
function Unregister-ScheduledTask { param($TaskName, $Confirm) }
$script:stopped = @()
function Get-Process { param($Name, $Id, $ErrorAction)
@([pscustomobject]@{ Id = 21; Path = "$exe.old" }, [pscustomobject]@{ Id = 22; Path = "$exe.bak" },
[pscustomobject]@{ Id = 23; Path = $exe }, [pscustomobject]@{ Id = 24; Path = 'C:\Users\a\AppData\Local\Programs\privatae-bastion\privatae-bastion.exe' }) }
function Stop-Process { param($Id, [switch]$Force, $ErrorAction) $script:stopped += $Id }
# A stand-in bastion whose `service install` succeeds.
Set-Content -LiteralPath $exe -Value "#!/bin/sh`nexit 0"
if ($IsLinux) { chmod +x $exe }
try { . $serviceBlock } catch { Fail "the service block failed: $($_.Exception.Message)" }
if (($script:stopped -join ',') -ne '24') { Fail "stopped as an earlier install: $($script:stopped -join ',')" }
Remove-Item -Recurse -Force $dir
Write-Output 'ok: the program is moved aside, put back on failure, the service restarted, and the tray restarted'
+38 -9
View File
@@ -33,11 +33,29 @@ const trayRefresh = 5 * time.Second
var (
shutdownOnce sync.Once
shutdownCh = make(chan struct{})
// shutdownWhy is who asked, as cmdRun logs it; set once, with the close.
shutdownWhy = "asked from the tray"
)
// requestShutdown asks cmdRun to stop, exactly as Ctrl-C or SIGTERM does. Once
// only — closing a closed channel panics, and a menu item can be clicked twice.
func requestShutdown() { shutdownOnce.Do(func() { close(shutdownCh) }) }
// The askers requestShutdownFor names.
const (
shutdownFromTray = "asked from the tray"
shutdownFromService = "the Windows service manager asked it to stop (Stop-Service, Restart-Service, an update or Windows shutting down)"
)
// requestShutdown asks cmdRun to stop, exactly as Ctrl-C or SIGTERM does, for
// the tray's "Stop the bastion". Once only — closing a closed channel panics,
// and a menu item can be clicked twice.
func requestShutdown() { requestShutdownFor(shutdownFromTray) }
// requestShutdownFor is requestShutdown for another asker, named in the log
// (the service manager's stop was logged as the tray's).
func requestShutdownFor(why string) {
shutdownOnce.Do(func() {
shutdownWhy = why
close(shutdownCh)
})
}
// openLocalUI is what every "take me to the page" affordance does: the tray's
// menu item, and a click on an approval notification (notify_windows.go).
@@ -74,16 +92,27 @@ func trayStateLine(s uiStatus) string {
case s.AnchorRequired:
return "Not anchored — answering nothing"
default:
n := 0
n, servable, waiting, who := 0, 0, 0, ""
for _, p := range s.Pairings {
if p.Listening {
n++
if p.WaitingFor != "" {
waiting, who = waiting+1, p.WaitingFor
continue
}
if p.State == "" || p.State == pairingActive {
servable++
if p.Listening {
n++
}
}
}
if n == len(s.Pairings) {
return fmt.Sprintf("Serving %d host(s)", n)
line := fmt.Sprintf("Serving %d of %d hosts", n, servable)
if n == servable {
line = fmt.Sprintf("Serving %d host(s)", n)
}
return fmt.Sprintf("Serving %d of %d hosts", n, len(s.Pairings))
if waiting > 0 {
line += fmt.Sprintf("; %d waiting for %s", waiting, who)
}
return line
}
}
+27 -9
View File
@@ -78,6 +78,9 @@ type uiPairing struct {
TunnelIP string
FirstSeen string
IsAnchor bool
// WaitingFor: pending, waiting for this Host (the one in charge of the
// machine) to admit it; nothing runs for it until then.
WaitingFor string
}
// uiApproval is one thing a host has asked this machine to allow, waiting for
@@ -240,7 +243,8 @@ func collectUIStatus() uiStatus {
s.StateNote = err.Error()
return s
}
running := 0
running, servable, waiting := 0, 0, 0
who := admitter()
for _, c := range cfgs {
p := uiPairing{
Host: c.HostName,
@@ -253,30 +257,44 @@ func collectUIStatus() uiStatus {
FirstSeen: c.FirstSeen,
IsAnchor: s.AnchorHost != "" && c.HostName == s.AnchorHost,
}
if p.Listening {
running++
if c.invalid == "" && c.State == pairingPending && who != "" {
p.WaitingFor = who
waiting++
}
if c.active() {
servable++
if p.Listening {
running++
}
}
s.Pairings = append(s.Pairings, p)
s.Approvals = append(s.Approvals, pendingApprovals(c)...)
}
wait := ""
if waiting > 0 {
wait = fmt.Sprintf(" %d more wait(s) for %s to admit them: %s asks you to approve each.", waiting, who, who)
}
switch {
case len(cfgs) == 0:
// No Host is a steady state, not a fault (run.go): the bastion watches
// this machine on its own.
s.State, s.StateClass = "ON ITS OWN", "ok"
s.StateNote = "No Host is connected. The bastion watches the AI agents on this machine on its own."
case running == len(cfgs):
case servable == 0 && waiting > 0:
s.State, s.StateClass = "WAITING", "warn"
s.StateNote = strings.TrimSpace(fmt.Sprintf("No Host is served yet.%s", wait))
case servable > 0 && running == servable:
s.State, s.StateClass = "RUNNING", "ok"
s.StateNote = fmt.Sprintf("Serving %d host(s) from this machine.", len(cfgs))
s.NoConsoleNote = runtime.GOOS == "windows"
s.StateNote = fmt.Sprintf("Serving %d host(s) from this machine.%s", servable, wait)
s.NoConsoleNote = runtime.GOOS == "windows" && !systemService.installed
case running > 0:
s.State, s.StateClass = "PARTLY RUNNING", "warn"
s.StateNote = fmt.Sprintf("%d of %d hosts are being served; the rest are not answering.",
running, len(cfgs))
s.StateNote = fmt.Sprintf("%d of %d hosts are being served; the rest are not answering.%s",
running, servable, wait)
default:
s.State, s.StateClass = "NOT RUNNING", "bad"
s.StateNote = "Nothing is serving these pairings. Start it with `" + runCommandHint() + "`."
s.StateNote = "Nothing is serving these pairings. Start it with `" + runCommandHint() + "`." + wait
}
return s
}
+4 -4
View File
@@ -24,9 +24,9 @@ func TestRenderAPreview(t *testing.T) {
d, _ := pairingsDir()
_ = os.MkdirAll(d, 0o700)
for _, p := range []struct{ host, addr, state string }{
{"259c83f46775ec66.host.privatae.ai", "127.0.0.1:7878", pairingActive},
{"8abb770bbfde71d0.host.privatae.ai", "127.0.0.1:7879", pairingActive},
{"99dab9c47d60edff.host.privatae.ai", "127.0.0.1:7880", pairingPending},
{"0a1b2c3d4e5f6071.host.privatae.ai", "127.0.0.1:7878", pairingActive},
{"1b2c3d4e5f607182.host.privatae.ai", "127.0.0.1:7879", pairingActive},
{"2c3d4e5f60718293.host.privatae.ai", "127.0.0.1:7880", pairingPending},
} {
b := []byte(`{"host_name":"` + p.host + `","listen_addr":"` + p.addr +
`","bastion_tunnel_ip":"fd00:9::a:1","state":"` + p.state +
@@ -38,7 +38,7 @@ func TestRenderAPreview(t *testing.T) {
// the page that is actually waiting on somebody.
dcn := filepath.Join(t.TempDir(), "dcn")
_ = os.MkdirAll(dcn, 0o700)
first := filepath.Join(d, "259c83f46775ec66.host.privatae.ai.json")
first := filepath.Join(d, "0a1b2c3d4e5f6071.host.privatae.ai.json")
if b, err := os.ReadFile(first); err == nil {
s := strings.TrimSuffix(string(b), "}") + `,"dcn_dir":` + jsonQuote(dcn) + `}`
_ = os.WriteFile(first, []byte(s), 0o600)
+3 -2
View File
@@ -62,14 +62,15 @@
<div class="host-name">
{{.Host}}
{{if .IsAnchor}}<span class="tag ok">anchor</span>{{end}}
{{if eq .State "pending"}}<span class="tag warn">waiting for approval</span>{{end}}
{{if eq .State "pending"}}<span class="tag warn">{{if .WaitingFor}}waiting for {{.WaitingFor}} to admit it{{else}}waiting for approval{{end}}</span>{{end}}
{{if eq .State "revoked"}}<span class="tag bad">revoked</span>{{end}}
{{if eq .State "invalid"}}<span class="tag bad">invalid, not served</span>{{end}}
</div>
<dl class="host-facts">
<dt>Local port</dt>
<dd>{{.ListenAddr}}
{{if .Listening}}<span class="tag ok">answering</span>
{{if .WaitingFor}}<span class="tag warn">not served until {{.WaitingFor}} admits it ({{.WaitingFor}} asks you to approve it)</span>
{{else if .Listening}}<span class="tag ok">answering</span>
{{else}}<span class="tag bad">not answering</span>{{end}}</dd>
{{if .TunnelIP}}<dt>Tunnel address</dt><dd>{{.TunnelIP}}</dd>{{end}}
{{if .FirstSeen}}<dt>Paired since</dt><dd>{{.FirstSeen}}</dd>{{end}}
+6 -3
View File
@@ -483,9 +483,12 @@ func planPairChange(before []string, after []*Config, cfg *Config) uiPairPlan {
}
}
if !cfg.active() {
p.Note = "Paired with " + cfg.HostName + " — PENDING. This machine is anchored, so a " +
"new host waits until the anchor host approves it. Nothing is started for it yet: " +
"no tunnel, no lanes. It shows as a pending pairing on the anchor host's Network app."
p.Note = "Paired with " + cfg.HostName + " — PENDING. Nothing is started for it yet: no tunnel, no lanes."
if who := admitter(); who != "" {
p.Note = "Paired with " + cfg.HostName + " — PENDING, waiting for " + who + " to admit it. The code was used, " +
"but nothing runs for it yet: " + who + " is the Host in charge of this machine, and it asks you to " +
"approve " + cfg.HostName + ". Once you do, it is served at once."
}
p.Class = "warn"
return p
}
+23 -1
View File
@@ -144,7 +144,28 @@ func cmdUninstall(deleteData bool) error {
cfgs, _ := loadPairings()
hadHosts := len(cfgs) > 0
failed := 0
for _, st := range uninstallSteps(runtime.GOOS, whole, home, os.Getuid()) {
// The door's own firewall rules (doorfw.go): once the first step has
// stopped the service, so nothing adds one back, and while the program a
// rule names is still in place.
fwDone := false
removeFirewall := func() {
if fwDone {
return
}
fwDone = true
fwSaid, fwErr := doorFirewallRemoveAll()
for _, l := range fwSaid {
fmt.Println(" " + l)
}
if fwErr != nil {
fmt.Fprintf(os.Stderr, " could not remove the bastion's own firewall rule: %v\n", fwErr)
failed++
}
}
for i, st := range uninstallSteps(runtime.GOOS, whole, home, os.Getuid()) {
if i == 1 {
removeFirewall()
}
switch {
case st.Call == "system-service":
if err := uninstallSystemService(); err != nil {
@@ -171,6 +192,7 @@ func cmdUninstall(deleteData bool) error {
fmt.Println(" " + st.Say)
}
}
removeFirewall() // a platform with one step
if deleteData {
if err := os.RemoveAll(dataDir); err != nil {
return fmt.Errorf("the bastion is removed, but its state could not be deleted: %w", err)
+1 -1
View File
@@ -1 +1 @@
0.62.0
0.63.1
+128 -7
View File
@@ -74,6 +74,70 @@ if ($PairFile) {
$arch = if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq 'Arm64') {'arm64'} else {'amd64'}
$key = "windows-$arch.exe"
# Puts the verified download $new at $target. The program already there may be
# running, and not only as the service: a whole-machine install starts the tray
# from the same file in every signed-in session (`"<exe>" tray`, tray.go
# trayRunValue). Windows will not replace or delete a running program, but it
# will rename one, so the old file is moved aside to <exe>.old and the new one
# takes its name, as the bastion's own self-update does (update.go
# swapBinaryAt). The bastion removes the .old at its next start
# (cleanupStaleBinaries). Without this, an install beside an open tray stopped
# with "Cannot create a file when that file already exists." If the new file
# cannot be moved in, the old one is put back, so there is always a program.
function Install-Program([string]$new, [string]$target) {
$aside = $null
if (Test-Path -LiteralPath $target) {
# An .old still running (a tray from before the last update) cannot be
# removed; .bak is the other name the bastion clears at its start.
foreach ($name in @("$target.old", "$target.bak")) {
Remove-Item -Force -LiteralPath $name -ErrorAction SilentlyContinue
if (-not (Test-Path -LiteralPath $name)) { $aside = $name; break }
}
if (-not $aside) {
throw "$target.old and $target.bak are both still in use by a running bastion, so the program could not be moved aside."
}
Move-Item -LiteralPath $target -Destination $aside
}
try {
Move-Item -LiteralPath $new -Destination $target
} catch {
if ($aside) { Move-Item -LiteralPath $aside -Destination $target -ErrorAction SilentlyContinue }
throw
}
}
# A failed install never leaves the machine's bastion stopped: the service this
# installer stopped is started again before the failure is reported.
function Stop-InstallFailed([bool]$stopped, [string]$why) {
if ($stopped) {
try {
Start-Service -Name 'PrivataeBastion' -ErrorAction Stop
$why += " The bastion's service is running again."
} catch {
$why += (" The bastion could not be started again ($($_.Exception.Message)); " +
"start it from an Administrator terminal: Start-Service PrivataeBastion")
}
}
throw $why
}
# After an install for the whole machine, the tray in this person's session is
# still the earlier program (it holds the file the install moved aside). It is
# stopped and started again from the new one, so the tray and the service are
# the same version. Only this session's trays, found by their command line.
function Restart-Tray([string]$exe) {
$session = (Get-Process -Id $PID).SessionId
$paths = @($exe, "$exe.old", "$exe.bak")
Get-CimInstance Win32_Process -Filter "Name='privatae-bastion.exe'" -ErrorAction SilentlyContinue |
Where-Object { $_.SessionId -eq $session -and $_.ExecutablePath -and ($paths -contains $_.ExecutablePath) -and
($_.CommandLine -match '\stray\s*$') } |
ForEach-Object {
Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue
Wait-Process -Id $_.ProcessId -Timeout 5 -ErrorAction SilentlyContinue
}
try { Start-Process -FilePath $exe -ArgumentList 'tray' | Out-Null; return $true } catch { return $false }
}
# Checked before anything is downloaded. Never echoed: it holds the token.
if ($Pair -and -not $Pair.StartsWith('skp_')) {
throw "-Pair takes the pairing code from your Host's card, which starts skp_. Nothing was installed."
@@ -141,6 +205,9 @@ if ($Service -and -not $isAdmin) {
Remove-Item -Force -LiteralPath $own -ErrorAction SilentlyContinue
Write-Host "Removed $own, an earlier copy for your account alone."
}
$installedExe = Join-Path $env:ProgramFiles 'Privatae\privatae-bastion.exe'
if (Restart-Tray $installedExe) { Write-Host "The tray was started again from the new program." }
else { Write-Host "The tray could not be started now; it starts at your next sign-in." }
$elevatedDone = $true
} elseif ($serviceInstalled) {
throw ("This machine's bastion is installed for the whole machine, and updating it needs an administrator's yes. " +
@@ -180,6 +247,7 @@ if (-not $elevatedDone) {
# the system. Made here, it has the folder's permissions from the start.
$tmpDir = if ($Service) { $dest } else { [System.IO.Path]::GetTempPath() }
$tmp = Join-Path $tmpDir ("privatae-bastion-" + [System.Guid]::NewGuid().ToString() + ".exe")
$stoppedService = $false
try {
# Take the URL FROM THE MANIFEST, never build one. The manifest's url now
# carries ?v=<version>, which is what stops Cloudflare handing back the
@@ -198,12 +266,19 @@ if (-not $elevatedDone) {
"The downloaded binary is not the one the platform published. Nothing was installed.")
}
Write-Host "Checksum verified ($expected)."
# A running service holds its program open, and Windows will not replace an
# open program. Stop it first; `service install` below starts it again.
# The service is stopped so that it restarts on the new program;
# `service install` below starts it again. Its program, and any tray's,
# is moved aside rather than replaced (Install-Program).
if ($Service -and (Get-Service -Name 'PrivataeBastion' -ErrorAction SilentlyContinue)) {
Stop-Service -Name 'PrivataeBastion' -Force -ErrorAction SilentlyContinue
$stoppedService = $true
}
try {
Install-Program $tmp $exe
} catch {
Stop-InstallFailed $stoppedService ("The new program could not be put in place of $exe ($($_.Exception.Message)). " +
"Nothing was installed.")
}
Move-Item -Force -Path $tmp -Destination $exe
} finally {
if (Test-Path $tmp) { Remove-Item -Force $tmp -ErrorAction SilentlyContinue }
}
@@ -244,11 +319,22 @@ if (-not $elevatedDone) {
if ($Service) {
$task = Get-ScheduledTask -TaskName 'PrivataeBastion' -ErrorAction SilentlyContinue
if ($task) {
Unregister-ScheduledTask -TaskName 'PrivataeBastion' -Confirm:$false
# The service is stopped by now, so a failure here starts it again
# before it is reported (Stop-InstallFailed).
try { Unregister-ScheduledTask -TaskName 'PrivataeBastion' -Confirm:$false }
catch {
Stop-InstallFailed $stoppedService ("The logon task from an earlier install could not be removed " +
"($($_.Exception.Message)), and it would start a second bastion beside the service.")
}
Write-Host "Removed the logon task from an earlier install; the service replaces it."
}
# Not the program Install-Program moved aside: every session's tray
# runs from it now, and Windows may report a running program under the
# name it was renamed to. Those trays are not an earlier install; they
# go on as the earlier program until their next sign-in.
$asideNames = @("$exe.old", "$exe.bak")
Get-Process -Name 'privatae-bastion' -ErrorAction SilentlyContinue |
Where-Object { $_.Path -and ($_.Path -ne $exe) } |
Where-Object { $_.Path -and ($_.Path -ne $exe) -and ($asideNames -notcontains $_.Path) } |
ForEach-Object {
Stop-Process -Id $_.Id -Force -ErrorAction SilentlyContinue
# Stop-Process returns before the process is gone, and its run
@@ -261,7 +347,9 @@ if (-not $elevatedDone) {
# as the machine's owner below.
$owner = if ($OwnerSid) { $OwnerSid } else { ([Security.Principal.WindowsIdentity]::GetCurrent()).User.Value }
& $exe service install --from-account $owner
if ($LASTEXITCODE -ne 0) { throw "could not register the Windows service (exit $LASTEXITCODE)" }
if ($LASTEXITCODE -ne 0) {
Stop-InstallFailed $stoppedService "The Windows service could not be registered (exit $LASTEXITCODE; the reason is above)."
}
} else {
try {
$taskName = 'PrivataeBastion'
@@ -342,7 +430,33 @@ if (-not $elevatedDone) {
if ($LASTEXITCODE -ne 0) { Write-Host "NOTE: your account was not recorded as this machine's owner (the reason is above)." }
Write-Host ""
Write-Host "The bastion runs for the whole machine as the Windows service 'PrivataeBastion' (LocalSystem), and starts with Windows."
if (-not $Pair -and $Shared) {
# Run elevated by hand, there is no ordinary-account step afterwards to
# start the tray again (Restart-Tray), and this window would start it
# elevated, so an open tray goes on as the earlier program until sign-in.
if (-not $OwnerSid) {
Write-Host " A tray already open keeps running the earlier program until you next sign in; it then starts from the new one."
}
# What the service serves now, and what waits for the Host in charge
# to admit it, from the service's own state (`status --json`, read
# with this terminal's machine authority): the move above carries
# every pairing this account's bastion had, and on a machine with a
# Host in charge (the anchor) each carried one waits for it.
$served = @(); $waits = @()
if (-not $Pair) {
try {
$st = (& $exe status --json 2>$null | Out-String) | ConvertFrom-Json
foreach ($p in @($st.pairings)) {
if ($p.waiting_for) { $waits += "$($p.host) waits for $($p.waiting_for) to admit it; $($p.waiting_for) asks you to approve it." }
elseif ($p.state -eq 'active') { $served += [string]$p.host }
}
} catch { }
}
if ($served -or $waits) {
if ($served) { Write-Host "It serves $($served -join ', ') from this computer, and watches the AI agents on it." }
else { Write-Host "It watches the AI agents on this computer. No Host is served yet:" }
foreach ($w in $waits) { Write-Host " $w Once you do, it is served at once, with no restart." }
Write-Host " Its approvals open from the tray. See what it serves any time: privatae-bastion status"
} elseif (-not $Pair -and $Shared) {
Write-Host "Next, anchor this machine to the Host that should govern it:"
Write-Host " privatae-bastion claim-anchor"
Write-Host " (add --host <name> if this machine already serves several Hosts)"
@@ -360,6 +474,13 @@ if (-not $elevatedDone) {
elseif ($fellBack) { Write-Host " To install it for the whole machine, run this installer again and answer Windows' prompt with an administrator's yes." }
if (-not $Pair) {
# Started now, not only at the next sign-in: it watches with no Host.
# A copy already running is the old program (moved aside above), and
# starting a task that is already running does nothing, so every copy
# is stopped first, as for -Pair below.
Stop-ScheduledTask -TaskName 'PrivataeBastion' -ErrorAction SilentlyContinue
Get-Process -Name 'privatae-bastion' -ErrorAction SilentlyContinue | ForEach-Object {
Stop-Process -Id $_.Id -Force -ErrorAction SilentlyContinue
}
try { Start-ScheduledTask -TaskName 'PrivataeBastion'; $now = 'It is running now, and' } catch { $now = 'It' }
Write-Host "$now starts each time you sign in. Open a NEW terminal, then: privatae-bastion ui"
Write-Host "Connect a Host, if you run one (optional): privatae-bastion pair <token>"
+1
View File
@@ -20,6 +20,7 @@ Everything below is in `client-go/` and compiled into the one binary.
- [backups.md](backups.md): this bastion's backup key, restore requests, backups kept here, and restoring a lost host.
- [audit.md](audit.md): the audit files on this machine and the same rows in its system log.
- [console.md](console.md): the local console.
- [host-connects.md](host-connects.md): a server your Host can reach but that can't reach it back: your Host connects to it, on one port, and the wire contract a Host implements for it.
- [host-on-this-machine.md](host-on-this-machine.md): pairing with a Host on the same machine over unix sockets, or in a VM over forwarded ports.
## Using and checking it
+1
View File
@@ -9,6 +9,7 @@
| `pair [--replace] --redeem-url https://<host>:<port>/<path> --host-ca <pem> <token>` | Pair with a Host in a VM on this machine over its enroll port, trusting that CA alone. Its CA is pinned for life. See [A Host in a VM on this machine](host-on-this-machine.md#a-host-in-a-vm-on-this-machine). |
| `pair <pairing code>` | Pair with the one code (`skp_…`) a Cloud Host's "Set up a bastion" card gives: it carries the enroll address and the fingerprint of the Host's CA, and the CA is taken from the enroll port only under that fingerprint. On a local machine that already serves a Host it asks, on this machine's terminal, whether to add this one; only a yes adds it. |
| | The card's one line installs the bastion, pairs it with that code and starts it: `curl -fsSL https://privatae.ai/bastion/install.sh \| PRIVATAE_PAIR_CODE='skp_…' sh` on macOS and Linux (the code goes in the environment, never in an argument, which the process list would show), and `& ([scriptblock]::Create((irm https://privatae.ai/bastion/install.ps1))) -Pair 'skp_…'` on Windows. Either hands the code to `pair -` on standard input. |
| `pair --open [--port 8812]` | On a server your Host can reach but that can't reach it back (your Host is at home, the server on the internet): opens a pairing window for one Host and shows a code, good once and for 15 minutes. Give your Host the server's address and the code; it connects to the server, and keeps connecting to it from then on. The server needs TCP 8812 open inbound, your home network nothing. Opening the window here is the yes a local machine asks before a second Host. See [Your Host connects to a server](host-connects.md). |
| | `pair` exits 0 when it paired, 3 when this machine is already paired with that Host (nothing changed), and anything else on a failure. |
| `run` | Start the bastion: it watches this machine whether or not a Host is paired, and brings up the tunnel and lanes for every paired host. A Host paired later is started without a restart. |
| `status [--json]` | Show every pairing and its tunnel status, or, with no Host paired, whether the bastion is running on its own. From the owner's account beside the bastion installed for the whole machine, it reads that bastion (see [Reads from the owner's account](#reads-from-the-owners-account)); when nothing answers, it says the service is not answering. |
+236
View File
@@ -0,0 +1,236 @@
# Your Host connects to a server
A Host on your own computer sits behind your home router. A server on the
internet can't open a connection to it, and nothing on your home network
should be opened for it. The server can be reached, though. So for a server
your Host can reach but that can't reach it back, your Host opens the
connections, to one port on the server, and the bastion there runs every lane
over them exactly as it runs them over a connection it opened itself.
Nothing changes but the direction of the TCP connection. Inside it the
bastion is still the TLS client, the Host's CA is still the only root it
trusts, the Host's certificate still names `host.cseries.internal`, and the
bastion still presents its operator certificate, which the Host signed at
pairing. The lanes, the bearer, the heartbeat and every op are the ones in
[lanes.md](lanes.md) and [host-on-this-machine.md](host-on-this-machine.md).
## For the person
On the server:
privatae-bastion pair --open
It shows a code, good once and for 15 minutes, and the port (8812). On your
Host, choose "Add a server" and give it the server's address and the code.
That is all: the pairing finishes by itself, the server's terminal says so,
and from then on your Host connects to the server whenever it runs. If a Host
is already in charge of the server, the new one waits for it to admit it, as
any other Host does ([approvals.md](approvals.md)).
**Who is in charge.** Run with machine authority (root, or an Administrator
terminal) on a server no Host is in charge of, `pair --open` puts the Host
that pairs through it in charge of the server: it becomes the anchor
([anchor_local.go](../client-go/anchor_local.go)), and the Host counts the
server as the person's own. A local-mode bastion is made networked for it,
unless it already serves another Host the person said yes to. A Host on the
server itself still wins (paired, and not revoked), and an organisation's
policy that names another Host still decides. With a Host in charge, each
terminal and write on the server waits for the person's approval: enrol a
second factor there (`privatae-bastion stepup-enroll`).
**Adding it again.** The same Host pairing the same server again, through a
fresh `pair --open` code, replaces its pairing in place: no `unpair` first.
It is the same Host when its connection presents the CA the pairing already
pins, whatever handle it gives. The pairing gets new credentials and keeps its
state, its grants and, when it is in charge, the anchor (under a new name too,
its pinned key kept). Another Host with the same name is refused.
Right after `unpair`, the running bastion holds the port until it sees the
pairing gone (within 15 seconds); `pair --open` waits for it, up to 25
seconds, and says so.
**Firewall.** The server needs one port open inbound: **TCP 8812** (or the
one given with `--port`). Nothing on your Host's network needs opening, and
nobody has to touch the server's own firewall: while the port is in use (a
pairing whose Host connects here is active, or a pairing window is open), the
bastion keeps a rule of its own that allows it, and removes that rule when the
last such pairing and window are gone, and on `uninstall`. Every 5 minutes the
running bastion also puts its rule back if it went while the port is still in
use (removed by hand, or the firewall reset), unless a rule of the machine's
own now allows the port. Each add and remove is in the audit, and `status` and
`doctor` show the rule.
- Windows: an inbound rule named "Privatae bastion: your Host connects (TCP
8812)", allowing TCP 8812 to the bastion's own program only, on every
profile, made with `netsh` from System32.
- Linux with ufw on: `8812/tcp` allowed, with the comment "privatae-bastion:
your Host connects". With firewalld running: the port added to the default
zone, now and permanently. The bastion never turns either on or off or
changes anything else in it, and a rule of the machine's own that already
allows the port in from anywhere is left alone (and never removed). ufw
tells the bastion's rule apart by its comment, so a ufw rule without it is
never removed. firewalld keeps no comment: a port the bastion added and you
later also add yourself reads as the bastion's, and goes when the door is no
longer in use. Running as an account that can't change them, it says what
to run.
- Rules of another kind (nftables or iptables of the machine's own) are not
changed: `status` and `doctor` say once that TCP 8812 needs allowing there.
- macOS: with the application firewall on, the bastion's program is allowed
(`socketfilterfw --add`, `--unblockapp`) and removed again after, by the
path it was allowed for (a bastion that moved removes its old path's
entry). `--listapps` is read entry by entry: only an entry for the
program's exact path that allows incoming connections counts. macOS 15
lists names without paths (`privatae-bastion (state: 1)`), so there the
program's name is matched. An entry the person set to block, and "Block all
incoming connections", are the person's settings: they are reported and
never changed. This has not been run on a Mac yet.
- A cloud provider's firewall or security group is outside the machine: allow
TCP 8812 there, from anywhere or from your home address.
## What the port answers
The port is open only while the bastion serves a pairing of this kind, or a
pairing window is open on it. Before the Host has proved itself it answers the
TLS handshake and nothing else, and it never forwards a byte anywhere: a
connection it keeps is a lane of a pairing it already serves.
- A connection whose certificate does not verify under the pinned CA of
exactly one pairing served there, for `host.cseries.internal`, is closed at
the handshake. So is one that names no protocol, or the pairing protocol
with no window open.
- At most 256 handshakes run at once, 8 from one address (an IPv6 address
counts as its /64), each with 10 seconds to finish. 64 of them are kept for
addresses a Host proved itself from in the last 7 days (its certificate
verified for a pairing this bastion serves, on this port or another, or it
paired through the window), so
that a flood from any number of addresses never holds them and your Host
gets in at its first try. Every other address is admitted only while fewer
than 192 are running, and, as OpenSSH's `MaxStartups` does, once 64 are
running it is refused at random, with a chance of 30% rising evenly to
certain at 192: a new Host pairing on a quiet port always gets in, and a
flood fills slowly. Those addresses are kept in memory only, at most 64 of
them, never written anywhere: a bastion that has just restarted knows none,
and a Host connecting from an address it has not connected from before
(a new home address) is admitted as any other until its first connection
succeeds.
- A wrong pairing code costs a wait of 1, 2, 4 and 8 seconds before the next
try, during which the window is not offered at all; the fifth closes the
window.
- The audit (`pairing_audit.jsonl`, and the system log, [audit.md](audit.md))
records the window opening and closing, each wrong code, each pairing made
or refused through it, each time a Host connects after having no connection
and each time its last connection closes, and refused connections (once per
address per 10 minutes, and at most 256 addresses per 10 minutes, so a scan
does not fill it).
- A connection offered the window during its handshake starts nothing if,
by the time the exchange begins, the window was closed at the terminal,
expired, closed by the fifth wrong code, or is inside a wrong code's wait;
and the Host is not asked to sign, nor the pairing saved, once the window
is closed.
## Wire contract (what the Host implements)
Version 1. All JSON is UTF-8, one object per body.
### Connections and roles
The Host opens a TCP connection to the server's address and port, and is the
**TLS server** on it; the bastion is the **TLS client**. TLS 1.3 only. The
bastion sends the server name `host.cseries.internal` and offers, by ALPN:
| ALPN | Offered | The Host chooses it to |
|---|---|---|
| `privatae-lane/1` | by a bastion serving pairings | carry a lane of an existing pairing |
| `privatae-pair/1` | only while a pairing window is open | pair through the window |
The Host chooses exactly one of them; a connection that negotiates neither is
closed by the bastion.
### Pairing (`privatae-pair/1`)
The person gives the Host the server's `host:port` and the code. The Host
normalises the code: upper case, dashes and spaces removed, `O` read as `0`,
`I` and `L` as `1`. It is then 20 characters of
`0123456789ABCDEFGHJKMNPQRSTVWXYZ` (100 bits). That string, as bytes, is the
HMAC key below; the code itself never crosses the wire.
TLS: the Host presents its certificate for `host.cseries.internal` **followed
by its CA certificate** in the chain (the bastion takes the CA from there and
pins it). It does not request a client certificate.
Then, over that one connection, kept open (HTTP/1.1 keep-alive), the bastion
sends three requests and the Host answers each. Host header
`host.cseries.internal`. Let
- `leaf` = SHA-256 of the DER of the Host's certificate as presented on this connection,
- `ca` = SHA-256 of the DER of its CA certificate,
- `proof(role)` = lowercase hex of HMAC-SHA256(key = code, message =
`"privatae-pair-v1 " + role + "\x00" + leaf + ca + nonce_b + nonce_h`), the
hashes and nonces as raw bytes.
1. `POST /v1/pair/hello` `{"v":1,"nonce":"<64 hex: nonce_b>"}` → 200
`{"v":1,"host_handle":"home","host_name":"home.cseries.localhost","nonce":"<64 hex: nonce_h, fresh random>","proof":"<proof(\"host\")>"}`.
The Host proves first, so a stranger who connects to an open window learns
nothing to test codes against. A wrong proof counts as a wrong code.
2. `POST /v1/pair/redeem`
`{"v":1,"proof":"<proof(\"bastion\")>","wg_public_key":"…","operator_csr":"<PEM>","hostname":"<server's hostname>","platform":"linux|windows|darwin","handle":"home"}`.
The Host checks the proof (constant time) **before signing anything**; a
wrong one is 403 `{"error":"…"}` and the Host should close the attempt. It
then answers 200 with the redeem answer of a TCP pairing, with a transport
of its own and no lane address:
`{"operator_cert":"<PEM, the CSR signed by its CA for client auth>","ca_cert":"<PEM>","bearer":"…","transport":"host_dials","host_handle":"home","host_name":"home.cseries.localhost"}`.
`ca_cert` must be the CA it presented; the CSR carries the device
fingerprint as URI SANs exactly as at a TCP redeem.
3. `POST /v1/pair/done`
`{"v":1,"state":"active|pending","host_name":"…","admitted_by":"<the Host in charge, when pending>"}`,
or `{"v":1,"state":"refused","error":"<why, in words for the person>"}` →
200 `{}`. The Host shows the person what it says. On `refused` it drops the
credentials it issued; on `pending` it says who admits it and connects
anyway (the server refuses its lane connections until it is admitted).
Both proofs bind the Host's certificate on this connection: a proof made on
one connection is worth nothing relayed onto another, and a machine the Host
was misdirected to can't pass step 2 without the code, so it never gets a
signed operator certificate. The bastion closes the connection after step 3.
### Lanes (`privatae-lane/1`)
TLS: the Host presents its lane certificate for `host.cseries.internal`
(signed by the CA pinned at pairing) and **requires and verifies** the
bastion's client certificate against its CA, as its lane port does. Then the
Host serves its lane routes on the connection, exactly as on its lane port:
HTTP/1.1, `Host: <the pairing's host name>:8443`, the WebSocket lanes
(`/bastion-harness`, `/bastion-mgmt`, `/bastion-egress`, `/bastion-serve`,
`/bastion-git`) and `/api/network/bastions/…`, every request with the bearer.
The Host sends nothing on a connection until the bastion has spoken on it (a
byte from the Host on an idle connection closes it), and each connection
carries one lane, or a run of requests, for as long as the bastion keeps it.
How many: the Host keeps **4 connections idle** (open, handshake done, no
request yet) at all times, and opens another each time one carries its first
request or closes, up to **64 open** to one server. The bastion holds at most
32 idle per pairing and closes any beyond that. A bastion with every lane
parked holds about 15 (2 harness, 4 session, 2 management, 4 egress and 2
serve workers, and the heartbeat), so a starting bastion draws a burst; a lane
waits up to 20 seconds for the next connection.
Keepalive and reconnect: TCP keepalive every 25 seconds on both sides (a home
router may forget an idle connection after 30). The server never connects to
the Host. When a connection fails to open or its handshake fails, the Host
tries again after 1 second, doubling to at most 30 seconds, with jitter, and
goes back to 1 second after a success. When the bastion restarts every
connection closes and the Host's redials bring the lanes back; when the Host
comes back after being away, the bastion beats at once.
### On the C-Series side
- "Add a server": the address and code fields, the three pairing requests on
a connection it opens (an `ssl` server context on a connected socket:
`asyncio`'s `loop.connect_accepted_socket(..., ssl=ctx)` or
`ctx.wrap_socket(sock, server_side=True)`), and the pairing kept with the
server's `host:port` and `transport: host_dials`.
- A dialler per such pairing that keeps the pool above, serving the existing
lane app on each connection as if accepted on the lane port.
- Showing `done`'s words, and the server as connected or not from its
heartbeat, as for every other bastion.
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env bash
# End to end: a Host behind a home router's NAT connects to a server on the
# internet, which can't reach it back (docs/host-connects.md).
#
# sudo scripts/e2e-host-connects.sh <workdir>
#
# Root, Linux, ip and nft. Everything runs in private PID, network and mount
# namespaces: no process, address, port or file of the machine it runs on is
# touched. ALL of /run and /tmp are empty mounts here, never a chosen
# subdirectory of them (a run that hid /run/privatae but not
# /run/privatae-bastion replaced a live bastion's ask.sock and context.sock),
# and so are the machine's bastion state, policy, machine marker and service
# units. The script refuses to go on if it can still see any of them. <workdir>
# must not be under /run or /tmp, which are hidden. Three namespaces:
#
# server 203.0.113.1 the real privatae-bastion binary, `pair --open` then `run`
# router 203.0.113.2 | 192.168.1.1 masquerades the home LAN; forwards nothing inbound
# home 192.168.1.10 the stand-in Host (TestStandinHostProcess)
#
# Steps: the server can't reach the home Host; pair through the window with the
# code; lanes, heartbeat and a harness job over the Host-opened connections;
# the bastion restarts; the Host restarts; each again over the Host's
# connections. Exits 0 only if every step held.
set -euo pipefail
if [ -z "${E2E_INNER:-}" ]; then
[ $# -eq 1 ] || { echo "usage: $0 <workdir>" >&2; exit 2; }
exec env E2E_INNER=1 unshare --pid --net --mount --fork --mount-proc "$0" "$1"
fi
# E2E_INNER set by a caller skips the unshare above, and every mount below
# would then land on this machine's own /run: go on only as PID 1 of the
# namespaces that line makes.
[ "$$" = 1 ] || { echo "refusing to run: not PID 1 of a private PID namespace (E2E_INNER is set outside this script?)" >&2; exit 2; }
mkdir -p "$1"
W=$(realpath "$1")
SRC=$(cd "$(dirname "$0")/../client-go" && pwd)
log() { printf '%s e2e %s\n' "$(date -u +%H:%M:%S.%3NZ)" "$*"; }
fail() { log "FAIL: $*"; exit 1; }
case "$W/" in /run/* | /tmp/*) fail "the workdir $W is under /run or /tmp, which this run hides; give one elsewhere" ;; esac
case "$SRC/" in /run/* | /tmp/*) fail "the source $SRC is under /run or /tmp, which this run hides" ;; esac
# The Go caches stay where they are; HOME moves below.
export GOMODCACHE="$(go env GOMODCACHE)" GOCACHE="$(go env GOCACHE)" GOFLAGS=-mod=mod GOPROXY=off
case "$GOCACHE/" in /run/* | /tmp/*) fail "GOCACHE $GOCACHE is under /run or /tmp, which this run hides" ;; esac
# Hide this machine's bastion, whole: every runtime directory and socket (all
# of /run and /tmp), its service units, its machine marker and lock, its
# policy, and its state in root's home.
mount --make-rprivate /
mount -t tmpfs tmpfs /run
mount -t tmpfs tmpfs /tmp
mount -t tmpfs tmpfs /etc/systemd/system
mkdir -p /var/lib/privatae && mount -t tmpfs tmpfs /var/lib/privatae
if [ -d /etc/privatae-bastion ]; then mount -t tmpfs tmpfs /etc/privatae-bastion; fi
for d in ~root/.config/privatae-bastion ~root/.local/state/privatae-bastion; do
if [ -d "$d" ]; then mount -t tmpfs tmpfs "$d"; fi
done
export TMPDIR=/tmp
mkdir -p /run/netns
# The guard: nothing of the machine's own bastion is visible from here.
for p in /run/privatae-bastion /run/privatae /run/systemd/private /run/dbus/system_bus_socket \
/etc/systemd/system/privatae-bastion.service /var/lib/privatae/machine.json; do
if [ -e "$p" ]; then fail "refusing to run: $p of this machine is still visible after hiding /run, /tmp and the bastion's state"; fi
done
for d in ~root/.config/privatae-bastion ~root/.local/state/privatae-bastion; do
if [ -d "$d" ] && [ -n "$(ls -A "$d")" ]; then fail "refusing to run: this machine's bastion state $d is still visible"; fi
done
# The bastion keeps its door's own rule in a firewall it manages (doorfw.go):
# ufw's configuration is a private copy here, and the system bus, through
# which firewall-cmd reaches firewalld, is under the hidden /run, so neither
# is this machine's.
if [ -d /etc/ufw ]; then rm -rf "$W/etc-ufw" && cp -a /etc/ufw "$W/etc-ufw" && mount --bind "$W/etc-ufw" /etc/ufw; fi
export HOME="$W/server-home" XDG_CONFIG_HOME="$W/server-home/.config"
rm -rf "$HOME" "$W/standin" && mkdir -p "$XDG_CONFIG_HOME"
log "build: the bastion and the stand-in Host"
(cd "$SRC" && CGO_ENABLED=0 go build -o "$W/privatae-bastion" . && go test -c -o "$W/standin.test" .)
log "network: server 203.0.113.1, router NAT, home 192.168.1.10"
ip link set lo up
ip netns add router
ip netns add home
ip link add srv0 type veth peer name wan0 netns router
ip addr add 203.0.113.1/24 dev srv0 && ip link set srv0 up
ip -n router link set lo up
ip -n router addr add 203.0.113.2/24 dev wan0 && ip -n router link set wan0 up
ip -n router link add lan0 type veth peer name eth0 netns home
ip -n router addr add 192.168.1.1/24 dev lan0 && ip -n router link set lan0 up
ip -n home link set lo up
ip -n home addr add 192.168.1.10/24 dev eth0 && ip -n home link set eth0 up
ip -n home route add default via 192.168.1.1
ip netns exec router sysctl -qw net.ipv4.ip_forward=1
ip netns exec router nft -f - <<'NFT'
table ip nat {
chain post {
type nat hook postrouting priority 100;
oifname "wan0" masquerade
}
}
table ip filter {
chain forward_home {
type filter hook forward priority 0; policy drop;
ct state established,related accept
iifname "lan0" oifname "wan0" accept
}
}
NFT
# Even with a route to the home LAN, the server can't open a connection to it.
ip route add 192.168.1.0/24 via 203.0.113.2
ip netns exec home python3 -I -c 'import socket;s=socket.socket();s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1);s.bind(("0.0.0.0",8811));s.listen();import time;time.sleep(3600)' &
sleep 0.5
if timeout 3 bash -c '</dev/tcp/192.168.1.10/8811' 2>/dev/null; then
fail "the server reached the home Host's port: the NAT is not in the way"
fi
log "ok: the server cannot reach the home Host (192.168.1.10:8811 behind the NAT)"
B="$W/privatae-bastion"
# exec all the way down, so the pid a kill gets is the stand-in itself.
STANDIN() { exec ip netns exec home env STANDIN_STATE="$W/standin" STANDIN_SERVER=203.0.113.1:8812 "$@" \
"$W/standin.test" -test.run '^TestStandinHostProcess$' -test.v -test.timeout 2h; }
log "pair: privatae-bastion pair --open on the server"
"$B" pair --open >"$W/open.out" 2>&1 &
OPEN=$!
for _ in $(seq 50); do grep -q 'code:' "$W/open.out" && break; sleep 0.1; done
CODE=$(sed -n 's/^ code: *//p' "$W/open.out")
[ -n "$CODE" ] || fail "pair --open showed no code: $(cat "$W/open.out")"
cat "$W/open.out"
(STANDIN STANDIN_MODE=pair STANDIN_CODE="$CODE") | grep standin || fail "the Host could not pair"
wait $OPEN || fail "pair --open: $(cat "$W/open.out")"
cat "$W/open.out" | sed -n '/Paired with/,$p'
grep -q '"host_dials": true' "$XDG_CONFIG_HOME/privatae-bastion/pairings/home.cseries.localhost.json" || fail "no host-dials pairing saved"
log "ok: paired through the window; the server dialled nothing"
# Opened here with machine authority on a machine nobody was in charge of:
# the Host that paired is in charge of it (the anchor).
grep -q '"anchor_host": "home.cseries.localhost"' "$XDG_CONFIG_HOME/privatae-bastion/anchor.json" || fail "the Host paired through the window is not in charge of the server"
grep -q 'is now the Host in charge of this machine' "$W/open.out" || fail "pair --open did not say who is in charge"
log "ok: the Host is in charge of the server (anchor.json)"
start_bastion() { "$B" run >>"$W/bastion.log" 2>&1 & BAST=$!; }
start_host() { (STANDIN STANDIN_MODE=serve) >>"$W/standin.log" 2>&1 & HOSTP=$!; }
beats() { grep -o 'beats=[0-9]*' "$W/standin.log" | tail -1 | cut -d= -f2; }
hellos() { grep -o 'worker-hellos=[0-9]*' "$W/standin.log" | tail -1 | cut -d= -f2; }
# A heartbeat goes once a minute, and at once when a run starts.
wait_beats_over() { local n=$1; for _ in $(seq 400); do [ "$(beats || echo 0)" -gt "$n" ] 2>/dev/null && return 0; sleep 0.2; done; return 1; }
log "serve: the bastion runs, the Host connects"
start_bastion
start_host
wait_beats_over 0 || fail "no heartbeat over the Host's connections"
for _ in $(seq 100); do grep -q 'HARNESS JOB' "$W/standin.log" && break; sleep 0.2; done
# The job reaches the bastion's harness over the Host's connections and its
# answer comes back. A Host in charge of a server has each job approved there,
# and this server has no second factor enrolled, so the answer is that refusal.
grep 'HARNESS JOB' "$W/standin.log" | grep -Eq 'exit=0|needs your approval' || fail "no harness job answered"
tail -3 "$W/standin.log"
ss -tn state established '( sport = :8812 )' | tail -n +2 | awk '{print " server side of a Host-opened connection:", $3, "<-", $4}' | head -3
if ss -tn state established | grep -q '192.168.1.10'; then fail "a connection names the home address: the NAT was bypassed"; fi
log "ok: lanes, heartbeat and a harness job answered over connections the Host opened (they arrive from the router's 203.0.113.2)"
log "restart: the bastion stops and starts; the Host redials"
n=$(beats); kill "$BAST"; wait "$BAST" 2>/dev/null || true
sleep 1; start_bastion
wait_beats_over "$n" || fail "no heartbeat after the bastion restarted"
log "ok: heartbeat again after the bastion restarted (beats $n -> $(beats))"
log "restart: the Host stops and starts (same CA from its state)"
kill "$HOSTP"; wait "$HOSTP" 2>/dev/null || true
: >"$W/standin.log"
start_host
for _ in $(seq 150); do [ "$(hellos || echo 0)" -gt 0 ] 2>/dev/null && break; sleep 0.2; done
[ "$(hellos || echo 0)" -gt 0 ] 2>/dev/null || fail "no lane came back after the Host restarted"
log "ok: the lanes are back after the Host restarted ($(hellos) workers)"
wait_beats_over 0 || fail "no heartbeat after the Host restarted"
log "ok: heartbeat again after the Host restarted"
grep -E '"host_connected"|"host_disconnected"|"window_(opened|paired)"' "$XDG_CONFIG_HOME/privatae-bastion/pairing_audit.jsonl" | sed 's/^/ audit: /'
kill "$BAST" "$HOSTP" 2>/dev/null || true
log "PASS"