feat: native HTTPS-proxy support and -transport=auto (Meta Muse / proxy-only sandboxes) - #470
Merged
Merged
Conversation
…rough `pilotctl daemon start` could not bring a node online from sandboxes whose only way out is an authenticating HTTPS proxy (Meta Muse): - config.json from `pilotctl init` holds the raw-TCP registry default 34.71.57.205:9000 and daemon start forwarded it as an explicit -registry, which stops pilot-daemon from switching to registry.pilotprotocol.network:443 (TLS) in compat mode. In compat mode the compiled-in registry/beacon defaults are now left off argv (a default-equal $PILOT_REGISTRY is dropped from the child env too). - PILOT_TRANSPORT never took effect: the daemon's -transport flag defaults to "udp", masking the env var. pilotctl now resolves --transport, then $PILOT_TRANSPORT, then config "transport" and passes an explicit -transport. New: --transport <udp|compat> and --proxy <auto|off|URL> (plus config keys "transport"/"proxy", validated by `config --set`). Both reach the daemon only when set; a pilot-daemon whose -help lacks them gets them dropped with a warning instead of a flag-parse crash. A proxy URL with credentials travels as $PILOT_PROXY (never argv, PILOT-290) and is redacted in config output. The child env explicitly keeps HTTPS_PROXY/HTTP_PROXY/ALL_PROXY/NO_PROXY (both cases), PILOT_PROXY, PILOT_TRANSPORT, SSL_CERT_FILE, SSL_CERT_DIR on both the fork and --foreground paths. --compat-beacon, --registry-trust, --registry-fingerprint, --tls-trust and the documented-but-dropped --endpoint/--motd-* are forwarded when given. Also: create ~/.pilot before the O_EXCL PID claim (a fresh HOME failed with "PID file locked" forever), and registry dial failures behind a proxy now say pilotctl's direct registry dials do not use the proxy yet (TODO(netproxy)). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
install.sh --transport compat (or PILOT_TRANSPORT=compat) merges "transport": "compat" and "proxy": "auto" into ~/.pilot/config.json via `pilotctl config --set` (atomic, 0600, other keys kept) and generates compat systemd/launchd units (-transport compat, raw-TCP registry/beacon defaults left to the daemon). A re-run without the flag keeps a compat config. Audited for a no-root, no-systemd VM whose only egress is an authenticating CONNECT-:443 proxy with poisoned local DNS for *.pilotprotocol.network: every network call is curl over HTTPS (proxy env honored, CONNECT by hostname, no wget / raw IP / non-443 / registry or beacon probes). Proxy URLs are only ever printed redacted. Download failures now name the proxy and the hosts it must allow instead of falling silently into "Go is required". With an older release (pilotctl without --transport) compat points config's registry at registry.pilotprotocol.network:443, and a pilot-daemon without -proxy gets an explicit warning when HTTPS_PROXY is set. No-systemd hosts get the `pilotctl daemon start` hint; service-manager units get a note that they do not inherit the shell's HTTPS_PROXY. Tested in docker (debian bookworm, non-root, temp HOME) on an --internal network whose only egress is an authenticating CONNECT-443 proxy, with poisoned /etc/hosts for the Pilot names: v1.13.9 download + SHA-256 verify + install succeeds; missing/wrong proxy credentials fail with a clear hint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Hosted agent sandboxes such as Meta Muse block outbound UDP, poison
local DNS for *.pilotprotocol.network, and allow only an authenticating
HTTP proxy that CONNECTs to :443. Until now the daemon ignored
HTTPS_PROXY entirely, so compat mode needed root, an SNI router and a
mount-namespace /etc/hosts override to get online.
New flag -proxy (env PILOT_PROXY, config key "proxy"), resolved once at
startup by daemon.ResolveProxy into a common/netproxy policy:
auto (default) -transport=compat: HTTPS_PROXY/https_proxy, falling
back to ALL_PROXY/all_proxy, honoring NO_PROXY.
-transport=udp: no policy, dialing exactly as before.
off never proxy (HTTP clients also stop following env).
http(s)://URL that proxy for every outbound TCP/HTTP connection.
The policy (daemon.Config.Proxy) is applied to every outbound
connection: the registry client (primary, pool and reconnects, via
registry.WithDialer), the compat WSS beacon (wss.Config.Proxy), the
MOTD fetch, and http.DefaultTransport for plugin HTTP clients
(catalogue pins, skillinject, trustedagents, webhook, enterprise
control). Targets are CONNECTed by host name and never resolved
locally; TLS, SNI and pinned-fingerprint checks stay end to end.
One startup line logs the transport and the redacted proxy.
Compat mode now treats the compiled-in raw-TCP registry default
(34.71.57.205:9000, which pilotctl passes on every daemon start) as not
explicit, so it still switches to registry.pilotprotocol.network:443.
-transport honors PILOT_TRANSPORT. -beacon-rtt-probe is skipped in
compat mode (its UDP probes cannot leave the host).
Pins github.com/pilot-protocol/common to the netproxy feature commit
(v0.5.14-0.20260923210943-65ea5b1a3d2d); move to a tagged release
before merge.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Replace the feature-branch pseudo-version with the tagged release that ships netproxy (HTTP CONNECT dialer, independent proxy env parsing, credential-safe URL handling) and registry/client WithDialer. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nboarding Unifies the daemon (-proxy) and pilotctl/install.sh (--transport/--proxy) branches and fixes every confirmed review finding on both. pilot-daemon - -transport=auto (also $PILOT_TRANSPORT, config "transport"): udp when the beacon answers a UDP discover (one round trip; two attempts within 1.5s), else compat when the compat beacon accepts TCP through the proxy compat would use, else udp as before. Logged once. Never picks compat for a private registry/beacon. The binary default stays udp. pkg/daemon exports SelectTransport/NormalizeTransport; the embedder path (TransportMode "" or "auto") uses the same check. - Precedence for -transport, -proxy, -registry-trust, -registry-fingerprint: flag, then $PILOT_TRANSPORT / $PILOT_PROXY / $PILOT_REGISTRY_TRUST / $PILOT_REGISTRY_FINGERPRINT, then config.json, then default. Literal flag defaults: -help never prints $PILOT_PROXY. - registry_trust/registry_fingerprint from config/env survive compat mode; a fingerprint alone selects pinned trust. compat keeps the raw registry with an explicit -registry-tls=false (TCP/9000 fallback) and a custom registry from config.json. registry.pilotprotocol.network:443 always gets TLS, also in udp mode. - internal/proxyconf (on common/netproxy): off also accepts none/no/false/direct; bare words and scheme-less values are errors, not proxy host names. Loopback targets are never proxied (DefaultTransport, daemon HTTP clients, registry and WSS dials), even with an explicit URL. - WSS beacon dials through a netproxy dialer (wss.Config.DialContext replaces Config.Proxy): an https:// proxy is verified with the system roots; the beacon TLS config applies to the beacon only. - Unusable HTTP_PROXY/ALL_PROXY no longer drop a valid HTTPS_PROXY (common v0.5.14); skipped variables are logged. Certificate errors against the system store name SSL_CERT_FILE and the fingerprint. pilotctl - daemon start asks a daemon that supports it for -transport=auto when no transport is configured; the daemon is probed once (-help) and flags/values it predates are dropped (auto -> udp) with a warning. The ready summary reports the transport the daemon chose. - --proxy, then $PILOT_PROXY, then config "proxy"; any '@' means credentials, which travel as $PILOT_PROXY only, and nothing on argv contradicts them. Redaction/validation via internal/proxyconf. - lookup/register/rotate-key, the auto-handshake visibility check and recovery dial the registry through the egress proxy (CONNECT by name), using registry.pilotprotocol.network:443 over TLS when proxied or in compat mode (pinned with registry_fingerprint when configured), with a one-shot TLS fallback when the raw registry is unreachable directly. - config --set transport accepts auto and normalizes; leaving compat restores the raw-TCP registry an older compat install saved. - withTempHomeFull clears PILOT_TRANSPORT/PILOT_PROXY/*_PROXY. install.sh - --transport auto|udp|compat; fresh installs save auto when the daemon supports it; no "proxy" key is written; units take the transport from config.json; --transport udp restores the raw registry. - Root is allowed in a Linux container/VM without systemd (hosted agent sandboxes); regular hosts still refuse without PILOT_ALLOW_ROOT. Docs: README compat/proxy section (precedence, SSL_CERT_FILE, pinned registry), env table, CHANGELOG, regenerated docs/cli-reference.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…review
Fixes the six re-review findings on feat/native-https-proxy.
muse-proxy-cred-rotation-unhandled (high)
- New -proxy-cmd / $PILOT_PROXY_CMD / config.json proxy_cmd: a command
whose stdout is the current proxy URL. internal/proxyconf.Policy wraps
it: re-run every 60s (Config.ProxyRefreshInterval) and whenever a
CONNECT gets 407 (or the malformed status line sandbox proxies send),
then the dial is retried once with the new URL. The registry client
(primary, pool, every redial), the WSS beacon (every reconnect),
daemon HTTP clients (retrying RoundTripper) and http.DefaultTransport
(refresh on 407 via OnProxyConnectResponse) all follow it. Output and
stderr are never logged; PILOT_ADMIN_TOKEN/PILOT_WEBHOOK_SECRET are
kept from the command. A failing first run falls back to the launch
environment's proxy.
- pkg/daemon: Config.ProxyPolicy (*ProxyPolicy), StaticProxyPolicy,
NewCommandProxyPolicy; Start runs the refresher until Stop.
- install.sh saves proxy_cmd=bash -c 'printf %s "${https_proxy:-$HTTPS_PROXY}"'
in a Linux container/VM without systemd whose HTTPS_PROXY carries
credentials (or PILOT_PROXY_CMD when set); README documents it.
version-skew-config-transport-auto-bricks-older-daemon (medium)
- install.sh never saves transport=auto (--transport auto removes a saved
transport); service units get Environment PILOT_TRANSPORT_DEFAULT=auto,
which a pre-auto daemon ignores. pilot-daemon honours
$PILOT_TRANSPORT_DEFAULT only when flag, $PILOT_TRANSPORT and config.json
choose nothing.
- install.sh --version <pre-auto tag> and pilotctl update --pin rewrite a
saved transport=auto to udp. pilotctl config --set transport= (proxy=,
proxy_cmd=) removes the key.
compat-explicit-registry-tls-now-pinned-fatal (low)
- applyRegistryDefaults defaults trust (pinned with a fingerprint, else
system) whenever TLS is on in compat or for the compat registry
address, also when -registry-tls was explicit.
auto-compat-check-tcp-only-fatal-on-beacon-outage (low)
- SelectTransport's compat check now does TLS + GET of the beacon path
and requires 426 Upgrade Required (live WebSocket endpoint); a TCP front
with the beacon down (502/503/close) keeps auto on udp.
daemon-proxied-udp-registry-stays-raw-9000 (low)
- When the registry dial goes through the proxy (any transport), the
compiled-in raw registry moves to registry.pilotprotocol.network:443
over TLS, the rule pilotctl already applied.
pilotctl-auto-proxy-regardless-of-transport (low)
- pilotctl's registry routes follow the daemon: an explicit proxy URL
always; the environment's proxy only when the transport is compat (the
running daemon's, from the new info "transport" field, else
$PILOT_TRANSPORT / config.json). udp hosts dial directly; unknown
transport tries direct first and the proxied TLS registry as fallback
for the production registry only; private raw registries are never sent
to the environment's proxy outside compat.
Tests: proxyconf policy unit tests (407 retry for dials and HTTP, NO_PROXY,
Run loop, command runner); pkg/daemon compat start through a rotating
proxy (registry + WSS reconnect); cmd/daemon end-to-end runs for
PILOT_PROXY_CMD rotation, command failure fallback, proxied udp registry,
explicit -registry-tls trust, PILOT_TRANSPORT_DEFAULT; SelectTransport
front-up/beacon-down; pilotctl route matrix, private registry direct,
fitTransportToDaemon, config key clearing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tl registry dial Fixes the two re-review findings on feat/native-https-proxy. web4-install-sh-edits-not-in-canonical-installer (high) - install.sh here is a synced copy of pilot-protocol/release:install.sh (the script https://pilotprotocol.network/install.sh serves; the canonical-drift job enforces byte equality). The installer changes are now ported onto the canonical script in pilot-protocol/release branch feat/installer-proxy-transport, keeping its managed-node mode (--managed-url, --no-start, PILOT_ENROLLMENT_TOKEN), and this copy is byte-identical to that branch. canonical-drift passes once the release change merges; merge it first. - Rotation no longer depends on the installer: on Linux without systemd, when $HTTPS_PROXY / $https_proxy carries credentials, the proxy setting is auto, no proxy_cmd is configured ($PILOT_PROXY_CMD, config.json / --config file) and the daemon supports -proxy-cmd, `pilotctl daemon start` hands the daemon PILOT_PROXY_CMD=bash -c 'printf %s "${https_proxy:-$HTTPS_PROXY}"'. A node set up by the currently served installer (PILOT_ALLOW_ROOT=1, no proxy_cmd) therefore re-reads rotated credentials too. The ready summary says so (JSON: "proxy_cmd"). - README/CHANGELOG describe both paths. web4-pilotctl-raw-route-defeated-by-muse-guard (low) - With the transport unknown (no daemon answering, nothing configured) and a proxy in the environment, pilotctl now tries the TLS registry through the proxy first and the raw registry directly second. The direct fallback is probed: a peer that sends data or hangs up within 300ms of connecting (a sandbox network guard) is rejected, so the proxied route's error is reported instead of a broken pipe. NO_PROXY exempting the registry keeps direct-first. Tests: route plan table (proxy first, probe flag, NO_PROXY), probedDirectDial (guard that talks, guard that closes, silent registry usable, dial error), dialRegistry against a local guard + authenticating CONNECT proxy + pinned TLS registry (proxy allowed, proxy refusing, raw fallback to a real registry), sandboxProxyCmdFor matrix, CLI daemon start passes/keeps PILOT_PROXY_CMD. Reviewer repro (muse-sim-vm, --network none, guard on 34.71.57.205:9000): CONNECT registry.pilotprotocol.network:443 is now the first attempt and the error names the proxy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Sep 24, 2026
…roxy Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… on the proxy; daemon start names the proxy error Credential refresh now comes from common v0.5.15's netproxy instead of the branch-local proxy policy (internal/proxyconf/policy.go is gone): - -proxy-cmd / $PILOT_PROXY_CMD / config.json proxy_cmd build the resolver with netproxy.WithRefreshCommand. The command runs at startup, again once 60s have passed (lookup-driven), and whenever a proxy answers 407; the rejected connection is then retried once. - Registry (primary, pool, every redial) and the compat WSS beacon (and its reconnects) dial through netproxy.Dialer; daemon-owned HTTP clients use netproxy.RefreshingTransport (proxyconf.RoundTripper, loopback direct); http.DefaultTransport is configured in place so plugin clients take the current credentials and a 407 refreshes them for the next request. - pkg/daemon: Config.Proxy is the only knob (ProxyPolicy, ProxyRefreshInterval, StaticProxyPolicy, NewCommandProxyPolicy removed); ResolveProxy takes netproxy options. - pilotctl daemon start --proxy-cmd (passed as $PILOT_PROXY_CMD, never on argv); pilotctl's own registry commands use $PILOT_PROXY_CMD, config proxy_cmd or the sandbox default too, so a stale HTTPS_PROXY in its environment is refreshed and a 407 is retried. E2E product gap (phase-2 scenario 2c): with a proxy configured and -transport=auto, a proxy error during the compat check (407, 403, garbled answer, proxy unreachable) no longer falls back to udp, which dialed the raw registry directly past the proxy. SelectTransport picks compat and returns the proxy error; the daemon logs it at WARN with a hint, and the registry/compat dial failures end with a hint naming the fix. Fatal startup errors are logged at ERROR (log.Fatalf printed them at INFO). pilotctl daemon start notices a daemon that exits during startup instead of polling until the deadline, and on exit or timeout prints the daemon's last error and last proxy error from its log with a hint, instead of only "did not become ready". gosec: annotate the daemon exec and log read, handle probe Close errors. Tests: in-process rotating-credential CONNECT proxies for proxyconf (dial retry, RoundTripper retry, DefaultTransport refresh), pkg/daemon (registry + WSS reconnect after rotation, timed refresh, MOTD client retry, auto stays compat on 407/403/unreachable), cmd/daemon (subprocess: auto + 407 stays on the proxy, mutation-checked), pilotctl (registry dial follows rotated credentials, proxy-first routes, daemon start failure reporting with fake daemons, and an end-to-end run of the real daemon behind a rejecting proxy). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…re HTTPS_PROXY Two phase-2 E2E onboarding papercuts: - `pilotctl --json trusted list` ignored --json and printed the table. - A new pilotctl paired with a daemon that predates -proxy (v1.13.9) in a shell with HTTPS_PROXY/ALL_PROXY started it silently; the failure then surfaced only as "did not become ready". daemon start now warns that the daemon will not use the proxy (not with --proxy off, and not twice when an explicit --proxy was already dropped). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… into feat/native-https-proxy Conflicts: - cmd/daemon/main.go: one fileConfig for both flagSources (recorded before ApplyToFlags) and #464's flagExplicit; logging.Setup stays early (the transport/proxy resolution logs), so #464's logcap.Watch moves up with it; the daemon start failure keeps #464's fatalAfterPluginStart. - cmd/daemon/shutdown.go: fatalAfterPluginStart logs at ERROR (slog), like the branch's fatalf, so pilotctl daemon start can report it. - CHANGELOG.md: both Fixed lists kept. - go.mod: tidy (golang.org/x/net indirect again, as on main, now that the branch-local refresh policy is gone). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
TestStartCompatModeThroughConnectProxy failed on ubuntu CI with "beacon authenticated node 0, want 1": the daemon's Start returns as soon as it reads auth_ok, and the fake beacon stored the node only after writing it. Store first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ctl update status) into feat/native-https-proxy Conflicts: - cmd/pilotctl/updates.go: #468's result fields and printUpdateResult, plus the branch's fitTransportToDaemon note for update --pin (JSON "note", text "Note:"), with a test that the two coexist. - CHANGELOG.md: both Fixed lists kept. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ay; Muse rejection diagnostics; credential-preferring sandbox command Phase-3 review findings on #470: web4-470-apps-inherit-stale-proxy-creds (high). App-store apps are processes the daemon spawns with its environment, so they kept the launch-time HTTPS_PROXY credentials: after one rotation every app that opened a new connection failed ("node online, all apps broken"), also after a respawn. New internal/proxyconf.Relay: a CONNECT proxy on 127.0.0.1:<random> that opens each tunnel upstream with a netproxy.Dialer (current credentials, refresh + one retry on a 407 or a garbled answer), CONNECT only, guarded by its own random credentials, never inside the TLS tunnel. A proxying daemon runs one; with -proxy-cmd it points HTTPS_PROXY/https_proxy (and PILOT_PROXY when it is a URL) at it before any plugin starts, so every app it spawns inherits the relay instead of credentials. The daemon itself never uses the relay: the refresh command now runs in the launch environment (proxyconf.CommandSource, same process-group/timeout/output-cap rules as netproxy's), a refresh that names the relay is refused, and a remote restart re-execs with the launch environment. web4-470-configuretransport-ignores-garbled-rejection. net/http never passes an unparseable CONNECT answer to OnProxyConnectResponse, so http.DefaultTransport clients never refreshed on Muse's rejection form and quoted the proxy's bytes. ConfigureTransport now takes the relay: https requests tunnel through it (refresh + retry, netproxy's wording in errors, the relay's upstream error surfaced to the client); http:// keeps going to the proxy. Doc comments and CHANGELOG corrected. web4-470-muse-rejection-diagnostics-misdirect. proxyconf.CredentialHint / UnreadableConnectReply recognise netproxy's "read CONNECT response: ... (response text withheld)"; daemon.ProxyRefusalHint, the auto-selected WARN and pilotctl's start-failure hint now name the credentials and proxy_cmd for it. No hint suggests -transport=udp unconditionally any more. web4-470-sandbox-cmd-variable-precedence. The sandbox proxy_cmd (pilotctl and install.sh) now prints whichever of $https_proxy / $HTTPS_PROXY carries credentials ($https_proxy when both do, the daemon's order when neither does), so it can never replace a credentialed URL with a bare one. install.sh differs from release#49 by that one line; #49 must take it. Tests: real daemon + app-store supervisor + sideloaded app across garbled rotations (fails with "malformed HTTP status code 4O7" without the relay), relay unit tests, the reviewer's ConfigureTransport repro, CommandSource, hints (daemon subprocess, pkg/daemon, pilotctl CLI), and the sandbox command through bash for 7 variable combinations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Byte-identical copy of the canonical installer on pilot-protocol/release feat/installer-proxy-transport, so canonical-drift passes once release#49 merges. Takes this branch's credential-preferring sandbox proxy_cmd, and adds: no raw-IP registry/beacon in config.json for compat or auto behind a proxy, downloads retried once after a proxy credential rotation, proxy_cmd saved for older daemons too, PILOT_PROXY downloads, the v1.13.x proxy warning pointing at the pilot-sandbox recipe, and onboarding text that reads replies from send-message --wait and drops routine appstore --force. Only install.sh changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Byte-identical to release@67e83fc (review fixes: root refused under sudo, no `pilotctl daemon start` advice on proxy-only hosts with a daemon that cannot use the proxy, the transport stated after the download, --version / --channel beta installable again). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…all.sh sync with release#49 (c36ad9b) - pilotctl daemon start: when the daemon re-reads its credentials with a proxy command and the proxy still rejects them, the hint names the command in use and says to check what it prints from a fresh shell, instead of telling the operator to set one (E2E scenario 3: the 407 hint said "give the daemon a command that prints the current proxy URL" with config.json proxy_cmd in place). The daemon's own CredentialHint is worded for both. - TestInstallerSavesTheSandboxProxyCmd: install.sh's SANDBOX_PROXY_CMD must evaluate to pilotctl's sandboxProxyCmd. A saved proxy_cmd turns off the one `daemon start` hands the daemon, so the two must never drift (web4-470 review: the installer kept the pre-change command). - gosec: #nosec G204 with the reason on proxyconf.CommandSource (the operator's proxy command run with sh -c is the function's purpose), and the new G104 warnings (unchecked Close/Remove) in relay.go and pilotctl. - install.sh: byte-identical to pilot-protocol/release#49 c36ad9b (sandbox proxy_cmd only for credentials a fresh shell sees and never over an explicit PILOT_PROXY; restart advice stops the running daemon before the sandbox recipe; macOS LaunchAgent line conditioned and never sent to the Linux-only recipe; truthful --transport auto note for pre-auto daemons). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… credential Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
TestConfigureTransportViaRelayHandlesGarbledRejection ranged over a map of the two transports and afterwards restored the proxy to "p-clone", the password the clone iteration rotates to. Go randomizes map order, so half the runs restored a password the relay no longer held and the 403 check saw a garbled 407 instead (architecture-gates race run on 30c034b). Iterate in a fixed order. -race -count=5 passes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Sep 24, 2026
Merged
TeoSlayer
marked this pull request as ready for review
September 24, 2026 10:16
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Pilot does not work inside Meta Muse or other hosted agent sandboxes. In a Muse VM:
HOME=/rootand no systemd;HTTPS_PROXYthat permitsCONNECT host:443;*.pilotprotocol.networkis poisoned to198.18.x.x;HTTPS_PROXYevery few minutes. A fresh shell sees the current ones; a long-running process keeps its launch-time copy and gets407on new CONNECTs while its old tunnels stay up.Baseline with released v1.13.9 in a local Muse simulator:
install.shgets through the proxy, but the daemon never registers. It dials34.71.57.205:9000directly, which is killed. The only recipe that works today is a root-only SNI router, which needsCAP_SYS_ADMIN.What changed
Built on
github.com/pilot-protocol/commonv0.5.15 (netproxywith credential refresh:WithRefreshCommand/WithRefreshFunc/PILOT_PROXY_CMD, rejection (407 or garbled answer) → refresh → retry-once in theDialer,RefreshingTransportfor HTTP; idempotent registry clientClose). Includesfeat/pilotctl-proxy-passthrough. Merged with current main (#464, #467, #468, #469, #471).pilot-daemon
-proxy <auto|off|URL>, also$PILOT_PROXYand config.jsonproxy.autowith compat sends the registry TLS, the WSS beacon and every plugin or daemon HTTP client through$HTTPS_PROXY/$ALL_PROXY, honoring$NO_PROXY.CONNECTby host name, so poisoned local DNS does not matter. TLS is end to end.off,none,no,falseanddirectall mean off. Any other bare word is rejected instead of being used as a host name.-proxy-cmd(also$PILOT_PROXY_CMD, config.jsonproxy_cmd) drives common v0.5.15's netproxy refresh (netproxy.WithRefreshFuncoverproxyconf.CommandSource, which runs the command exactly likeWithRefreshCommand—sh -c, own process group killed on timeout, output capped and never logged — but in the environment the daemon was launched with, see the relay below). There is no branch-local refresh logic (internal/proxyconf/policy.gois gone). The command prints the current proxy URL; it runs at startup, again once 60s have passed, and whenever the proxy rejects the credentials — a407, or a CONNECT answer so garbled it cannot be parsed, which is how Meta Muse's proxy answers them (malformed HTTP status code) — after which that connection is retried once:netproxy.Dialer;netproxy.RefreshingTransport(loopback always direct);http.DefaultTransport(configured in place, since plugins use or clone it): https requests are tunnelled through the daemon's proxy relay (below), so a rejected CONNECT, including Muse's garbled form, is refreshed and retried instead of failing, and the error never quotes the proxy's bytes (net/http's own error did:malformed HTTP status code "4O7"). Plainhttp://requests keep going to the proxy directly.app-storesupervisor,execwith the daemon's environment). Before, they kept the launch-timeHTTPS_PROXY: within minutes every app that opened a new connection failed withmalformed HTTP status code, also after a respawn — "node online, all apps broken". A daemon that proxies now runsproxyconf.Relay, a CONNECT proxy on127.0.0.1:<random>:netproxy.Dialerwith the resolver's current credentials (refresh + one retry on a 407 or garbled answer); it never sees inside the tunnel (TLS end to end);pilot-relay:<random 192-bit token>, constant-time check), so another local user cannot borrow the daemon's proxy credentials; loopback targets go direct, NO_PROXY is honored;502(403if the proxy refused the target,504on a timeout) whose reason says why in netproxy's words — never credentials or proxy-supplied text — and a rate-limited WARN with the credential hint;-proxy-cmd, the daemon pointsHTTPS_PROXY/https_proxy(and$PILOT_PROXYwhen it holds a URL) at the relay before any plugin starts, so every app it spawns (and respawns) inherits the relay instead of credentials.HTTP_PROXYis left alone (the relay only tunnels). Without-proxy-cmdthe apps' environment is left as it is (it is exactly what the daemon uses);syscall.Exec) re-execs with the launch environment.-transport=autopicks UDP if the beacon answers a UDP discover. Otherwise it picks compat when the compat beacon answers a TLS GET with426 Upgrade Required. New: it also picks compat when a proxy is configured and it refuses the check (407,403, a garbled answer, or the proxy is unreachable). Before, auto settled on udp and the daemon dialed the raw registry directly, past the proxy, which Muse kills (E2E scenario 2c). Now every connection stays on the proxy, and the choice is logged at WARN with the proxy's answer and a hint. The registry dial retries with refreshed credentials and, if the proxy keeps refusing, fails with a hint that names the fix. With no proxy involved and nothing reachable, auto still stays on udp. The daemon's own default is stilludp.pilotctl daemon startasks forautoonly when nothing is configured, andautois never saved to config.json.daemon.ProxyRefusalHint/proxyconf.CredentialHintrecognise netproxy's report of an unparseable CONNECT answer (read CONNECT response: malformed HTTP status code (response text withheld)) and say it is how Muse rejects wrong or expired credentials, naming-proxy-cmd. Thetransport auto-selectedWARN and the finalregistry dial (after 10 attempts)error carry that hint. No hint suggests-transport=udpunconditionally any more (it dials past the proxy, which proxy-only sandboxes kill); an unreachable proxy's hint mentions-proxy=off/-transport=udponly for a host that can reach the internet without the proxy.log.Fatalf. fix(daemon): graceful watchdog exit (no orphaned apps) + built-in log size cap #464'sfatalAfterPluginStartalso logs at ERROR now.-transport,-proxy,-proxy-cmd,-registry-trustand-registry-fingerprint: flag, then$PILOT_*, then config.json, then the default.registry.pilotprotocol.network:443over TLS, in any transport.-registry-tls=falseand a custom registry are kept as they are.pkg/daemonAPI:Config.Proxy *netproxy.Resolveris the only knob (the resolver carries the refresh).ResolveProxy(spec, transport, ...netproxy.Option).SelectTransportalso returns the proxy error that kept it on compat.pilotctl
daemon start --transport <udp|compat|auto> --proxy <auto|off|URL> --proxy-cmd <command>forwards the proxy and TLS environment. A credentialed proxy URL travels as$PILOT_PROXYand--proxy-cmdas$PILOT_PROXY_CMD; neither goes on argv.daemon startsays why a start failed. It notices a daemon that exits during startup right away instead of polling until the deadline. On an exit or a timeout it prints the daemon's last error and its last proxy error from the log, with a hint. Example:did not become ready within 30s; last proxy error: proxy CONNECT registry.pilotprotocol.network:443: 407 Proxy Authentication Required, with a hint about credentials andproxy_cmd. Muse's garbled form (... read CONNECT response: malformed HTTP status code (response text withheld)) gets the same credentials/proxy_cmdhint (it used to get "the proxy must allow CONNECT to registry... and beacon..."). Before, it said only "did not become ready".pilot-daemon -help. Flags an older daemon lacks are dropped with a warning, andautobecomesudp. New: a daemon that predates proxy support, started from a shell withHTTPS_PROXY, gets a warning that it will not use the proxy.HTTPS_PROXYorhttps_proxycarries credentials, it setsPILOT_PROXY_CMD=bash -c 'case $https_proxy in *@*) printf %s "$https_proxy";; *) printf %s "${HTTPS_PROXY:-$https_proxy}";; esac'unless aproxy_cmdis already configured: the variable that carries credentials wins ($https_proxy, which Muse's guidance reads from a fresh shell, when both do; the daemon's own order when neither does), so the command can never swap the credentialed URL the daemon started with for one without credentials (the old${https_proxy:-$HTTPS_PROXY}did, withHTTPS_PROXY=http://u:p@proxyandhttps_proxy=http://proxy). The ready summary shows where the refresh command came from, but only when the daemon reports that it uses one.lookup,register,rotate-key, the auto-handshake check andrecovery recoverfollow the daemon's network (cmd/pilotctl/registry_dial.go):TestRegistryRoutesProxyFirstWhenProxyConfigured).$PILOT_PROXY_CMD, configproxy_cmdor the sandbox command too, so a staleHTTPS_PROXYin pilotctl's own environment is refreshed and a407is retried.config --set transport=|proxy=|proxy_cmd=deletes the key. Switching back to udp restores the raw registry.--json trusted listnow returns JSON.install.sh. This is a byte-identical copy of pilot-protocol/release#49 at c36ad9b (sha256
4384f0cb2890177cb87a2e2bc49e1b814d2850434d24c54d1730c9955fbbb671), and of the website fallback in pilot-protocol/website#263 (a7f94bc). It:--transport auto|udp|compat;proxy_cmd(the credential-preferringSANDBOX_PROXY_CMDabove) in sandboxes whose proxy environment carries credentials, never for credentials that come fromPILOT_PROXYand never next to an explicitPILOT_PROXY(release#49 round 2);pilotctl daemon start; restart advice there stops the running daemon first; macOS is never sent to the Linux-only recipe and its LaunchAgent start line carries the same condition;--transport autowith a pre-auto daemon);--version/--channel betainstall again;TestInstallerSavesTheSandboxProxyCmdfails if install.sh'sSANDBOX_PROXY_CMDever differs from pilotctl'ssandboxProxyCmd(a savedproxy_cmdturns pilotctl's off, so they must not drift).Docs: README, CHANGELOG,
docs/cli-reference.md(regenerated, no change).Round 3 (review + E2E follow-ups, this push)
sandboxProxyCmd; install.sh here is synced; the new test above pins it.#nosec G204with the reason onproxyconf.CommandSource(running the operator's proxy command withsh -cis its purpose), uncheckedClose/Removehandled inrelay.goandpilotctl; a hint constant renamed so G101 does not read it as a credential. Thegoseccheck passes.proxy_cmd(E2E scenario 3): when the daemon re-reads its credentials with a command and the proxy still rejects them,pilotctl daemon startnames that command and says to check what it prints from a fresh shell, instead of telling the operator to set one (TestProxyErrorHintWithProxyCommandInUse); the daemon's ownCredentialHintis worded for both cases.TestConfigureTransportViaRelayHandlesGarbledRejectionranged over a map and then restored a password named after one iteration; Go's random map order failed it half the time (architecture-gates race run). Fixed order.Compatibility
HTTPS_PROXYexported on a host where UDP works, the registry and beacon still go direct.-transport=udpforces the old behavior.HTTPS_PROXYis set. With--transport compat, the default--registryis not forced, and v1.13.9 registers in compat.proxy_cmdkey is ignored.autois never saved, andinstall.sh --version <old>andpilotctl update --pinrewrite a savedautotoudp.transport=compatand keepsregistry=34.71.57.205:9000, the node does not register.update --pindoes not rewrite the registry the wayinstall.sh --versiondoes. v1.13.9 cannot work in Muse anyway.Security
-help, the daemon argv, the pilotctl output or pilotctl's excerpt of the daemon log. netproxy never quotes a proxy's reply or the credentials;ConfigureTransportturns every refused CONNECT into anetproxy.ConnectErrorfor the same reason, and with the relay a garbled answer no longer reaches net/http's quoting error at all.TestHelpNeverPrintsProxyCredentials,TestProxyCredentialsWithReservedCharactersStayOffArgv, and password checks in every new test.CommandSource, G204) are annotated with their reasons; uncheckedClose/Removeerrors in the new code are handled. Thegoseccheck passes.Tests
All commands run with
GOWORK=off, on macOS. Round 3 at headb01d0c45(main through #468/#469/#471 merged, 0 behind):go test -short -count=1 ./internal/proxyconf ./cmd/pilotctl ./cmd/daemon ./pkg/daemonpass;go test -race -count=5 -run TestConfigureTransportViaRelayHandlesGarbledRejection ./internal/proxyconfpass; gofmt/vet clean. Earlier rounds at5097bc22:gofmtis clean andgo vet ./...is clean;GOOS=linux go vetof the changed packages is clean.go test -count=1 ./cmd/... ./internal/... ./pkg/daemon/...(full, not-short): all 23 packages pass.go test -race -shortofinternal/proxyconfandcmd/daemon(whole packages) and the proxy/transport/registry/sandbox tests incmd/pilotctlandpkg/daemon: pass.pilotctl --helpcapture is unchanged (docs/cli-reference.mdneeds no regeneration).New tests for the phase-3 review findings (each fails without its fix):
cmd/daemonTestAppsFollowProxyRotationThroughTheRelay: the real daemon (main()in a subprocess) and the real app-store supervisor spawn a sideloaded app (a stdlibnet/httpclient built in the test, like plainweb) behind a proxy that rotates its credentials and garbles stale ones the Muse way. The app'sHTTPS_PROXYis the relay; across two rotations every request succeeds and the daemon output holds no password. With the relay disabled the same test fails withmalformed HTTP status code "4O7"on every request after the rotation (checked).cmd/daemon:TestAppRelayEnvironment(exported variables,HTTP_PROXYuntouched,PILOT_PROXYonly when it is a URL, the refresh still runs in the launch environment, a refresh printing the relay is refused),TestRelayExportedOnlyWithProxyCommand,TestProxyErrorHintGarbledRejection,TestAutoTransportGarbledRefusalHintsCredentials(real daemon, wrong password, garbled answers: the WARN hint names the credentials and-proxy-cmd, never-transport=udp; the output never contains4O7).internal/proxyconf:TestRelayFollowsRotationForApps(3 garbled rotations, one refresh and one retry each),TestRelayRequiresItsCredentials,TestRelayRefusals,TestRelayServesAndLoopGuard,TestRelayCloseEndsTunnels,TestConfigureTransportViaRelayHandlesGarbledRejection(the reviewer's repro: the first GET after a garbled rotation succeeds with exactly 1 refresh, forDefaultTransportand a clone; unrefreshable credentials give the relay's 502 with netproxy's wording; http:// and loopback keep their routes),TestConfigureTransportWithoutRelayGarbled(documents the limit without the relay),TestCredentialHint,TestCommandSourceUsesGivenEnvironment,TestCommandSourceFailures,TestCommandSourceTimeoutKillsTheGroup.pkg/daemon:TestProxyRefusalHintGarbledAnswer.cmd/pilotctl:TestSandboxProxyCmdKeepsTheCredentialedProxy(runs the command through bash for 7 variable combinations, and through a real resolver in the reviewer's case),TestCLIDaemonStartGarbledProxyAnswerHint.Earlier tests. Each uses in-process rotating-credential CONNECT proxies, which accept one password at a time and answer anything else with
407:internal/proxyconf:TestDialContextRetriesAfterRotation: 407 → the command runs again → one retry; credentials that stay wrong get no retry.TestRoundTripperRetriesAfterRotation: POST with a replayable body, a body that cannot be replayed, loopback direct. The base transport is already configured in place, ashttp.DefaultTransportis.TestConfigureTransportRefreshesOn407:DefaultTransportand a clone of it.TestResolveWithRefreshCommand.pkg/daemon:TestProxyRefreshCommandFollowsCredentialRotation: realStartin compat through the proxy; after a rotation, the registry reconnect and the WSS beacon reconnect both work.TestProxyRefreshCommandPeriodicRefresh: the timed refresh, 0 × 407.TestNewHTTPClientRetriesAfterRotation: the MOTD client.TestSelectTransportProxyRefusalStaysCompat:407,403and an unreachable proxy all give compat plus the proxy error; the direct path and a proxied beacon outage still give udp.cmd/daemon:TestAutoTransportProxyRefusalStaysCompat: real daemon subprocess with a wrong proxy password. It stays on compat, with a WARN and a hint, and makes no raw-registry CONNECT and no direct dial.TestResolveProxy(-proxy-cmdrules).TestProxyCmd*: subprocess rotation.cmd/pilotctl:TestRegistryDialFollowsRotatedProxyCredentials: staleHTTPS_PROXY→ 407 reported; the command supplies current credentials; a rotation between resolve and dial → 407, refresh, retry.TestRegistryRoutesProxyFirstWhenProxyConfigured.TestExtractProxyErrorFromDaemonLog.TestCLIDaemonStartReportsDaemonProxyError: a timeout, and a daemon exit noticed early.TestCLIDaemonStartWarnsOldDaemonIgnoresProxy.TestCLITrustedListJSON.TestE2EDaemonStartBehindRejectingProxybuilds the realpilot-daemon. It is skipped under-short, so CI does not run it.Mutation checks: each of these makes the tests above fail:
ProxyForout of the auto probe (the old udp fallback);RefreshingTransportwith a transport configured in place;ConfigureTransport's CONNECT hook stay on theRefreshingTransportbase.Binaries:
darwin/linux×amd64/arm64(pilot-daemon,pilotctl,pilot-updater), built withCGO_ENABLED=0 -trimpath -ldflags "-s -w -X main.version=v1.13.9-muse-<sha>".End-to-end
Round 3, Muse sim (rotating-credential CONNECT-443-only proxy, password rotates every 60s and only the current one is accepted; root,
HOME=/root, no systemd, Pilot names poisoned, default route to a packet sink that logs every direct dial; branch binaries at30c034bbfor linux/arm64, installed by release#49 c36ad9b through pilot-skills#34 a366fe8):curl … muse/install.sh | bashas rootnative path, -transport=compat, proxy credentials: cmd; daemon argv has no credentials; logproxy=auto: http://***@e2e3-proxy:3128 … (credentials refreshed by command); registered 5s after start.config.json:transport=compat,proxy_cmd=sandboxProxyCmd, no registry key.plainweb.fetch5/5 (the app's proxy is the daemon's relay; every stale-generation 407 was followed by an ALLOW), list-agents 0/5 — the production list-agents problem below, not proxy related.skills_dir=/root/workspace/skills,skill_format=muse) the gatedmuserow is active:drifted → rewrite, and after deletionabsent → create, Muse frontmatter.muse/install.shnames the 407 and says to rerun from a fresh shell (rc=1). The outercurl -fsSL … | bashstill shows onlycurl: (22)(documented in pilot-sandbox troubleshooting).-proxy, so upgrading does not help yet (no more loop). The release#49 warning names the recipe.Phase-3 review fixes (macOS, the reviewer's harness:
sandbox-execprofile allowing only localhost egress; a Muse-like CONNECT proxy on 127.0.0.1,:443only, Basic auth, stale credentials answered withHTTP/1.1 4O7 ...; a "fresh shell" viaBASH_ENV; config.jsonproxy_cmd= the new sandbox command; real registry, beacon, catalogue and plainweb through the proxy):proxy relay listening addr=127.0.0.1:<port>plainweb.fetchps -E) isHTTPS_PROXY=http://pilot-relay:***@127.0.0.1:<port>, no proxy credentialsDENY 407 plainweb-...run.app:443thenALLOW(the relay refreshed and retried); before the fix this wasmalformed HTTP status code "4O7"DENY→ALLOWretry. An earlier 16-round run (8 app kills) was also clean except while the supervisor's crash backoff kept the app downBASH_ENV=/dev/null), fresh HOME,daemon startlast proxy error: ... read CONNECT response: malformed HTTP status code (response text withheld)with hint: "the proxy's answer to CONNECT could not be parsed ... reject wrong or expired credentials ... proxy_cmd"; daemon WARNtransport auto-selectedhint names the credentials and-proxy-cmd, no-transport=udp; plugin errors via the relay, never4O7list-agentsqueryd80ac75e) and the fixed binaries (rekey retransmit gave up ... session desyncwith node 179172 over the compat relay), while it worked an hour earlier in the reviewer's run: an overlay issue outside these findings, see StatusPhase-2 evidence (Muse sim and macOS substitutes, head
1dd9e8e7):pilot-skillsmuse/install.shproxy_cmd, and only:443ALLOW lines in the proxy log. The firstlist-agentsquery failed withconnection_failed, and the same failure hit v1.13.9 over UDP at the same time. #469 (merged here) fixes the "dropping frame with spoofed source" pong drops seen in that run. Re-run: see Round 3 above (registration and rotation fine; list-agents is a separate production problem).pilotctl daemon start(auto)TestE2EDaemonStartBehindRejectingProxy(real daemon, in-process proxy that answers407): auto stays on compat, the proxy seesCONNECT registry.pilotprotocol.network:443and never the raw registry, the daemon makes no direct dial, and pilotctl printslast proxy error: proxy CONNECT registry.pilotprotocol.network:443: 407 Proxy Authentication Requiredwith a hint.daemon startproxy_cmdStatus
Ready for review. Required checks (Go ubuntu/macos, Analyze Go, Architecture gates, gitleaks, govulncheck) and gosec pass.
install.sh matches pilot-protocol/releasefails by design until release#49 merges (it compares against releasemain); re-run it after that merge.Open items outside this PR:
cannot connect to 0:0000.0002.BBE4 (port 1001)or a 30s no-reply, and fresh UDP identities fail even the first. pilot-mom works (5/5 here, 18/19 in phase 3). daemon: path probes and legacy keepalives prove liveness (stop spurious path resets) #469 (in v1.13.10 and in this branch) does not change it. This needs a look at the list-agents node/session path before onboarding can be called clean; it does not block this PR.-transport=autoprobe sends 2 UDP datagrams to the beacon; in Muse those go around the proxy. The Muse installers savetransport=compat(no probe) and pilot-mcp skips its own probe there./proc/<pid>/environstill holds the launch-timeHTTPS_PROXY(not argv, not a file).Deploy order (all six PRs)
notify-install-sh-syncdispatches to pilot-protocol/website →sync-install-sh.ymlputsinstall.shinto R2pilot-release-assets/install.sh(thepilot-releaseWorker reads R2 per request; no Worker deploy). Check:curl -fsSL https://pilotprotocol.network/install.sh | shasum -a 256→4384f0cb2890177cb87a2e2bc49e1b814d2850434d24c54d1730c9955fbbb671(up to 5 min cache).public/install.sh, same bytes) → the sync workflow's drift step goes green.install.sh matches pilot-protocol/release, merge.v1.13.11on main (release workflow: binaries,checksums.txt, signedlatest.json). v1.13.10 shipped without feat: native HTTPS-proxy support and -transport=auto (Meta Muse / proxy-only sandboxes) #470, so until this tag every install gets a daemon without-proxy. Checkpilot-daemon -hlists-proxyand-proxy-cmd.skills.json/setups.jsonandskills/pilotctl/SKILL.md, dispatches the website deploy; the Muse one-shot (raw.githubusercontent.com/TeoSlayer/pilot-skills/main/muse/install.sh) is live at merge.0.3.0(behaviour change; last published 0.2.13): bumppackage.json,package-lock.json,server.json(both fields),.well-known/mcp/server-card.json,src/version.js,src/openclaw-plugin/openclaw.plugin.json,src/openclaw-plugin/evaluate.jsand the pinnedpilotprotocol-mcp@0.2.13strings intest/{hermes-setup,openclaw-plugin,native-harness-setup,harness-config-contracts}.test.js(test/release-contract.test.jslocks them), move[Unreleased]to[0.3.0], push tagv0.3.0→publish.yml(tag must equalv+version; lint + tests; npm via OIDC, MCP Registryserver.json,ghcr.io/pilot-protocol/pilot-mcp).v0.2.5→ pilotprotocol follow-up PRgo get github.com/pilot-protocol/skillinject@v0.2.5 && go mod tidy(GOWORK=off) → next daemon release (v1.13.12; or fold the bump in before tagging v1.13.11).gatedToolsmuse row). Released daemons drop the key (CIreleased-skillinjectv0.2.2–v0.2.4). No manifest signing step:DefaultManifestPublicKeyHexis empty; only hosts that installed~/.pilot/skillinject.pubverifyinject-manifest.json.sig.🤖 Generated with Claude Code