Repository navigation
feat(cluster): let nodes on one LAN reach each other directly over the host network - #79
Conversation
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
Security findingsFinding details are still loading. Check the individual review comments. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Mearman
changed the title
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
a6266fa to
a2bf347
Compare
|
@codex security review |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
a2bf347 to
302048d
Compare
ca5df0f to
b728aec
Compare
302048d to
445a987
Compare
|
@codex security review |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
b728aec to
c2b9c25
Compare
445a987 to
dfdfe6d
Compare
c2b9c25 to
3c9fe72
Compare
dfdfe6d to
f549691
Compare
|
@codex security review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
🛡️ Codex Security Review
Here are some automated security review suggestions for this pull request.
Reviewed commit: f5496919f4
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
f549691 to
8fa841a
Compare
…e up The entrypoint said k3s's StartVPN checks the backend state and skips its own tailscale up once the daemon is running. It does not: on every start it runs tailscale up with --reset, adding only --login-server and the --vpn-auth extraArgs. The identity survives, but every other preference is reset, so a Tailscale setting a node needs has to be a tailscaled flag or an extraArgs entry.
A node container on Compose's bridge network sits behind the Docker host's NAT, so its Tailscale client can offer its peers no address they can reach and every pair of nodes is relayed, even on one LAN. github_runner_cluster_network_mode host merges docker-compose.host-network.yml last, which puts the server and agent services on the host's network and empties the port list; the client then offers the host's own addresses and nodes that share a network connect directly, falling back to a relay otherwise. bridge stays the default. Nothing k3s uses changes. In host mode the role checks that the API server answers on the host's loopback before it uses the kubeconfig there, since no port is published.
…ir own A host_network scenario in the mesh integration test gives each node a Docker daemon of its own, a docker:dind container at a fixed address on the shared test network standing in for a VM with an address on the LAN, and runs the role with github_runner_cluster_network_mode host against a Headscale server the test starts. Besides the usual cluster checks, every node's Tailscale client must reach every other directly at that daemon's address, the node containers must publish no port, and a second run must restart nothing. The workflow runs it on pull requests.
… node Before the role changes anything on a host it checks the node's network mode, and with github_runner_cluster_network_mode host that the Docker host has an address on github_runner_cluster_lan_cidr, read from a throwaway container on the host network, and no Tailscale device of another client there: the host's own on a Linux machine with no VM, or a second node on the same Docker host. Moving a running node between the bridge and the host network recreates its container, so the role first reads the cluster through each server of the inventory in turn, this one first, and refuses unless the other servers whose node is Ready and whose API server reports etcd healthy still form a majority, and no ARC runner pod is on the node. Whatever cannot be told refuses the move, saying why: no server that answers, servers without their role labels, a missing or unreachable server, one the inventory does not hold, or no server in the inventory at all. Only one node may move per run. Every check only reads.
… bridged Colima VM
A native node runs k3s under systemd on the host's own network with the host's own Tailscale client and has no container, so github_runner_cluster_network_mode: host means nothing for it. The role now refuses the combination with the other native settings checks, before it changes anything on the host.
…ontainers With network_mode: host the node's Tailscale client, k3s routes and kube-proxy rules live in the Docker host's network namespace, so every other container on that engine gets a route into the mesh. The role now reads the running containers before it changes anything, under --check too, and refuses host mode while any is not part of the node's own Compose project, naming each, which makes the engine dedicated to the node by construction.
8fa841a to
e4b16a7
Compare
… node's own container The single-quoted expansions in that command are for the shell inside the node's container, as for the other commands in the file that carry the same directive.
…mon in host_network assert_cluster read the joining servers' logs from the machine's own daemon, which holds none of the host_network scenario's node containers, so the check could never find the wait line there. The diagnosis dump had the same gap for the regrade container.
The failure dump showed the regrade service's status but not why a container that restarts in a loop exits, which left a regrade timeout without its cause.
… container template The checks registered the result of docker ps as github_runner_cluster_node_container, the role default that names the node's container, so everything rendered after them (the regrade service's environment file above all) received a dict instead of the name and the service crashed on start. They now register under a name of their own, and a test fails if any variable they register or set is a role default.
|
@codex security review |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.13.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
Stacked on #80 (version pins), which it needs: the Compose renders here set both pinned versions, and the role's checks run after the version checks.
Lets cluster nodes that share a LAN reach each other over it directly, with the mesh's relays as the fallback, without changing anything k3s uses. The role adds a network mode and the checks around it; moving a node onto a Docker VM with its own LAN address is a documented runbook, not automation.
Today every node's k3s container sits on Compose's bridge network behind the Docker host's NAT, and on macOS behind the Docker VM's too. So the Tailscale client inside it (k3s
--vpn-auth) has no address to offer its peers, and every pair of nodes is relayed, even on one LAN.Changes
github_runner_cluster_network_mode(bridge, the default, orhost).hostmergesdocker-compose.host-network.ymllast, putting thek3sandk3s-agentservices on the Docker host's network with no published ports. The node's Tailscale client then offers the host's own addresses. Node IP, etcd peer URLs, TLS SANs and the mesh identity are unchanged. In host mode the role checks that the API server answers on the host's loopback before using the kubeconfig there.--checktoo:bridgeorhost;github_runner_cluster_lan_cidr, and the Docker host must have an address in it, read from a throwaway container on the host network;github_runner_cluster_cluster_read_timeoutas a whole and as kubectl's--request-timeout, etcd's included, using the first that answers, and refuses unless the other servers whose node is Ready and whose API server reports/readyz/etcdok (read through each server's container on its host) still form a majority, and no unfinished ARC runner pod is on the node. Whatever cannot be told refuses, saying why: no server that answers (each named with its reason), no server in the inventory at all (an agentbootstrap.shjoins to an outside cluster), an unreachable Docker daemon, servers without their role labels, a missing or unreachable server, a server the inventory does not hold. Only one node may move per run, since each host's checks see the others as healthy before any is recreated.etcdctl, moving leadership off the node, an etcd snapshot, cordon and drain, carrying the node password, the bridged interface (and that it can change on a later restart, seen with a test profile only),COLIMA_HOME,--activate=falseand sizing, volumes created with Compose's labels, moving the engine first on the bridge network and then the network through the role, so the guard runs on the real move, identity checks after each (InternalIP, etcd member IDs and peer URLs, mesh device, no node-password event, the kubeconfig rewritten through the new VM's mount), starting the profile at login (whatbrew servicescovers and what it does not), and the rollback limits.StartVPNdoes:tailscale up ... --reseton every start.tailscaledUDP port: the client offers its real local endpoint whatever port it chose, and the mesh scenario reaches every peer directly on random ports.Tests
tests/unit/test_cluster_network.pyrenders the Compose project in both modes.tests/unit/test_cluster_guard.pycovers the guard: quorum for three and five servers, etcd unhealthy or unreachable, servers without labels, servers missing from the node list or from the inventory, runner pods, and LAN address matching.tests/unit/test_host_network_role.pyrunstasks/host_network.ymlagainst a stand-indocker: the mode check, the LAN and Tailscale device checks, a healthy move (including under--check), refusals for a not-Ready server, an unhealthy etcd, unlabelled servers, a runner pod and two nodes moving at once, a read through another server when this one cannot list the cluster, a refusal naming each server when none can, an agent reading through the servers of a group with no bootstrap host, an agent whose servers are outside the inventory, an unreachable daemon, and the skips for a node already on its network, stopped or missing. It also checks fromtasks/main.ymlthat the loopback API wait runs only on the host network and that the checks come before any change.tests/mesh/run.sh host_network(Linux only) runs each node in adocker:dinddaemon of its own, standing in for a VM with a LAN address: the usual cluster checks, every node reaching every other directly at its daemon's address, no published port, and a second run restarting nothing. Then the role must refuse to move two running nodes to the bridge network in one run without restarting anything, and move one node to the bridge network and back through its guard against the real cluster, the node keeping its mesh address and the cluster its three etcd members.--check,set -ein it, the loopback API wait's condition and the mode check.Not tested here
These need a bridged Colima VM on a real host: the runbook end to end, Lima forwarding the API server's port to the host's loopback, and the address probe on a real bridged VM. The role does not read which server leads etcd, so leadership is moved by hand in the runbook.