Replace actor eth0 move with veth networking by EItanya · Pull Request #1 · kagent-dev/substrate

EItanya · 2026-05-29T13:38:46Z

Summary

This replaces the ateom-gvisor networking path that moved the worker pod's Kubernetes-provided eth0 into the actor/gVisor network namespace.

Instead, the worker pod keeps its real eth0, and ateom-gvisor creates a point-to-point veth pair between the worker pod namespace and the actor namespace. The actor-side peer is renamed to eth0, receives the actor-side address, and uses the worker-side veth as its default gateway.

The PR also adds temporary nftables compatibility rules so existing inbound and outbound behavior continues to work while preserving the worker pod's own network connectivity.

Why

Moving the pod's real eth0 makes the worker pod lose normal Kubernetes network connectivity while an actor is active. That blocks pod-local networking components, including the planned transparent egress capture and AgentGateway integration, because those components remain in the worker pod namespace while actor traffic leaves through an interface that was moved elsewhere.

Keeping eth0 in the worker pod namespace gives Substrate a stable worker-owned networking boundary for future transparent egress policy enforcement.

Validation

go test ./cmd/ateom-gvisor ./cmd/atelet ./internal/ateompath ./internal/controllers
go test ./cmd/ateom-gvisor ./internal/serverboot
NO_DEV_ENV=true BUCKET_NAME=ate-snapshots KO_DOCKER_REPO=localhost:5001 KUBECTL_CONTEXT=kind-kind ./hack/run-e2e.sh ./internal/e2e/suites/demo -run TestDemo3 -count=1

Checklist

Tests pass
Appropriate changes to documentation are included in the PR

What was happening: 1. runsc checkpoint pause succeeded. 2. After that, substrate tried to run runsc state and runsc delete for the app container. 3. But after checkpointing the root sandbox container, the runsc control server was no longer usable, so those post-checkpoint commands failed with connection refused. 4. That made CheckpointWorkload return an error even though the actual checkpoint had already succeeded. 5. The ActorTemplate stayed stuck in WaitGoldenActor. The change removes the post-checkpoint state/delete calls from ateom-gvisor. After checkpoint succeeds, it only cleans up the actor network and returns success. This matches the existing contract in the code: atelet owns resetting the actor directories after checkpoint/upload, and it already calls resetActorDirs(...). Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

* Replace actor eth0 move with veth networking * Group nftables rules with chain definitions * Address actor inbound networking TODOs * Trim actor networking comment * Update licenses for nftables dependency * Fix gvisor cleanup issue What was happening: 1. runsc checkpoint pause succeeded. 2. After that, substrate tried to run runsc state and runsc delete for the app container. 3. But after checkpointing the root sandbox container, the runsc control server was no longer usable, so those post-checkpoint commands failed with connection refused. 4. That made CheckpointWorkload return an error even though the actual checkpoint had already succeeded. 5. The ActorTemplate stayed stuck in WaitGoldenActor. The change removes the post-checkpoint state/delete calls from ateom-gvisor. After checkpoint succeeds, it only cleans up the actor network and returns success. This matches the existing contract in the code: atelet owns resetting the actor directories after checkpoint/upload, and it already calls resetActorDirs(...). Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io> * Address actor network cleanup review * Fix ateom cleanup failure handling --------- Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>

…-substrate#1002) Fixes agent-substrate#1001 `TestK8sResolverEndpointSliceUpdates` intermittently fails at `resolver_test.go:215` — 8 of the 30 most recent failed `pr-workflow` runs (~27%): ``` updated state.Addresses = [{Addr: "10.0.0.1:443", ServerName: "", }], want [{Addr: "10.0.0.1:443", ServerName: "", } {Addr: "10.0.0.2:443", ServerName: "", }] ``` `Build` starts a goroutine that calls `updateState` once `WaitForCacheSync` returns. That report is deliberate: a service with no EndpointSlices never fires `AddFunc`, so without it the resolver would stay silent instead of telling gRPC the answer is an empty set. But `WaitForCacheSync` polls at `syncedPollPeriod = 100ms`, so it fires roughly 100ms after Build — and if the test has not yet created the second slice by then, that update still carries only `10.0.0.1` and sits in the channel ahead of the real one. The second `select` took whatever came next, so it asserted against the stale update. Locally Build-to-Create is 0.6ms, well ahead of the timer, which is why this only shows up on loaded runners. The resolver is not at fault — it promises eventual convergence, not that the first update after a change is final, and a duplicate update costs gRPC nothing. So both waits now go through one `waitForAddrs` helper that consumes updates until the set matches, with a timeout so a genuinely broken resolver still fails and reports the last set it saw. Note this changes the first wait as well: it asserted the *first* update equals `[10.0.0.1]`, and now waits for that set instead. The same argument applies there — nothing promises the first update is final. ## Verification `-count=N` proves nothing here: the unfixed test passes locally at any count because the window is never hit. A/B with the Build→Create delay as the only variable: | delay | old assertion | new assertion | |---|---|---| | 0ms | 5/5 pass | 5/5 pass | | 150ms | **0/5 pass** | **5/5 pass** | Measured timeline with the 150ms stall in place: ``` [ 0.5ms] update #1: [10.0.0.1] <- AddFunc for slice1 first select takes it ... 150ms stall ... [101.1ms] update #2: [10.0.0.1] <- the WaitForCacheSync goroutine test creates slice2 [151.4ms] update #3: [10.0.0.1, 10.0.0.2] ``` The probe tests used for this are not included. Rebased over agent-substrate#1013. That fixes a different bug — concurrent `updateState` calls letting an older address set win — and does not close this one: the two updates here are ~100ms apart, so the queue has nothing to coalesce. Re-measured on top of it, unchanged: at a 150ms delay the old assertion is 0/5 and the new one 5/5.

EItanya force-pushed the transparent-egress-veth-networking branch from b66b5d1 to 2e8c8fd Compare May 29, 2026 13:59

EItanya force-pushed the main branch 2 times, most recently from e12128d to 727d8aa Compare June 4, 2026 16:25

EItanya force-pushed the transparent-egress-veth-networking branch from 2e5e6c7 to 3f5cbae Compare June 5, 2026 15:54

EItanya force-pushed the main branch from a6d1403 to 63ea2b0 Compare June 8, 2026 13:16

EItanya added 7 commits June 8, 2026 22:26

Replace actor eth0 move with veth networking

1f954be

Group nftables rules with chain definitions

c927a1f

Address actor inbound networking TODOs

64a52a5

Trim actor networking comment

6210b69

Update licenses for nftables dependency

d5722a1

Address actor network cleanup review

326110a

EItanya force-pushed the transparent-egress-veth-networking branch from 3f5cbae to 326110a Compare June 8, 2026 22:27

EItanya added 2 commits June 9, 2026 11:59

Fix ateom cleanup failure handling

7cffadd

Merge upstream main into transparent egress branch

e8b03ee

EItanya force-pushed the main branch from 63ea2b0 to 3c8f321 Compare June 9, 2026 12:13

EItanya merged commit ef1159f into main Jun 9, 2026

Navigation Menu

Uh oh!

Replace actor eth0 move with veth networking - #1

Replace actor eth0 move with veth networking#1
EItanya merged 9 commits into
mainfrom
transparent-egress-veth-networking

EItanya commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

EItanya commented May 29, 2026

Summary

Why

Validation

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant