Skip to content

supervisor: route.closing and route.closed so a reload's outcome is knowable - #31

Merged
ualtinok merged 7 commits into
cortexkit:masterfrom
iceteaSA:feat/route-closing
Aug 17, 2026
Merged

ualtinok merged 7 commits into
cortexkit:masterfrom
iceteaSA:feat/route-closing

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Closes #22, and closes the buildable half of #5. subc's first daemon→client channel-0 push family.

The bool wait_for_forwarding_quiescence returns now reaches the client instead of only a warn!.

The two emissions, in the drain block

if let Some(target) = drain_target.as_ref() {
    let routes = forwarding.endpoint_routes(target.endpoint)?;
    send_route_control_pushes(forwarding, routes.clone(),
        ClientControlPush::RouteClosing { module_id, reason });

    send_route_goodbyes(forwarding, target.abandoned_bindings.clone());

    let drained = wait_for_forwarding_quiescence(forwarding, target.endpoint, drain_timeout).await?;

    send_route_control_pushes(forwarding, routes,
        ClientControlPush::RouteClosed { module_id, reason, drained,
                                         abandoned: target.abandoned_bindings.len() as u32 });

    let released_routes = forwarding.release_module_endpoint_routes(target.endpoint)?;
    send_route_goodbyes(forwarding, released_routes);
    send_module_goodbye(&spec.module_id, forwarding, target);
}

drained is bound once from the wait and passed straight through — your criterion, and it's structural rather than promised: there is no second source it could come from.

endpoint_routes lost its #[allow(dead_code)]; the consumer it was reserved for is here.

Your abandoned-set requirement, and what the build found

The finding is sharper than my brief's premise, and it's worth reading before the rest.

endpoint_routes covers exactly the committed routes that release_module_endpoint_routes later releases — but it excludes pending route.bind relays, which are the abandoned set. Those have a module-only GoodbyeTarget, and their clients already receive module_reloading rather than a route GOODBYE. So they were never addressable by this push in the first place.

That makes abandoned: u32 the honest shape rather than the fallback you allowed: the count says "N binds were forced down before the wait, and drained says nothing about them." A drained: true cannot paint over them because they are structurally outside the set the push addresses.

Crash asymmetry

Emitted from cleanup_connection, not the drain path:

ClientControlPush::RouteClosed { module_id, reason: RouteCloseReason::Crash,
                                 drained: false, abandoned: 0 }

No RouteClosing precedes it — there are only two RouteClosing emission sites and both are in the planned drain. So closed with no prior closing means nobody planned this, as normative text rather than convention.

Contract text

route.closing { module_id, reason } is enqueued before a planned drain starts.
  reason is reload, restart, or disable; it makes no completion claim.

route.closed { module_id, reason, drained, abandoned } is enqueued after the
  forwarding-quiescence wait and before the released-route GOODBYEs. drained is
  the exact boolean returned by that wait, not a later route-state inference.
  abandoned counts pending route.bind relays forced down before the wait; they
  are never covered by drained.

Enqueue-order is the ONLY claim. FrameSink::send enqueues; it does not complete
  the write. So "pushed before the GOODBYE" can only ever mean "enqueued before".

Plus the daemon-originated-only property pinned, per #6: there is no module→client push relay to forge through, and the push constructor lives daemon-side, so a module cannot express one on its control channel. First instance of this direction is the right moment to write that down.

Zero structural changes

No ctx.egress threading — git diff origin/master -- crates/ | grep '^+.*ctx\.egress' returns nothing. Emission goes through per-connection FrameSinks exactly as send_route_goodbyes already did three lines above the wait. supervisor.restart keeps its completion semantics and the supervisor op mutex is untouched, per your #5 disposition.

The ordering test

route_lifecycle_enqueues_closing_drain_closed_then_released_goodbyes reads frames sequentially from one client connection:

closing  →  the in-flight response completing (the drain)  →  closed  →  GOODBYE

Single ordered stream, so any reorder fails an assertion rather than needing a separate order check. 10/10 in a loop.

Also covered: timed-out drain yields drained: false, quiesced yields drained: true, both from the real wait; crash yields closed with no closing; and a disable path asserting the route.closing body field-by-field.

Gates

cargo test --workspace              35 binaries, 0 failures
cargo clippy --all-targets          clean
cargo fmt --check                   clean
cargo check --workspace --locked    clean
check-wire-crate-versions.sh        6 examined, 0 violations

Golden diff is three op-list insertions, nothing else. subc-control → 0.4.0, subc-core → 0.6.0, subc-client-rs caret pin → "0.4".

Windows unverified — twin whenever convenient.

One note

I previously told you ctx.egress threading was the structural change this needed, you accepted it, and I retracted it. This PR is the retraction being true: the brief for it said do not build that, and stop if you find yourself needing it. Nothing needed it.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith with what you need. Autofix is disabled.


Summary by cubic

Adds daemon→client channel‑0 pushes for route teardown so clients can observe reload outcomes. Previously clients only saw GOODBYE and the quiescence result was log‑only; now they receive route.closing before a planned drain and route.closed after, including whether live routes drained and how many pending binds were abandoned. Crashes emit route.closed { reason: crash, drained: false, abandoned: 0 } with no prior route.closing.

  • Emits for reload/restart/disable: route.closing then route.closed; for crash: route.closed only. Both are enqueued on channel 0 before released‑route GOODBYEs.
  • drained is the exact wait_for_forwarding_quiescence result; abandoned counts pending route.bind relays forced down and is not covered by drained.
  • Pushes are serialized once and enqueued per affected route via existing FrameSinks; on enqueue failure, the target connection may be closed per sink policy.
  • Crash path sends route.closed during connection cleanup; no structural forwarding changes.

Review notes

  • supervise.rs: begins drain by sending ClientControlPush::RouteClosing, waits, then sends RouteClosed and GOODBYEs; reason threaded via RouteCloseReason.
  • control.rs: send_route_control_pushes helper; crash emission in handle_connection_closed.
  • subc-control: adds ClientControlPush and RouteCloseReason; ops list exposes route.closing/route.closed; docs updated.
  • Tests: ordering and semantics added in forwarding.rs and daemon_config.rs; stub gains FAKE_AFT_BIND_NEVER_REPLY_AFTER to exercise abandoned binds.

Rollout

  • Update dependencies: subc-control0.4.0, subc-core0.6.0; subc-client-rs pins subc-control "0.4".
  • Client action optional: channel‑0 pushes are additive; existing clients may ignore them. To surface better UX, handle route.closing/route.closed.

Written for commit 7dbda42. Summary will update on new commits.

Review in cubic

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GOODBYE cannot say whether the drain finished, so every client must treat a clean reload as outcome-unknown

2 participants