Skip to content

[STACKED on #891] fix(guest): make KMS failover ordering deterministic - #946

Closed
kvinwang wants to merge 2 commits into
masterfrom
codex/fix-guest-kms-failover-order
Closed

[STACKED on #891] fix(guest): make KMS failover ordering deterministic#946
kvinwang wants to merge 2 commits into
masterfrom
codex/fix-guest-kms-failover-order

Conversation

@kvinwang

@kvinwang kvinwang commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

STACKED PR — merge #891 first.

Problem

KMS endpoint failover depended on map/inventory iteration order. Different runs could choose different first endpoints, causing nondeterministic latency and making recovery behavior change after restart.

Root cause and fix

Sort/retain an explicit endpoint order and apply failover in that deterministic sequence.

Implementation

The branch records the following focused implementation work:

  • fix(guest): make KMS failover ordering testable
  • style(guest): format KMS failover tests

Changed paths:

  • dstack/dstack-util/src/system_setup.rs

Scope

This PR addresses one logical guest finding. It intentionally excludes the acceptance-test infrastructure from #841 and unrelated product fixes from #840.

Dependency and merge order

Verification

  • git diff --check origin/codex/fix-guest-gateway-private-state..origin/codex/fix-guest-kms-failover-order: passed.
  • The declared base was verified as an ancestor of the PR head.
  • Focused compile/check verification was run for the changed component where applicable; non-Rust packaging or configuration changes were reviewed against their exact branch delta.

Copilot AI review requested due to automatic review settings July 31, 2026 03:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@kvinwang kvinwang changed the title fix(guest): make KMS failover ordering deterministic [STACKED on #891] fix(guest): make KMS failover ordering deterministic Jul 31, 2026
@kvinwang
kvinwang force-pushed the codex/fix-guest-kms-failover-order branch from 4c66ddd to 15efa30 Compare July 31, 2026 03:25
@kvinwang
kvinwang changed the base branch from master to codex/fix-guest-gateway-private-state July 31, 2026 03:26
@kvinwang
kvinwang force-pushed the codex/fix-guest-gateway-private-state branch from 8715ee1 to 73a7ff8 Compare August 4, 2026 08:49
Base automatically changed from codex/fix-guest-gateway-private-state to master August 5, 2026 00:51
@kvinwang
kvinwang force-pushed the codex/fix-guest-kms-failover-order branch from 15efa30 to 7adae4d Compare August 5, 2026 09:02
@kvinwang

kvinwang commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

Closing because the existing implementation already performs deterministic, sequential failover in configured URL order. This change primarily refactors the loop for testing and does not address the claimed inventory-ordering root cause. URL normalization, if needed, should be proposed separately as a focused change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants