Skip to content

Use bounded rate-limit-aware backoff for real Vuforia HTTP 429 responses #3768

Open

Description

@adamtheturtle

The real-Vuforia fixture retry policy treats HTTP 429 like other transient failures and sleeps a fixed 10 seconds between attempts. Make rate-limit recovery explicitly bounded and responsive to the service's retry guidance.

Evidence:

  • CI job on PR #3766 exhausted the outer fixture policy at attempt 10 after 90 seconds of retry sleep. The final GET /summary/{target_id} returned HTTP 429 with an empty body and x-envoy-ratelimited: true; it had no Retry-After header. Earlier attempt outcomes are not all shown in the traceback.
  • tests/mock_vws/utils/retries.py explicitly includes TooManyRequestsError and uses stop_after_attempt(10) with wait_fixed(10). This policy is also present on current main.
  • The PR description records a separate isolated rerun that still exhausted retries with the same throttle. This does not establish that local job concurrency caused the rate limit, or that changing backoff alone will clear it.

Retrying a 429 after waiting is reasonable. However, a fixed interval neither uses a supplied cooldown nor spreads retry timing across workers. RFC 6585 section 4 permits a Retry-After header specifying how long to wait.

Acceptance criteria:

  • Honor valid Retry-After values on 429 responses, including delay-seconds and HTTP-date forms. When the requested cooldown exceeds the remaining retry deadline, stop and surface the throttle rather than retrying earlier than requested.
  • When the header is absent or invalid, use capped exponential backoff with jitter and an explicit overall retry budget. Avoid multiplying nested budgets into unexpectedly long fixture setup.
  • Preserve final exception and response details on exhaustion, and report useful attempt/delay information without credentials, so CI shows whether a throttle persisted.
  • Keep real-service retry policy owned by the real backend as the shared suite is decoupled under Decouple shared API tests from concrete mock implementations #3738. Preserve deterministic mock tests of intentional error responses.
  • Verify delay selection, header parsing, jitter bounds, and exhaustion without live credentials or real sleeping. Document the chosen delay limits and overall budget.
  • Treat persistent throttling as a failed live-service check. Any CI concurrency or request-pacing change should follow evidence about the limiter's scope; a successful rerun must not be assumed.

Separate from #3767, which fixes retries recreating targets after read failures. This issue concerns cooldown and retry timing, not widening which operations are safe to repeat.

Historical context: #3540 originally proposed a backoff change, but its closing comment explicitly corrected that diagnosis: those malformed-request delays were server response latency, not 429 retry sleep. The failure linked above provides direct 429 evidence; #3540's original timing inference should not be reused.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions