fix(control_plane): retry transient errors on startup POST to backend (404/connect-refused)

[zzylol]

Race we are patching

In the multinode harness (asap arm from ASAPCollector PR #394), the controller and backend start concurrently. We observed:

backend logs HTTP server listening on port 9091 at 15:06:51 but the streaming-config route registers a few hundred ms later; controller POSTs land at 15:06:56 right after with_hot_reload_config but apparently before the route is reachable from across the host. All 7 startup POST /api/v1/streaming-config calls returned 404.

Convergence today depends on OpampServer::on_connect (added in PR #290) re-firing replan_all() when the first agent connects. That's a workaround — the first POST is wasted, and any deployment without OpAMP connects (or where the controller restarts before agents reconnect) hangs.

Fix

Add retry-with-backoff for transient errors in control_plane/src/emit/backend_push.rs::post_typed_backend_for_role (the single helper PR #290 unified both call sites through).

Transient (retry): 404 (route not yet registered), 5xx, TCP connection-refused / DNS / connect timeout, reqwest IsTimeout/IsConnect/IsRequest.
Permanent (no retry): 4xx other than 404.

Policy: 5 attempts, exponential 100ms -> 300ms -> 900ms -> 2.7s (capped) with full jitter, total span ~5-8s. Bounded by RETRY_MAX_ATTEMPTS=5 and RETRY_DELAY_CAP=2.7s. The lock on backend_routing_cache is dropped before any HTTP call (preserved from PR #290's pattern), so retries never stall sibling replan cycles.

Public-interface preservation: BackendClient's existing methods are untouched. New typed variants post_streaming_config_json_typed / post_storage_routing_json_typed returning Result<(), BackendPostError> sit alongside; the retry layer uses only those. OpampServer::on_connect re-fire is preserved as the secondary convergence mechanism.

Tests

• cargo build -p control_plane — clean
• cargo test -p control_plane --lib — 748 passed, 0 failed
• New tests in emit::backend_push::tests:
  • retry_transient_recovers_after_first_404 — closure returns 404 then 200; asserts attempts > 1 and final Ok (the explicit ask in the task)
  • retry_transient_does_not_retry_permanent_errors — closure returns 400; asserts attempts == 1
  • retry_transient_exhausts_and_reports_attempts — closure returns connection-refused 5x; asserts attempts == 5 and final Transient Err
  • retry_transient_no_retry_on_first_success — protects the smoke-test happy-path invariant
  • backoff_delay_respects_cap — every attempt's delay <= cap
• New tests in backend_client::tests:
  • typed_streaming_config_404_is_transient
  • typed_streaming_config_500_is_transient
  • typed_streaming_config_400_is_permanent
  • typed_connection_refused_is_transient
  • typed_streaming_config_success_is_ok

All five retry-loop tests use tokio::test(start_paused = true) so they execute in microseconds despite the multi-second nominal backoff window.

Test plan

• cargo build -p control_plane: clean
• cargo test -p control_plane --lib: all pass
• Unit test: 404 then 200 -> attempts > 1, final Ok
• Multinode verification (optional, requires bring-up of asap arm) — confirm controller's first POST eventually succeeds without relying on OpAMP-on-connect re-fire
• Smoke test regression (optional) — confirm fire-and-forget happy path is unchanged when backend is up before controller's first POST