Skip to content

tune: fake-exporter cardinality down (5K aggregate target, was 25K+) - #352

Merged
zzylol merged 1 commit into
mainfrom
tune/fake-exporter-cardinality-down
May 9, 2026
Merged

zzylol merged 1 commit into
mainfrom
tune/fake-exporter-cardinality-down

Conversation

@zzylol

@zzylol zzylol commented May 9, 2026

Copy link
Copy Markdown
Contributor

Summary

The post-PR-#338 five-sketch workload was emitting ~2.46 MB/s SDK output, blowing up agent -> gateway bandwidth and amplifying every other bug. The HLL inner-label fan-out (user_id x outer label set) was the dominant contributor. This PR lowers two binary defaults so the demo lands at the documented ~5K aggregate-series gateway target.

Changes (current -> new)

knob file current default new default rationale
EXPORTER_FIVE_SKETCH_USER_POOL deploy/fake-exporter/five_sketch_workload.go 1000 (floor 500) 100 (floor 50) HLL still has a non-trivial active-user cardinality to estimate; per-min rotate semantic intact.
EXPORTER_CARDINALITY deploy/fake-exporter/main.go 1000 500 Direct-binary use now matches run_mvp_demo.sh's PER_AGENT_CARDINALITY=500 default; N_PRODUCERS=10 x 500 = 5K aggregate.
EXPORTER_FIVE_SKETCH_ENDPOINTS deploy/fake-exporter/five_sketch_workload.go 50 50 (unchanged) Already small; Zipfian heavy hitters dominate top-K anyway.
request_size_bytes scalar n/a n/a No cardinality knob.

Aggregate gateway cardinality (math)

  • 10 producers x PER_AGENT_CARDINALITY=500 outer label sets = 5,000 outer series (target).
  • HLL unique_users_per_min: ~5,000 outer x ~100 active user ids per rotation window, with rotation moving the active window through the pool — bounded inflation.
  • CountSketch / CountMinSketch endpoints: 5,000 x ~50 = ~250K event keys but Zipfian heavy hitters concentrate on ~5 endpoints, so series count is bounded by 50 endpoint suffixes per outer.
  • KLL request_size_bytes: scalar per outer = 5,000 series.

Pre-tune we were at 10K-25K+ series; this brings the gateway closer to the runbook's 5K target line.

Driver / config sanity

  • deploy/scripts/run_mvp_demo.sh line 111: PER_AGENT_CARDINALITY=500 already matches.
  • deploy/docker-compose/mvp-multi-stage.yml: per-producer env propagates PER_AGENT_CARDINALITY:-500. No override above 500.
  • deploy/configs/mvp-workload.yaml: no cardinality pin.
  • EXPORTER_FIVE_SKETCH_USER_POOL / _ENDPOINTS are not set externally anywhere — the new binary defaults take effect.

Don't / scope guard

  • Metric SHAPES preserved: still request_size_bytes (KLL gauge), unique_users_per_min (HLL counter w/ user_id), top_endpoint_qps / endpoint_request_freq (CountSketch / CMS counter w/ endpoint).
  • gorillas3processor untouched (parallel agent owns OOB fix).
  • Static placeholder yaml untouched (parallel agent).

Test plan

  • go test ./deploy/fake-exporter/... — pass (all 4 metrics still emit; existing test uses explicit USER_POOL=500 so floor change is non-breaking).
  • go build ./deploy/fake-exporter/... — pass.
  • Local binary smoke — startup log confirms cardinality=500 users=100 endpoints=50 outer_card=500 (was cardinality=1000 users=1000).
  • Full demo run (deferred — out of scope per task instructions).

Refs #46.

Generated with Claude Code (1M context).

The five-sketch workload's HLL inner-label fan-out (user_id × outer
label set) was the dominant SDK output contributor, pushing the
post-PR-#338 demo to ~2.46 MB/s and blowing up agent → gateway
bandwidth. Lower the two defaults that drive that fan-out:

- EXPORTER_FIVE_SKETCH_USER_POOL: 1000 -> 100 (floor 500 -> 50).
  HLL still has a non-trivial active-user cardinality to estimate
  (100 distinct ids), and the rotate window keeps the per-minute
  semantic intact.
- EXPORTER_CARDINALITY (binary default): 1000 -> 500 so direct-binary
  use also matches the demo script's PER_AGENT_CARDINALITY=500
  default. With N_PRODUCERS=10 the aggregate gateway cardinality
  lands at ~5K series (was 10K-25K+).

top_endpoint_qps / endpoint_request_freq stay at 50 endpoints
(already small); request_size_bytes is scalar.

PER_AGENT_CARDINALITY=500 in run_mvp_demo.sh and the
mvp-multi-stage compose env propagate already match the new
binary default; mvp-workload.yaml does not pin cardinality.

Verified with go test ./deploy/fake-exporter/... and a local
binary smoke (startup log shows cardinality=500 users=100
endpoints=50).

Refs #46.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant