eval(real-workload): google cluster trace fetcher + OTLP mapper + queries

[zzylol]

Summary

• Adds datasets_eval/google_cluster/ — fetcher + OTLP mapper + matching PromQL log for the public Google cluster traces (2011 task_usage + 2019 instance_usage). Closes paper blocker add CountMinSKetch in processor #6.
• Streaming HTTPS fetch from public GCS buckets (no auth, no gsutil); sha256-checksummed cache + manifest for idempotence; documented head-subsample so the smoke flow stays under 50 MB of network traffic.
• OTLP mapper produces deterministic JSONL matching deploy/fake-exporter/'s {zone, rack, host, service, task} attribute schema; --cardinality-cap N projects tuples onto an N-element hashed subset for the 1k/10k/100k sweep matrix.
• 10-entry queries.json covering all five evaluation claims (quantile / topk / sum / count_unique), schema-compatible with deploy/scripts/queries-e2e.json plus an expected_ground_truth_query per entry for the accuracy reducer.
• 16 unit tests: golden mapper output across both trace years + queries.json schema cross-check against the reference file.

Subset selection rationale

• 2011: first part of task_usage (gs://clusterdata-2011-2/task_usage/part-00000-of-00500.csv.gz). Per-(machine, job, task) 5-min CPU/memory samples — directly maps to OTLP gauges keyed by the natural workload-resource shape for ASAP.
• 2019: first part of instance_usage cell-a (gs://clusterdata_2019_a/instance_usage-000000000000.json.gz). Per-Borg-instance CPU/memory sampling, same shape concern at larger scale.
• Default --max-rows 100_000 is the documented paper-experiment subsample; smoke --max-rows 1_000 runs fit on a laptop in ~10 s.

Cardinality-cap projection bias

• For trace cardinality U and cap N: U ≤ N -> identity (zero bias); U > N -> ~U/N collisions per cell, ~√(N/U) RSD on per-cell sample counts (Poisson approx). At default N=1000 with U≈1e7, RSD is ~1 %.
• count_unique queries saturate at N when capped — accuracy reducer rescales by U/N to recover true cardinality (documented in queries.json).

Mapper output shape vs fake-exporter

• Compatible. Same five attribute keys; same per-event two-family pattern (cpu_rate gauge + memory_usage gauge mirror fake-exporter's counter + latency gauge).
• zone and rack are synthesized from machine_id because the trace doesn't expose those columns directly; deterministic salted blake2b hash, documented in README.md's "OTLP wire shape" section.
• No new ingest endpoint required — replay uses the existing OTLP/gRPC port on the agent.

Files added (only datasets_eval/google_cluster/ touched)

• README.md, fetcher.py, otlp_mapper.py, queries.json, run.py
• tests/test_otlp_mapper.py, tests/test_queries_schema.py
• tests/fixtures/{2011,2019}_*.csv + *_expected_*.jsonl golden files

Verification (all pass)

• python3 datasets_eval/google_cluster/fetcher.py --year 2019 --max-rows 1000 --out-dir /tmp/gct -> 1000 rows, sha256 e2ec23b0...
• python3 datasets_eval/google_cluster/otlp_mapper.py --year 2019 --in-dir /tmp/gct --cardinality-cap 1000 --out /tmp/gct-otlp.jsonl -> 2000 OTLP rows
• python3 datasets_eval/google_cluster/run.py validate --queries datasets_eval/google_cluster/queries.json -> OK
• pytest datasets_eval/google_cluster/tests/ -> 16 passed

Disk + wall-time budget

Run	Disk	Wall
smoke (--max-rows 1k, either year)	<2 MB	~5–10 s
default (--max-rows 100k)	~30–80 MB	~30–60 s
full 2011 (all 500 parts)	~40 GB	~6 h
full 2019 cell-a	~250 GB	~24 h
full 2019 (8 cells)	~2 TB	~1 wk (not recommended)

Open questions / interpretation choices

• zone/rack are synthesized from machine_id. The 2011 trace's machine_events and 2019's cluster-cell columns could supply real placement, but joining them adds another fetch. Hash-from-machine-id is reproducible and roughly uniform; flagged for future improvement if the paper requires real placement.
• 2011 timestamps are µs-since-trace-start (relative); we convert /1000 -> ms. Inter-arrival structure is preserved; absolute wall times are not real-world dates. Same convention for 2019.
• "Natural" Google-trace queries chosen: per-instance CPU/mem quantiles, per-zone roll-ups, top-K busy hosts, distinct-service cardinality. If the paper wants service-runtime quantiles or job-scheduling metrics, those would need the task_events join (out of scope for this PR).

Test plan

• 4 verification commands from the prompt all pass
• 16 unit tests pass (10 mapper + 6 schema)
• Idempotence: re-running fetcher hits the sha256 cache check, no network IO
• Determinism: byte-stable mapper output across runs at fixed flags
• Cardinality-cap saturation: cap=2 with U=4 -> ≤2 distinct cells
• File domain: only datasets_eval/google_cluster/ modified

🤖 Generated with Claude Code