mvp/v5: postings index + concat-only compactor + agent + demo (issue #46) - #295
Merged
Merged
Conversation
…issue #46 v5) Adds the four code pieces the v5 MVP needs from the collector side: 1. **Postings index** (`asap-gorilla::postings`). Per-block sidecar keyed by `(label_name, label_value) → series_ids`, JSON body framed by `POSTING1` magic + version + CRC. `Postings::merge_many` is load-bearing for the compactor — set-union per key, sorted + deduped output. 2. **Chunk-manifest extension** (`asap-gorilla::IndexEntry`). Adds `object_key` / `byte_offset` / `byte_length` so the backend can issue `Range:` partial reads against compactor-merged blocks. Backward-compat: legacy fixtures parse with all three at `None` and resolve to "fetch whole object at `key`". 3. **Concat-only compactor** (`compactor/`). New `asap-compactor` binary + lib. Walks the bucket, finds 6+ adjacent ≥ 6h-old per-hour blocks, byte-concatenates the source `part-*.gor` files into one merged-block object, merges postings via `Postings::merge_many`. Chunks are NEVER decoded or re-encoded — concat-only is the entire design (see `src/lib.rs` for rationale). Idempotent re-run, `--dry-run`, verify partial-read. 4. **Agent emission** (`gorillas3processor`). `postings.go` writes the `postings-v1.json` sidecar per flush. `s3_sink.go` extends `chunkSink` with `PutPostings` and stamps the new `object_key/byte_offset/byte_length` fields on the agent-side `index.json`. 5. **Demo updates** (`deploy/scripts/run_mvp_demo.sh`, `mvp_report.py`). v5 phases: ad-hoc PromQL with label predicates (criterion ⑦), compactor dry-run + live run with before/after MinIO listing, per-baseline `s3_cost.csv`. Report grows §5/§6/§7 plus criterion ⑦ + cost-projection annex. Test summary: * asap-gorilla: 39 tests pass (release build — debug-mode smoke_bench has a hardware-dependent threshold). * compactor: 11 tests pass (plan + run + idempotency + partial-read + concat-only). * gorillas3processor: 4 new postings tests pass; pre-existing `decodeChunk` tests fail same as origin/main (separate bug). Cross-repo dependency: `asap_gorilla::Postings` is consumed by the matching `mvp/v5-postings-aware-engine` PR in ASAPQuery-backend. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compactor binary is specifically for Gorilla chunks (concat-merge of Gorilla chunks into larger blocks + postings merge). Rename to gorilla-compactor for clarity — leaves room for any future non-Gorilla compaction tooling. Mechanical rename: - compactor/Cargo.toml: package + [[bin]] name + bin path - compactor/src/bin/asap-compactor.rs → src/bin/gorilla-compactor.rs - compactor/Cargo.lock: package name - compactor/**/*.rs: asap_compactor:: → gorilla_compactor:: - deploy/scripts/run_mvp_demo.sh: COMPACTOR_BIN default path cargo check passes. Tests still pass (run agent will re-verify). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
marked this pull request as ready for review
May 6, 2026 21:22
This was referenced May 6, 2026
zzylol
added a commit
that referenced
this pull request
May 6, 2026
* mvp v6 phase E: add controller-driven multi-stage demo driver DRAFT — does not run the demo. Phase F brings up the stack and posts numbers once v5 PRs (#295 ASAPCollector, #90 ASAPQuery-backend) land. run_mvp_demo_v6.sh is independent from v5's run_mvp_demo.sh (which is in active use by the v5 demo run, PID 1945011) — separate file, separate output dir, separate report. Eight phases: 0. preflight (docker, docker-compose, compactor binary check) 1. stack up via base.yml + mvp-v6-multi-stage.yml + USE_TYPED_STAGE_SPLIT=1 2. measurements (replay + measure_stages.py + measure_per_edge_bandwidth.py) 3. freshness via run_freshness_phase.sh (raw/warm/archive) 4. ad-hoc postings exercise (count + topk-5xx) 5. cold-fallback verification (gorilla_archive marker) 6. compactor concat-only (dry-run + live) 7. teardown 8. mvp_report_v6.py reduce → MVP_REPORT_v6.md Captures controller-emitted runtime configs from /api/v1/collector-config/ and /api/v1/config/<metric>; falls back to placeholder gateway+agent configs when typed-stage-split returns None (logged as fallback-placeholder STATUS marker for the report). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * mvp v6 phase E: add per-edge bandwidth measurement script Extends measure_stages.py's docker stats sampling with per-edge labelling for the v6 fan-in topology: edge_sdk_to_agent (10 producers TX) edge_agent_to_gateway (2 agents TX) edge_gateway_to_backend (backend RX — distinguishes the gateway's split egress) edge_gateway_to_s3 (minio RX — same rationale) Output schema: edge,sample_ts_ms,window_s,bytes_total,bytes_per_s. One row per (edge, sample) for a per-edge time-series; mvp_report_v6.py reduces to per-edge mean B/s for criterion ① bandwidth. Stdlib only; same `docker stats --no-stream` mechanism as measure_stages.py so two probes can run in parallel without docker-daemon contention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * mvp v6 phase E: add MVP_REPORT_v6.md generator Independent from v5's mvp_report.py (which is in active use, PID 1945011). Eight sections: §1 Stage-separated resource table (per-stage TOTAL across containers) §2 Six-criterion verdict table (bandwidth/latency/resource/accuracy/ cold-fallback/freshness) §3 Per-query-class breakdown (window/label/combined latency + accuracy) §4 Postings filtering effect (gated on v5 merge — renders v5-merge-pending markers if response fields absent) §5 Compaction effect (before/after MinIO object count + bytes) §6 S3-ops cost (gated on v5 cost tracker endpoint) §7 Honest caveats (no replan, no hot reconfig, 10K not 1M, single host) §8 Controller-emitted runtime configs status (live / fallback-placeholder / not-exercised) Pure stdlib, idempotent. Reads the v6 results dir layout written by run_mvp_demo_v6.sh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * mvp v6 phase E: add unit tests for mvp_report_v6.py Hermetic — no compose stack, no docker. Synthesises a minimal v6 results directory layout (with mock CSVs / JSON / JSONL fixtures clearly marked synthetic) and asserts: * happy-path: all 8 sections present, all 6 criteria rows in §2, all 5 stages in §1, all 3 query classes in §3 * idempotency: re-running over the same CSVs produces byte-identical MD * sparse fixture (only stages.csv): renderer still produces complete MD with UNKNOWN verdicts, v5-merge-pending markers in §4 / §6 * fallback emitter status renders correctly in §8 * missing --results-dir produces self-explanatory MD rather than crashing * helper unit tests (per-edge mean, query-class classifier, minio-listing counter, postings field extractor) 9 tests, all pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 7, 2026
The original 3-phase plan (always-archive at gateway / delete JSONL / PromQL completeness) is partially superseded by what shipped via v5/v7. Updated to reflect what's already on main and what genuinely remains: - "Always-archive at gateway" effectively shipped via dual-routing (PR #91): BackendStorageRouting now allows multi-target per metric - The archive-tier engine has gained postings filtering (PR #295), partial-S3 reads (PR #295), concat-only compactor (PR #295), and freshness-pattern registration (PR #91) Two outstanding items remain: - Delete the JSONL cold-fallback (now safely unreachable under normal routing) - PromQL completeness on GorillaQueryEngine (Path A: vendor prometheus/promql via sidecar; Path B: pure-Rust evaluator; Path C: curated subset extension) Adds a "what's already on main" diff table at the top so reviewers immediately see what's done vs. what's outstanding. Adds a sentence about Prometheus-block-compatible layout potentially letting Thanos store-gateway answer queries directly (cross-references the comparison doc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
3 tasks
zzylol
added a commit
that referenced
this pull request
May 7, 2026
… on GorillaQueryEngine (#292) * docs: design for JSONL deprecation + always-archive Gorilla-S3 + PromQL completeness Three-phase proposal to collapse ASAP's cold-fallback tier (JSONL) into the Gorilla-S3 archive tier: - Phase 1: always-archive every metric at the gateway (mirrors Databricks' Hydra always-streaming pattern); ~1-2 days - Phase 2: delete the JSONL path (LocalFsColdStore, parse_jsonl, raw-tee exporter, StorageBackend::ColdJsonlFallback enum variant, paper §Cold- fallback tier prose, cost-model "cold-tier scan bytes" line item); ~1-2 days - Phase 3: PromQL completeness on GorillaQueryEngine. Three sub-paths: Path A (vendor Prometheus' promql package, ~2 weeks, recommended for correctness), Path B (pure-Rust evaluator, ~4-6 weeks, correctness risk), Path C (extend curated subset, ~1 week, ships fast but reviewers may push back). Recommends Phase 1+2 combined for paper deadline (drops cold-fallback prose); Phase 3 ships as Path C bridge with Path A as post-deadline follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: rewrite to reflect post-v7 state The original 3-phase plan (always-archive at gateway / delete JSONL / PromQL completeness) is partially superseded by what shipped via v5/v7. Updated to reflect what's already on main and what genuinely remains: - "Always-archive at gateway" effectively shipped via dual-routing (PR #91): BackendStorageRouting now allows multi-target per metric - The archive-tier engine has gained postings filtering (PR #295), partial-S3 reads (PR #295), concat-only compactor (PR #295), and freshness-pattern registration (PR #91) Two outstanding items remain: - Delete the JSONL cold-fallback (now safely unreachable under normal routing) - PromQL completeness on GorillaQueryEngine (Path A: vendor prometheus/promql via sidecar; Path B: pure-Rust evaluator; Path C: curated subset extension) Adds a "what's already on main" diff table at the top so reviewers immediately see what's done vs. what's outstanding. Adds a sentence about Prometheus-block-compatible layout potentially letting Thanos store-gateway answer queries directly (cross-references the comparison doc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v5 MVP — collector side. DRAFT, do not merge. Run agent will execute the demo and post numbers.
asap-gorilla::postings) — per-block sidecar(label_name, label_value) → series_idswithPostings::merge_manyfor the compactor (set-union, sorted, deduped).IndexEntry::object_key / byte_offset / byte_length) — back-compat additive; legacy fixtures still parse.compactor/asap-compactor) — byte-concatenates 6+ adjacent ≥6h-old per-hour blocks into one merged block. Chunks are NEVER decoded or re-encoded. Seecompactor/src/lib.rsfor the full concat-only design rationale.gorillas3processornow writespostings-v1.jsonper flush and stamps the new manifest fields.run_mvp_demo.shadds ad-hoc PromQL with label predicates (criterion ⑦), compactor dry-run + live-run with before/after MinIO listing, per-baselines3_cost.csv.mvp_report.pygrows §5 (S3 cost) / §6 (compaction effect, explicitly notes "no decode/re-encode") / §7 (postings filtering) / §8 (criterion ⑦) plus a cost-projection annex.Test plan
cargo test --releaseinasap-gorilla/(39 pass)cargo testincompactor/(11 pass — plan/discovery, run/idempotency, partial-read decode, dry-run, concat-only)go test -run "TestFlushWindow_EmitsPostingsSidecar|TestBuildPostings"ingorillas3processor/(4 pass)bash deploy/scripts/run_mvp_demo.sh→MVP_REPORT_v5.md.gorbyte length equals sum of source chunk byte lengthsCross-repo dependency
Pairs with
mvp/v5-postings-aware-enginePR in ASAPQuery-backend.🤖 Generated with Claude Code