Skip to content

Add a strict multi-fit ERP Figure 1 runner - #592

Merged
zzylol merged 6 commits into
mainfrom
eval/erp-figure1
Sep 10, 2026
Merged

zzylol merged 6 commits into
mainfrom
eval/erp-figure1

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Figure 1 needs to compare five independently executed methods without allowing dataset, candidate, or ERP-fit drift between arms. Fixed sketch parameters must not be relabeled as a Planner ERP selection.

The stack pins merged Planner #380 (a9651cc) across every backend crate. It also adds a strict runner for AutoSketch-PerQuery, Planner Analytical, Planner ERP, ASAP-NoSharing, and Exact. A manifest pins the dataset SHA-256 and one candidate space, total-memory bound, accuracy contract, window workload, multi-family ERP observation, and complete fit/divergence/confidence selection policy from Planner #380. Each arm runs in a separate process and must echo the identical contract and actual selected candidates.

The runner measures wall time, user/system CPU, and peak RSS. It verifies candidate membership, state size, and measured error; invalid/ambiguous fits, constraint drift, missing arms, and dataset changes fail closed. Failed processes remain in the artifact with resource counters and stderr.

Before this PR, sequential in-process prototypes could not isolate CPU/RSS and the fixed --width/--depth configuration could be mislabeled as ERP. After this PR, an ERP result requires an actual selected-plan record under the same immutable inputs as all baselines.

Verification:

  • python3 -m unittest tools/erp-figure1/test_run.py -v — 7 passed
  • cargo check -p control_plane -p data_plane

No Figure 1 numbers are claimed here. Production arm adapters must implement the documented JSON boundary; publishing partial or relabeled smoke output remains disallowed.

@zzylol
zzylol merged commit 791f7d7 into main Sep 10, 2026
1 check passed
@zzylol
zzylol deleted the eval/erp-figure1 branch September 10, 2026 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant