CF-06: harden oracle execution cost and corpus qualification runtime

Purpose

Track a performance/reliability hardening gap discovered while qualifying the HL7 comparator fix against the full frozen CF-10 corpus.

This issue does not change CF-06 semantics, the production HL7 pin, the frozen corpus, or PR #11. It is not authorization to parallelize or batch oracle work without preserving determinism and fail-closed behavior.

Observed execution shape

The current CF-10 runner invokes the HL7 Java adapter once per changed matched canonical StructureDefinition. Each invocation has a bounded 60-second timeout and process-tree termination on timeout, which is correct fail-closed behavior.

The qualification workflow hl7-upstream-fix-qualification runs the complete frozen corpus twice sequentially (A then B) under a 180-minute job timeout. Once the two known upstream comparator crashes are fixed, the run reaches substantially more oracle comparisons instead of failing early, making full A/B qualification materially more expensive.

Current qualification run:

run: 31918255417
job: 95093683595
step: Run frozen CF-10 corpus twice with combined candidate

Required invariants

Any optimization must preserve:

CF-03 as semantic authority;
HL7 as advisory oracle only;
exact per-resource identity and evidence;
bounded stdout/stderr;
bounded execution time;
fail-closed subprocess errors/timeouts;
deterministic report ordering and byte output;
no hidden network/package-cache dependency;
unchanged frozen CF-10 case selection and package bytes;
no silent skipping of failed resources.

Candidate directions to evaluate after CF-10 qualification

reuse a long-lived validated JVM/oracle worker with an explicit request/response protocol;
bounded deterministic worker-pool concurrency with canonical result reordering;
batch multiple StructureDefinition comparisons inside one adapter invocation while retaining per-resource failure identity;
cache immutable parsed package/context state within a single corpus run;
separate per-case or per-side workflow jobs only if retained evidence and A/B determinism remain straightforward to audit.

No direction is pre-authorized by this issue.

Acceptance

A hardening change should demonstrate, on the frozen corpus:

equivalent normalized oracle evidence to the current one-process-per-resource baseline;
byte-identical deterministic summaries;
explicit regression coverage for timeout/process failure behavior;
bounded concurrency/resource use;
no weakening of CF-06 failure semantics;
materially lower full-corpus runtime measured from retained CI evidence.

Scheduling

Do not modify the currently running qualification proof. Evaluate this only after run 31918255417 has completed and its evidence has been classified.

Measured runtime evidence is now available from successful qualification run 31918255417 / job 95093683595.

job started:     2026-08-16T00:52:19Z
job completed:   2026-08-16T02:55:29Z
total job time:  2h 03m 10s

full frozen A/B step started:   2026-08-16T01:18:53Z
full frozen A/B step completed: 2026-08-16T02:55:25Z
A/B step time:                  1h 36m 32s

The expensive A/B step nevertheless completed successfully inside the current 180-minute job timeout, and final evidence enforcement passed. This gives #16 a concrete baseline rather than a theoretical performance concern.

The hardening target should therefore be measured against a 1h 36m 32s full frozen A/B baseline on this GitHub runner while preserving byte-identical summaries, per-resource failure identity, bounded subprocess behavior, and all existing CF-06 fail-closed invariants. No runtime optimization is being applied to the already-proven CF-10 qualification.

Hardening sequence after successful CF-10 proof

Code inspection and the successful 31918255417 runtime baseline make the safest optimization order clearer.

H0 — workflow-only parallel A/B qualification (first)

The current experiment runs frozen corpus A and then frozen corpus B serially inside one job, then compares the two summaries. A and B are intentionally independent executions, so the first optimization should move them into separate bounded CI jobs that run concurrently and retain their evidence independently, followed by a small enforcement job that downloads both artifacts and requires:

status_a == 0
status_b == 0
cmp(summary_a, summary_b) == 0

This changes no commandF production code, no CF-06 semantics, no oracle source identity, no per-resource timeout behavior, and no frozen corpus inputs. Each A/B job must independently verify exact commandF/HL7/adapter identities and run with network isolation for the corpus execution.

Acceptance for H0:

both sides use the exact same pinned inputs and adapter bytes;
each side retains the complete case evidence and status independently;
the comparison/enforcement job fails closed if either artifact/status is missing or non-zero;
byte-identical deterministic summaries remain mandatory;
measured wall-clock improvement is recorded against the 1h 36m 32s serial A/B step baseline.

H1 — bounded deterministic process concurrency (only if H0 is insufficient)

The current Rust path invokes run_hl7_oracle_adapter once per eligible canonical StructureDefinition. Every invocation preserves its own 60-second timeout, bounded stdout/stderr, and process-tree termination. If additional runtime reduction is still needed after H0, evaluate a small fixed worker bound around these existing invocations rather than changing the adapter protocol first.

Required H1 properties:

fixed explicit concurrency bound, not unbounded available_parallelism() fan-out;
canonical input order assigned before execution;
results reordered to canonical input order before reconciliation/serialization;
no new work scheduled after a failure is known;
already-running workers are joined/terminated safely;
reported failure is the earliest failing canonical input that the sequential baseline would encounter, not whichever worker finishes first;
existing per-resource timeout/output/process-tree behavior remains unchanged;
baseline and concurrent modes produce equivalent normalized per-resource oracle evidence and byte-identical final summaries.

Defer — long-lived JVM and batch protocol

Do not start with a long-lived JVM or multi-resource batch adapter. Main.compare() currently loads core, side, and context packages fresh for each comparison, so those designs may eventually yield the largest CPU/startup savings, but they also change important failure boundaries: per-resource timeout identity, subprocess isolation, partial-output handling, and which resource is attributable when a worker hangs or exits. Those should be considered only after H0/H1 measurements show they are necessary.

Current recommendation

H0 workflow-only parallel A/B
    -> measure retained CI runtime/evidence
    -> H1 bounded deterministic process concurrency only if needed
    -> batch/long-lived JVM deferred

No production mutation or CF-06 pin/source change is authorized by this sequencing note.

H0 runtime experiment is now running on an isolated workflow-only branch. No production CF-06 code, oracle identity, frozen corpus, PR #11 content, or main branch state is changed.

Exact H0 identity:

branch: exp/cf06-oracle-runtime-h0-parallel-ab
base:   c95cc7665b0dce6b91d2dcff31791361215abe82
head:   edb57179eb719e9c0ef4edd2b1676eaeffd77e47
run:    32050152005

The diff from the proven experiment base is workflow-only. The H0 benchmark runs the two complete frozen-corpus executions concurrently with independent work roots, then requires all of the existing deterministic/evidence properties before accepting any runtime result:

runner A/B success;
each emitted summary equals its retained internal summary.json;
A and B byte-identical;
both summaries equal the already-proven serial summary SHA-256 074594e630c02f10c4bec9bcb903e153edaeb88a04c65636ff11c11301547409;
both retained closure bindings verify;
all three frozen cases remain complete;
metadata-only repository boundary remains clean;
retained per-side and wall-clock runtime evidence is uploaded.

Baseline for comparison remains the proven serial A/B step from run 31918255417: 5792 seconds (1h 36m 32s).

H0 deliberately does not introduce long-lived JVM state, batching, per-resource concurrency, altered timeout semantics, or altered failure identity. Those remain deferred until this lowest-risk workflow-only experiment is measured.

H0 isolation preflight was checked against the exact experiment head before interpreting runtime results. corpus run creates all mutable acquisition/cache/lock/evidence state under the supplied work root, and resolve_state() creates PackageCache under work_root/states/<case>/<before|after>/cache. prepare_fresh_work_root() fails if the root already exists. H0 supplies distinct /tmp/cf10-h0-a and /tmp/cf10-h0-b roots, so A/B do not share commandF package cache, lockfiles, or evidence state. The shared oracle JAR and Java executable are read-only inputs. This makes concurrent A/B a valid workflow-level isolation experiment; any external registry/network contention remains part of runner wall-time behavior and cannot silently alter accepted package bytes because existing digest/closure/equality gates still apply.

H0 acceptance is tightened before any result is classified as adopted.

A single GitHub-hosted-runner timing sample is not sufficient to accept runtime hardening because host CPU/network variance can materially affect wall time. The first successful H0 run should therefore be treated only as a single-run candidate proof.

For H0_ACCEPT_FOR_RUNTIME_HARDENING, require at least 3 independent successful runs on the exact same workflow/head (or an equivalent replicated benchmark set) with all semantic/evidence gates passing every time:

runner A = 0 and runner B = 0;
A/B internal summaries match retained summaries;
A and B are byte-identical;
both equal the proven serial summary SHA-256 074594e630c02f10c4bec9bcb903e153edaeb88a04c65636ff11c11301547409;
closure binding passes for both sides;
C001/C002/C003 remain complete;
repository boundary remains clean;
no change to CF-06 semantics, production pin, frozen corpus, or per-resource fail-closed behavior.

Record per-run A seconds, B seconds, concurrent wall seconds, runner/image identity when available, then compare median and worst-case wall time against the proven serial A/B baseline of 5792 seconds. A material improvement should be repeatable, not a one-off favorable runner.

The currently running benchmark 32050152005 remains untouched and is the first candidate sample only.

CF-06 runtime hardening is now canonical and proven.

Canonical integration:

PR perf(cf06): parallelize deterministic oracle validation #17 — MERGED
canonical main at integration: 3df371773d9f8ba284a4ff5938e710aa63bccec8
canonical tree: f0f009fe8abcab99b3a1d339d400b63af4a7b486
candidate head: 098f644c15d894c109711f432cbaac720d138fb8
candidate-to-merge tree equivalence: no changed files
exact-head ci: SUCCESS
exact-head cf06-oracle: SUCCESS
CodeRabbit: SUCCESS / no actionable findings

Production topology now runs the independent oracle-self-smoke and oracle-changed-profile suites on separate runners, followed by fail-closed oracle-proof enforcement. Existing CF-06 assertions, failure-path coverage, deterministic behavior, frozen corpus boundaries, and the production HL7 identity (6.10.2, source d06577dbc5c62c74a2a8823fbc4830a3024d5b0b) remain unchanged.

Measured production-candidate wall-time improvement was approximately 13.6% versus the comparable canonical single-job baseline. The earlier H0 benchmark remains experimental provenance and is not a product performance guarantee.

Classification: CF06_RUNTIME_HARDENING_CLOSED_CANONICAL_PROVEN.

This closeout does not authorize a production HL7 pin change, CF-10 merge, semantic reinterpretation, or CF-12.

Navigation Menu

Repository navigation

CF-06: harden oracle execution cost and corpus qualification runtime #16

Purpose

Observed execution shape

Required invariants

Candidate directions to evaluate after CF-10 qualification

Acceptance

Scheduling

Hardening sequence after successful CF-10 proof

H0 — workflow-only parallel A/B qualification (first)

H1 — bounded deterministic process concurrency (only if H0 is insufficient)

Defer — long-lived JVM and batch protocol

Current recommendation

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

CF-06: harden oracle execution cost and corpus qualification runtime #16

Description

Purpose

Observed execution shape

Required invariants

Candidate directions to evaluate after CF-10 qualification

Acceptance

Scheduling

Activity

TheHalfMoon commented on Aug 16, 2026

TheHalfMoon commented on Aug 17, 2026

Hardening sequence after successful CF-10 proof

H0 — workflow-only parallel A/B qualification (first)

H1 — bounded deterministic process concurrency (only if H0 is insufficient)

Defer — long-lived JVM and batch protocol

Current recommendation

TheHalfMoon commented on Aug 17, 2026

TheHalfMoon commented on Aug 17, 2026

TheHalfMoon commented on Aug 17, 2026

TheHalfMoon commented on Aug 18, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions