Add machine-readable review orchestration contract - #182
Jesper Schulz-Wedde (JesperSchulz) merged 2 commits into
Conversation
Generate a deterministic action-skill index from frontmatter, publish structural schemas for orchestration and findings, and validate flat review composition in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 76eb42c4-2acd-4f9c-a898-f4a44f9d46f7
|
👋 Heads up dayland — and cc maintainers — this PR introduces new top-level entries that aren't part of BCQuality's known repository structure:
This isn't a block — just a flag. 🚩 New top-level folders and files are usually unintended (a stray export, a tool's scratch dir, or content that meant to land inside an existing layer). BCQuality keeps a deliberately small root: plugin metadata, If this was intentional and the new entry genuinely belongs at the repo root, a maintainer can review and merge as normal — no action needed beyond a quick sanity check. If it wasn't, please move the content into the right existing layer (or drop it) and push an update. 🙏 A maintainer will take a look before merging. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 76eb42c4-2acd-4f9c-a898-f4a44f9d46f7
Jesper Schulz-Wedde (JesperSchulz)
left a comment
Jesper Schulz-Wedde (JesperSchulz)
left a comment
There was a problem hiding this comment.
Reviewed at 0a018fd9fbe8e84b6f6e043edb8198f66c4b09ee (exact merge tree c8c7225939c9a6904b6db5c3593ecf6da7c9182b). The versioned index and findings schema are consistent with current skill producers, preserve declared leaf order and flat composition, and remain structurally backward-compatible. All 32 report examples validate; generator output matches canonical YAML across all 18 action skills. Full local contract/index/retrieval/fixture validation passed, all PR checks are green, and downstream orchestration/E2E evidence is green. The new root schemas/ directory is intentional and appropriate.
## Summary - replace prompt-driven review leaf creation with engine-owned process orchestration - resolve the ordered leaf plan from the BCQuality machine-readable contract - bind an explicit model to every leaf process and a separate model to root consolidation - support deterministic serial and bounded-parallel scheduling over the same plan - pin Copilot CLI 1.0.83 and resolve BCQuality from the engine-owned configuration; derive its commit from the resulting checkout - emit a schema-validated `_run-manifest.json` with resolved configuration and per-process model, usage, timing, status, and report evidence - fail closed on substitutions, missing/reordered leaves, malformed reports, incomplete telemetry, CLI mismatch, timeouts, and process failures ## Why Previous model-selection experiments could not prove that requested leaf models were actually used. Mechanical process creation and per-process telemetry make model placement and scheduling reproducible before benchmark results are accepted. ## Validation - 137 Pester tests passed - zero PSScriptAnalyzer errors or warnings - JSON schema, workflow YAML, and diff validation passed - fake-Copilot serial and parallel engine runs produced 16 leaf processes plus one root process with exact requested models - two live BC-Bench E2E smoke runs completed successfully: - run 34844099841: 16 GPT-5.4 leaves + one Claude Sonnet 5 root - run 34846127419: 16 GPT-5.6 Luna leaves + one GPT-5.4 root - both live manifests reported CLI 1.0.83, complete usage, zero malformed records, zero failed processes, and engine-produced BCQuality provenance ## Dependencies - Depends on microsoft/BCQuality#182. - BC-Bench integration will be opened as the next draft PR and will remain dependent on both changes. ## Scope This PR establishes deterministic configuration and evidence. It does not select the final production model combination; controlled rebaselining will occur after the dependency chain is accepted. --------- Co-authored-by: dayland <dayland@microsoft.com> Copilot-Session: 76eb42c4-2acd-4f9c-a898-f4a44f9d46f7
3 participants
Summary
Why
BC-ALAgents needs a stable contract that defines exactly which review leaves exist and their required order. This keeps BCQuality frontmatter as the source of truth while allowing orchestration to be owned mechanically by the consuming engine rather than inferred by a root model.
Validation
feature/deterministic-leaf-orchestrationDependency order
This is PR 1 of 3. BC-ALAgents depends on this contract; BC-Bench then pins and validates both repositories.
E2E evidence
Both runs resolved this BCQuality commit (
25accf021ed2e5d9c1f776c4dc13580bf072b678) and produced 17 ordered successful processes with exact requested/observed models and complete telemetry.