Skip to content

Test workflow builder experiments through the product API - #1

Draft
deyton wants to merge 8 commits into
mainfrom
codex/workflow-prototype-client
Draft

Test workflow builder experiments through the product API#1
deyton wants to merge 8 commits into
mainfrom
codex/workflow-prototype-client

Conversation

@deyton

@deyton deyton commented Sep 8, 2026

Copy link
Copy Markdown

Allow the existing Monarch competitor to test the workflow builder experiments through the product's authenticated HTTP API. Setting builder_experiment selects the configuration and records durable authoring evidence; leaving it unset preserves the existing request and stream behavior.

Shared product client

  • Poll persisted status, verify the resolved configuration, and export complete cursor history with raw structured failures. Preserve selection in configuration hashes and preserve the configured model identity. Hosted testing requires its source checkout to match the deployed revision.

  • Cancel authoring or execution when prototype polling fails or times out, retain cancellation responses and partial attempt evidence, and leave unsettled usage explicitly unknown.

  • Release the fixed synthetic HTTP port between attempts on POSIX while preserving Windows ownership checks and existing connection-closing behavior. Direct regressions cover restart after a request and refusal over an active listener.

Deferred evaluation scaffold

  • Produce a deterministic, keyless 66-attempt candidate manifest with paired configuration order and independent grader controls. Two simple cases qualify; one candidate demonstrates false positives and seven remain pending. Frozen tasks and vendor code are unchanged.
  • Provide optional reservation/accounting and evidence checks for a later campaign. The reservation ledger is not a provider spend limiter, and the proposed campaign is not qualified or ready to execute. The shared client can be used independently of this runner.

Validation

Full offline suite: 768 passed, 3 skipped. Earlier independent review also passed 175 focused keyless tests. Regression tests cover actual progress/lifecycle event shapes, repeated build keys after clarification, missing or duplicate receipts, cancellation confirmation, model identity and stored failure evidence. The regenerated manifest matches the implementation.

The original fixed-port failure reproduced on baseline locally and in Linux CI; the lifecycle correction passes the full local suite and refreshed Linux CI, including corpus validation. Graphify could not be refreshed because its module is unavailable. No paid model run, live product execution, or benchmark improvement is claimed. Private dogfood workflow content is excluded.

Product counterpart: https://github.com/TestBoxLab/monarch/pull/702

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant