[ab-advisor] Experiment campaign for plan: A/B test reasoning_depth

[github-actions]

🧪 Experiment Campaign: plan

Workflow file: .github/workflows/plan.md
Selected dimension: reasoning_depth
Triggered by: ab-testing-advisor on 2026-07-02

---

Background

The plan workflow turns an issue or discussion comment into a small set of actionable, agent-ready sub-issues, with built-in grouping and discussion-resolution behavior. I chose reasoning_depth because this workflow is prompt-dense, structurally important, and already shows strong reliability; the most valuable unknown is whether a shorter one-pass planning prompt can preserve task quality while reducing latency and token use.

Hypothesis

H0: Changing reasoning depth does not improve sub-issue quality or workflow efficiency versus the current baseline.

H1: A lighter planning prompt (shallow) will reduce token/latency cost without materially reducing sub-issue usefulness, while a more explicit reflective prompt (deep) may improve task decomposition quality by at least 8 percentage points on a manual quality score.

Experiment Configuration

Add the following experiments: block to the workflow frontmatter (use the rich object form so all metadata is self-documenting):

experiments:
  reasoning_depth:
    variants: [shallow, baseline, deep]
    description: "Tests whether lighter or more reflective planning instructions produce better agent-ready sub-issues for /plan requests."
    hypothesis: "H0: no change in plan quality score or success rate. H1: deep improves quality score by >=8%, or shallow reduces run cost/latency by >=15% with no material quality loss."
    metric: plan_quality_score
    secondary_metrics: [run_duration_ms, issue_creation_count, output_token_estimate]
    guardrail_metrics:
      - name: workflow_success_rate
        direction: min
        threshold: 0.95
      - name: empty_plan_rate
        direction: min
        threshold: 0.02
    min_samples: 137
    weight: [34, 33, 33]
    start_date: "2026-07-02"
    issue: <this_issue_number>

Variant descriptions:

• shallow: Use a shorter one-pass planning instruction set focused on extracting 3-5 direct tasks with minimal reflection.
• baseline: Preserve the current workflow behavior and prompt detail as the control.
• deep: Add an explicit intermediate reflection step to verify sequencing, independence, and acceptance criteria before drafting sub-issues.

Workflow Changes Required

List the exact changes needed in the workflow markdown body to implement the experiment using handlebars conditional blocks. Always compare against a specific variant value — the correct syntax is {{#if experiments.<name> == "<variant>" }}...{{else}}...{{/if}}. The compiler automatically expands experiments.<name> references at compile time; never write the internal env-var form (__GH_AW_EXPERIMENTS__<NAME>___<variant>) directly.

View Details

Concrete prompt splice near the “## Begin Planning” section:

 ## Begin Planning
 
 {{#if github.event.issue.number}}
-1. First, analyze the current issue (#${{ github.event.issue.number }}) and the user's comment for context and any additional guidance
-2. Create sub-issues (at most 5) - they will be automatically grouped
+1. First, analyze the current issue (#${{ github.event.issue.number }}) and the user's comment for context and any additional guidance
+{{#if experiments.reasoning_depth == "shallow" }}
+2. In one pass, draft 3-5 concrete sub-issues directly from the highest-confidence work items.
+3. Keep rationale brief; prioritize speed, specificity, and implementability.
+{{else}}{{#if experiments.reasoning_depth == "deep" }}
+2. Before drafting issues, briefly reason about dependencies, ordering, and whether each task is independently completable in one PR.
+3. Remove redundant or overly broad tasks, then draft the best 3-5 sub-issues.
+4. For each task, verify acceptance criteria and affected files/components are explicit when knowable.
+{{else}}
+2. Create sub-issues (at most 5) - they will be automatically grouped
+{{/if}}{{/if}}
 {{/if}}
 
 {{#if github.event.discussion.number}}
-1. First, analyze the discussion (#${{ github.event.discussion.number }}) and the user's comment for context and any additional guidance
-2. Create sub-issues (at most 5) - they will be automatically grouped
-3. After creating all issues successfully, if this was triggered from a discussion in the "Ideas" category, close the discussion with a comment summarizing the plan and resolution reason "RESOLVED"
+1. First, analyze the discussion (#${{ github.event.discussion.number }}) and the user's comment for context and any additional guidance
+{{#if experiments.reasoning_depth == "shallow" }}
+2. In one pass, create concise, high-confidence sub-issues (at most 5) - they will be automatically grouped
+{{else}}{{#if experiments.reasoning_depth == "deep" }}
+2. Briefly reason about dependencies, grouping, and which tasks are best separated before drafting sub-issues (at most 5)
+3. Ensure each resulting sub-issue has explicit scope and acceptance criteria before finalizing
+{{else}}
+2. Create sub-issues (at most 5) - they will be automatically grouped
+{{/if}}{{/if}}
+3. After creating all issues successfully, if this was triggered from a discussion in the "Ideas" category, close the discussion with a comment summarizing the plan and resolution reason "RESOLVED"
 {{/if}}

Operationally, instrument post-run evaluation by scoring created issues on: specificity, independence, sequencing quality, and acceptance-criteria clarity. Store that score in analytics derived from the created issue bodies plus a lightweight reviewer rubric.

Success Metrics

Metric	Type	Target
plan_quality_score	Primary	deep improves by >=8% or shallow stays within -2% while saving cost
run_duration_ms	Secondary	shallow improves median runtime by >=15%
workflow_success_rate	Guardrail	Must remain >=95%

Statistical Design

• Variants: shallow, baseline, deep
• Assignment: Round-robin via gh-aw experiments runtime (cache-based)
• Minimum runs per variant: 137
• Expected experiment duration: ~45-70 days at recent observed cadence (~1-1.5 runs/day)
• Analysis approach: proportion test for quality pass/fail threshold, plus Mann-Whitney U on rubric scores and duration

Implementation Steps

• Add experiments: section to frontmatter
• Add conditional blocks to workflow prompt body using {{#if experiments.reasoning_depth == "shallow" }} / "deep" (value-comparison form — never use the internal __GH_AW_EXPERIMENTS__ env-var syntax)
• Run gh aw compile plan to regenerate lock file
• Monitor experiment artifact uploaded per run to /tmp/gh-aw/agent/experiments/state.json
• After sufficient runs, analyze variant distribution via workflow run artifacts
• Document findings and promote winning variant

References

• A/B Testing in gh-aw
• Workflow file: .github/workflows/plan.md

Infrastructure Note

Using the field-presence-checker results for analysis_type, tags, and notify, the frontmatter schema surface appears complete enough to avoid opening a separate infrastructure sub-issue in this run. Remaining work is better framed as future reporting/analytics follow-up after the first campaign lands.

Generated by 🧪 Daily A/B Testing Advisor · 34 AIC · ⌖ 23.2 AIC · ⊞ 7.2K · ◷

• expires on Jul 16, 2026, 3:24 AM UTC-08:00