Skip to content

[ab-advisor] Experiment campaign for agent-persona-explorer: A/B test sub_agent_strategy #33744

Description

@github-actions

🧪 Experiment Campaign: agent-persona-explorer

Workflow file: .github/workflows/agent-persona-explorer.md
Selected dimension: sub_agent_strategy
Triggered by: ab-testing-advisor on 2026-05-21


Background

The agent-persona-explorer workflow explores custom agent behavior by generating software personas and testing them against automation scenarios. Currently it runs a single agent that sequentially tests 3-4 scenarios out of 10 possible ones (5 personas × 2 tasks each). Each scenario invocation calls the "agentic-workflows" custom agent sub-agent, which adds latency and token costs.

This experiment tests whether consolidating all scenario testing into a single agent turn (batch mode) vs. the current approach of invoking the sub-agent for each scenario individually produces better cost/quality outcomes.

Hypothesis

H0: Sub-agent strategy does not significantly affect effective token count or run duration.

H1: Batch mode (single agent turn testing multiple scenarios) reduces effective token count by ≥20% and run duration by ≥15% compared to per-scenario sub-agent invocations, without degrading output quality.

Experiment Configuration

Add the following experiments: block to the workflow frontmatter:

experiments:
  sub_agent_strategy:
    variants: [per_scenario, batch]
    description: "Test whether batch scenario testing reduces token costs vs. per-scenario sub-agent calls"
    hypothesis: "H0: no change in effective_tokens or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss"
    metric: effective_tokens
    secondary_metrics: [run_duration_minutes, scenarios_tested, output_quality_score]
    guardrail_metrics:
      - name: discussion_created
        direction: "=="
        threshold: 1
      - name: scenarios_analyzed
        direction: ">="
        threshold: 3
    min_samples: 14
    weight: [50, 50]
    start_date: "2026-05-22"
    issue: #aw_campaign
    analysis_type: t_test
    tags: [cost_optimization, token_efficiency, sub_agents]

Variant descriptions:

  • per_scenario: Current behavior — invoke "agentic-workflows" sub-agent for each of the 3-4 selected scenarios individually (one task call per scenario)
  • batch: Consolidated mode — describe all selected scenarios in a single prompt to the "agentic-workflows" sub-agent and ask it to analyze all at once

Workflow Changes Required

Modify Phase 3 in the workflow body to conditionally use batch vs. per-scenario strategy. Add conditional blocks using {{#if experiments.sub_agent_strategy == "per_scenario"}} and {{#if experiments.sub_agent_strategy == "batch"}}.

Per-scenario variant (current behavior):

  • Invoke "agentic-workflows" sub-agent separately for each of 3-4 scenarios
  • Each invocation presents one scenario context and captures one response
  • Store analysis per scenario with quality assessment (1-5 scale)

Batch variant (consolidated):

  • Invoke "agentic-workflows" sub-agent once with all 3-4 scenarios in a structured list
  • Parse consolidated response to extract per-scenario assessments
  • Store analysis with same structure as per-scenario mode for fair comparison

Success Metrics

Metric Type Target
effective_tokens Primary Batch reduces by ≥20%
run_duration_minutes Secondary Batch reduces by ≥15%
scenarios_tested Secondary Both ≥3
output_quality_score Secondary No degradation (≥4.0/5.0 avg)
discussion_created Guardrail Must equal 1
scenarios_analyzed Guardrail Must be ≥3

Statistical Design

  • Variants: per_scenario (current), batch (consolidated)
  • Assignment: Round-robin via gh-aw experiments runtime (cache-based)
  • Minimum runs per variant: 14 (total 28 runs needed)
  • Expected experiment duration: 28 days (workflow runs daily)
  • Analysis approach: Two-sample t-test for effective_tokens and run_duration; Mann-Whitney U for quality score comparison

Implementation Steps

  • Add experiments: section to frontmatter
  • Add conditional blocks to Phase 3 using handlebars syntax
  • Run gh aw compile agent-persona-explorer to regenerate lock file
  • Monitor experiment artifact uploaded per run to /tmp/gh-aw/experiments/state.json
  • After 28 runs (14 per variant), analyze token usage and duration distributions
  • Compare output quality scores between variants using Mann-Whitney U test
  • Document findings and promote winning variant

References

  • A/B Testing in gh-aw
  • Workflow file: .github/workflows/agent-persona-explorer.md
  • Related: Token budget optimization goals (lines 11-16 of workflow)

Generated by 🧪 Daily A/B Testing Advisor · ● 1.1M ·

  • expires on Jun 4, 2026, 11:40 AM UTC

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions