Skip to content

feat: Add LLM council sample to workflow patterns#6445

Open
simonraj79 wants to merge 1 commit into
google:mainfrom
simonraj79:feat/llm-council-sample
Open

feat: Add LLM council sample to workflow patterns#6445
simonraj79 wants to merge 1 commit into
google:mainfrom
simonraj79:feat/llm-council-sample

Conversation

@simonraj79

Copy link
Copy Markdown

Link to Issue or Description of Change

2. Or, if no issue exists, describe the change:

Problem:
The contributing/samples catalog demonstrates fan-out/fan-in with plain
function nodes (workflows/fan_out_fan_in) but has no sample showing a
complete multi-LLM deliberation pattern: parallel Agent nodes, two chained
fan-out/fan-in rounds, structured output contracts between LLM stages, and
deterministic post-processing of LLM votes. The "LLM council" pattern
(popularized by Andrej Karpathy's llm-council)
is a natural fit and a frequently requested multi-agent shape.

Solution:
Add contributing/samples/patterns/llm_council — a self-contained,
three-stage council in a single Workflow:

  1. First opinions — three persona agents (pragmatist, skeptic,
    visionary) answer in parallel via a fan-out tuple.
  2. Anonymized peer review — a function node relabels the JoinNode
    output as "Response A/B/C"; a second fan-out has every member rank all
    responses blind, ending with a machine-parseable
    FINAL RANKING: B > A > C ballot line.
  3. Tally + synthesistally_ballots (plain Python) parses the ballots
    and computes the average-rank leaderboard deterministically; a chairman
    agent synthesizes the final answer and de-anonymizes the verdict via a
    roster only it can see.

Design points the sample teaches:

  • Chained fan-out/fan-in rounds with two JoinNodes in one graph.
  • One shared static_instruction across all seven agents, so every LLM call
    in a session carries a byte-identical system instruction (keeps the
    cacheable prefix stable; the adk web System Instruction Performance
    Analysis reports no cache-breaking instruction changes). Persona and stage
    tasks ride in the dynamic instruction, which ADK sends as user content
    when static_instruction is set.
  • Selective state exposure via instruction templating: reviewers see the
    anonymized {transcript}; only the chairman sees {roster}, which keeps
    the peer review blind.
  • LLM output → code contract: reviewers must emit an exact ballot line;
    arithmetic is done in a function node, never by the model.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

The sample is covered by the existing sample harness in
tests/unittests/test_samples.py (test_sample_loads auto-discovers it).

pytest tests/unittests/test_samples.py -k llm_council
1 passed, 278 deselected in 2.78s

Full local unit suite (Python 3.11, Windows):

19 failed, 9056 passed, 64 skipped, 18 xfailed, 1 xpassed in 559.57s (0:09:19)

All 19 failures are pre-existing Windows line-ending issues in
tests/unittests/tools/environment/ file-tool tests, unrelated to this
change — the branch is main plus a single commit that only adds files
under contributing/samples/ and touches no source code.

scripts/compliance_checks.py and all pre-commit hooks (isort, pyink,
addlicense, mdformat) pass on both files.

Manual End-to-End (E2E) Tests:

Setup: adk web contributing/samples/patterns --port 8000 with a Gemini API
key in the sample folder's .env, select llm_council, using the three
Sample Inputs from the README.

Three full runs verified via the dev UI event stream and trace viewer:

  • All 13 nodes execute per invocation; the two fan-out stages run their three
    agents concurrently (stage wall-clock = slowest member, confirmed in
    Traces).
  • All 3 reviewer ballots parsed in every run; tally_ballots leaderboard
    matches a hand-computed average of the ballots each time.
  • Chairman reply contains the four required sections and reproduces the
    leaderboard table exactly, de-anonymized via the roster.
  • Total invocation latency across the three runs: 48.8s → 33.9s → 32.2s
    (improvements during development: shared static_instruction, 500-word
    chairman cap).
  • With the shared static_instruction, the dev UI shows no
    System-Instruction cache-miss performance warnings on any event
    (previously flagged on every LLM event).

Screenshots of the executed graph, event stream, and chairman verdict
attached below.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

The sample follows the structure documented in the repo's sample-authoring
guidance (agent.py + README with Overview / Sample Inputs / Graph / How To /
Related Guides) and does not set an explicit model=, per sample guidelines.
The README notes how to reproduce Karpathy's original one-model-per-seat
configuration.

Adds contributing/samples/patterns/llm_council implementing the
three-stage LLM council pattern (Karpathy's llm-council) as a single
ADK Workflow: parallel first opinions from three personas, anonymized
peer review with machine-parseable FINAL RANKING ballots, a
deterministic leaderboard tally computed in a function node, and a
chairman synthesis with a de-anonymized verdict.

Demonstrates chained fan-out/fan-in rounds with JoinNode, a shared
static_instruction across all seats for context cache alignment, and
selective state exposure via instruction templating (reviewers see the
anonymized transcript; only the chairman sees the identity roster).
@adk-bot adk-bot added the core [Component] This issue is related to the core interface and implementation label Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core [Component] This issue is related to the core interface and implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants