feat: Add LLM council sample to workflow patterns#6445
Open
simonraj79 wants to merge 1 commit into
Open
Conversation
Adds contributing/samples/patterns/llm_council implementing the three-stage LLM council pattern (Karpathy's llm-council) as a single ADK Workflow: parallel first opinions from three personas, anonymized peer review with machine-parseable FINAL RANKING ballots, a deterministic leaderboard tally computed in a function node, and a chairman synthesis with a de-anonymized verdict. Demonstrates chained fan-out/fan-in rounds with JoinNode, a shared static_instruction across all seats for context cache alignment, and selective state exposure via instruction templating (reviewers see the anonymized transcript; only the chairman sees the identity roster).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
2. Or, if no issue exists, describe the change:
Problem:
The
contributing/samplescatalog demonstrates fan-out/fan-in with plainfunction nodes (
workflows/fan_out_fan_in) but has no sample showing acomplete multi-LLM deliberation pattern: parallel
Agentnodes, two chainedfan-out/fan-in rounds, structured output contracts between LLM stages, and
deterministic post-processing of LLM votes. The "LLM council" pattern
(popularized by Andrej Karpathy's llm-council)
is a natural fit and a frequently requested multi-agent shape.
Solution:
Add
contributing/samples/patterns/llm_council— a self-contained,three-stage council in a single
Workflow:pragmatist,skeptic,visionary) answer in parallel via a fan-out tuple.JoinNodeoutput as "Response A/B/C"; a second fan-out has every member rank all
responses blind, ending with a machine-parseable
FINAL RANKING: B > A > Cballot line.tally_ballots(plain Python) parses the ballotsand computes the average-rank leaderboard deterministically; a chairman
agent synthesizes the final answer and de-anonymizes the verdict via a
roster only it can see.
Design points the sample teaches:
JoinNodes in one graph.static_instructionacross all seven agents, so every LLM callin a session carries a byte-identical system instruction (keeps the
cacheable prefix stable; the
adk webSystem Instruction PerformanceAnalysis reports no cache-breaking instruction changes). Persona and stage
tasks ride in the dynamic
instruction, which ADK sends as user contentwhen
static_instructionis set.anonymized
{transcript}; only the chairman sees{roster}, which keepsthe peer review blind.
arithmetic is done in a function node, never by the model.
Testing Plan
Unit Tests:
The sample is covered by the existing sample harness in
tests/unittests/test_samples.py(test_sample_loadsauto-discovers it).Full local unit suite (Python 3.11, Windows):
All 19 failures are pre-existing Windows line-ending issues in
tests/unittests/tools/environment/file-tool tests, unrelated to thischange — the branch is
mainplus a single commit that only adds filesunder
contributing/samples/and touches no source code.scripts/compliance_checks.pyand all pre-commit hooks (isort, pyink,addlicense, mdformat) pass on both files.
Manual End-to-End (E2E) Tests:
Setup:
adk web contributing/samples/patterns --port 8000with a Gemini APIkey in the sample folder's
.env, selectllm_council, using the threeSample Inputs from the README.
Three full runs verified via the dev UI event stream and trace viewer:
agents concurrently (stage wall-clock = slowest member, confirmed in
Traces).
tally_ballotsleaderboardmatches a hand-computed average of the ballots each time.
leaderboard table exactly, de-anonymized via the roster.
(improvements during development: shared
static_instruction, 500-wordchairman cap).
static_instruction, the dev UI shows noSystem-Instruction cache-miss performance warnings on any event
(previously flagged on every LLM event).
Screenshots of the executed graph, event stream, and chairman verdict
attached below.
Checklist
Additional context
The sample follows the structure documented in the repo's sample-authoring
guidance (agent.py + README with Overview / Sample Inputs / Graph / How To /
Related Guides) and does not set an explicit
model=, per sample guidelines.The README notes how to reproduce Karpathy's original one-model-per-seat
configuration.