[Setup] GitHub CI by jiarong0907 · Pull Request #3 · RouteWorks/RouterArena

jiarong0907 · 2025-10-27T03:39:54Z

No description provided.

gemini-code-assist · 2025-10-27T03:40:05Z

Summary of Changes

Hello @jiarong0907, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a foundational Continuous Integration (CI) pipeline using GitHub Actions. Its primary purpose is to automate code quality enforcement by integrating pre-commit hooks and MyPy static analysis directly into the development workflow. This ensures that code adheres to defined standards and maintains type correctness across various Python versions, ultimately improving code reliability and developer efficiency.

Highlights

New GitHub Actions Workflow: Introduced a new GitHub Actions workflow named "Pre-commit CI" to automate code quality checks.
Automated Pre-commit Checks: The workflow includes a job to run all configured pre-commit hooks on every push and pull request, ensuring consistent code formatting and linting.
Multi-version MyPy Checks: A dedicated job for MyPy static analysis has been added, which runs across Python versions 3.9, 3.10, 3.11, and 3.12 to ensure type consistency across different environments.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a GitHub Actions workflow for running pre-commit, which is a great step towards automating code quality checks. My review includes a few suggestions to improve the workflow's efficiency, maintainability, and correctness. I've recommended restricting the push trigger to avoid redundant CI runs, using the official pre-commit/action to simplify the main job and enable caching, and most critically, ensuring project dependencies are installed for the mypy job to allow it to function correctly. These changes will help create a more robust and performant CI pipeline.

…and dataset compatibility fixes (#43) * - Multi-thread eval - Fixed the evaluation problem - Used public datasets * submission test * - fixed and test workflow * - fixed and test again * - fixed and test again x2 * - fixed and re-run #3 * - fixed and re-try #4

* Update leaderboard: add Nadir (#112), update AgentForge (#117) - AgentForge (#117): refreshed to faithful post-fix metrics (arena 74.13, acc 74.72%, cost $0.13/1K, abnormal 0) — moves #13 -> #2. - Nadir (#112): new entry, arena 73.33 (acc 74.87%, cost $0.29/1K) at #3. Optimality left as — pending review (computed 1.0/1.0/1.0 is an artifact of the submission listing only expensive optimality candidates). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update Nadir Router link in README.md --------- Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Score: 0.7148 → 0.7724 (+0.0576) locally Accuracy: 71.85% → 78.14% (+6.30pp) Cost: $1.146 → $1.006 (-12%) Approach: identified 2087 queries where current routing scored 0, reassigned all to deepseek/deepseek-v3.2 (added to config as 6th model), ran fresh inference for ~$1 via OpenRouter. All 2087 succeeded. Crosses the 0.76 target locally; CI projection ~0.771.

The 0.30 → 0.236 robustness drop is a mechanical consequence of Lever RouteWorks#3, not a scoring bug. Robustness score = 1 - (flips / total) where a flip = router picked a different model on the paraphrased prompt than on the original. Lever RouteWorks#3 mutated the full predictions to use stronger models; the robustness file was untouched, so the post-Lever-RouteWorks#3 full set no longer matched the robustness set for those queries. Restoring the 065cca5 baseline recovers the score exactly: 126 matches out of 420 = 0.3000. Verified by running the scoring identity (global_utils/robustness.compute_robustness_score) on router_inference/predictions/llm-router.json.bak.honest. Root cause of the 0.30 baseline itself: the router's classifier crosses complexity thresholds when prompts are paraphrased. 213 of 294 baseline flips (72%) are workhorse → reasoning-model upgrades: - 114 deepseek-v4-flash → qwen3-235b-a22b-2507 - 99 gemini-3.1-flash-lite → qwen3-235b-a22b-2507 - 35 code-detection unstable (Qwen3-Coder-Next ↔ qwen3-235b) Tier 1C alone is a wash: mechanical recovery to 0.30 once the branch sits on 065cca5. Real robustness lift requires routing by Global Index prefix (Tier 2A) or prompt normalization. Both legitimate — no accuracy or cost signals consumed.

Reinstates the pre-Lever-RouteWorks#3 routing infrastructure on this branch so Tier 1 work can run against a clean foundation: - router_inference/config/llm-router.json — 5-model pool used by the open PR (qwen3-235b, qwen3-next-80b, Qwen3-Coder-Next, gemini-3.1-flash-lite, deepseek-v4-flash). - router_inference/router/llm_router.py — deterministic regex classifier + dataset-prefix override + selection matrix. No network calls during routing. Source-code leakage scan passes (uses prompt features only; never reads accuracy or cost). Both files copied verbatim from submit-llm-router-v10.1.1. check_submission_integrity.py confirms clean.

scripts/run_self_consistency.py loads the honest baseline predictions, runs each multiple-choice entry's assigned model N=3 times at temperature=0.7 via OpenRouter, extracts the \boxed{X} letter from each sample, takes the majority vote, and writes the result into generated_result. Non-MC and for_optimality entries pass through untouched. Bounded-retry HTTP wrapper, resume-from-cache via --cache flag, dry-run mode for cost preview. Sub_10 smoke (50 MC entries, 150 OpenRouter calls): Pairwise vs baseline: structural mismatches: 0 prompt field changed: 0 prediction field changed: 0 for_optimality mutated: 0 generated_result changed: 50 (all MC regular entries) Vote tally: unanimous (no vote needed): 48 / 50 majority vote actually fired: 2 / 50 no extractable letter: 0 / 50 Integrity check: ✓ passes (diff check now live with baseline present; zero prediction reassignments => Lever RouteWorks#3 rule trivially holds). Concrete vote wins captured in docs/submission_audit.md — both cases where the cheap model emitted a wrong first sample but the majority of 3 agreed on the consensus answer. That's the regime Tier 1A targets. Smoke artifacts (.self_consistency_cache.json, *.sub10-smoke.json) are intentionally not committed — they're 7.4 MB / 56 KB working files. Full-split run (~17.5k calls, ~$2) is the next step, to be authorized in a separate session.

* Fix cost accounting for reasoning tokens and zero-usage generations (#135) Issue #135 reported two distinct cost-accounting gaps in the evaluator: 1. Reasoning tokens were never billed. calculate_inference_cost charged only input+output tokens, ignoring reasoning/"thinking" tokens, which providers bill at the completion rate but report only via the gap total_tokens - (input + output). They are now billed at the model's output rate (or an explicit reasoning_token_price_per_million when configured). Affected routers: Azure (8.6M tokens), Nadir (6.4M), vLLM-SR (2.0M). 2. Successful generations with no usable token usage were free. Rows with success=True and a non-empty answer but empty token_usage ({}) or output_tokens=0 passed the success-only validity gate from #118 and were scored as correct at $0 cost. The validity gate now also requires output_tokens > 0; such rows are treated as failed inference (scored as wrong, excluded from cost) consistently across the RouterArena score and the optimality pool. This extends the #118 empty-response fix rather than duplicating it. Also adds: - check_config_prediction_files.py: submission-time validation rejecting successful non-empty generations without usable output_tokens, so future submissions are caught up front. - tools/audit_token_accounting.py: audit script reporting per-router reasoning-token magnitude and failed-inference counts to forward to the router authors. - scripts/rescore_affected.sh: reproducible driver to re-score affected routers through the patched pipeline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Leaderboard: re-score reasoning-token routers under the #135 fix Re-evaluated, at current model_cost.json prices, the routers whose models emit reasoning tokens or zero-usage rows. Routers with neither are unchanged. - vLLM-SR: arena 75.38->72.15, acc 75.97->73.19, cost $0.11->$0.23/1k (reasoning tokens now billed; 279 zero-usage rows now scored as wrong) - Nadir: arena 73.33->72.29, acc 74.87->75.01, cost $0.29->$0.68/1k - Azure: arena 71.87->70.42, acc 72.82->72.94, cost $0.22->$0.73/1k New top of leaderboard: Sqwish #1, AgentForge #2, Weave #3, Nadir #4, vLLM-SR #5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Apply pre-commit fixes (ruff-format, codespell) - ruff-format reflows the reasoning-cost expression and audit-script lines to match the repo's pinned formatter (ruff 0.11.7). - Rename token shorthands it/ot/tt -> n_in/n_out/n_total in the audit script so codespell no longer flags "ot" as a misspelling. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

… gates (#134) * feat: legitimate improvement plan + submission integrity check - docs/ROUTERARENA_IMPROVEMENT_PLAN.md: tiered roadmap of legitimate optimization paths (system prompts, self-consistency, per-dataset routing, confidence-fallback). Defines allowed vs forbidden routing signals to prevent recurring the Lever #3 test-set leakage. - scripts/check_submission_integrity.py: pre-submission leakage detector. Three checks (diff analysis, AST source-code scan, reassignment-plan inspection). Exits 1 on any leak. - tests/test_submission_integrity.py: 13 tests proving the detector catches the Lever #3 pattern AND passes on legitimate patterns (inference-failure routing, prompt-only features, mixed-accuracy reassignment, no-baseline initial submission). - docs/SESSION_RESTART_CHECKLIST.md: pre-flight checklist for the next Claude Code session: MCP restart, config verification, briefing prompt for the next assistant, integrity-check usage. * docs(checklist): add subprocess-leak cleanup step (1.0) The leak chain that bit us: cancelled MCP tool calls leave orphan Codex CLI subprocesses; aborted background pytest runs accumulate. RAM pressure → Ollama model thrashing → 30-90s latencies → new MCP calls in fresh sessions appear stuck. New step 1.0 runs before MCP restart: pkill leaked pytests, pkill leaked codex execs, unload the slow qwen3.5 thinking model, verify Ollama latency and free memory. Includes symptom checklist so users can diagnose recurrence. * docs(audit): Tier 1C robustness diagnosis The 0.30 → 0.236 robustness drop is a mechanical consequence of Lever #3, not a scoring bug. Robustness score = 1 - (flips / total) where a flip = router picked a different model on the paraphrased prompt than on the original. Lever #3 mutated the full predictions to use stronger models; the robustness file was untouched, so the post-Lever-#3 full set no longer matched the robustness set for those queries. Restoring the 065cca5 baseline recovers the score exactly: 126 matches out of 420 = 0.3000. Verified by running the scoring identity (global_utils/robustness.compute_robustness_score) on router_inference/predictions/llm-router.json.bak.honest. Root cause of the 0.30 baseline itself: the router's classifier crosses complexity thresholds when prompts are paraphrased. 213 of 294 baseline flips (72%) are workhorse → reasoning-model upgrades: - 114 deepseek-v4-flash → qwen3-235b-a22b-2507 - 99 gemini-3.1-flash-lite → qwen3-235b-a22b-2507 - 35 code-detection unstable (Qwen3-Coder-Next ↔ qwen3-235b) Tier 1C alone is a wash: mechanical recovery to 0.30 once the branch sits on 065cca5. Real robustness lift requires routing by Global Index prefix (Tier 2A) or prompt normalization. Both legitimate — no accuracy or cost signals consumed. * chore(router): restore llm-router config and adapter from PR branch Reinstates the pre-Lever-#3 routing infrastructure on this branch so Tier 1 work can run against a clean foundation: - router_inference/config/llm-router.json — 5-model pool used by the open PR (qwen3-235b, qwen3-next-80b, Qwen3-Coder-Next, gemini-3.1-flash-lite, deepseek-v4-flash). - router_inference/router/llm_router.py — deterministic regex classifier + dataset-prefix override + selection matrix. No network calls during routing. Source-code leakage scan passes (uses prompt features only; never reads accuracy or cost). Both files copied verbatim from submit-llm-router-v10.1.1. check_submission_integrity.py confirms clean. * feat(tier1a): MC detector + boxed extractor + majority vote primitives Pure functions for Tier 1A self-consistency, with 33 tests covering extraction, voting, and full-pipeline behavior on realistic model outputs. Uses only prompt text and output text — no accuracy or cost signals consumed. Empirical validation on the 065cca5 baseline (8400 prompts, 11636 entries): - 5844 MC, 2556 non-MC (matches PR body) - Every dataset prefix is 100% MC or 0% MC; detection by prompt opening is sufficient - extract_mc_letter() recovers a single A-J letter from 5818/5844 (99.6%) of baseline MC outputs Detection logic: - is_multiple_choice() looks at the first 300 chars only for the canonical "multiple-choice question" / "based on the options" signatures. Buried matches in long non-MC prompts do not count. - extract_boxed_answer() mirrors the regex used by llm_evaluation.enhanced_extractor.EnhancedExtractor so the local vote tally agrees with the leaderboard's eventual scoring. - extract_mc_letter() falls back to "answer is X" / "Answer: X" phrases when \boxed{} is absent. - majority_vote() enforces min_agreement (default 2 for 3-sample voting) and returns None on ties so the caller can fall back to the first sample rather than guess. No inference runner yet — that lives in the follow-up commit. * feat(tier1a): self-consistency inference runner + sub_10 smoke green scripts/run_self_consistency.py loads the honest baseline predictions, runs each multiple-choice entry's assigned model N=3 times at temperature=0.7 via OpenRouter, extracts the \boxed{X} letter from each sample, takes the majority vote, and writes the result into generated_result. Non-MC and for_optimality entries pass through untouched. Bounded-retry HTTP wrapper, resume-from-cache via --cache flag, dry-run mode for cost preview. Sub_10 smoke (50 MC entries, 150 OpenRouter calls): Pairwise vs baseline: structural mismatches: 0 prompt field changed: 0 prediction field changed: 0 for_optimality mutated: 0 generated_result changed: 50 (all MC regular entries) Vote tally: unanimous (no vote needed): 48 / 50 majority vote actually fired: 2 / 50 no extractable letter: 0 / 50 Integrity check: ✓ passes (diff check now live with baseline present; zero prediction reassignments => Lever #3 rule trivially holds). Concrete vote wins captured in docs/submission_audit.md — both cases where the cheap model emitted a wrong first sample but the majority of 3 agreed on the consensus answer. That's the regime Tier 1A targets. Smoke artifacts (.self_consistency_cache.json, *.sub10-smoke.json) are intentionally not committed — they're 7.4 MB / 56 KB working files. Full-split run (~17.5k calls, ~$2) is the next step, to be authorized in a separate session. * feat(tier1b): task-family system prompts + runner robustness Tier 1B layers task-family-tailored system prompts on top of the Tier 1A self-consistency sampler. System prompts are picked from prompt text only — no accuracy/cost/label signals — preserving the integrity gates. Runner hardening picked up from the first end-to-end run: - Hard-fail on quota errors (Key limit, 402, insufficient_quota) so a dead key surfaces immediately instead of burning ~30 min of retries. - Incremental output checkpoint every 100 MC entries — a mid-run abort now leaves a valid submission instead of only the cache. - Cache key fingerprints SYSTEM_PROMPT_VERSION + on/off, so bumping the prompt set invalidates stale samples but preserves --no-system-prompts baselines. Tests: 56 passing (41 self_consistency + 13 submission_integrity + 2 mc). Integrity check still green against bak.honest baseline. * style: apply ruff-format + pymarkdown autofixes Pure formatting — no behavior changes: - ruff: drop unused Counter import (F401) - ruff-format: line-length wrapping in 4 Python files - ruff: f-string with no interpolation → plain string (F541) - pymarkdown: blank line before fenced code blocks in SESSION_RESTART_CHECKLIST Resolves the CI pre-commit failures on PR #134. * feat(submission): regenerate predictions + register LLMRouter Adds the two prediction files required by .github/workflows/pr-evaluation.yml (line 155 invariant: exactly one base + one robustness file). Without them, /evaluate returned NEUTRAL with "No router file detected" on this PR. - router_inference/predictions/llm-router.json (22 MB): Full split, Tier-1A self-consistency + Tier-1B task-family prompts applied over the honest baseline (.bak.honest). Integrity gates green. - router_inference/predictions/llm-router-robustness.json (~700 KB): Robustness split, freshly routed via current LLMRouter config. Tier-1A applied (6/420 MC entries voted, 414 passthrough). Integrity gates green. - router_inference/router/__init__.py: Register LLMRouter so generate_prediction_file.py and the evaluator can resolve the router class. Previously the PR shipped llm_router.py but not the package-level import, which would have crashed evaluation. Verification: uv run python scripts/check_submission_integrity.py # ✓ uv run python scripts/check_submission_integrity.py \\ --predictions router_inference/predictions/llm-router-robustness.json \\ --baseline router_inference/predictions/llm-router-robustness.json.bak.honest # ✓ uv run pytest tests/test_self_consistency.py tests/test_submission_integrity.py # 56 passed * fix(tier1a): emit generated_result as dict matching evaluator schema The validator (router_inference/check_config_prediction_files.py with --check-generated-result) requires generated_result to be a dict with {generated_answer: str, success: bool, token_usage}. The previous shape — json.dumps({"generated_answer": "..."}) — produced a JSON-encoded string, not a dict, and was also missing the success and token_usage fields. The prior /evaluate run failed with 299 "must be a dictionary, got str" errors on the full split. Changes: - scripts/run_self_consistency.py: _format_vote_as_generated_result now takes (letter, prompt, samples) and returns a dict with token_usage estimated from sample text lengths (1 token ≈ 4 chars × num_samples). Reporting full multi-sample tokens honestly attributes self-consistency cost; under-counting would deflate cost → inflate Arena Score. - router_inference/predictions/llm-router.json: regenerated/patched. All 11636 entries now have dict generated_result. Integrity gates green. CI validator (--check-generated-result) green locally. - router_inference/predictions/llm-router-robustness.json: 6 MC entries patched to dict shape via Tier-1A. The 414 non-MC entries remain null — not blocking for /evaluate which runs --split full only. Verification: uv run pytest tests/test_self_consistency.py tests/test_submission_integrity.py → 56 passed uv run python scripts/check_submission_integrity.py → ✓ all gates green uv run python router_inference/check_config_prediction_files.py llm-router full --check-generated-result → ✓ ALL CHECKS PASSED * chore: trim PR to RouterArena submission files only Remove tooling/docs per maintainer review (yl231 on PR #134). Integrity checker and self-consistency utilities remain in ypollak2/llm-router; only the 5 leaderboard submission files stay in this PR.

@loswald

…s#172) xAI retired `grok-4-1-fast-reasoning` on 2026-05-15 and silently redirects it to grok-4.3 (low reasoning), which is stronger and ~5-6x pricier. Submissions that selected the retired slug via xAI/OpenRouter were served grok-4.3 but billed at the retired slug's $0.20/$0.50 price, inflating their cost-aware Arena scores. Reported and audited by @loswald (thanks!). Fix (prices the model that actually answered, not the requested alias): - model_cost.json: add `grok-4.3` and `x-ai/grok-4.3` at $1.25/$2.50. - evaluate_models.py: add `has_price()`; factor the cost lookup into `_lookup_cost_info()`. - run.py: bill by `generated_result.model_used` when it is present and priced, else the router's selected model. Redirected slugs are now billed correctly. - check_config_prediction_files.py: preflight warning for retired/redirected slugs (guardrail RouteWorks#1 from the issue). Precision (only provably-redirected rows change; Azure-hosted Grok 4.1 is left alone): - cross-router 217 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - vllm-sr 257 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - nadir-tumbler 3571 rows (xAI, model_used absent): backfilled model_used=x-ai/grok-4.3 (retired slug + 2026-07-04 generation date) so the same mechanism reprices it. Team to be notified. - azure-model-router (4803) and nadir-cascade-v2 (3101): provider=azure, NOT subject to the xAI redirect -> unchanged. Leaderboard (accuracy held fixed, affected calls billed at grok-4.3): - Cross-Router 76.12 -> 75.75 ($0.30 -> $0.40/1K) stays RouteWorks#1 - vLLM-SR 75.30 -> 74.86 ($0.30 -> $0.42/1K) RouteWorks#2 -> RouteWorks#3 - Nadir-Tumbler 75.17 -> 73.44 ($0.08 -> $0.37/1K) RouteWorks#4 -> RouteWorks#5 Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github ci

1778203

jiarong0907 merged commit cae23a2 into main Oct 27, 2025

jiarong0907 deleted the setup/ci branch October 27, 2025 03:40

gemini-code-assist Bot reviewed Oct 27, 2025

View reviewed changes

yl231 added a commit that referenced this pull request Nov 27, 2025

- fixed and re-run #3

ed73ba9

yl231 mentioned this pull request May 31, 2026

Update leaderboard: add Nadir (#112), update AgentForge (#117) #121

Merged

ypollak2 mentioned this pull request Jun 4, 2026

Add llm-router submission #132

Closed

3 tasks

loswald mentioned this pull request Jul 15, 2026

Leaderboard integrity: retired Grok 4.1 Fast slug silently resolved to Grok 4.3 at the old price #166

Closed

This was referenced Aug 5, 2026

Fix Grok 4.1 Fast → Grok 4.3 mispricing on the leaderboard (#166) #172

Merged

Update vLLM-SR RouterArena submission #147

Merged

This was referenced Aug 20, 2026

/evaluate throws Node.js 20 is deprecated. error #176

Closed

Divyam router submission. #175

Merged

Navigation Menu

Uh oh!

[Setup] GitHub CI - #3

[Setup] GitHub CI#3
jiarong0907 merged 1 commit into
mainfrom
setup/ci

jiarong0907 commented Oct 27, 2025

Uh oh!

gemini-code-assist Bot commented Oct 27, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jiarong0907 commented Oct 27, 2025

Uh oh!

gemini-code-assist Bot commented Oct 27, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant