[Setup] Gemini code review - #2
Conversation
Summary of ChangesHello @jiarong0907, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request establishes the initial setup for a Gemini-powered code review system by introducing its configuration file. Concurrently, it enhances repository cleanliness by updating the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request sets up Gemini code review by adding a configuration file and updating the .gitignore. The changes look good, but I have a few suggestions for the configuration to ensure it behaves as expected and to follow common file formatting practices. Specifically, there's a contradictory comment and a potentially misconfigured setting that would prevent reviews on PR creation. I've also suggested adding trailing newlines to both new/modified files and improving the patterns in .gitignore.
| .mypy_cache/* | ||
| .ruff_cache/* No newline at end of file |
There was a problem hiding this comment.
For ignoring directories, it's more conventional to use a trailing slash (e.g., .mypy_cache/) instead of /*. Also, text files should ideally end with a newline. The suggestion below applies both of these improvements.
.mypy_cache/
.ruff_cache/
Weave Router v0.62 retook #1 on the leaderboard with Acc-Cost Arena 76.09 (up from 74.61 on v0.27), accuracy 79.32%, cost/1K $0.61. Sqwish Router moves to #2 at 75.27. New v0.62 row: - Acc-Cost Arena: 74.61 → 76.09 - Accuracy: 78.43 → 79.32 - Cost/1K: $0.92 → $0.61 - Opt. Selection: 1.27 → 5.60 - Opt. Cost: 12.10 → 17.19 - Opt. Accuracy: 89.56 → 89.21 - Robustness: 79.05 → 79.76 Submission PR: #102 Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Weave Router v0.54 (commit bd64f8a), evaluated on the full split: arena 0.7282, accuracy 76.32%, cost/1K $0.9442, robustness 100.00. Ranks #2 by Acc-Cost Arena; existing entries shift down one rank. Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update leaderboard: add Nadir (#112), update AgentForge (#117) - AgentForge (#117): refreshed to faithful post-fix metrics (arena 74.13, acc 74.72%, cost $0.13/1K, abnormal 0) — moves #13 -> #2. - Nadir (#112): new entry, arena 73.33 (acc 74.87%, cost $0.29/1K) at #3. Optimality left as — pending review (computed 1.0/1.0/1.0 is an artifact of the submission listing only expensive optimality candidates). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update Nadir Router link in README.md --------- Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix cost accounting for reasoning tokens and zero-usage generations (#135) Issue #135 reported two distinct cost-accounting gaps in the evaluator: 1. Reasoning tokens were never billed. calculate_inference_cost charged only input+output tokens, ignoring reasoning/"thinking" tokens, which providers bill at the completion rate but report only via the gap total_tokens - (input + output). They are now billed at the model's output rate (or an explicit reasoning_token_price_per_million when configured). Affected routers: Azure (8.6M tokens), Nadir (6.4M), vLLM-SR (2.0M). 2. Successful generations with no usable token usage were free. Rows with success=True and a non-empty answer but empty token_usage ({}) or output_tokens=0 passed the success-only validity gate from #118 and were scored as correct at $0 cost. The validity gate now also requires output_tokens > 0; such rows are treated as failed inference (scored as wrong, excluded from cost) consistently across the RouterArena score and the optimality pool. This extends the #118 empty-response fix rather than duplicating it. Also adds: - check_config_prediction_files.py: submission-time validation rejecting successful non-empty generations without usable output_tokens, so future submissions are caught up front. - tools/audit_token_accounting.py: audit script reporting per-router reasoning-token magnitude and failed-inference counts to forward to the router authors. - scripts/rescore_affected.sh: reproducible driver to re-score affected routers through the patched pipeline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Leaderboard: re-score reasoning-token routers under the #135 fix Re-evaluated, at current model_cost.json prices, the routers whose models emit reasoning tokens or zero-usage rows. Routers with neither are unchanged. - vLLM-SR: arena 75.38->72.15, acc 75.97->73.19, cost $0.11->$0.23/1k (reasoning tokens now billed; 279 zero-usage rows now scored as wrong) - Nadir: arena 73.33->72.29, acc 74.87->75.01, cost $0.29->$0.68/1k - Azure: arena 71.87->70.42, acc 72.82->72.94, cost $0.22->$0.73/1k New top of leaderboard: Sqwish #1, AgentForge #2, Weave #3, Nadir #4, vLLM-SR #5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Apply pre-commit fixes (ruff-format, codespell) - ruff-format reflows the reasoning-cost expression and audit-script lines to match the repo's pinned formatter (ruff 0.11.7). - Rename token shorthands it/ot/tt -> n_in/n_out/n_total in the audit script so codespell no longer flags "ot" as a misspelling. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…s#172) xAI retired `grok-4-1-fast-reasoning` on 2026-05-15 and silently redirects it to grok-4.3 (low reasoning), which is stronger and ~5-6x pricier. Submissions that selected the retired slug via xAI/OpenRouter were served grok-4.3 but billed at the retired slug's $0.20/$0.50 price, inflating their cost-aware Arena scores. Reported and audited by @loswald (thanks!). Fix (prices the model that actually answered, not the requested alias): - model_cost.json: add `grok-4.3` and `x-ai/grok-4.3` at $1.25/$2.50. - evaluate_models.py: add `has_price()`; factor the cost lookup into `_lookup_cost_info()`. - run.py: bill by `generated_result.model_used` when it is present and priced, else the router's selected model. Redirected slugs are now billed correctly. - check_config_prediction_files.py: preflight warning for retired/redirected slugs (guardrail RouteWorks#1 from the issue). Precision (only provably-redirected rows change; Azure-hosted Grok 4.1 is left alone): - cross-router 217 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - vllm-sr 257 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - nadir-tumbler 3571 rows (xAI, model_used absent): backfilled model_used=x-ai/grok-4.3 (retired slug + 2026-07-04 generation date) so the same mechanism reprices it. Team to be notified. - azure-model-router (4803) and nadir-cascade-v2 (3101): provider=azure, NOT subject to the xAI redirect -> unchanged. Leaderboard (accuracy held fixed, affected calls billed at grok-4.3): - Cross-Router 76.12 -> 75.75 ($0.30 -> $0.40/1K) stays RouteWorks#1 - vLLM-SR 75.30 -> 74.86 ($0.30 -> $0.42/1K) RouteWorks#2 -> RouteWorks#3 - Nadir-Tumbler 75.17 -> 73.44 ($0.08 -> $0.37/1K) RouteWorks#4 -> RouteWorks#5 Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
No description provided.