[Setup] License - #4
Conversation
Summary of ChangesHello @jiarong0907, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request establishes the licensing terms for the project by adding the Apache License, Version 2.0. This is a foundational step for any open-source project, ensuring clarity on how the software can be used, shared, and contributed to by the community. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds an Apache 2.0 license file to the project, which is a great step towards clarifying the terms of use for your software. My review focuses on ensuring the license is fully and correctly applied. I've added one comment suggesting the next step to complete the license setup by specifying the copyright holder, which is a crucial part of applying the Apache 2.0 license.
| same "printed page" as the copyright notice for easier | ||
| identification within third-party archives. | ||
|
|
||
| Copyright [yyyy] [name of copyright owner] |
There was a problem hiding this comment.
To complete the license setup, the copyright notice needs to be defined. The appendix of the license provides a template with placeholders [yyyy] and [name of copyright owner]. It's important to provide this information for the project.
A common way to do this is by creating a NOTICE file in the project root with the copyright statement, for example:
Copyright 2024 The RouterArena Authors
Alternatively, since a tools/addlicense.sh script exists in the repository, you could configure and use it to apply the appropriate license headers, including the filled-out copyright notice, to all your source files. This would ensure the license is fully applied across the project.
* Fix cost accounting for reasoning tokens and zero-usage generations (#135) Issue #135 reported two distinct cost-accounting gaps in the evaluator: 1. Reasoning tokens were never billed. calculate_inference_cost charged only input+output tokens, ignoring reasoning/"thinking" tokens, which providers bill at the completion rate but report only via the gap total_tokens - (input + output). They are now billed at the model's output rate (or an explicit reasoning_token_price_per_million when configured). Affected routers: Azure (8.6M tokens), Nadir (6.4M), vLLM-SR (2.0M). 2. Successful generations with no usable token usage were free. Rows with success=True and a non-empty answer but empty token_usage ({}) or output_tokens=0 passed the success-only validity gate from #118 and were scored as correct at $0 cost. The validity gate now also requires output_tokens > 0; such rows are treated as failed inference (scored as wrong, excluded from cost) consistently across the RouterArena score and the optimality pool. This extends the #118 empty-response fix rather than duplicating it. Also adds: - check_config_prediction_files.py: submission-time validation rejecting successful non-empty generations without usable output_tokens, so future submissions are caught up front. - tools/audit_token_accounting.py: audit script reporting per-router reasoning-token magnitude and failed-inference counts to forward to the router authors. - scripts/rescore_affected.sh: reproducible driver to re-score affected routers through the patched pipeline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Leaderboard: re-score reasoning-token routers under the #135 fix Re-evaluated, at current model_cost.json prices, the routers whose models emit reasoning tokens or zero-usage rows. Routers with neither are unchanged. - vLLM-SR: arena 75.38->72.15, acc 75.97->73.19, cost $0.11->$0.23/1k (reasoning tokens now billed; 279 zero-usage rows now scored as wrong) - Nadir: arena 73.33->72.29, acc 74.87->75.01, cost $0.29->$0.68/1k - Azure: arena 71.87->70.42, acc 72.82->72.94, cost $0.22->$0.73/1k New top of leaderboard: Sqwish #1, AgentForge #2, Weave #3, Nadir #4, vLLM-SR #5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Apply pre-commit fixes (ruff-format, codespell) - ruff-format reflows the reasoning-cost expression and audit-script lines to match the repo's pinned formatter (ruff 0.11.7). - Rename token shorthands it/ot/tt -> n_in/n_out/n_total in the audit script so codespell no longer flags "ot" as a misspelling. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ap (#156) Cross-Router (#149, @JiaHg) merged and evaluated: Arena 73.71, Accuracy 75.13, Cost/1K $0.26, Opt 40.18/44.03/89.58, Robustness 59.05. Enters at #4 (between AgentForge and Weave); ranks 5-23 shift down one. Numbers are the official /evaluate metrics; an independent adversarial audit reproduced Arena/Accuracy/Cost/Robustness exactly and found the submission genuine (real inference, ground-truth exact-match 13.5%, valid token usage, real 420-row robustness file). Also adds the missing manifest entries for Cross-Router AND LLM Router (#139) so the website sync stops skipping them. Verified build_site_data.py regenerates 23 leaderboard rows including both. Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…er (#159); Cross-Router update to #1 (#163) (#165) Adds three newly-merged router submissions and refreshes the Cross-Router row from its updated submission. Official /evaluate metrics: - Cross-Router (#163): Arena 76.12 · Acc 78.14% · $0.30/1K · Robust 67.14 -> now #1 - Nadir-Tumbler (#159): Arena 75.17 · Acc 75.34% · $0.08/1K · Robust 66.43 -> #4 - chuzom-solo-v32 (#161): Arena 70.61 · Acc 70.59% · $0.10/1K · Robust 100.00 -> #12 - BARouter (#157): Arena 67.09 · Acc 68.80% · $0.63/1K · Robust 52.38 -> #15 Manifest entries added for the three new routers so the website sync regenerates their derived data. Co-authored-by: yl231 <jamesbond19925@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s#172) xAI retired `grok-4-1-fast-reasoning` on 2026-05-15 and silently redirects it to grok-4.3 (low reasoning), which is stronger and ~5-6x pricier. Submissions that selected the retired slug via xAI/OpenRouter were served grok-4.3 but billed at the retired slug's $0.20/$0.50 price, inflating their cost-aware Arena scores. Reported and audited by @loswald (thanks!). Fix (prices the model that actually answered, not the requested alias): - model_cost.json: add `grok-4.3` and `x-ai/grok-4.3` at $1.25/$2.50. - evaluate_models.py: add `has_price()`; factor the cost lookup into `_lookup_cost_info()`. - run.py: bill by `generated_result.model_used` when it is present and priced, else the router's selected model. Redirected slugs are now billed correctly. - check_config_prediction_files.py: preflight warning for retired/redirected slugs (guardrail RouteWorks#1 from the issue). Precision (only provably-redirected rows change; Azure-hosted Grok 4.1 is left alone): - cross-router 217 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - vllm-sr 257 rows (openrouter, model_used=x-ai/grok-4.3) -> repriced - nadir-tumbler 3571 rows (xAI, model_used absent): backfilled model_used=x-ai/grok-4.3 (retired slug + 2026-07-04 generation date) so the same mechanism reprices it. Team to be notified. - azure-model-router (4803) and nadir-cascade-v2 (3101): provider=azure, NOT subject to the xAI redirect -> unchanged. Leaderboard (accuracy held fixed, affected calls billed at grok-4.3): - Cross-Router 76.12 -> 75.75 ($0.30 -> $0.40/1K) stays RouteWorks#1 - vLLM-SR 75.30 -> 74.86 ($0.30 -> $0.42/1K) RouteWorks#2 -> RouteWorks#3 - Nadir-Tumbler 75.17 -> 73.44 ($0.08 -> $0.37/1K) RouteWorks#4 -> RouteWorks#5 Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
No description provided.