Skip to content

Update BARouter submission - #180

Merged
yl231 merged 1 commit into
RouteWorks:mainfrom
zulk2002:resubmit-barouter
Aug 20, 2026
Merged

Update BARouter submission#180
yl231 merged 1 commit into
RouteWorks:mainfrom
zulk2002:resubmit-barouter

Conversation

@zulk2002

Copy link
Copy Markdown
Contributor

This PR updates the BARouter submission artifacts for RouterArena leaderboard evaluation.
BARouter is a budget-adaptive, cost-aware router. For this submission, the router selects among:

google/gemini-3.1-flash-lite
deepseek/deepseek-v4-flash
deepseek/deepseek-v4-pro

The submitted artifacts contain the final routing decisions and generated outputs. No local predictor artifacts or training files are included.

@zulk2002

Copy link
Copy Markdown
Contributor Author

/evaluate

@zulk2002

Copy link
Copy Markdown
Contributor Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: BARouter
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.7375
Accuracy 75.33%
Total Cost $2.444208
Avg Cost per Query $0.000291
Avg Cost per 1K Queries $0.2910
Number of Queries 8400
Abnormal Entries 25
Robustness Score 0.8310

⚠️ 25 of 8400 queries (0.3%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.7966
Opt.Cost (Cost Efficiency) 0.7207
Opt.Acc (Accuracy vs Optimal) 0.9245

Evaluation completed by RouterArena automated workflow

@zulk2002

Copy link
Copy Markdown
Contributor Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: BARouter
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.7379
Accuracy 75.72%
Total Cost $3.058621
Avg Cost per Query $0.000364
Avg Cost per 1K Queries $0.3641
Number of Queries 8400
Abnormal Entries 21
Robustness Score 0.6881

⚠️ 21 of 8400 queries (0.3%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.6441
Opt.Cost (Cost Efficiency) 0.6713
Opt.Acc (Accuracy vs Optimal) 0.9384

Evaluation completed by RouterArena automated workflow

@yl231
yl231 merged commit d513354 into RouteWorks:main Aug 20, 2026
6 checks passed
@yl231

yl231 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Merged — the BARouter update lands at #8 (Arena 67.09 → 73.79, accuracy 75.72%, $0.36/1K) on the refreshed deepseek-v4 / gemini-3.1 pool. Clean audit. Thanks @zulk2002!

ypollak2 pushed a commit to ypollak2/RouterArena that referenced this pull request Aug 23, 2026
… Divyam, nadir-caliper, AsiaInfo (RouteWorks#184)

- Paix2 (RouteWorks#164): new 🥇 arena 77.63 / acc 79.69% / $0.27
- Sqwish update (RouteWorks#177): 🥈 75.27 -> 76.21 (sqwish-fvcta8 pool)
- Divyam (RouteWorks#175): new 🥉 arena 75.85 / acc 78.59%
- nadir-caliper (RouteWorks#173): 74.55, supersedes Nadir-Tumbler (73.44)
- BARouter update (RouteWorks#180): 67.09 -> 73.79
- AsiaInfo (RouteWorks#181): arena 65.87 (high-cost 2-model)

All audited: 8400 unique rows, real tokens, faithfulness gaps within trusted band.

Co-authored-by: Louie Lu <yl231@datalab2.cs.rice.edu>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants