Divyam router submission.

[samikd]

new file:   ../config/divyam-router.json
new file:   divyam-router-robustness.json
new file:   divyam-router.json

[samikd]

/evaluate

[samikd]

Can someone please look into this: /evaluate throws the following error:-

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/github-script@v7, actions/setup-python@v5. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

[samikd]

Can someone please look into this: /evaluate throws the following error:-

Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/github-script@v7, actions/setup-python@v5. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Issue: #176 (comment)

[samikd]

/evaluate

[samikd]

/evaluate

[samikd]

/evaluate

[github-actions]

Router Evaluation Results

Router: divyam-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7596
Accuracy	78.31%
Total Cost	$3.194976
Avg Cost per Query	$0.000380
Avg Cost per 1K Queries	$0.3804
Number of Queries	8400
Abnormal Entries	4
Robustness Score	0.9833

⚠️ 4 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.0534
Opt.Cost (Cost Efficiency)	0.1832
Opt.Acc (Accuracy vs Optimal)	0.9318

---

Evaluation completed by RouterArena automated workflow

[loswald]

Really interesting submission. The routing pattern is striking: 8,280 of 8,400 queries, or 98.57%, go to gemini-3-flash-preview.

On the complete 809-query optimality subset included in the artifact, simply choosing Gemini 3 Flash every time is actually slightly better:

• Fixed Gemini: 82.097% accuracy, 79.196 Arena
• Submitted routing: 81.823% accuracy, 78.992 Arena
• Additional fixed-arm cost: only $0.0067/1K

However more importantly, the Gemini cost accounting appears incorrect. RouterArena recalculates cost using the submitted token_usage, but does not verify those counts against provider metadata. The [generation code used here](https://github.com/Divyam-AI/RouterArena/blob/c8effda3eafe8ba0bbafaa4c7937d15b22b70ced/llm_inference/model_inference.py#L436-L462) estimates both prompt and completion tokens as 1.3 × whitespace words, then sets total tokens to their sum. This necessarily reports no hidden thinking tokens.

Google’s current price is correctly registered at $0.50/M input and $3/M output, but that output price explicitly [includes thinking tokens](https://ai.google.dev/gemini-api/docs/pricing#gemini-3-flash-preview). The issue is therefore the token quantities, not the price.

There is a useful public control in the current #1 submission [Cross Router](#163). It used the same Gemini model on the same 148 ChessInstruct queries:

• Cross Router: 81/148 correct, 54.73%
• This submission: 83/148 correct, 56.08%
• Cross reported 1,084.9 mean total tokens per query
• This submission reported 385.9

The answers are independently generated, with no exact response matches. However, the very similar accuracy makes the 2.8× difference in reported total usage worth resolving. Cross’s counts are the normal provider-style/tokenizer-style values rather than the whitespace estimator used here, although its artifact alone does not prove identical thinking settings.

Could the submission be regenerated using Google’s returned promptTokenCount, candidatesTokenCount, thoughtsTokenCount, totalTokenCount, and modelVersion, together with the exact thinking configuration? The current $0.3804/1K or resulting Arena score does not seem reliable until that is fixed.

[samikd]

Really interesting submission. The routing pattern is striking: 8,280 of 8,400 queries, or 98.57%, go to gemini-3-flash-preview.

On the complete 809-query optimality subset included in the artifact, simply choosing Gemini 3 Flash every time is actually slightly better:

• Fixed Gemini: 82.097% accuracy, 79.196 Arena
• Submitted routing: 81.823% accuracy, 78.992 Arena
• Additional fixed-arm cost: only $0.0067/1K

However more importantly, the Gemini cost accounting appears incorrect. RouterArena recalculates cost using the submitted token_usage, but does not verify those counts against provider metadata. The [generation code used here](https://github.com/Divyam-AI/RouterArena/blob/c8effda3eafe8ba0bbafaa4c7937d15b22b70ced/llm_inference/model_inference.py#L436-L462) estimates both prompt and completion tokens as 1.3 × whitespace words, then sets total tokens to their sum. This necessarily reports no hidden thinking tokens.

Google’s current price is correctly registered at $0.50/M input and $3/M output, but that output price explicitly [includes thinking tokens](https://ai.google.dev/gemini-api/docs/pricing#gemini-3-flash-preview). The issue is therefore the token quantities, not the price.

There is a useful public control in the current #1 submission [Cross Router](#163). It used the same Gemini model on the same 148 ChessInstruct queries:

• Cross Router: 81/148 correct, 54.73%
• This submission: 83/148 correct, 56.08%
• Cross reported 1,084.9 mean total tokens per query
• This submission reported 385.9

The answers are independently generated, with no exact response matches. However, the very similar accuracy makes the 2.8× difference in reported total usage worth resolving. Cross’s counts are the normal provider-style/tokenizer-style values rather than the whitespace estimator used here, although its artifact alone does not prove identical thinking settings.

Could the submission be regenerated using Google’s returned promptTokenCount, candidatesTokenCount, thoughtsTokenCount, totalTokenCount, and modelVersion, together with the exact thinking configuration? The current $0.3804/1K or resulting Arena score does not seem reliable until that is fixed.

Thank you for your insightful comments – I really appreciate the rigour.

The evidence you've provided regarding cost calculation (2.8x cost difference) is compelling.

I would take another look at the cost calculation and circle back. Until then, let me move this PR to a draft state.

[nilam-divyam]

Summary. Google-path token counts in our earlier submission were estimated as 1.3 × whitespace_words rather than read from the API. They are now taken from the response's usage_metadata. The correction is validated against an independent external reference on ChessInstruct (agreement within 0.8%) and raises reported cost by 1.27× on Gemini-routed queries. ChessInstruct is the extreme case at 2.84× total tokens.

1. What changed

Before — inherited verbatim from the RouterArena reference implementation, which sets no thinking configuration and estimates both counts:

input_tokens  = len(prompt.split()) * 1.3
output_tokens = len(text.split()) * 1.3 if text else 0

After — provider counts, with thought tokens folded into output to match Google's thinking-inclusive output price:

input_tokens  = usage.prompt_token_count
output_tokens = usage.candidates_token_count + usage.thoughts_token_count

Each row now also records promptTokenCount, candidatesTokenCount, thoughtsTokenCount, totalTokenCount, model_version, and the literal thinking_config, so counts can be reconciled against the provider without trusting our arithmetic. We have explicitly set thinking_budget=0 from the start, unchanged across both runs.

2. Metering cross-checked against Cross Router submission for ChessInstruct

The Cross Router submission ran the same Gemini model over the same 148 queries with its own implementation and its own generations.

ChessInstruct, mean total tokens/query	value	vs Cross Router
Cross Router (independent, reported)	1,084.9	—
this submission, latest	1,093.5	+0.8%
this submission, previously estimated	385.9	−64.5%

Two independently written accounting paths, over independently generated responses, agree to within 0.8%.

3. Magnitude across splits

input × is a pure re-count over byte-identical prompts and isolates the metering change. total × and cost × also absorb output-length variance, since responses were regenerated.

split	n	input ×	total ×	cost ×
ChessInstruct	148	3.50	2.84	1.86
MMLUPro_engineering	280	1.62	1.69	1.77
MATH	55	1.41	1.53	1.59
MMLUPro_business	166	1.39	1.48	1.58
MMLUPro_math	205	1.37	1.46	1.54
AIME	39	1.35	1.68	1.78
LiveCodeBench	385	1.35	1.48	1.62
MMLUPro_computer science	392	1.31	1.33	1.36
PubMedQA	466	1.18	1.16	1.11
ArcMMLU	396	1.12	1.10	1.08
GeoBench	330	1.07	1.06	1.04
MMLUPro_history	325	1.02	1.03	1.03
NarrativeQA	380	1.01	1.01	1.01
SuperGLUE-RC	82	1.00	1.01	1.03
all Gemini-routed splits	8,369	1.20	1.23	1.27

ChessInstruct and few other datasets with symbol-dense prompts affect the metering most.

[nilam-divyam]

/evaluate

[samikd]

/evaluate

[github-actions]

Router Evaluation Results

Router: divyam-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7585
Accuracy	78.59%
Total Cost	$4.036801
Avg Cost per Query	$0.000481
Avg Cost per 1K Queries	$0.4806
Number of Queries	8400
Abnormal Entries	4
Robustness Score	0.9833

⚠️ 4 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.0493
Opt.Cost (Cost Efficiency)	0.1603
Opt.Acc (Accuracy vs Optimal)	0.9343

---

Evaluation completed by RouterArena automated workflow

[yl231]

Merged and live — Divyam is #3 🥉 (Arena 75.85, accuracy 78.59%, $0.48/1K). The re-metered Google-path token counts resolved the earlier cost-accounting concern, and the faithfulness gap (+0.031) is clean. Thanks @samikd / @nilam-divyam — and your Node.js report is addressed in #176.