Skip to content

feat: add A3M Router MCTS with free-tier models - #189

Open
Das-rebel wants to merge 1 commit into
RouteWorks:mainfrom
Das-rebel:a3m-mcts-v3
Open

feat: add A3M Router MCTS with free-tier models#189
Das-rebel wants to merge 1 commit into
RouteWorks:mainfrom
Das-rebel:a3m-mcts-v3

Conversation

@Das-rebel

Copy link
Copy Markdown

A3M Router MCTS - Free-tier Model Submission

Adds A3M Router with MCTS algorithm using free-tier models for RouterArena evaluation.

Models Used

  • openai/gpt-oss-120b
  • openai/gpt-oss-20b
  • google/gemma-4-31b-it
  • meta-llama/llama-3.3-70b-instruct

Files Added

  • router_inference/config/a3m-router-mcts.json
  • router_inference/predictions/a3m-router-mcts.json (159,601 predictions)
  • router_inference/predictions/a3m-router-mcts-robustness.json

Based on latest upstream/main

- Uses free-tier models: gpt-oss-120b, gpt-oss-20b, gemma-4-31b-it, llama-3.3-70b-instruct
- 159,601 predictions for full split evaluation
- 9.9M robustness test predictions
- Based on latest upstream/main
@Das-rebel

Copy link
Copy Markdown
Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: a3m-router-mcts
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.9404
Accuracy 96.77%
Total Cost $0.645530
Avg Cost per Query $0.000077
Avg Cost per 1K Queries $0.0768
Number of Queries 8400
Abnormal Entries 0
Robustness Score 1.0000

Evaluation completed by RouterArena automated workflow

@Das-rebel

Copy link
Copy Markdown
Author

Hi @yl231 @xufan866! This is a resubmission with a clean, single commit based on latest main.

Evaluation passed with strong results:

  • Accuracy: 96.77% (highest in benchmark)
  • RouterArena Score: 0.9404
  • Robustness: 1.0000

Would be great to get this reviewed when you have time. Happy to make any changes needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant