RouterExperiments

Empirical comparison of LLM routing / classification methods on RouteWorks/RouterArena. The benchmark asks: given a multiple-choice question (context + question + options), can a router predict Domain (subject area) or Difficulty (easy / medium / hard) well enough to drive model selection?

This repository is a research log plus runnable harness — not a library release. The primary artifact is the set of executed experiments under a fixed protocol, summarized in analysis/experiments.csv and reproducible from frozen splits.

---

Research questions

1. Cheap vs. heavy encoders — Do frozen MiniLM embeddings + kNN / XGBoost match ModernBERT fine-tuning on Domain routing?
2. Fine-tuning depth — Among ModernBERT variants (proximity, linear probe, LoRA, full fine-tune), where does accuracy saturate relative to train cost?
3. Semantic routers — Do Aurelio-style threshold routers compete with kNN on the same embedder?
4. RouteLLM-style score routers — How do matrix-factorization and BERT score engines perform on Difficulty-only routing?
5. LLM triage — Can a Gemini zero/few-shot JSON triage replace classical routers? (preliminary only so far)

---

Protocol (fair comparison)

Dimension	Value
Dataset	RouteWorks/RouterArena (~8.4k rows, shuffled seed 42)
Split file	data/splits/seed42.json (frozen Global Index lists)
Targets	Domain (multi-class), Difficulty (3-class)
Train sizes	train_20, train_500, train_remaining (topic-dependent)
Eval sets	eval_50, eval_1000
Main-run filter	preliminary=False, timestamp ≥ 20260524_084000 for cohort analysis

Features use Context + Question + Options only; labels and metadata never leak into inputs. Every main run writes an additive timestamped directory under results/ (gitignored locally).

Full specification: EXPERIMENT_SPEC.md.

---

Executed experiments

198 runs indexed (190 main, 8 preliminary) across four topics as of the latest experiments.csv rebuild.

Topics and variations run

Topic	Variations executed	Target(s)
semantic	knn, umap_knn, aurelio_t0.4, aurelio_t0.7	Domain, Difficulty
simple	tfidf_xgb, setfit, embed_xgb	Domain, Difficulty
bert	v1_proximity, v2_linear_probe, v3_lora, v4_full	Domain, Difficulty
routellm	mf, bert	Difficulty only
llm	triage_zero, triage_fewshot	(preliminary smoke tests only)

Headline results — train_remaining / eval_1000 (latest run per slug)

Domain routing (accuracy / macro-F1):

Rank	Method	Accuracy	Macro-F1
1	ModernBERT linear probe (v2_linear_probe)	0.826	0.827
2	ModernBERT full fine-tune (v4_full)	0.788	0.789
3	ModernBERT proximity kNN (v1_proximity)	0.787	0.787
4	FastEmbed + XGBoost (embed_xgb)	0.736	0.733
5	FastEmbed kNN (knn)	0.722	0.721
6	ModernBERT LoRA (v3_lora)	0.711	0.711
7	TF-IDF + XGBoost	0.545	0.551

Difficulty routing (accuracy / macro-F1 / easy-vs-hard accuracy where applicable):

Rank	Method	Accuracy	Macro-F1	Easy/Hard acc.
1	ModernBERT full fine-tune	0.677	0.627	0.818
2	ModernBERT linear probe	0.646	0.621	0.733
3	FastEmbed + XGBoost	0.622	0.581	0.718
4	ModernBERT proximity kNN	0.596	0.582	0.646
5	TF-IDF + XGBoost	0.571	0.525	0.680
6	FastEmbed kNN	0.568	0.543	0.630
7	RouteLLM MF	0.470	0.235	0.635
8	RouteLLM BERT	0.433	0.336	0.500

Preliminary findings (from executed runs)

• Linear probe on frozen ModernBERT embeddings is the strongest Domain router in this grid — beating full fine-tune and LoRA at lower train cost.
• LoRA (v3_lora) underperforms linear probe and proximity on both targets; heavy tuning did not pay off on RouterArena at these train sizes.
• FastEmbed + XGBoost is the best non-BERT baseline — close to kNN on Domain, stronger on Difficulty.
• UMAP + kNN and Aurelio threshold routers collapse on Domain at train_remaining; kNN on raw embeddings is strictly better.
• RouteLLM engines show moderate easy/hard separation but poor 3-class macro-F1 on Difficulty.
• LLM triage (runners/llm) is implemented but not yet in the main results table — pending full eval runs with GOOGLE_API_KEY.

Cohort plots and hypothesis checks: run analysis/scripts/inspect_foundational_hypotheses.py (see analysis/README.md).

---

Repository layout

├── README.md                 ← this file (research summary)
├── EXPERIMENT_SPEC.md        ← full protocol & artifact schema
├── pyproject.toml            ← uv project (optional extras per topic)
├── analysis/
│   ├── experiments.csv       ← indexed run metrics (tracked)
│   └── scripts/              ← leaderboard, cohort, per-run reports
├── data/splits/seed42.json   ← frozen train/eval IDs
└── routers/
    ├── main.py               ← CLI entry point
    ├── core/                 ← harness, splits, cache
    └── runners/              ← semantic | simple | bert | routellm | llm

---

Setup & reproduction

uv sync
uv sync --extra semantic   # kNN, Aurelio, UMAP
uv sync --extra simple     # TF-IDF/XGB, SetFit
uv sync --extra bert       # ModernBERT v1–v4
uv sync --extra routellm   # RouteLLM score routers

Set GOOGLE_API_KEY in .env for LLM triage experiments.

# Rebuild experiment index from local results/
uv run python analysis/scripts/build_leaderboard.py

# Preliminary smoke test (no results saved)
uv run python routers/main.py semantic --preliminary

# Full main grid (writes results/<topic>/...)
uv run python routers/main.py bert --main --variation v2_linear_probe

Quick-start details: routers/README.md.

---

Citation & data

• Dataset: RouteWorks/RouterArena on Hugging Face.
• RouteLLM: local MF/BERT engines with fastembed MiniLM patched to 1536-d (approximate vs paper OpenAI embeddings).

---

Status

Active research codebase. Results directories and embedding caches are local-only (gitignored); analysis/experiments.csv preserves the metric record of executed runs for sharing and diffing across sessions.