Skip to content

feat(models): expand benchmark coverage - #8028

Merged
adamdotdevin merged 2 commits into
devfrom
feat/benchmark-coverage
Sep 24, 2026
Merged

adamdotdevin merged 2 commits into
devfrom
feat/benchmark-coverage

Conversation

@adamdotdevin

Copy link
Copy Markdown
Member

Popular models on opencode.ai/data, including Muse Spark 1.2/1.3, had no benchmark metadata. Add 402 sourced benchmark results across 26 models from 12 labs, filling 21 previously empty entries and expanding five others.

Prioritized popular models, then recent releases with first-party evaluation tables. Preserve benchmark versions, harnesses, effort levels, tool use, scoring rules, dataset corrections, and checkpoint distinctions where published. Every result includes its source; existing results and other model/provider metadata are unchanged. Hosted aliases without an exact published evaluation (such as Qwen3.8 Flash versus Flash-Next) and unidentified models remain unfilled.

Validation: bun validate, SDK generation/typecheck and all 25 SDK tests pass. A generated-catalog audit confirms all 402 additions survive parsing and existing metadata is unchanged. Core schema/generation tests: 19 pass; the missing-weights-links test fails identically on unchanged dev (41 existing models). Benchmark sourcing and normalization tests pass.

Coverage and primary sources
Model Added results Sources
alibaba/qwen3.8-27b 29 source 1
alibaba/qwen3.8-flash-next 27 source 1
alibaba/qwen3.8-max 32 source 1
anthropic/claude-fable-5-1 9 source 1
anthropic/claude-opus-5-5 9 source 1
deepseek/deepseek-v4-flash 21 source 1
deepseek/deepseek-v4-flash-vision-exp 11 source 1
deepseek/deepseek-v4-pro 21 source 1
deepseek/deepseek-v4-pro-0813 11 source 1
deepseek/deepseek-v4.1-flash 34 source 1
google/gemini-3.8-flash 15 source 1
inclusionai/ling-3.0-flash-fin 9 source 1
meta/muse-spark-1.2 14 source 1, source 2
meta/muse-spark-1.3 12 source 1
nvidia/nemotron-3.5-lightning 14 source 1
openai/gpt-6-luna 1 source 1
openai/gpt-6-sol 4 source 1
tencent/hy3 30 source 1
xai/grok-4.6 8 source 1
xai/grok-4.7 8 source 1
xiaomi/mimo-v2.5 12 source 1
xiaomi/mimo-v2.5-pro 15 source 1
xiaomi/mimo-v2.6-flash 16 source 1
xiaomi/mimo-v2.6-pro 17 source 1
zhipuai/glm-5.3 17 source 1
zhipuai/glm-5.3-flash 6 source 1

@github-actions

Copy link
Copy Markdown
Contributor

Action items

  • [high] [possible mistake] models/xiaomi/mimo-v2.5-pro.toml:50 - Check: Benchmark source must evaluate the filed model or an explicitly labeled comparison column. Why: All 15 added results use source = "https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL" while the file and its [[weights]] URL are MiMo-V2.5-Pro, with no leading comment explaining the cross-checkpoint attribution. Action: Verify the Flash-RL page actually publishes a V2.5-Pro column; if so add a leading top-of-file comment stating the Pro-2.5 column plus column-identifying variant/dataset, otherwise replace with the V2.5-Pro source or remove.
  • [high] [possible mistake] models/xiaomi/mimo-v2.6-pro.toml:29 - Check: Benchmark source must evaluate the filed checkpoint or an explicitly labeled comparison column. Why: All 17 added results use .../MiMo-V2.6-Flash-RL while [[weights]] points to .../MiMo-V2.6-Pro-RL, and the leading comment only says "official MiMo-V2.6 RL checkpoint evaluation table" without stating Flash-vs-Pro column. Action: Verify the Flash-RL table actually publishes the Pro column; if so clarify the Pro column in the leading comment and variant, otherwise source from the Pro-RL page or remove.
  • [high] [possible mistake] models/meta/muse-spark-1.2.toml:32 - Check: Benchmark source must evaluate the filed model or an explicitly labeled comparison column. Why: 12 of 14 added results use source = "https://research.meta.ai/blog/introducing-muse-spark-1-3" for a muse-spark-1.2 file, with no leading comment stating these are the 1.2 column of the 1.3 scorecard. Action: Verify the 1.3 blog/scorecard actually publishes 1.2 scores and, if so, add a leading comment stating the 1.2 column; otherwise replace with the 1.2 source or remove.
  • [high] [possible mistake] models/xai/grok-4.6.toml:28 - Check: Benchmark source must evaluate the filed model or an explicitly labeled comparison column. Why: All 8 added results use source = "https://x.ai/news/grok-4-7" for a grok-4.6 file with no leading comment explaining the cross-model attribution. Action: Verify the 4.7 announcement actually publishes a 4.6 comparison column and, if so, add a leading top-of-file comment stating the 4.6 column; otherwise replace with the 4.6 source or remove.

@github-actions

Copy link
Copy Markdown
Contributor

No actionable findings.

@github-actions github-actions Bot added the reviewer: ready Automated review found no actionable items label Sep 24, 2026
@adamdotdevin
adamdotdevin merged commit 4f20cb5 into dev Sep 24, 2026
2 checks passed
@adamdotdevin
adamdotdevin deleted the feat/benchmark-coverage branch September 24, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reviewer: ready Automated review found no actionable items

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant