diff --git a/.github/scripts/resolve-model.sh b/.github/scripts/resolve-model.sh old mode 100644 new mode 100755 diff --git a/README.md b/README.md index 89385d2..3bb9374 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Main repository form my Opencode Github Actions to share to multiple repository ## Model Recommendations by Task Type > Automatically updated by `opencode-maintenance` workflow. -> Last updated: **2026-08-03 22:03 UTC**. +> Last updated: **2026-08-03 22:06 UTC**. > LiveBench data: **325 models scored**. > LiveBench snapshot: **2026_06_25**. > Source: https://livebench.ai/table_2026_06_25.csv @@ -17,16 +17,16 @@ Main repository form my Opencode Github Actions to share to multiple repository | Task Type | Description | Best Zen | Best Free | Best Go | |-----------|-------------|----------|-----------|---------| -| `Plan` | Planning, architecture decisions, task decomposition | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/kimi-k3` (87.0) โš ๏ธ x2.0 ยท alt: `opencode-go/grok-4.5` (86.9) | -| `Ask` | General Q&A, explanations, analysis | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | ๐Ÿ† `opencode-go/qwen3.7-max` (76.6) | -| `Code` | Code generation, implementation, refactoring | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/big-pickle` (67.0) | -| `issue-triage` | Triage, label, categorize, route issues | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | ๐Ÿ† `opencode-go/qwen3.7-max` (76.6) | -| `issue-implementation` | Implement, fix, resolve issues | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/big-pickle` (67.0) | -| `pr-review` | Review PRs, pull requests, diffs | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/kimi-k3` (87.0) โš ๏ธ x2.0 ยท alt: `opencode-go/grok-4.5` (86.9) | -| `code-implementation` | Generate code, refactor, implement features | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/big-pickle` (67.0) | -| `frontend-design` | UI design, components, layouts, mockups | `opencode/gpt-5.6-sol` (79.1) | `opencode/mimo-v2.5-free` (54.0) | ๐Ÿ† `opencode-go/kimi-k3` (72.6) โš ๏ธ x2.0 ยท alt: `opencode-go/grok-4.5` (66.4) | -| `frontend-testing` | Playwright, Cypress, E2E, frontend tests | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/big-pickle` (67.0) | -| `api-testing` | API testing, integration tests, OpenAPI, Postman | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/big-pickle` (67.0) | +| `Plan` | Planning, architecture decisions, task decomposition | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/qwen3.8-max` (95.5) | +| `Ask` | General Q&A, explanations, analysis | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | ๐Ÿ† `opencode-go/qwen3.8-max` (93.9) | +| `Code` | Code generation, implementation, refactoring | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0) | +| `issue-triage` | Triage, label, categorize, route issues | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | ๐Ÿ† `opencode-go/qwen3.8-max` (93.9) | +| `issue-implementation` | Implement, fix, resolve issues | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0) | +| `pr-review` | Review PRs, pull requests, diffs | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/qwen3.8-max` (95.5) | +| `code-implementation` | Generate code, refactor, implement features | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0) | +| `frontend-design` | UI design, components, layouts, mockups | `opencode/gpt-5.6-sol` (79.1) | `opencode/mimo-v2.5-free` (54.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (88.1) | +| `frontend-testing` | Playwright, Cypress, E2E, frontend tests | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0) | +| `api-testing` | API testing, integration tests, OpenAPI, Postman | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0) | | `other` | Everything else | `opencode/claude-fable-5` (83.4) | `opencode/deepseek-v4-flash-free` (66.1) | ๐Ÿ† `opencode-go/kimi-k3` (79.5) โš ๏ธ x2.0 ยท alt: `opencode-go/grok-4.5` (77.1) | ### LiveBench Score Reference @@ -64,22 +64,22 @@ Main repository form my Opencode Github Actions to share to multiple repository | `qwen3.6-plus` | Go (Paid) | โœ… LiveBench | Plan, Review | โ€” | 69.0 | 56.1 | 77.0 | 52.5 | 67.7 | | `qwen3.7-max` | Go (Paid) | โœ… LiveBench | Plan, Review | โ€” | 74.1 | 55.8 | 82.4 | 58.7 | 76.6 | | `qwen3.7-plus` | Go (Paid) | ๐Ÿ“‹ Fallback | Plan, Ask | โ€” | 66.0 | 62.0 | 72.0 | 62.0 | 72.0 | -| `qwen3.8-max` | Go (Paid) | โŒ Missing | โ€” | โ€” | โ€” | โ€” | โ€” | โ€” | โ€” | +| `qwen3.8-max` | Go (Paid) | ๐Ÿ“‹ Fallback | Plan, Review | โ€” | 78.2 | 77.0 | 95.5 | 88.1 | 93.9 | ## Workflow Model Audit -> Audited: **2026-08-03 22:03 UTC** +> Audited: **2026-08-03 22:06 UTC** > Workflows checked: **4** > OpenCode steps found: **7** | Workflow | Job | Step | Task Type | Current Model | Recommended Zen | Recommended Free | Recommended Go | Status | |----------|-----|------|-----------|---------------|-----------------|------------------|----------------|--------| -| `opencode-issue-handler` | `process-4` | `Run opencode (Process 4 โ€” Issue Review & Refinement)` | `issue-triage` | `opencode-go/qwen3.7-max` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5.5-pro` (81.6 (+7%)) | `opencode/deepseek-v4-flash-free` (69.4) | ๐Ÿ† `opencode-go/qwen3.7-max` (76.6 (+10%)) | โœ… | -| `opencode-issue-handler` | `process-5` | `Run opencode (Process 5 โ€” Issue Work & PR Creation)` | `issue-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) โš™๏ธ | `opencode/claude-fable-5` (71.7 (+14%)) | ๐Ÿ† `opencode/big-pickle` (67.0) | `opencode-go/big-pickle` (67.0) | โ— | -| `OpenCode Maintenance` | `Handle Checked Tasks` | `Run OpenCode for checked tasks` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) โš™๏ธ | `opencode/claude-fable-5` (71.7 (+14%)) | ๐Ÿ† `opencode/big-pickle` (67.0) | `opencode-go/big-pickle` (67.0) | โ— | -| `opencode-pr-comment` | `process-2` | `Run opencode (Process 2 โ€” Bot thread reply)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/kimi-k3` (87.0 (+19%)) โš ๏ธ x2.0 ยท alt: `opencode-go/grok-4.5` (86.9 (+19%)) | โœ… | -| `opencode-pr-comment` | `process-3` | `Run opencode (Process 3 โ€” User-owned thread takeover)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/kimi-k3` (87.0 (+19%)) โš ๏ธ x2.0 ยท alt: `opencode-go/grok-4.5` (86.9 (+19%)) | โœ… | -| `opencode-pr-comment` | `process-6` | `Run opencode (Process 6 โ€” PR Task Execution)` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) โš™๏ธ | `opencode/claude-fable-5` (71.7 (+14%)) | ๐Ÿ† `opencode/big-pickle` (67.0) | `opencode-go/big-pickle` (67.0) | โ— | -| `opencode-pr-review` | `review` | `Run opencode (PR code review)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/kimi-k3` (87.0 (+19%)) โš ๏ธ x2.0 ยท alt: `opencode-go/grok-4.5` (86.9 (+19%)) | โœ… | +| `opencode-issue-handler` | `process-4` | `Run opencode (Process 4 โ€” Issue Review & Refinement)` | `issue-triage` | `opencode-go/qwen3.7-max` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5.5-pro` (81.6 (+7%)) | `opencode/deepseek-v4-flash-free` (69.4) | ๐Ÿ† `opencode-go/qwen3.8-max` (93.9 (+35%)) | โŒ | +| `opencode-issue-handler` | `process-5` | `Run opencode (Process 5 โ€” Issue Work & PR Creation)` | `issue-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) โš™๏ธ | `opencode/claude-fable-5` (71.7 (+14%)) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0 (+15%)) | โŒ | +| `OpenCode Maintenance` | `Handle Checked Tasks` | `Run OpenCode for checked tasks` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) โš™๏ธ | `opencode/claude-fable-5` (71.7 (+14%)) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0 (+15%)) | โŒ | +| `opencode-pr-comment` | `process-2` | `Run opencode (Process 2 โ€” Bot thread reply)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/qwen3.8-max` (95.5 (+31%)) | โŒ | +| `opencode-pr-comment` | `process-3` | `Run opencode (Process 3 โ€” User-owned thread takeover)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/qwen3.8-max` (95.5 (+31%)) | โŒ | +| `opencode-pr-comment` | `process-6` | `Run opencode (Process 6 โ€” PR Task Execution)` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) โš™๏ธ | `opencode/claude-fable-5` (71.7 (+14%)) | `opencode/big-pickle` (67.0) | ๐Ÿ† `opencode-go/qwen3.8-max` (77.0 (+15%)) | โŒ | +| `opencode-pr-review` | `review` | `Run opencode (PR code review)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) โš™๏ธ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | ๐Ÿ† `opencode-go/qwen3.8-max` (95.5 (+31%)) | โŒ | _Legend: โœ… Optimal ยท โš ๏ธ Warn (free, not best) ยท โ— Alert (paid when free is preferred) ยท โŒ Error (wrong model) ยท ๐Ÿ’€ Fatal (model not set). ๐Ÿ† marks the preferred model after free-first policy (free within 5% of best Go โ†’ prefer free). โš ๏ธ xN marks models with elevated token consumption. โš™๏ธ marks steps resolved at runtime from the central config (`data/model-config.json`). Recommended Zen shows best Zen model with score difference vs current model (e.g., `model (+15%)`)._ \ No newline at end of file diff --git a/config/model-scores.yaml b/config/model-scores.yaml index 9de9efe..e516016 100644 --- a/config/model-scores.yaml +++ b/config/model-scores.yaml @@ -47,6 +47,12 @@ model_scores: overall: 66.0 reasoning: 72.0 vision: 62.0 + qwen3.8-max: + coding: 77.0 # BenchLM weighted coding category (#6/130, 96th pct) + instruction_following: 93.9 # BenchLM IFBench weighted (#2/33, 97th pct) + overall: 78.2 # BenchLM overall score (#6/215) + reasoning: 95.5 # BenchLM reasoning category (#1/2, 100th pct) + vision: 88.1 # BenchLM multimodal category (#2/34, 97th pct) laguna-s-2.1-free: coding: 52.0 instruction_following: 54.0 diff --git a/data/audit_results.json b/data/audit_results.json index 7ccd4da..05dfec6 100644 --- a/data/audit_results.json +++ b/data/audit_results.json @@ -1,5 +1,5 @@ { - "timestamp": "2026-08-03T22:03:20.211349+00:00", + "timestamp": "2026-08-03T22:06:09.009286+00:00", "livebench_snapshot": "2026_06_25", "livebench_source": "https://livebench.ai/table_2026_06_25.csv", "livebench_models": 325, @@ -9,7 +9,7 @@ "steps_audited": 7, "model_config_drift": { "detected": true, - "timestamp": "2026-08-03T22:03:20.211358+00:00", + "timestamp": "2026-08-03T22:06:09.009300+00:00", "livebench_snapshot": "2026_06_25", "current_workflows": { "opencode-issue-handler": { @@ -61,14 +61,14 @@ "current_model_free": "opencode/deepseek-v4-flash-free", "auto": true, "recommended_free": "deepseek-v4-flash-free", - "recommended_go": "qwen3.7-max", + "recommended_go": "qwen3.8-max", "recommended_free_alt": null, "recommended_go_alt": null, "recommended_free_multiplier": 1.0, "recommended_go_multiplier": 1.0, "preferred_tier": "go", - "preferred_diff": 10, - "status": "\u2705" + "preferred_diff": 35, + "status": "\u274c" }, { "file": ".github/workflows/opencode-issue-handler.yml", @@ -80,14 +80,14 @@ "current_model_free": "opencode/nemotron-3-ultra-free", "auto": true, "recommended_free": "big-pickle", - "recommended_go": "big-pickle", + "recommended_go": "qwen3.8-max", "recommended_free_alt": null, "recommended_go_alt": null, "recommended_free_multiplier": 1.0, "recommended_go_multiplier": 1.0, - "preferred_tier": "free", - "preferred_diff": null, - "status": "\u2757" + "preferred_tier": "go", + "preferred_diff": 15, + "status": "\u274c" }, { "file": ".github/workflows/opencode-maintenance.yaml", @@ -99,14 +99,14 @@ "current_model_free": "opencode/nemotron-3-ultra-free", "auto": true, "recommended_free": "big-pickle", - "recommended_go": "big-pickle", + "recommended_go": "qwen3.8-max", "recommended_free_alt": null, "recommended_go_alt": null, "recommended_free_multiplier": 1.0, "recommended_go_multiplier": 1.0, - "preferred_tier": "free", - "preferred_diff": null, - "status": "\u2757" + "preferred_tier": "go", + "preferred_diff": 15, + "status": "\u274c" }, { "file": ".github/workflows/opencode-pr-comment.yml", @@ -118,14 +118,14 @@ "current_model_free": "opencode/deepseek-v4-flash-free", "auto": true, "recommended_free": "deepseek-v4-flash-free", - "recommended_go": "kimi-k3", + "recommended_go": "qwen3.8-max", "recommended_free_alt": null, - "recommended_go_alt": "grok-4.5", + "recommended_go_alt": null, "recommended_free_multiplier": 1.0, - "recommended_go_multiplier": 2.0, + "recommended_go_multiplier": 1.0, "preferred_tier": "go", - "preferred_diff": 19, - "status": "\u2705" + "preferred_diff": 31, + "status": "\u274c" }, { "file": ".github/workflows/opencode-pr-comment.yml", @@ -137,14 +137,14 @@ "current_model_free": "opencode/deepseek-v4-flash-free", "auto": true, "recommended_free": "deepseek-v4-flash-free", - "recommended_go": "kimi-k3", + "recommended_go": "qwen3.8-max", "recommended_free_alt": null, - "recommended_go_alt": "grok-4.5", + "recommended_go_alt": null, "recommended_free_multiplier": 1.0, - "recommended_go_multiplier": 2.0, + "recommended_go_multiplier": 1.0, "preferred_tier": "go", - "preferred_diff": 19, - "status": "\u2705" + "preferred_diff": 31, + "status": "\u274c" }, { "file": ".github/workflows/opencode-pr-comment.yml", @@ -156,14 +156,14 @@ "current_model_free": "opencode/nemotron-3-ultra-free", "auto": true, "recommended_free": "big-pickle", - "recommended_go": "big-pickle", + "recommended_go": "qwen3.8-max", "recommended_free_alt": null, "recommended_go_alt": null, "recommended_free_multiplier": 1.0, "recommended_go_multiplier": 1.0, - "preferred_tier": "free", - "preferred_diff": null, - "status": "\u2757" + "preferred_tier": "go", + "preferred_diff": 15, + "status": "\u274c" }, { "file": ".github/workflows/opencode-pr-review.yml", @@ -175,14 +175,14 @@ "current_model_free": "opencode/deepseek-v4-flash-free", "auto": true, "recommended_free": "deepseek-v4-flash-free", - "recommended_go": "kimi-k3", + "recommended_go": "qwen3.8-max", "recommended_free_alt": null, - "recommended_go_alt": "grok-4.5", + "recommended_go_alt": null, "recommended_free_multiplier": 1.0, - "recommended_go_multiplier": 2.0, + "recommended_go_multiplier": 1.0, "preferred_tier": "go", - "preferred_diff": 19, - "status": "\u2705" + "preferred_diff": 31, + "status": "\u274c" } ] } \ No newline at end of file diff --git a/data/benchmark_results.json b/data/benchmark_results.json index f6ea8b7..ea4fd52 100644 --- a/data/benchmark_results.json +++ b/data/benchmark_results.json @@ -1,5 +1,5 @@ { - "timestamp": "2026-08-03T22:03:20.190932+00:00", + "timestamp": "2026-08-03T22:06:08.988584+00:00", "livebench_snapshot": "2026_06_25", "livebench_source": "https://livebench.ai/table_2026_06_25.csv", "zen_models_count": 60, @@ -283,8 +283,8 @@ "rank": 27 }, { - "model": "gpt-5.3-codex", - "score": 79.3, + "model": "gemini-3-flash", + "score": 80.0, "source": "livebench", "rank": 28 }, @@ -366,58 +366,58 @@ "source": "livebench", "rank": 41 }, + { + "model": "gpt-5.3-codex", + "score": 72.6, + "source": "livebench", + "rank": 42 + }, { "model": "minimax-m2.7", "score": 72.4, "source": "livebench", - "rank": 42 + "rank": 43 }, { "model": "gpt-5.1-codex-mini", "score": 66.3, "source": "livebench", - "rank": 43 + "rank": 44 }, { "model": "claude-haiku-4-5", "score": 66.0, "source": "livebench", - "rank": 44 + "rank": 45 }, { "model": "gemini-3.5-flash-lite", "score": 65.2, "source": "livebench", - "rank": 45 + "rank": 46 }, { "model": "minimax-m2.5", "score": 62.3, "source": "livebench", - "rank": 46 + "rank": 47 }, { "model": "qwen3.5-plus", "score": 62.0, "source": "fallback", - "rank": 47 + "rank": 48 }, { "model": "big-pickle", "score": 61.5, "source": "fallback", - "rank": 48 + "rank": 49 }, { "model": "mimo-v2.5-free", "score": 60.0, "source": "fallback", - "rank": 49 - }, - { - "model": "gemini-3-flash", - "score": 54.8, - "source": "livebench", "rank": 50 }, { @@ -518,280 +518,280 @@ "source": "livebench", "rank": 7 }, + { + "model": "gemini-3-flash", + "score": 77.8, + "source": "livebench", + "rank": 8 + }, { "model": "gemini-3.5-flash", "score": 77.6, "source": "livebench", - "rank": 8 + "rank": 9 }, { "model": "gpt-5.5", "score": 76.8, "source": "livebench", - "rank": 9 + "rank": 10 }, { "model": "gemini-3.6-flash", "score": 76.5, "source": "livebench", - "rank": 10 + "rank": 11 }, { "model": "gpt-5.6-sol", "score": 75.0, "source": "livebench", - "rank": 11 + "rank": 12 }, { "model": "claude-opus-4-8", "score": 74.9, "source": "livebench", - "rank": 12 + "rank": 13 }, { "model": "kimi-k3", "score": 74.9, "source": "livebench", - "rank": 13 + "rank": 14 + }, + { + "model": "gpt-5.3-codex", + "score": 74.0, + "source": "livebench", + "rank": 15 }, { "model": "grok-4.5", "score": 73.9, "source": "livebench", - "rank": 14 + "rank": 16 }, { "model": "gpt-5.6-terra", "score": 72.2, "source": "livebench", - "rank": 15 + "rank": 17 }, { "model": "claude-opus-5", "score": 71.9, "source": "livebench", - "rank": 16 + "rank": 18 }, { "model": "claude-opus-4-7", "score": 71.1, "source": "livebench", - "rank": 17 + "rank": 19 }, { "model": "gpt-5.1-codex-max", "score": 71.1, "source": "livebench", - "rank": 18 - }, - { - "model": "gpt-5.3-codex", - "score": 70.8, - "source": "livebench", - "rank": 19 + "rank": 20 }, { "model": "claude-opus-4-6", "score": 70.7, "source": "livebench", - "rank": 20 + "rank": 21 }, { "model": "deepseek-v4-pro", "score": 70.3, "source": "livebench", - "rank": 21 + "rank": 22 }, { "model": "grok-build-0.1", "score": 70.2, "source": "livebench", - "rank": 22 + "rank": 23 }, { "model": "gpt-5.2", "score": 70.1, "source": "livebench", - "rank": 23 + "rank": 24 }, { "model": "gpt-5.2-codex", "score": 70.1, "source": "livebench", - "rank": 24 + "rank": 25 }, { "model": "kimi-k2.6", "score": 69.7, "source": "livebench", - "rank": 25 + "rank": 26 }, { "model": "claude-opus-4-5", "score": 69.5, "source": "livebench", - "rank": 26 + "rank": 27 }, { "model": "glm-5.1", "score": 69.5, "source": "livebench", - "rank": 27 + "rank": 28 }, { "model": "deepseek-v4-flash", "score": 69.4, "source": "livebench", - "rank": 28 + "rank": 29 }, { "model": "deepseek-v4-flash-free", "score": 69.4, "source": "livebench", - "rank": 29 + "rank": 30 }, { "model": "gemini-3.5-flash-lite", "score": 68.9, "source": "livebench", - "rank": 30 + "rank": 31 }, { "model": "claude-sonnet-5", "score": 68.5, "source": "livebench", - "rank": 31 + "rank": 32 }, { "model": "glm-5.2", "score": 68.2, "source": "livebench", - "rank": 32 + "rank": 33 }, { "model": "qwen3.6-plus", "score": 67.7, "source": "livebench", - "rank": 33 + "rank": 34 }, { "model": "gpt-5.4", "score": 67.4, "source": "livebench", - "rank": 34 + "rank": 35 }, { "model": "minimax-m2.7", "score": 67.4, "source": "livebench", - "rank": 35 + "rank": 36 }, { "model": "claude-sonnet-4-6", "score": 67.2, "source": "livebench", - "rank": 36 + "rank": 37 }, { "model": "claude-sonnet-4", "score": 67.2, "source": "livebench", - "rank": 37 + "rank": 38 }, { "model": "minimax-m3", "score": 66.9, "source": "livebench", - "rank": 38 + "rank": 39 }, { "model": "gpt-5.1-codex", "score": 66.8, "source": "livebench", - "rank": 39 + "rank": 40 }, { "model": "gpt-5.6-luna", "score": 66.6, "source": "livebench", - "rank": 40 + "rank": 41 }, { "model": "kimi-k2.7-code", "score": 65.9, "source": "livebench", - "rank": 41 + "rank": 42 }, { "model": "kimi-k2.5", "score": 65.3, "source": "livebench", - "rank": 42 + "rank": 43 }, { "model": "glm-5", "score": 65.0, "source": "livebench", - "rank": 43 + "rank": 44 }, { "model": "gpt-5.1-codex-mini", "score": 63.1, "source": "livebench", - "rank": 44 + "rank": 45 }, { "model": "nemotron-3-ultra-free", "score": 62.5, "source": "livebench", - "rank": 45 + "rank": 46 }, { "model": "minimax-m2.5", "score": 62.2, "source": "livebench", - "rank": 46 + "rank": 47 }, { "model": "mimo-v2.5-free", "score": 62.0, "source": "fallback", - "rank": 47 + "rank": 48 }, { "model": "claude-haiku-4-5", "score": 60.7, "source": "livebench", - "rank": 48 + "rank": 49 }, { "model": "qwen3.5-plus", "score": 60.0, "source": "fallback", - "rank": 49 + "rank": 50 }, { "model": "big-pickle", "score": 60.0, "source": "fallback", - "rank": 50 + "rank": 51 }, { "model": "gpt-5-nano", "score": 56.0, "source": "livebench", - "rank": 51 + "rank": 52 }, { "model": "laguna-s-2.1-free", "score": 54.0, "source": "fallback", - "rank": 52 - }, - { - "model": "gemini-3-flash", - "score": 50.2, - "source": "livebench", "rank": 53 }, { @@ -844,50 +844,50 @@ "source": "livebench", "rank": 2 }, + { + "model": "gpt-5.3-codex", + "score": 71.0, + "source": "livebench", + "rank": 3 + }, { "model": "kimi-k3", "score": 69.9, "source": "livebench", - "rank": 3 + "rank": 4 }, { "model": "claude-sonnet-5", "score": 67.9, "source": "livebench", - "rank": 4 + "rank": 5 }, { "model": "gpt-5.6-sol", "score": 67.3, "source": "livebench", - "rank": 5 + "rank": 6 }, { "model": "big-pickle", "score": 67.0, "source": "fallback", - "rank": 6 + "rank": 7 }, { "model": "gpt-5.1-codex-max", "score": 66.6, "source": "livebench", - "rank": 7 + "rank": 8 }, { "model": "gpt-5.5", "score": 65.3, "source": "livebench", - "rank": 8 - }, - { - "model": "gpt-5.6-terra", - "score": 64.3, - "source": "livebench", "rank": 9 }, { - "model": "gpt-5.3-codex", + "model": "gpt-5.6-terra", "score": 64.3, "source": "livebench", "rank": 10 @@ -1018,76 +1018,76 @@ "source": "livebench", "rank": 31 }, - { - "model": "gemini-3-flash", - "score": 57.4, - "source": "livebench", - "rank": 32 - }, { "model": "claude-sonnet-4-6", "score": 57.3, "source": "livebench", - "rank": 33 + "rank": 32 }, { "model": "claude-sonnet-4", "score": 57.3, "source": "livebench", - "rank": 34 + "rank": 33 }, { "model": "gemini-3.6-flash", "score": 57.2, "source": "livebench", - "rank": 35 + "rank": 34 }, { "model": "gemini-3.1-pro", "score": 57.1, "source": "livebench", - "rank": 36 + "rank": 35 }, { "model": "kimi-k2.7-code", "score": 57.0, "source": "livebench", - "rank": 37 + "rank": 36 }, { "model": "gpt-5-codex", "score": 56.8, "source": "livebench", - "rank": 38 + "rank": 37 }, { "model": "nemotron-3-ultra-free", "score": 56.5, "source": "livebench", - "rank": 39 + "rank": 38 }, { "model": "qwen3.6-plus", "score": 56.1, "source": "livebench", - "rank": 40 + "rank": 39 }, { "model": "mimo-v2.5-free", "score": 56.0, "source": "fallback", - "rank": 41 + "rank": 40 }, { "model": "claude-opus-4-5", "score": 55.7, "source": "livebench", - "rank": 42 + "rank": 41 }, { "model": "claude-haiku-4-5", "score": 54.1, "source": "livebench", + "rank": 42 + }, + { + "model": "gemini-3-flash", + "score": 53.6, + "source": "livebench", "rank": 43 }, { @@ -1230,280 +1230,280 @@ "source": "livebench", "rank": 7 }, + { + "model": "gemini-3-flash", + "score": 77.8, + "source": "livebench", + "rank": 8 + }, { "model": "gemini-3.5-flash", "score": 77.6, "source": "livebench", - "rank": 8 + "rank": 9 }, { "model": "gpt-5.5", "score": 76.8, "source": "livebench", - "rank": 9 + "rank": 10 }, { "model": "gemini-3.6-flash", "score": 76.5, "source": "livebench", - "rank": 10 + "rank": 11 }, { "model": "gpt-5.6-sol", "score": 75.0, "source": "livebench", - "rank": 11 + "rank": 12 }, { "model": "claude-opus-4-8", "score": 74.9, "source": "livebench", - "rank": 12 + "rank": 13 }, { "model": "kimi-k3", "score": 74.9, "source": "livebench", - "rank": 13 + "rank": 14 + }, + { + "model": "gpt-5.3-codex", + "score": 74.0, + "source": "livebench", + "rank": 15 }, { "model": "grok-4.5", "score": 73.9, "source": "livebench", - "rank": 14 + "rank": 16 }, { "model": "gpt-5.6-terra", "score": 72.2, "source": "livebench", - "rank": 15 + "rank": 17 }, { "model": "claude-opus-5", "score": 71.9, "source": "livebench", - "rank": 16 + "rank": 18 }, { "model": "claude-opus-4-7", "score": 71.1, "source": "livebench", - "rank": 17 + "rank": 19 }, { "model": "gpt-5.1-codex-max", "score": 71.1, "source": "livebench", - "rank": 18 - }, - { - "model": "gpt-5.3-codex", - "score": 70.8, - "source": "livebench", - "rank": 19 + "rank": 20 }, { "model": "claude-opus-4-6", "score": 70.7, "source": "livebench", - "rank": 20 + "rank": 21 }, { "model": "deepseek-v4-pro", "score": 70.3, "source": "livebench", - "rank": 21 + "rank": 22 }, { "model": "grok-build-0.1", "score": 70.2, "source": "livebench", - "rank": 22 + "rank": 23 }, { "model": "gpt-5.2", "score": 70.1, "source": "livebench", - "rank": 23 + "rank": 24 }, { "model": "gpt-5.2-codex", "score": 70.1, "source": "livebench", - "rank": 24 + "rank": 25 }, { "model": "kimi-k2.6", "score": 69.7, "source": "livebench", - "rank": 25 + "rank": 26 }, { "model": "claude-opus-4-5", "score": 69.5, "source": "livebench", - "rank": 26 + "rank": 27 }, { "model": "glm-5.1", "score": 69.5, "source": "livebench", - "rank": 27 + "rank": 28 }, { "model": "deepseek-v4-flash", "score": 69.4, "source": "livebench", - "rank": 28 + "rank": 29 }, { "model": "deepseek-v4-flash-free", "score": 69.4, "source": "livebench", - "rank": 29 + "rank": 30 }, { "model": "gemini-3.5-flash-lite", "score": 68.9, "source": "livebench", - "rank": 30 + "rank": 31 }, { "model": "claude-sonnet-5", "score": 68.5, "source": "livebench", - "rank": 31 + "rank": 32 }, { "model": "glm-5.2", "score": 68.2, "source": "livebench", - "rank": 32 + "rank": 33 }, { "model": "qwen3.6-plus", "score": 67.7, "source": "livebench", - "rank": 33 + "rank": 34 }, { "model": "gpt-5.4", "score": 67.4, "source": "livebench", - "rank": 34 + "rank": 35 }, { "model": "minimax-m2.7", "score": 67.4, "source": "livebench", - "rank": 35 + "rank": 36 }, { "model": "claude-sonnet-4-6", "score": 67.2, "source": "livebench", - "rank": 36 + "rank": 37 }, { "model": "claude-sonnet-4", "score": 67.2, "source": "livebench", - "rank": 37 + "rank": 38 }, { "model": "minimax-m3", "score": 66.9, "source": "livebench", - "rank": 38 + "rank": 39 }, { "model": "gpt-5.1-codex", "score": 66.8, "source": "livebench", - "rank": 39 + "rank": 40 }, { "model": "gpt-5.6-luna", "score": 66.6, "source": "livebench", - "rank": 40 + "rank": 41 }, { "model": "kimi-k2.7-code", "score": 65.9, "source": "livebench", - "rank": 41 + "rank": 42 }, { "model": "kimi-k2.5", "score": 65.3, "source": "livebench", - "rank": 42 + "rank": 43 }, { "model": "glm-5", "score": 65.0, "source": "livebench", - "rank": 43 + "rank": 44 }, { "model": "gpt-5.1-codex-mini", "score": 63.1, "source": "livebench", - "rank": 44 + "rank": 45 }, { "model": "nemotron-3-ultra-free", "score": 62.5, "source": "livebench", - "rank": 45 + "rank": 46 }, { "model": "minimax-m2.5", "score": 62.2, "source": "livebench", - "rank": 46 + "rank": 47 }, { "model": "mimo-v2.5-free", "score": 62.0, "source": "fallback", - "rank": 47 + "rank": 48 }, { "model": "claude-haiku-4-5", "score": 60.7, "source": "livebench", - "rank": 48 + "rank": 49 }, { "model": "qwen3.5-plus", "score": 60.0, "source": "fallback", - "rank": 49 + "rank": 50 }, { "model": "big-pickle", "score": 60.0, "source": "fallback", - "rank": 50 + "rank": 51 }, { "model": "gpt-5-nano", "score": 56.0, "source": "livebench", - "rank": 51 + "rank": 52 }, { "model": "laguna-s-2.1-free", "score": 54.0, "source": "fallback", - "rank": 52 - }, - { - "model": "gemini-3-flash", - "score": 50.2, - "source": "livebench", "rank": 53 }, { @@ -1556,50 +1556,50 @@ "source": "livebench", "rank": 2 }, + { + "model": "gpt-5.3-codex", + "score": 71.0, + "source": "livebench", + "rank": 3 + }, { "model": "kimi-k3", "score": 69.9, "source": "livebench", - "rank": 3 + "rank": 4 }, { "model": "claude-sonnet-5", "score": 67.9, "source": "livebench", - "rank": 4 + "rank": 5 }, { "model": "gpt-5.6-sol", "score": 67.3, "source": "livebench", - "rank": 5 + "rank": 6 }, { "model": "big-pickle", "score": 67.0, "source": "fallback", - "rank": 6 + "rank": 7 }, { "model": "gpt-5.1-codex-max", "score": 66.6, "source": "livebench", - "rank": 7 + "rank": 8 }, { "model": "gpt-5.5", "score": 65.3, "source": "livebench", - "rank": 8 - }, - { - "model": "gpt-5.6-terra", - "score": 64.3, - "source": "livebench", "rank": 9 }, { - "model": "gpt-5.3-codex", + "model": "gpt-5.6-terra", "score": 64.3, "source": "livebench", "rank": 10 @@ -1730,76 +1730,76 @@ "source": "livebench", "rank": 31 }, - { - "model": "gemini-3-flash", - "score": 57.4, - "source": "livebench", - "rank": 32 - }, { "model": "claude-sonnet-4-6", "score": 57.3, "source": "livebench", - "rank": 33 + "rank": 32 }, { "model": "claude-sonnet-4", "score": 57.3, "source": "livebench", - "rank": 34 + "rank": 33 }, { "model": "gemini-3.6-flash", "score": 57.2, "source": "livebench", - "rank": 35 + "rank": 34 }, { "model": "gemini-3.1-pro", "score": 57.1, "source": "livebench", - "rank": 36 + "rank": 35 }, { "model": "kimi-k2.7-code", "score": 57.0, "source": "livebench", - "rank": 37 + "rank": 36 }, { "model": "gpt-5-codex", "score": 56.8, "source": "livebench", - "rank": 38 + "rank": 37 }, { "model": "nemotron-3-ultra-free", "score": 56.5, "source": "livebench", - "rank": 39 + "rank": 38 }, { "model": "qwen3.6-plus", "score": 56.1, "source": "livebench", - "rank": 40 + "rank": 39 }, { "model": "mimo-v2.5-free", "score": 56.0, "source": "fallback", - "rank": 41 + "rank": 40 }, { "model": "claude-opus-4-5", "score": 55.7, "source": "livebench", - "rank": 42 + "rank": 41 }, { "model": "claude-haiku-4-5", "score": 54.1, "source": "livebench", + "rank": 42 + }, + { + "model": "gemini-3-flash", + "score": 53.6, + "source": "livebench", "rank": 43 }, { @@ -2063,8 +2063,8 @@ "rank": 27 }, { - "model": "gpt-5.3-codex", - "score": 79.3, + "model": "gemini-3-flash", + "score": 80.0, "source": "livebench", "rank": 28 }, @@ -2146,58 +2146,58 @@ "source": "livebench", "rank": 41 }, + { + "model": "gpt-5.3-codex", + "score": 72.6, + "source": "livebench", + "rank": 42 + }, { "model": "minimax-m2.7", "score": 72.4, "source": "livebench", - "rank": 42 + "rank": 43 }, { "model": "gpt-5.1-codex-mini", "score": 66.3, "source": "livebench", - "rank": 43 + "rank": 44 }, { "model": "claude-haiku-4-5", "score": 66.0, "source": "livebench", - "rank": 44 + "rank": 45 }, { "model": "gemini-3.5-flash-lite", "score": 65.2, "source": "livebench", - "rank": 45 + "rank": 46 }, { "model": "minimax-m2.5", "score": 62.3, "source": "livebench", - "rank": 46 + "rank": 47 }, { "model": "qwen3.5-plus", "score": 62.0, "source": "fallback", - "rank": 47 + "rank": 48 }, { "model": "big-pickle", "score": 61.5, "source": "fallback", - "rank": 48 + "rank": 49 }, { "model": "mimo-v2.5-free", "score": 60.0, "source": "fallback", - "rank": 49 - }, - { - "model": "gemini-3-flash", - "score": 54.8, - "source": "livebench", "rank": 50 }, { @@ -2268,50 +2268,50 @@ "source": "livebench", "rank": 2 }, + { + "model": "gpt-5.3-codex", + "score": 71.0, + "source": "livebench", + "rank": 3 + }, { "model": "kimi-k3", "score": 69.9, "source": "livebench", - "rank": 3 + "rank": 4 }, { "model": "claude-sonnet-5", "score": 67.9, "source": "livebench", - "rank": 4 + "rank": 5 }, { "model": "gpt-5.6-sol", "score": 67.3, "source": "livebench", - "rank": 5 + "rank": 6 }, { "model": "big-pickle", "score": 67.0, "source": "fallback", - "rank": 6 + "rank": 7 }, { "model": "gpt-5.1-codex-max", "score": 66.6, "source": "livebench", - "rank": 7 + "rank": 8 }, { "model": "gpt-5.5", "score": 65.3, "source": "livebench", - "rank": 8 - }, - { - "model": "gpt-5.6-terra", - "score": 64.3, - "source": "livebench", "rank": 9 }, { - "model": "gpt-5.3-codex", + "model": "gpt-5.6-terra", "score": 64.3, "source": "livebench", "rank": 10 @@ -2442,76 +2442,76 @@ "source": "livebench", "rank": 31 }, - { - "model": "gemini-3-flash", - "score": 57.4, - "source": "livebench", - "rank": 32 - }, { "model": "claude-sonnet-4-6", "score": 57.3, "source": "livebench", - "rank": 33 + "rank": 32 }, { "model": "claude-sonnet-4", "score": 57.3, "source": "livebench", - "rank": 34 + "rank": 33 }, { "model": "gemini-3.6-flash", "score": 57.2, "source": "livebench", - "rank": 35 + "rank": 34 }, { "model": "gemini-3.1-pro", "score": 57.1, "source": "livebench", - "rank": 36 + "rank": 35 }, { "model": "kimi-k2.7-code", "score": 57.0, "source": "livebench", - "rank": 37 + "rank": 36 }, { "model": "gpt-5-codex", "score": 56.8, "source": "livebench", - "rank": 38 + "rank": 37 }, { "model": "nemotron-3-ultra-free", "score": 56.5, "source": "livebench", - "rank": 39 + "rank": 38 }, { "model": "qwen3.6-plus", "score": 56.1, "source": "livebench", - "rank": 40 + "rank": 39 }, { "model": "mimo-v2.5-free", "score": 56.0, "source": "fallback", - "rank": 41 + "rank": 40 }, { "model": "claude-opus-4-5", "score": 55.7, "source": "livebench", - "rank": 42 + "rank": 41 }, { "model": "claude-haiku-4-5", "score": 54.1, "source": "livebench", + "rank": 42 + }, + { + "model": "gemini-3-flash", + "score": 53.6, + "source": "livebench", "rank": 43 }, { @@ -2679,8 +2679,8 @@ "rank": 11 }, { - "model": "gpt-5.3-codex", - "score": 64.3, + "model": "gemini-3-flash", + "score": 65.7, "source": "livebench", "rank": 12 }, @@ -2720,28 +2720,28 @@ "source": "livebench", "rank": 18 }, - { - "model": "gemini-3-flash", - "score": 59.4, - "source": "livebench", - "rank": 19 - }, { "model": "gpt-5.2", "score": 58.4, "source": "livebench", - "rank": 20 + "rank": 19 }, { "model": "gpt-5.1-codex-max", "score": 58.2, "source": "livebench", - "rank": 21 + "rank": 20 }, { "model": "kimi-k2.6", "score": 58.1, "source": "livebench", + "rank": 21 + }, + { + "model": "gpt-5.3-codex", + "score": 57.5, + "source": "livebench", "rank": 22 }, { @@ -2980,50 +2980,50 @@ "source": "livebench", "rank": 2 }, + { + "model": "gpt-5.3-codex", + "score": 71.0, + "source": "livebench", + "rank": 3 + }, { "model": "kimi-k3", "score": 69.9, "source": "livebench", - "rank": 3 + "rank": 4 }, { "model": "claude-sonnet-5", "score": 67.9, "source": "livebench", - "rank": 4 + "rank": 5 }, { "model": "gpt-5.6-sol", "score": 67.3, "source": "livebench", - "rank": 5 + "rank": 6 }, { "model": "big-pickle", "score": 67.0, "source": "fallback", - "rank": 6 + "rank": 7 }, { "model": "gpt-5.1-codex-max", "score": 66.6, "source": "livebench", - "rank": 7 + "rank": 8 }, { "model": "gpt-5.5", "score": 65.3, "source": "livebench", - "rank": 8 - }, - { - "model": "gpt-5.6-terra", - "score": 64.3, - "source": "livebench", "rank": 9 }, { - "model": "gpt-5.3-codex", + "model": "gpt-5.6-terra", "score": 64.3, "source": "livebench", "rank": 10 @@ -3154,76 +3154,76 @@ "source": "livebench", "rank": 31 }, - { - "model": "gemini-3-flash", - "score": 57.4, - "source": "livebench", - "rank": 32 - }, { "model": "claude-sonnet-4-6", "score": 57.3, "source": "livebench", - "rank": 33 + "rank": 32 }, { "model": "claude-sonnet-4", "score": 57.3, "source": "livebench", - "rank": 34 + "rank": 33 }, { "model": "gemini-3.6-flash", "score": 57.2, "source": "livebench", - "rank": 35 + "rank": 34 }, { "model": "gemini-3.1-pro", "score": 57.1, "source": "livebench", - "rank": 36 + "rank": 35 }, { "model": "kimi-k2.7-code", "score": 57.0, "source": "livebench", - "rank": 37 + "rank": 36 }, { "model": "gpt-5-codex", "score": 56.8, "source": "livebench", - "rank": 38 + "rank": 37 }, { "model": "nemotron-3-ultra-free", "score": 56.5, "source": "livebench", - "rank": 39 + "rank": 38 }, { "model": "qwen3.6-plus", "score": 56.1, "source": "livebench", - "rank": 40 + "rank": 39 }, { "model": "mimo-v2.5-free", "score": 56.0, "source": "fallback", - "rank": 41 + "rank": 40 }, { "model": "claude-opus-4-5", "score": 55.7, "source": "livebench", - "rank": 42 + "rank": 41 }, { "model": "claude-haiku-4-5", "score": 54.1, "source": "livebench", + "rank": 42 + }, + { + "model": "gemini-3-flash", + "score": 53.6, + "source": "livebench", "rank": 43 }, { @@ -3336,50 +3336,50 @@ "source": "livebench", "rank": 2 }, + { + "model": "gpt-5.3-codex", + "score": 71.0, + "source": "livebench", + "rank": 3 + }, { "model": "kimi-k3", "score": 69.9, "source": "livebench", - "rank": 3 + "rank": 4 }, { "model": "claude-sonnet-5", "score": 67.9, "source": "livebench", - "rank": 4 + "rank": 5 }, { "model": "gpt-5.6-sol", "score": 67.3, "source": "livebench", - "rank": 5 + "rank": 6 }, { "model": "big-pickle", "score": 67.0, "source": "fallback", - "rank": 6 + "rank": 7 }, { "model": "gpt-5.1-codex-max", "score": 66.6, "source": "livebench", - "rank": 7 + "rank": 8 }, { "model": "gpt-5.5", "score": 65.3, "source": "livebench", - "rank": 8 - }, - { - "model": "gpt-5.6-terra", - "score": 64.3, - "source": "livebench", "rank": 9 }, { - "model": "gpt-5.3-codex", + "model": "gpt-5.6-terra", "score": 64.3, "source": "livebench", "rank": 10 @@ -3510,76 +3510,76 @@ "source": "livebench", "rank": 31 }, - { - "model": "gemini-3-flash", - "score": 57.4, - "source": "livebench", - "rank": 32 - }, { "model": "claude-sonnet-4-6", "score": 57.3, "source": "livebench", - "rank": 33 + "rank": 32 }, { "model": "claude-sonnet-4", "score": 57.3, "source": "livebench", - "rank": 34 + "rank": 33 }, { "model": "gemini-3.6-flash", "score": 57.2, "source": "livebench", - "rank": 35 + "rank": 34 }, { "model": "gemini-3.1-pro", "score": 57.1, "source": "livebench", - "rank": 36 + "rank": 35 }, { "model": "kimi-k2.7-code", "score": 57.0, "source": "livebench", - "rank": 37 + "rank": 36 }, { "model": "gpt-5-codex", "score": 56.8, "source": "livebench", - "rank": 38 + "rank": 37 }, { "model": "nemotron-3-ultra-free", "score": 56.5, "source": "livebench", - "rank": 39 + "rank": 38 }, { "model": "qwen3.6-plus", "score": 56.1, "source": "livebench", - "rank": 40 + "rank": 39 }, { "model": "mimo-v2.5-free", "score": 56.0, "source": "fallback", - "rank": 41 + "rank": 40 }, { "model": "claude-opus-4-5", "score": 55.7, "source": "livebench", - "rank": 42 + "rank": 41 }, { "model": "claude-haiku-4-5", "score": 54.1, "source": "livebench", + "rank": 42 + }, + { + "model": "gemini-3-flash", + "score": 53.6, + "source": "livebench", "rank": 43 }, { @@ -3831,8 +3831,8 @@ "rank": 25 }, { - "model": "gpt-5.3-codex", - "score": 73.2, + "model": "gemini-3-flash", + "score": 73.0, "source": "livebench", "rank": 26 }, @@ -3854,130 +3854,130 @@ "source": "livebench", "rank": 29 }, + { + "model": "gpt-5.3-codex", + "score": 72.0, + "source": "livebench", + "rank": 30 + }, { "model": "kimi-k2.6", "score": 70.9, "source": "livebench", - "rank": 30 + "rank": 31 }, { "model": "glm-5.1", "score": 70.6, "source": "livebench", - "rank": 31 + "rank": 32 }, { "model": "gpt-5.1-codex", "score": 69.3, "source": "livebench", - "rank": 32 + "rank": 33 }, { "model": "kimi-k2.5", "score": 69.2, "source": "livebench", - "rank": 33 + "rank": 34 }, { "model": "qwen3.6-plus", "score": 69.0, "source": "livebench", - "rank": 34 + "rank": 35 }, { "model": "kimi-k2.7-code", "score": 68.8, "source": "livebench", - "rank": 35 + "rank": 36 }, { "model": "glm-5", "score": 68.7, "source": "livebench", - "rank": 36 + "rank": 37 }, { "model": "grok-build-0.1", "score": 68.6, "source": "livebench", - "rank": 37 + "rank": 38 }, { "model": "minimax-m3", "score": 67.5, "source": "livebench", - "rank": 38 + "rank": 39 }, { "model": "gpt-5.4", "score": 66.6, "source": "livebench", - "rank": 39 + "rank": 40 }, { "model": "deepseek-v4-flash", "score": 66.1, "source": "livebench", - "rank": 40 + "rank": 41 }, { "model": "deepseek-v4-flash-free", "score": 66.1, "source": "livebench", - "rank": 41 + "rank": 42 }, { "model": "minimax-m2.7", "score": 65.0, "source": "livebench", - "rank": 42 + "rank": 43 }, { "model": "gemini-3.5-flash-lite", "score": 63.8, "source": "livebench", - "rank": 43 + "rank": 44 }, { "model": "big-pickle", "score": 61.5, "source": "fallback", - "rank": 44 + "rank": 45 }, { "model": "claude-haiku-4-5", "score": 61.0, "source": "livebench", - "rank": 45 + "rank": 46 }, { "model": "gpt-5.1-codex-mini", "score": 60.8, "source": "livebench", - "rank": 46 + "rank": 47 }, { "model": "minimax-m2.5", "score": 60.3, "source": "livebench", - "rank": 47 + "rank": 48 }, { "model": "qwen3.5-plus", "score": 58.0, "source": "fallback", - "rank": 48 + "rank": 49 }, { "model": "mimo-v2.5-free", "score": 58.0, "source": "fallback", - "rank": 49 - }, - { - "model": "gemini-3-flash", - "score": 54.4, - "source": "livebench", "rank": 50 }, { diff --git a/data/coverage_issues.json b/data/coverage_issues.json index 51f7ef9..88fa783 100644 --- a/data/coverage_issues.json +++ b/data/coverage_issues.json @@ -1,9 +1,4 @@ { "stale_fallback": [], - "missing_scores": [ - { - "model": "qwen3.8-max", - "tier": "Go (Paid)" - } - ] + "missing_scores": [] } \ No newline at end of file diff --git a/data/go_models.json b/data/go_models.json index 5f05a6f..ce173b2 100644 --- a/data/go_models.json +++ b/data/go_models.json @@ -3,151 +3,151 @@ { "id": "minimax-m3", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "minimax-m2.7", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "minimax-m2.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "kimi-k3", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "kimi-k2.7-code", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "kimi-k2.6", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "kimi-k2.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "glm-5.2", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "glm-5.1", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "glm-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "deepseek-v4-pro", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "deepseek-v4-flash", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "qwen3.7-max", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "qwen3.8-max", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "qwen3.7-plus", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "qwen3.6-plus", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "qwen3.5-plus", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "mimo-v2-pro", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "mimo-v2-omni", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "mimo-v2.5-pro", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "mimo-v2.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "hy3", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "hy3-preview", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "gpt-5.6-luna", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "grok-4.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" } ] diff --git a/data/livebench.json b/data/livebench.json index 0b8ad4a..619de94 100644 --- a/data/livebench.json +++ b/data/livebench.json @@ -261,54 +261,19 @@ "instruction_following": 72.4, "overall": 74.8 }, - "grok-4.20-beta-0309-reasoning": { - "coding": 52.4, - "reasoning": 78.3, - "vision": 51.2, - "instruction_following": 68.8, - "overall": 69.0 - }, - "gpt-5.1-2025-11-13-nothinking": { - "coding": 48.0, - "reasoning": 32.8, - "vision": 42.9, - "instruction_following": 44.9, - "overall": 39.7 - }, - "qwen3-235b-a22b-instruct-2507": { - "coding": 35.8, - "reasoning": 56.6, - "vision": 39.2, - "instruction_following": 43.9, - "overall": 48.0 - }, - "gpt-5.3-codex-high": { - "coding": 64.3, - "reasoning": 79.3, - "vision": 64.3, - "instruction_following": 70.8, - "overall": 73.2 - }, - "gemini-3.1-flash-lite-preview-high": { - "coding": 47.4, - "reasoning": 64.9, - "vision": 46.5, - "instruction_following": 71.9, - "overall": 62.1 - }, - "gemini-3-flash-preview-minimal": { - "coding": 57.4, - "reasoning": 54.8, - "vision": 59.4, - "instruction_following": 50.2, - "overall": 54.4 + "kimi-k2-thinking": { + "coding": 50.0, + "reasoning": 66.6, + "vision": 43.4, + "instruction_following": 67.7, + "overall": 62.3 }, - "claude-opus-4-5-20251101-thinking-64k-medium-effort": { - "coding": 66.0, - "reasoning": 82.0, - "vision": 62.7, - "instruction_following": 67.5, - "overall": 73.9 + "gpt-5-mini-low": { + "coding": 49.8, + "reasoning": 52.3, + "vision": 28.7, + "instruction_following": 58.6, + "overall": 52.4 }, "claude-opus-4-5-20251101-medium-effort": { "coding": 69.4, @@ -317,61 +282,26 @@ "instruction_following": 50.6, "overall": 57.0 }, - "claude-4-sonnet-20250514-thinking-64k": { - "coding": 55.0, - "reasoning": 64.6, - "vision": 55.9, - "instruction_following": 58.5, - "overall": 60.6 - }, - "gpt-5.4-mini-high": { - "coding": 51.7, - "reasoning": 71.8, - "vision": 40.5, - "instruction_following": 62.4, - "overall": 63.7 - }, - "gpt-5.3-codex-xhigh": { - "coding": 71.0, - "reasoning": 72.6, - "vision": 57.5, - "instruction_following": 74.0, - "overall": 72.0 - }, - "qwen3-next-80b-a3b-thinking": { - "coding": 29.3, - "reasoning": 59.4, - "vision": 40.8, - "instruction_following": 55.2, - "overall": 51.0 - }, - "qwen3-next-80b-a3b-instruct": { - "coding": 33.3, - "reasoning": 56.3, - "vision": 40.0, - "instruction_following": 44.3, - "overall": 47.4 - }, - "gpt-5.4-mini": { - "coding": 40.1, - "reasoning": 28.8, - "vision": 30.6, - "instruction_following": 38.6, - "overall": 33.9 + "deepseek-v3.2-exp-thinking": { + "coding": 47.0, + "reasoning": 65.9, + "vision": 47.5, + "instruction_following": 57.9, + "overall": 58.9 }, - "gpt-5.5-medium": { - "coding": 41.5, - "reasoning": 78.6, - "vision": 70.7, - "instruction_following": 73.8, - "overall": 69.0 + "gemini-2.5-pro-06-05-highthinking": { + "coding": 50.3, + "reasoning": 62.8, + "vision": 63.0, + "instruction_following": 52.8, + "overall": 57.5 }, - "kimi-k2-thinking": { - "coding": 50.0, - "reasoning": 66.6, - "vision": 43.4, - "instruction_following": 67.7, - "overall": 62.3 + "gpt-5-mini": { + "coding": 51.4, + "reasoning": 62.7, + "vision": 37.4, + "instruction_following": 69.7, + "overall": 61.0 }, "grok-4.20-beta-0309-non-reasoning": { "coding": 46.4, @@ -380,96 +310,54 @@ "instruction_following": 45.3, "overall": 37.9 }, - "gemma-4-31b-it": { - "coding": 48.1, - "reasoning": 65.2, - "vision": 47.5, - "instruction_following": 71.6, - "overall": 62.4 - }, - "grok-4-1-fast-reasoning": { - "coding": 46.8, - "reasoning": 73.7, - "vision": 47.6, - "instruction_following": 48.3, - "overall": 60.1 - }, - "claude-haiku-4-5-20251001-thinking-64k": { - "coding": 54.1, - "reasoning": 66.0, - "vision": 42.5, - "instruction_following": 60.7, - "overall": 61.0 - }, - "gpt-5.3-instant": { - "coding": 48.5, - "reasoning": 62.9, - "vision": 53.0, - "instruction_following": 64.8, - "overall": 59.8 - }, - "claude-opus-4-8-xhigh-effort": { - "coding": 67.7, - "reasoning": 86.0, - "vision": 70.4, - "instruction_following": 71.5, - "overall": 77.6 - }, - "gpt-5.4-nano": { - "coding": 41.0, - "reasoning": 23.4, - "vision": 14.4, - "instruction_following": 35.2, - "overall": 29.9 - }, - "claude-opus-4-8-medium-effort": { - "coding": 66.6, - "reasoning": 82.0, - "vision": 61.3, - "instruction_following": 67.7, - "overall": 74.0 + "gpt-5.1-2025-11-13-nothinking": { + "coding": 48.0, + "reasoning": 32.8, + "vision": 42.9, + "instruction_following": 44.9, + "overall": 39.7 }, - "qwen3.6-flash": { - "coding": 54.0, - "reasoning": 65.8, - "vision": 43.1, - "instruction_following": 59.1, - "overall": 60.5 + "gpt-5.2-2025-12-11-nothinking": { + "coding": 54.6, + "reasoning": 44.0, + "vision": 45.1, + "instruction_following": 46.2, + "overall": 46.9 }, - "gpt-5-mini": { - "coding": 51.4, - "reasoning": 62.7, - "vision": 37.4, - "instruction_following": 69.7, - "overall": 61.0 + "gpt-5.4-nano-high": { + "coding": 56.8, + "reasoning": 73.4, + "vision": 34.8, + "instruction_following": 56.3, + "overall": 63.6 }, - "gemini-3-pro-preview-11-2025-high": { - "coding": 62.8, - "reasoning": 79.0, - "vision": 69.9, - "instruction_following": 73.2, - "overall": 73.5 + "gpt-5.1-2025-11-13-medium": { + "coding": 62.3, + "reasoning": 74.9, + "vision": 56.0, + "instruction_following": 66.4, + "overall": 69.1 }, - "claude-4-1-opus-20250805-base": { - "coding": 62.4, - "reasoning": 48.1, - "vision": 60.3, - "instruction_following": 49.3, - "overall": 52.0 + "grok-4.20-beta-0309-reasoning": { + "coding": 52.4, + "reasoning": 78.3, + "vision": 51.2, + "instruction_following": 68.8, + "overall": 69.0 }, - "gpt-5-nano": { - "coding": 44.0, - "reasoning": 48.4, - "vision": 16.5, - "instruction_following": 56.0, - "overall": 48.0 + "devstral-2512": { + "coding": 52.7, + "reasoning": 33.8, + "vision": 38.7, + "instruction_following": 36.4, + "overall": 38.8 }, - "kimi-k2-instruct": { - "coding": 48.7, - "reasoning": 45.7, - "vision": 45.9, - "instruction_following": 44.0, - "overall": 45.9 + "glm-4.6v": { + "coding": 27.7, + "reasoning": 43.4, + "vision": 29.7, + "instruction_following": 41.4, + "overall": 38.9 }, "claude-opus-4-7-low-effort": { "coding": 66.6, @@ -478,75 +366,26 @@ "instruction_following": 58.9, "overall": 68.4 }, - "deepseek-v3.2-exp": { - "coding": 51.3, - "reasoning": 49.3, - "vision": 45.5, - "instruction_following": 43.1, - "overall": 47.9 - }, - "gpt-5.1-2025-11-13-low": { - "coding": 56.1, - "reasoning": 59.7, - "vision": 53.6, - "instruction_following": 60.6, - "overall": 58.9 - }, - "claude-opus-4-5-20251101-low-effort": { - "coding": 61.3, - "reasoning": 51.5, - "vision": 61.5, - "instruction_following": 50.0, - "overall": 53.6 - }, - "deepseek-v3.2-thinking": { - "coding": 49.8, - "reasoning": 72.6, - "vision": 45.7, - "instruction_following": 59.7, - "overall": 63.1 - }, - "gpt-5-pro-2025-10-06": { - "coding": 59.8, - "reasoning": 78.0, - "vision": 64.1, - "instruction_following": 69.7, - "overall": 71.3 - }, - "nemotron-3-ultra-550b-a55b": { - "coding": 56.5, - "reasoning": 42.9, - "vision": 36.5, - "instruction_following": 62.5, - "overall": 50.7 - }, - "gpt-5.2-2025-12-11-low": { - "coding": 59.6, - "reasoning": 72.5, - "vision": 47.0, - "instruction_following": 61.1, - "overall": 65.6 - }, - "claude-4-1-opus-20250805-thinking-32k": { - "coding": 58.9, - "reasoning": 65.9, - "vision": 60.5, - "instruction_following": 55.3, - "overall": 61.4 + "qwen3-next-80b-a3b-instruct": { + "coding": 33.3, + "reasoning": 56.3, + "vision": 40.0, + "instruction_following": 44.3, + "overall": 47.4 }, - "gpt-5.1-codex-max-high": { - "coding": 64.3, - "reasoning": 80.7, - "vision": 57.4, - "instruction_following": 73.9, - "overall": 74.4 + "claude-opus-4-5-20251101-thinking-64k-low-effort": { + "coding": 61.0, + "reasoning": 69.8, + "vision": 60.9, + "instruction_following": 60.7, + "overall": 65.1 }, - "claude-opus-4-5-20251101-high-effort": { - "coding": 68.1, - "reasoning": 54.9, - "vision": 62.3, - "instruction_following": 49.0, - "overall": 56.5 + "claude-opus-4-8-high-effort": { + "coding": 66.6, + "reasoning": 85.0, + "vision": 64.9, + "instruction_following": 69.9, + "overall": 76.2 }, "gpt-5.5-high": { "coding": 51.0, @@ -555,40 +394,47 @@ "instruction_following": 76.0, "overall": 77.1 }, - "glm-5": { - "coding": 62.5, - "reasoning": 74.0, - "vision": 63.6, - "instruction_following": 65.0, - "overall": 68.7 + "gpt-5.3-codex-xhigh": { + "coding": 71.0, + "reasoning": 72.6, + "vision": 57.5, + "instruction_following": 74.0, + "overall": 72.0 }, - "glm-5.1": { - "coding": 63.1, - "reasoning": 75.6, - "vision": 60.3, - "instruction_following": 69.5, - "overall": 70.6 + "gpt-5.4-mini-high": { + "coding": 51.7, + "reasoning": 71.8, + "vision": 40.5, + "instruction_following": 62.4, + "overall": 63.7 }, - "gemini-2.5-flash-lite-preview-09-2025-highthinking": { - "coding": 27.2, - "reasoning": 45.4, - "vision": 32.8, - "instruction_following": 47.6, - "overall": 41.5 + "qwen3-235b-a22b-instruct-2507": { + "coding": 35.8, + "reasoning": 56.6, + "vision": 39.2, + "instruction_following": 43.9, + "overall": 48.0 }, - "gpt-5-nano-low": { - "coding": 27.1, - "reasoning": 36.1, - "vision": 13.9, - "instruction_following": 38.0, - "overall": 33.7 + "claude-4-sonnet-20250514-thinking-64k": { + "coding": 55.0, + "reasoning": 64.6, + "vision": 55.9, + "instruction_following": 58.5, + "overall": 60.6 }, - "claude-opus-4-8-low-effort": { - "coding": 53.6, - "reasoning": 81.0, - "vision": 63.0, - "instruction_following": 66.5, - "overall": 70.5 + "gpt-5.1-codex-max": { + "coding": 66.6, + "reasoning": 77.0, + "vision": 58.2, + "instruction_following": 71.1, + "overall": 72.4 + }, + "gemini-3-pro-preview-11-2025-high": { + "coding": 62.8, + "reasoning": 79.0, + "vision": 69.9, + "instruction_following": 73.2, + "overall": 73.5 }, "gemini-3-flash-preview-high": { "coding": 53.6, @@ -597,6 +443,20 @@ "instruction_following": 77.8, "overall": 73.0 }, + "claude-4-sonnet-20250514-base": { + "coding": 55.3, + "reasoning": 45.5, + "vision": 54.2, + "instruction_following": 47.0, + "overall": 48.4 + }, + "nemotron-3-super-120b-a12b": { + "coding": 38.5, + "reasoning": 31.0, + "vision": 36.1, + "instruction_following": 28.1, + "overall": 32.0 + }, "qwen3-32b-thinking": { "coding": 28.4, "reasoning": 49.7, @@ -604,12 +464,47 @@ "instruction_following": 42.6, "overall": 42.7 }, - "claude-opus-4-5-20251101-thinking-64k-low-effort": { - "coding": 61.0, - "reasoning": 69.8, - "vision": 60.9, + "qwen3.6-flash": { + "coding": 54.0, + "reasoning": 65.8, + "vision": 43.1, + "instruction_following": 59.1, + "overall": 60.5 + }, + "gpt-5.4-mini": { + "coding": 40.1, + "reasoning": 28.8, + "vision": 30.6, + "instruction_following": 38.6, + "overall": 33.9 + }, + "claude-haiku-4-5-20251001-thinking-64k": { + "coding": 54.1, + "reasoning": 66.0, + "vision": 42.5, "instruction_following": 60.7, - "overall": 65.1 + "overall": 61.0 + }, + "gpt-5.1-2025-11-13-low": { + "coding": 56.1, + "reasoning": 59.7, + "vision": 53.6, + "instruction_following": 60.6, + "overall": 58.9 + }, + "claude-opus-4-7-high-effort": { + "coding": 70.3, + "reasoning": 83.6, + "vision": 58.1, + "instruction_following": 64.7, + "overall": 74.7 + }, + "kimi-k2.5-thinking": { + "coding": 60.1, + "reasoning": 76.7, + "vision": 55.0, + "instruction_following": 65.3, + "overall": 69.2 }, "gpt-5.4-mini-low": { "coding": 43.3, @@ -618,117 +513,75 @@ "instruction_following": 50.7, "overall": 48.2 }, - "gpt-5.1-2025-11-13-medium": { - "coding": 62.3, - "reasoning": 74.9, - "vision": 56.0, - "instruction_following": 66.4, - "overall": 69.1 - }, - "deepseek-v3.2-exp-thinking": { - "coding": 47.0, - "reasoning": 65.9, - "vision": 47.5, - "instruction_following": 57.9, - "overall": 58.9 - }, - "gpt-5-nano-high": { - "coding": 39.0, - "reasoning": 52.8, - "vision": 17.4, - "instruction_following": 55.1, - "overall": 48.8 - }, - "gpt-5.2-2025-12-11-medium": { - "coding": 59.8, - "reasoning": 83.8, - "vision": 51.3, - "instruction_following": 66.3, - "overall": 72.6 + "glm-4.6": { + "coding": 49.4, + "reasoning": 61.5, + "vision": 50.0, + "instruction_following": 47.2, + "overall": 54.7 }, - "gpt-5.4-high": { - "coding": 59.3, - "reasoning": 85.6, - "vision": 66.0, - "instruction_following": 72.4, + "claude-sonnet-4-6-thinking-auto-high-effort": { + "coding": 66.0, + "reasoning": 84.8, + "vision": 59.7, + "instruction_following": 69.3, "overall": 75.6 }, - "claude-opus-4-8-high-effort": { - "coding": 66.6, - "reasoning": 85.0, - "vision": 64.9, - "instruction_following": 69.9, - "overall": 76.2 - }, - "gpt-5-mini-minimal": { - "coding": 41.3, - "reasoning": 35.6, - "vision": 25.9, - "instruction_following": 40.7, - "overall": 37.7 + "claude-opus-4-5-20251101-thinking-64k-medium-effort": { + "coding": 66.0, + "reasoning": 82.0, + "vision": 62.7, + "instruction_following": 67.5, + "overall": 73.9 }, - "grok-4-0709": { - "coding": 47.3, - "reasoning": 74.9, - "vision": 60.5, + "claude-sonnet-4-5-20250929": { + "coding": 59.4, + "reasoning": 47.8, + "vision": 57.3, "instruction_following": 49.9, - "overall": 61.8 - }, - "gemini-2.5-flash-06-05-highthinking": { - "coding": 36.4, - "reasoning": 49.9, - "vision": 52.7, - "instruction_following": 49.1, - "overall": 46.9 + "overall": 51.3 }, - "glm-4.6": { - "coding": 49.4, - "reasoning": 61.5, - "vision": 50.0, - "instruction_following": 47.2, - "overall": 54.7 + "deepseek-v3.2": { + "coding": 58.3, + "reasoning": 49.1, + "vision": 44.9, + "instruction_following": 44.6, + "overall": 49.8 }, - "glm-5v-turbo": { - "coding": 31.6, - "reasoning": 58.1, - "vision": 46.8, - "instruction_following": 46.4, - "overall": 48.8 + "gpt-5-nano": { + "coding": 44.0, + "reasoning": 48.4, + "vision": 16.5, + "instruction_following": 56.0, + "overall": 48.0 }, - "gemini-2.5-flash-preview-09-2025-highthinking": { + "gpt-5.4-nano": { "coding": 41.0, - "reasoning": 59.0, - "vision": 52.4, - "instruction_following": 49.2, - "overall": 52.3 - }, - "elephant-alpha": { - "coding": 23.7, - "reasoning": 41.4, - "vision": 13.7, - "instruction_following": 39.8, - "overall": 35.9 + "reasoning": 23.4, + "vision": 14.4, + "instruction_following": 35.2, + "overall": 29.9 }, - "claude-opus-4-7-high-effort": { - "coding": 70.3, - "reasoning": 83.6, - "vision": 58.1, - "instruction_following": 64.7, - "overall": 74.7 + "claude-opus-4-7-medium-effort": { + "coding": 66.0, + "reasoning": 80.7, + "vision": 60.3, + "instruction_following": 63.0, + "overall": 72.0 }, - "gpt-5-mini-high": { - "coding": 55.3, - "reasoning": 71.2, - "vision": 41.9, - "instruction_following": 71.8, - "overall": 66.6 + "mimo-v2-pro": { + "coding": 45.5, + "reasoning": 65.8, + "vision": 43.6, + "instruction_following": 57.8, + "overall": 58.4 }, - "grok-code-fast-1-0825": { - "coding": 45.8, - "reasoning": 43.5, - "vision": 38.0, - "instruction_following": 43.2, - "overall": 43.6 + "glm-4.7": { + "coding": 54.3, + "reasoning": 60.8, + "vision": 53.2, + "instruction_following": 54.2, + "overall": 57.3 }, "gpt-5.1-codex": { "coding": 60.7, @@ -737,6 +590,41 @@ "instruction_following": 66.8, "overall": 69.3 }, + "gpt-5.4-mini-medium": { + "coding": 44.8, + "reasoning": 65.0, + "vision": 36.9, + "instruction_following": 60.1, + "overall": 58.1 + }, + "nemotron-3-ultra-550b-a55b": { + "coding": 56.5, + "reasoning": 42.9, + "vision": 36.5, + "instruction_following": 62.5, + "overall": 50.7 + }, + "gemini-2.5-flash-preview-09-2025-highthinking": { + "coding": 41.0, + "reasoning": 59.0, + "vision": 52.4, + "instruction_following": 49.2, + "overall": 52.3 + }, + "gemini-3-pro-preview-11-2025-low": { + "coding": 61.3, + "reasoning": 70.8, + "vision": 61.5, + "instruction_following": 50.1, + "overall": 62.9 + }, + "gpt-5-nano-low": { + "coding": 27.1, + "reasoning": 36.1, + "vision": 13.9, + "instruction_following": 38.0, + "overall": 33.7 + }, "gpt-5.1-codex-mini": { "coding": 52.0, "reasoning": 66.3, @@ -744,33 +632,47 @@ "instruction_following": 63.1, "overall": 60.8 }, - "deepseek-v3.2": { - "coding": 58.3, - "reasoning": 49.1, - "vision": 44.9, - "instruction_following": 44.6, - "overall": 49.8 + "gpt-5.4-high": { + "coding": 59.3, + "reasoning": 85.6, + "vision": 66.0, + "instruction_following": 72.4, + "overall": 75.6 }, - "gpt-5.2-2025-12-11-nothinking": { - "coding": 54.6, - "reasoning": 44.0, - "vision": 45.1, - "instruction_following": 46.2, + "gemma-4-31b-it": { + "coding": 48.1, + "reasoning": 65.2, + "vision": 47.5, + "instruction_following": 71.6, + "overall": 62.4 + }, + "gemini-2.5-flash-06-05-highthinking": { + "coding": 36.4, + "reasoning": 49.9, + "vision": 52.7, + "instruction_following": 49.1, "overall": 46.9 }, - "claude-sonnet-4-6-thinking-auto-low-effort": { - "coding": 67.7, - "reasoning": 78.5, - "vision": 53.3, - "instruction_following": 59.9, - "overall": 70.2 + "claude-opus-4-5-20251101-low-effort": { + "coding": 61.3, + "reasoning": 51.5, + "vision": 61.5, + "instruction_following": 50.0, + "overall": 53.6 }, - "claude-sonnet-4-6-thinking-auto-high-effort": { - "coding": 66.0, - "reasoning": 84.8, - "vision": 59.7, - "instruction_following": 69.3, - "overall": 75.6 + "gpt-5-mini-minimal": { + "coding": 41.3, + "reasoning": 35.6, + "vision": 25.9, + "instruction_following": 40.7, + "overall": 37.7 + }, + "qwen3-235b-a22b-thinking-2507": { + "coding": 31.6, + "reasoning": 60.0, + "vision": 47.6, + "instruction_following": 58.6, + "overall": 52.9 }, "gemini-2.5-flash-lite-highthinking": { "coding": 29.6, @@ -779,103 +681,110 @@ "instruction_following": 44.2, "overall": 41.5 }, - "glm-4.7": { - "coding": 54.3, - "reasoning": 60.8, - "vision": 53.2, - "instruction_following": 54.2, - "overall": 57.3 - }, - "glm-4.6v": { - "coding": 27.7, - "reasoning": 43.4, - "vision": 29.7, - "instruction_following": 41.4, - "overall": 38.9 + "gpt-5.3-instant": { + "coding": 48.5, + "reasoning": 62.9, + "vision": 53.0, + "instruction_following": 64.8, + "overall": 59.8 }, - "minimax-m2.5": { - "coding": 59.3, - "reasoning": 62.3, - "vision": 31.3, - "instruction_following": 62.2, - "overall": 60.3 + "glm-5": { + "coding": 62.5, + "reasoning": 74.0, + "vision": 63.6, + "instruction_following": 65.0, + "overall": 68.7 }, - "arcee-trinity-large-preview": { - "coding": 28.3, - "reasoning": 28.6, - "vision": 31.6, - "instruction_following": 35.4, - "overall": 30.4 + "gpt-5.1-2025-11-13-high": { + "coding": 61.0, + "reasoning": 80.4, + "vision": 60.4, + "instruction_following": 69.9, + "overall": 72.6 }, - "qwen3-235b-a22b-thinking-2507": { - "coding": 31.6, - "reasoning": 60.0, + "grok-4-1-fast-reasoning": { + "coding": 46.8, + "reasoning": 73.7, "vision": 47.6, - "instruction_following": 58.6, - "overall": 52.9 + "instruction_following": 48.3, + "overall": 60.1 }, - "claude-opus-4-7-medium-effort": { - "coding": 66.0, - "reasoning": 80.7, - "vision": 60.3, - "instruction_following": 63.0, - "overall": 72.0 + "gpt-5.2-2025-12-11-low": { + "coding": 59.6, + "reasoning": 72.5, + "vision": 47.0, + "instruction_following": 61.1, + "overall": 65.6 }, - "gemini-2.5-pro-06-05-highthinking": { - "coding": 50.3, - "reasoning": 62.8, + "kimi-k2-instruct": { + "coding": 48.7, + "reasoning": 45.7, + "vision": 45.9, + "instruction_following": 44.0, + "overall": 45.9 + }, + "gpt-5.5-medium": { + "coding": 41.5, + "reasoning": 78.6, + "vision": 70.7, + "instruction_following": 73.8, + "overall": 69.0 + }, + "claude-opus-4-8-low-effort": { + "coding": 53.6, + "reasoning": 81.0, "vision": 63.0, - "instruction_following": 52.8, - "overall": 57.5 + "instruction_following": 66.5, + "overall": 70.5 }, - "gpt-oss-120b": { - "coding": 34.1, - "reasoning": 51.8, - "vision": 18.6, - "instruction_following": 51.2, - "overall": 46.4 + "gpt-5-mini-high": { + "coding": 55.3, + "reasoning": 71.2, + "vision": 41.9, + "instruction_following": 71.8, + "overall": 66.6 }, - "gemini-3-pro-preview-11-2025-low": { - "coding": 61.3, - "reasoning": 70.8, - "vision": 61.5, - "instruction_following": 50.1, - "overall": 62.9 + "gpt-5-pro-2025-10-06": { + "coding": 59.8, + "reasoning": 78.0, + "vision": 64.1, + "instruction_following": 69.7, + "overall": 71.3 }, - "gpt-5-mini-low": { - "coding": 49.8, - "reasoning": 52.3, - "vision": 28.7, - "instruction_following": 58.6, - "overall": 52.4 + "glm-5.1": { + "coding": 63.1, + "reasoning": 75.6, + "vision": 60.3, + "instruction_following": 69.5, + "overall": 70.6 }, - "devstral-2512": { - "coding": 52.7, - "reasoning": 33.8, - "vision": 38.7, - "instruction_following": 36.4, - "overall": 38.8 + "gemini-3-flash-preview-minimal": { + "coding": 57.4, + "reasoning": 54.8, + "vision": 59.4, + "instruction_following": 50.2, + "overall": 54.4 }, - "claude-4-sonnet-20250514-base": { - "coding": 55.3, - "reasoning": 45.5, - "vision": 54.2, - "instruction_following": 47.0, - "overall": 48.4 + "elephant-alpha": { + "coding": 23.7, + "reasoning": 41.4, + "vision": 13.7, + "instruction_following": 39.8, + "overall": 35.9 }, - "gpt-5.1-2025-11-13-high": { - "coding": 61.0, - "reasoning": 80.4, - "vision": 60.4, - "instruction_following": 69.9, - "overall": 72.6 + "arcee-trinity-large-preview": { + "coding": 28.3, + "reasoning": 28.6, + "vision": 31.6, + "instruction_following": 35.4, + "overall": 30.4 }, - "gpt-5.4-mini-medium": { - "coding": 44.8, - "reasoning": 65.0, - "vision": 36.9, - "instruction_following": 60.1, - "overall": 58.1 + "gpt-5.1-codex-max-high": { + "coding": 64.3, + "reasoning": 80.7, + "vision": 57.4, + "instruction_following": 73.9, + "overall": 74.4 }, "grok-4-1-fast-non-reasoning": { "coding": 27.7, @@ -884,54 +793,26 @@ "instruction_following": 38.6, "overall": 31.6 }, - "mimo-v2-pro": { - "coding": 45.5, - "reasoning": 65.8, - "vision": 43.6, - "instruction_following": 57.8, - "overall": 58.4 - }, - "minimax-m2.7": { - "coding": 52.0, - "reasoning": 72.4, - "vision": 34.0, - "instruction_following": 67.4, - "overall": 65.0 - }, - "nemotron-3-super-120b-a12b": { - "coding": 38.5, - "reasoning": 31.0, - "vision": 36.1, - "instruction_following": 28.1, - "overall": 32.0 - }, - "gpt-5.4-nano-high": { - "coding": 56.8, - "reasoning": 73.4, - "vision": 34.8, - "instruction_following": 56.3, - "overall": 63.6 - }, - "qwen3-30b-a3b-thinking": { - "coding": 20.6, - "reasoning": 45.3, - "vision": 31.2, - "instruction_following": 43.5, - "overall": 38.8 + "claude-opus-4-5-20251101-high-effort": { + "coding": 68.1, + "reasoning": 54.9, + "vision": 62.3, + "instruction_following": 49.0, + "overall": 56.5 }, - "kimi-k2.5-thinking": { - "coding": 60.1, - "reasoning": 76.7, - "vision": 55.0, - "instruction_following": 65.3, - "overall": 69.2 + "gpt-5.2-2025-12-11-medium": { + "coding": 59.8, + "reasoning": 83.8, + "vision": 51.3, + "instruction_following": 66.3, + "overall": 72.6 }, - "claude-sonnet-4-5-20250929": { - "coding": 59.4, - "reasoning": 47.8, - "vision": 57.3, - "instruction_following": 49.9, - "overall": 51.3 + "gpt-5.4-nano-medium": { + "coding": 52.5, + "reasoning": 66.5, + "vision": 29.7, + "instruction_following": 55.3, + "overall": 59.0 }, "claude-sonnet-4-5-20250929-thinking-64k": { "coding": 64.1, @@ -940,166 +821,166 @@ "instruction_following": 64.2, "overall": 67.9 }, - "claude-haiku-4-5-20251001": { - "coding": 48.9, - "reasoning": 41.9, - "vision": 37.8, - "instruction_following": 41.1, - "overall": 43.0 - }, - "gpt-5.4-nano-medium": { - "coding": 52.5, - "reasoning": 66.5, - "vision": 29.7, - "instruction_following": 55.3, - "overall": 59.0 + "gpt-oss-120b": { + "coding": 34.1, + "reasoning": 51.8, + "vision": 18.6, + "instruction_following": 51.2, + "overall": 46.4 }, - "gpt-5.4-nano-low": { - "coding": 50.1, - "reasoning": 49.0, - "vision": 17.6, - "instruction_following": 48.4, - "overall": 47.7 + "glm-5v-turbo": { + "coding": 31.6, + "reasoning": 58.1, + "vision": 46.8, + "instruction_following": 46.4, + "overall": 48.8 }, - "gpt-5.1-codex-max": { + "claude-opus-4-8-medium-effort": { "coding": 66.6, - "reasoning": 77.0, - "vision": 58.2, - "instruction_following": 71.1, - "overall": 72.4 + "reasoning": 82.0, + "vision": 61.3, + "instruction_following": 67.7, + "overall": 74.0 }, - "gpt-5.1-codex-max-xhigh": { - "coding": 64.0, - "reasoning": 85.4, - "vision": 58.2, - "instruction_following": 75.3, - "overall": 76.1 + "minimax-m2.5": { + "coding": 59.3, + "reasoning": 62.3, + "vision": 31.3, + "instruction_following": 62.2, + "overall": 60.3 }, - "deepseek-v3.2-speciale": { - "coding": 32.3, - "reasoning": 82.7, - "vision": 55.6, - "instruction_following": 62.6, - "overall": 62.7 + "claude-4-1-opus-20250805-base": { + "coding": 62.4, + "reasoning": 48.1, + "vision": 60.3, + "instruction_following": 49.3, + "overall": 52.0 }, - "gpt-5-minimal": { - "coding": 45.0, - "reasoning": 59.8, - "vision": 42.0, - "instruction_following": 70.2, - "overall": 58.2 + "grok-4-0709": { + "coding": 47.3, + "reasoning": 74.9, + "vision": 60.5, + "instruction_following": 49.9, + "overall": 61.8 }, - "gpt-5-low": { - "coding": 50.7, - "reasoning": 89.3, - "vision": 52.7, - "instruction_following": 80.9, - "overall": 74.6 + "deepseek-v3.2-thinking": { + "coding": 49.8, + "reasoning": 72.6, + "vision": 45.7, + "instruction_following": 59.7, + "overall": 63.1 }, - "gpt-5-codex": { - "coding": 56.8, - "reasoning": 96.1, - "vision": 54.0, - "instruction_following": 80.1, - "overall": 78.5 + "minimax-m2.7": { + "coding": 52.0, + "reasoning": 72.4, + "vision": 34.0, + "instruction_following": 67.4, + "overall": 65.0 }, - "deepseek-v3.1-terminus-thinking": { - "coding": 46.6, - "reasoning": 87.7, - "vision": 50.5, - "instruction_following": 76.4, - "overall": 71.3 + "qwen3-next-80b-a3b-thinking": { + "coding": 29.3, + "reasoning": 59.4, + "vision": 40.8, + "instruction_following": 55.2, + "overall": 51.0 }, - "claude-opus-4-5-20251101-thinking-low-effort": { - "coding": 61.0, - "reasoning": 88.9, - "vision": 60.9, - "instruction_following": 75.5, - "overall": 75.8 + "gemini-2.5-flash-lite-preview-09-2025-highthinking": { + "coding": 27.2, + "reasoning": 45.4, + "vision": 32.8, + "instruction_following": 47.6, + "overall": 41.5 }, - "deepseek-v3.1-terminus": { - "coding": 42.8, - "reasoning": 77.8, - "vision": 47.1, - "instruction_following": 75.2, - "overall": 66.2 + "grok-code-fast-1-0825": { + "coding": 45.8, + "reasoning": 43.5, + "vision": 38.0, + "instruction_following": 43.2, + "overall": 43.6 }, - "minimax-m2": { - "coding": 36.1, - "reasoning": 83.5, - "vision": 26.6, - "instruction_following": 71.5, - "overall": 64.2 + "gpt-5-nano-high": { + "coding": 39.0, + "reasoning": 52.8, + "vision": 17.4, + "instruction_following": 55.1, + "overall": 48.8 }, - "gpt-5": { - "coding": 60.0, - "reasoning": 93.0, - "vision": 57.0, - "instruction_following": 81.6, - "overall": 78.8 + "deepseek-v3.2-exp": { + "coding": 51.3, + "reasoning": 49.3, + "vision": 45.5, + "instruction_following": 43.1, + "overall": 47.9 }, - "qwen3-max-2025-09-23": { - "coding": 47.7, - "reasoning": 86.5, - "vision": 42.3, - "instruction_following": 72.9, - "overall": 69.7 + "gpt-5.3-codex-high": { + "coding": 64.3, + "reasoning": 79.3, + "vision": 64.3, + "instruction_following": 70.8, + "overall": 73.2 }, - "command-a-03-2025": { - "coding": 24.1, - "reasoning": 45.3, - "vision": 34.8, - "instruction_following": 66.4, - "overall": 45.8 + "claude-opus-4-8-xhigh-effort": { + "coding": 67.7, + "reasoning": 86.0, + "vision": 70.4, + "instruction_following": 71.5, + "overall": 77.6 }, - "gpt-5-chat": { - "coding": 38.4, - "reasoning": 71.8, - "vision": 44.2, - "instruction_following": 70.3, - "overall": 61.1 + "claude-4-1-opus-20250805-thinking-32k": { + "coding": 58.9, + "reasoning": 65.9, + "vision": 60.5, + "instruction_following": 55.3, + "overall": 61.4 }, - "qwen3-235b-a22b-thinking": { - "coding": 28.0, - "reasoning": 83.4, - "vision": 37.3, - "instruction_following": 75.7, - "overall": 64.0 + "gpt-5.4-nano-low": { + "coding": 50.1, + "reasoning": 49.0, + "vision": 17.6, + "instruction_following": 48.4, + "overall": 47.7 }, - "grok-4-fast-reasoning-2511": { - "coding": 40.6, - "reasoning": 90.2, - "vision": 51.8, - "instruction_following": 71.3, - "overall": 69.2 + "claude-sonnet-4-6-thinking-auto-low-effort": { + "coding": 67.7, + "reasoning": 78.5, + "vision": 53.3, + "instruction_following": 59.9, + "overall": 70.2 + }, + "gemini-3.1-flash-lite-preview-high": { + "coding": 47.4, + "reasoning": 64.9, + "vision": 46.5, + "instruction_following": 71.9, + "overall": 62.1 }, - "claude-3-7-sonnet-20250219-base": { - "coding": 50.4, - "reasoning": 61.3, - "vision": 53.7, - "instruction_following": 71.7, - "overall": 61.3 + "qwen3-30b-a3b-thinking": { + "coding": 20.6, + "reasoning": 45.3, + "vision": 31.2, + "instruction_following": 43.5, + "overall": 38.8 }, - "claude-opus-4-5-20251101-thinking-high-effort": { - "coding": 64.9, - "reasoning": 94.2, - "vision": 66.5, - "instruction_following": 77.1, - "overall": 79.6 + "claude-haiku-4-5-20251001": { + "coding": 48.9, + "reasoning": 41.9, + "vision": 37.8, + "instruction_following": 41.1, + "overall": 43.0 }, - "gpt-5-high": { - "coding": 58.8, - "reasoning": 94.9, - "vision": 59.5, - "instruction_following": 81.2, - "overall": 79.2 + "gpt-5.1-codex-max-xhigh": { + "coding": 64.0, + "reasoning": 85.4, + "vision": 58.2, + "instruction_following": 75.3, + "overall": 76.1 }, - "grok-4-fast-non-reasoning-2511": { - "coding": 28.4, - "reasoning": 46.3, - "vision": 39.8, - "instruction_following": 63.8, - "overall": 46.8 + "deepseek-v3.2-speciale": { + "coding": 32.3, + "reasoning": 82.7, + "vision": 55.6, + "instruction_following": 62.6, + "overall": 62.7 }, "claude-opus-4-5-20251101-thinking-medium-effort": { "coding": 66.0, @@ -1108,6 +989,13 @@ "instruction_following": 76.5, "overall": 79.9 }, + "glm-4.5-air": { + "coding": 34.1, + "reasoning": 75.1, + "vision": 33.2, + "instruction_following": 70.6, + "overall": 60.7 + }, "qwen3-coder-480b-a35b-instruct": { "coding": 49.0, "reasoning": 66.7, @@ -1122,12 +1010,19 @@ "instruction_following": 73.8, "overall": 65.8 }, - "glm-4.5-air": { - "coding": 34.1, - "reasoning": 75.1, - "vision": 33.2, - "instruction_following": 70.6, - "overall": 60.7 + "gpt-5-high": { + "coding": 58.8, + "reasoning": 94.9, + "vision": 59.5, + "instruction_following": 81.2, + "overall": 79.2 + }, + "gpt-5-minimal": { + "coding": 45.0, + "reasoning": 59.8, + "vision": 42.0, + "instruction_following": 70.2, + "overall": 58.2 }, "mistral-medium-2505": { "coding": 31.6, @@ -1150,61 +1045,166 @@ "instruction_following": 76.9, "overall": 69.1 }, - "deepseek-v3-0324": { - "coding": 68.9, - "reasoning": 61.9, + "claude-opus-4-5-20251101-thinking-high-effort": { + "coding": 64.9, + "reasoning": 94.2, + "vision": 66.5, + "instruction_following": 77.1, + "overall": 79.6 + }, + "gpt-5-codex": { + "coding": 56.8, + "reasoning": 96.1, + "vision": 54.0, + "instruction_following": 80.1, + "overall": 78.5 + }, + "grok-4-fast-non-reasoning-2511": { + "coding": 28.4, + "reasoning": 46.3, + "vision": 39.8, + "instruction_following": 63.8, + "overall": 46.8 + }, + "gpt-5-chat": { + "coding": 38.4, + "reasoning": 71.8, + "vision": 44.2, + "instruction_following": 70.3, + "overall": 61.1 + }, + "gpt-5-low": { + "coding": 50.7, + "reasoning": 89.3, + "vision": 52.7, + "instruction_following": 80.9, + "overall": 74.6 + }, + "deepseek-v3.1-terminus-thinking": { + "coding": 46.6, + "reasoning": 87.7, + "vision": 50.5, + "instruction_following": 76.4, + "overall": 71.3 + }, + "qwen3-max-2025-09-23": { + "coding": 47.7, + "reasoning": 86.5, "vision": 42.3, + "instruction_following": 72.9, + "overall": 69.7 + }, + "command-a-03-2025": { + "coding": 24.1, + "reasoning": 45.3, + "vision": 34.8, + "instruction_following": 66.4, + "overall": 45.8 + }, + "claude-3-7-sonnet-20250219-base": { + "coding": 50.4, + "reasoning": 61.3, + "vision": 53.7, "instruction_following": 71.7, - "overall": 65.3 + "overall": 61.3 }, - "o4-mini-2025-04-16-medium": { - "coding": 74.2, + "qwen3-235b-a22b-thinking": { + "coding": 28.0, "reasoning": 83.4, - "vision": 28.2, - "instruction_following": 75.0, - "overall": 75.7 + "vision": 37.3, + "instruction_following": 75.7, + "overall": 64.0 }, - "gemini-2.5-pro-preview-05-06": { - "coding": 72.9, - "reasoning": 86.3, - "vision": 59.1, - "instruction_following": 77.3, - "overall": 79.5 + "grok-4-fast-reasoning-2511": { + "coding": 40.6, + "reasoning": 90.2, + "vision": 51.8, + "instruction_following": 71.3, + "overall": 69.2 }, - "gpt-4.1-nano-2025-04-14": { - "coding": 63.2, - "reasoning": 40.6, - "vision": 12.7, - "instruction_following": 54.3, - "overall": 46.8 + "gpt-5": { + "coding": 60.0, + "reasoning": 93.0, + "vision": 57.0, + "instruction_following": 81.6, + "overall": 78.8 }, - "amazon.nova-pro-v1:0": { - "coding": 49.6, - "reasoning": 36.9, - "vision": 29.0, - "instruction_following": 58.2, - "overall": 46.0 + "claude-opus-4-5-20251101-thinking-low-effort": { + "coding": 61.0, + "reasoning": 88.9, + "vision": 60.9, + "instruction_following": 75.5, + "overall": 75.8 }, - "deepseek-r1-distill-llama-70b": { - "coding": 46.6, - "reasoning": 56.0, - "vision": 26.0, - "instruction_following": 63.2, - "overall": 55.7 + "deepseek-v3.1-terminus": { + "coding": 42.8, + "reasoning": 77.8, + "vision": 47.1, + "instruction_following": 75.2, + "overall": 66.2 + }, + "minimax-m2": { + "coding": 36.1, + "reasoning": 83.5, + "vision": 26.6, + "instruction_following": 71.5, + "overall": 64.2 + }, + "step-2-16k-202411": { + "coding": 57.6, + "reasoning": 46.0, + "vision": 33.7, + "instruction_following": 68.9, + "overall": 55.3 + }, + "qwen2.5-max": { + "coding": 66.8, + "reasoning": 54.2, + "vision": 45.9, + "instruction_following": 72.7, + "overall": 62.2 + }, + "command-r-plus-08-2024": { + "coding": 27.1, + "reasoning": 28.0, + "vision": 22.6, + "instruction_following": 52.9, + "overall": 36.9 + }, + "qwq-32b": { + "coding": 61.4, + "reasoning": 77.0, + "vision": 28.3, + "instruction_following": 73.7, + "overall": 70.8 + }, + "gpt-4.1-2025-04-14": { + "coding": 73.2, + "reasoning": 58.7, + "vision": 42.8, + "instruction_following": 71.1, + "overall": 64.2 + }, + "gemini-2.5-flash-preview-04-17": { + "coding": 60.3, + "reasoning": 76.2, + "vision": 54.3, + "instruction_following": 71.7, + "overall": 71.1 }, "gpt-4o-2024-11-20": { "coding": 69.3, "reasoning": 46.3, - "vision": 35.4, - "instruction_following": 63.5, - "overall": 54.9 - }, - "gpt-4o-mini-2024-07-18": { - "coding": 55.0, - "reasoning": 35.9, - "vision": 19.1, - "instruction_following": 55.1, - "overall": 44.5 + "vision": 35.4, + "instruction_following": 63.5, + "overall": 54.9 + }, + "grok-3-mini-beta-high": { + "coding": 54.5, + "reasoning": 78.0, + "vision": 47.4, + "instruction_following": 71.4, + "overall": 70.7 }, "gemini-2.5-flash-preview-05-20": { "coding": 62.8, @@ -1213,26 +1213,12 @@ "instruction_following": 73.9, "overall": 72.5 }, - "phi-4-reasoning-plus": { - "coding": 60.6, - "reasoning": 60.0, - "vision": 18.4, - "instruction_following": 61.1, - "overall": 57.9 - }, - "deepseek-r1": { - "coding": 76.1, - "reasoning": 76.5, - "vision": 38.8, - "instruction_following": 72.7, - "overall": 72.7 - }, - "qwen3-14b-thinking": { - "coding": 58.2, - "reasoning": 74.6, - "vision": 33.4, - "instruction_following": 73.2, - "overall": 69.5 + "o4-mini-2025-04-16-high": { + "coding": 80.0, + "reasoning": 89.1, + "vision": 34.7, + "instruction_following": 75.7, + "overall": 79.5 }, "qwen2.5-7b-instruct-turbo": { "coding": 34.3, @@ -1241,13 +1227,6 @@ "instruction_following": 43.2, "overall": 35.8 }, - "gpt-4.5-preview-2025-02-27": { - "coding": 76.1, - "reasoning": 66.7, - "vision": 48.1, - "instruction_following": 68.6, - "overall": 67.4 - }, "claude-3-5-haiku-20241022": { "coding": 53.2, "reasoning": 37.7, @@ -1255,6 +1234,20 @@ "instruction_following": 58.9, "overall": 46.7 }, + "gpt-4.1-nano-2025-04-14": { + "coding": 63.2, + "reasoning": 40.6, + "vision": 12.7, + "instruction_following": 54.3, + "overall": 46.8 + }, + "chatgpt-4o-latest-2025-03-27": { + "coding": 77.5, + "reasoning": 55.1, + "vision": 41.0, + "instruction_following": 69.1, + "overall": 62.2 + }, "amazon.nova-micro-v1:0": { "coding": 28.9, "reasoning": 30.1, @@ -1262,19 +1255,19 @@ "instruction_following": 45.8, "overall": 34.8 }, - "gemini-2.5-flash-preview-04-17": { - "coding": 60.3, - "reasoning": 76.2, - "vision": 54.3, - "instruction_following": 71.7, - "overall": 71.1 + "claude-4-opus-20250514-thinking-32k": { + "coding": 73.3, + "reasoning": 88.1, + "vision": 60.5, + "instruction_following": 75.1, + "overall": 79.7 }, - "mistral-small-2503": { - "coding": 49.6, - "reasoning": 40.3, - "vision": 26.8, - "instruction_following": 57.5, - "overall": 47.1 + "dracarys2-72b-instruct": { + "coding": 58.7, + "reasoning": 47.0, + "vision": 29.3, + "instruction_following": 55.6, + "overall": 50.6 }, "deepseek-r1-distill-qwen-32b": { "coding": 47.0, @@ -1283,41 +1276,6 @@ "instruction_following": 51.7, "overall": 48.4 }, - "grok-3-mini-beta-high": { - "coding": 54.5, - "reasoning": 78.0, - "vision": 47.4, - "instruction_following": 71.4, - "overall": 70.7 - }, - "learnlm-2.0-flash-experimental": { - "coding": 64.3, - "reasoning": 53.7, - "vision": 34.9, - "instruction_following": 70.6, - "overall": 60.2 - }, - "hunyuan-turbos-20250313": { - "coding": 50.4, - "reasoning": 47.3, - "vision": 21.4, - "instruction_following": 65.7, - "overall": 52.9 - }, - "dracarys2-llama-3.1-70b-instruct": { - "coding": 41.1, - "reasoning": 41.3, - "vision": 34.9, - "instruction_following": 59.6, - "overall": 47.7 - }, - "command-r-08-2024": { - "coding": 26.1, - "reasoning": 24.8, - "vision": 17.8, - "instruction_following": 49.5, - "overall": 33.8 - }, "grok-3-beta": { "coding": 73.6, "reasoning": 62.7, @@ -1325,47 +1283,33 @@ "instruction_following": 71.0, "overall": 66.1 }, - "llama4-maverick-instruct-basic": { - "coding": 54.2, - "reasoning": 57.5, - "vision": 34.6, - "instruction_following": 63.7, - "overall": 58.0 - }, - "qwq-32b": { - "coding": 61.4, - "reasoning": 77.0, - "vision": 28.3, - "instruction_following": 73.7, - "overall": 70.8 - }, - "chatgpt-4o-latest-2025-03-27": { - "coding": 77.5, - "reasoning": 55.1, - "vision": 41.0, - "instruction_following": 69.1, - "overall": 62.2 + "amazon.nova-pro-v1:0": { + "coding": 49.6, + "reasoning": 36.9, + "vision": 29.0, + "instruction_following": 58.2, + "overall": 46.0 }, - "dracarys2-72b-instruct": { - "coding": 58.7, - "reasoning": 47.0, - "vision": 29.3, - "instruction_following": 55.6, - "overall": 50.6 + "deepseek-r1": { + "coding": 76.1, + "reasoning": 76.5, + "vision": 38.8, + "instruction_following": 72.7, + "overall": 72.7 }, - "claude-3-5-sonnet-20241022": { - "coding": 73.9, - "reasoning": 52.8, - "vision": 50.3, - "instruction_following": 64.5, - "overall": 59.7 + "gpt-4.5-preview-2025-02-27": { + "coding": 76.1, + "reasoning": 66.7, + "vision": 48.1, + "instruction_following": 68.6, + "overall": 67.4 }, - "step-2-16k-202411": { - "coding": 57.6, - "reasoning": 46.0, - "vision": 33.7, - "instruction_following": 68.9, - "overall": 55.3 + "mistral-small-2503": { + "coding": 49.6, + "reasoning": 40.3, + "vision": 26.8, + "instruction_following": 57.5, + "overall": 47.1 }, "gemini-2.5-pro-preview-03-25": { "coding": 71.1, @@ -1374,6 +1318,20 @@ "instruction_following": 72.0, "overall": 77.3 }, + "command-r-08-2024": { + "coding": 26.1, + "reasoning": 24.8, + "vision": 17.8, + "instruction_following": 49.5, + "overall": 33.8 + }, + "deepseek-v3-0324": { + "coding": 68.9, + "reasoning": 61.9, + "vision": 42.3, + "instruction_following": 71.7, + "overall": 65.3 + }, "gemma-3-27b-it": { "coding": 48.9, "reasoning": 46.7, @@ -1381,68 +1339,40 @@ "instruction_following": 61.3, "overall": 51.3 }, - "amazon.nova-lite-v1:0": { - "coding": 45.0, - "reasoning": 33.2, - "vision": 19.9, - "instruction_following": 48.2, - "overall": 39.5 - }, - "o3-2025-04-16-medium": { - "coding": 77.9, - "reasoning": 87.8, - "vision": 45.4, - "instruction_following": 77.4, - "overall": 80.0 - }, - "gpt-4.1-2025-04-14": { - "coding": 73.2, - "reasoning": 58.7, - "vision": 42.8, - "instruction_following": 71.1, - "overall": 64.2 - }, - "command-r-plus-08-2024": { - "coding": 27.1, - "reasoning": 28.0, - "vision": 22.6, - "instruction_following": 52.9, - "overall": 36.9 - }, - "o4-mini-2025-04-16-high": { - "coding": 80.0, - "reasoning": 89.1, - "vision": 34.7, - "instruction_following": 75.7, - "overall": 79.5 + "phi-4-reasoning-plus": { + "coding": 60.6, + "reasoning": 60.0, + "vision": 18.4, + "instruction_following": 61.1, + "overall": 57.9 }, - "llama-3.3-70b-instruct-turbo": { - "coding": 51.8, - "reasoning": 44.8, - "vision": 35.2, - "instruction_following": 65.0, - "overall": 52.6 + "gpt-4o-mini-2024-07-18": { + "coding": 55.0, + "reasoning": 35.9, + "vision": 19.1, + "instruction_following": 55.1, + "overall": 44.5 }, - "o3-2025-04-16-high": { - "coding": 76.7, - "reasoning": 89.8, - "vision": 51.0, - "instruction_following": 78.2, - "overall": 81.4 + "learnlm-2.0-flash-experimental": { + "coding": 64.3, + "reasoning": 53.7, + "vision": 34.9, + "instruction_following": 70.6, + "overall": 60.2 }, - "qwen2.5-72b-instruct-turbo": { - "coding": 57.3, - "reasoning": 45.4, - "vision": 31.6, - "instruction_following": 57.3, - "overall": 50.5 + "llama4-maverick-instruct-basic": { + "coding": 54.2, + "reasoning": 57.5, + "vision": 34.6, + "instruction_following": 63.7, + "overall": 58.0 }, - "mistral-large-2411": { - "coding": 62.9, - "reasoning": 41.4, - "vision": 36.5, - "instruction_following": 61.3, - "overall": 51.3 + "o3-2025-04-16-medium": { + "coding": 77.9, + "reasoning": 87.8, + "vision": 45.4, + "instruction_following": 77.4, + "overall": 80.0 }, "claude-4-opus-20250514-base": { "coding": 72.9, @@ -1451,12 +1381,19 @@ "instruction_following": 74.8, "overall": 73.6 }, - "qwen2.5-max": { - "coding": 66.8, - "reasoning": 54.2, - "vision": 45.9, - "instruction_following": 72.7, - "overall": 62.2 + "learnlm-1.5-pro-experimental": { + "coding": 58.9, + "reasoning": 48.6, + "vision": 38.1, + "instruction_following": 53.8, + "overall": 51.2 + }, + "o3-2025-04-16-high": { + "coding": 76.7, + "reasoning": 89.8, + "vision": 51.0, + "instruction_following": 78.2, + "overall": 81.4 }, "gpt-4.1-mini-2025-04-14": { "coding": 72.1, @@ -1465,12 +1402,26 @@ "instruction_following": 64.2, "overall": 59.2 }, - "learnlm-1.5-pro-experimental": { - "coding": 58.9, - "reasoning": 48.6, - "vision": 38.1, - "instruction_following": 53.8, - "overall": 51.2 + "mistral-large-2411": { + "coding": 62.9, + "reasoning": 41.4, + "vision": 36.5, + "instruction_following": 61.3, + "overall": 51.3 + }, + "amazon.nova-lite-v1:0": { + "coding": 45.0, + "reasoning": 33.2, + "vision": 19.9, + "instruction_following": 48.2, + "overall": 39.5 + }, + "claude-3-5-sonnet-20241022": { + "coding": 73.9, + "reasoning": 52.8, + "vision": 50.3, + "instruction_following": 64.5, + "overall": 59.7 }, "gemini-2.0-flash-lite-001": { "coding": 59.3, @@ -1479,60 +1430,61 @@ "instruction_following": 68.8, "overall": 56.0 }, - "claude-4-opus-20250514-thinking-32k": { - "coding": 73.3, - "reasoning": 88.1, - "vision": 60.5, - "instruction_following": 75.1, - "overall": 79.7 - }, - "phi-4": { - "reasoning": 42.0, - "vision": 20.0, - "instruction_following": 50.6, - "overall": 44.1 + "hunyuan-turbos-20250313": { + "coding": 50.4, + "reasoning": 47.3, + "vision": 21.4, + "instruction_following": 65.7, + "overall": 52.9 }, - "gemini-2.5-pro-exp-03-25": { - "reasoning": 86.0, - "vision": 60.1, - "instruction_following": 78.4, - "overall": 80.9 + "llama-3.3-70b-instruct-turbo": { + "coding": 51.8, + "reasoning": 44.8, + "vision": 35.2, + "instruction_following": 65.0, + "overall": 52.6 }, - "grok-2-1212": { - "reasoning": 50.5, - "vision": 36.4, - "instruction_following": 63.3, - "overall": 55.0 + "gemini-2.5-pro-preview-05-06": { + "coding": 72.9, + "reasoning": 86.3, + "vision": 59.1, + "instruction_following": 77.3, + "overall": 79.5 }, - "o1-mini-2024-09-12": { - "reasoning": 61.4, - "vision": 18.7, - "instruction_following": 59.1, - "overall": 57.4 + "dracarys2-llama-3.1-70b-instruct": { + "coding": 41.1, + "reasoning": 41.3, + "vision": 34.9, + "instruction_following": 59.6, + "overall": 47.7 }, - "o3-mini-2025-01-31-high": { - "reasoning": 80.4, - "vision": 23.1, - "instruction_following": 77.4, - "overall": 75.0 + "qwen3-14b-thinking": { + "coding": 58.2, + "reasoning": 74.6, + "vision": 33.4, + "instruction_following": 73.2, + "overall": 69.5 }, - "gemini-2.0-pro-exp-02-05": { - "reasoning": 59.7, - "vision": 47.7, - "instruction_following": 78.2, - "overall": 66.8 + "o4-mini-2025-04-16-medium": { + "coding": 74.2, + "reasoning": 83.4, + "vision": 28.2, + "instruction_following": 75.0, + "overall": 75.7 }, - "o3-mini-2025-01-31-low": { - "reasoning": 65.4, - "vision": 15.4, - "instruction_following": 72.0, - "overall": 64.7 + "qwen2.5-72b-instruct-turbo": { + "coding": 57.3, + "reasoning": 45.4, + "vision": 31.6, + "instruction_following": 57.3, + "overall": 50.5 }, - "o1-2024-12-17-high": { - "reasoning": 84.4, - "vision": 47.5, - "instruction_following": 73.9, - "overall": 77.3 + "deepseek-r1-distill-llama-70b": { + "coding": 46.6, + "reasoning": 56.0, + "vision": 26.0, + "instruction_following": 63.2, + "overall": 55.7 }, "gemma-3-12b-it": { "reasoning": 42.0, @@ -1540,29 +1492,41 @@ "instruction_following": 58.0, "overall": 47.6 }, + "azerogpt": { + "reasoning": 33.0, + "vision": 26.5, + "instruction_following": 45.2, + "overall": 37.7 + }, "o3-mini-2025-01-31-medium": { "reasoning": 77.1, "vision": 16.5, "instruction_following": 75.1, "overall": 71.9 }, - "gemini-2.0-flash-thinking-exp-01-21": { - "reasoning": 70.5, - "vision": 36.4, - "instruction_following": 76.7, - "overall": 70.7 + "meta-llama-3.1-70b-instruct-turbo": { + "reasoning": 35.3, + "vision": 36.2, + "instruction_following": 61.6, + "overall": 46.6 }, - "meta-llama-3.1-8b-instruct-turbo": { - "reasoning": 19.9, - "vision": 12.8, - "instruction_following": 43.4, - "overall": 29.5 + "gemini-1.5-flash-8b-001": { + "reasoning": 31.1, + "vision": 18.8, + "instruction_following": 55.6, + "overall": 40.8 }, - "gpt-4o-2024-08-06": { - "reasoning": 48.8, - "vision": 37.2, - "instruction_following": 65.0, - "overall": 54.9 + "phi-4": { + "reasoning": 42.0, + "vision": 20.0, + "instruction_following": 50.6, + "overall": 44.1 + }, + "o1-2024-12-17-high": { + "reasoning": 84.4, + "vision": 47.5, + "instruction_following": 73.9, + "overall": 77.3 }, "llama-4-maverick-17b-128e-instruct": { "reasoning": 57.5, @@ -1570,23 +1534,53 @@ "instruction_following": 68.5, "overall": 60.6 }, + "gpt-4o-2024-08-06": { + "reasoning": 48.8, + "vision": 37.2, + "instruction_following": 65.0, + "overall": 54.9 + }, + "grok-2-1212": { + "reasoning": 50.5, + "vision": 36.4, + "instruction_following": 63.3, + "overall": 55.0 + }, + "gemini-2.0-flash-thinking-exp-01-21": { + "reasoning": 70.5, + "vision": 36.4, + "instruction_following": 76.7, + "overall": 70.7 + }, + "o3-mini-2025-01-31-high": { + "reasoning": 80.4, + "vision": 23.1, + "instruction_following": 77.4, + "overall": 75.0 + }, + "gemini-2.5-pro-exp-03-25": { + "reasoning": 86.0, + "vision": 60.1, + "instruction_following": 78.4, + "overall": 80.9 + }, "meta-llama-3.1-405b-instruct-turbo": { "reasoning": 44.9, "vision": 41.8, "instruction_following": 64.5, "overall": 53.1 }, - "meta-llama-3.1-70b-instruct-turbo": { - "reasoning": 35.3, - "vision": 36.2, - "instruction_following": 61.6, - "overall": 46.6 + "claude-3-opus-20240229": { + "reasoning": 43.5, + "vision": 45.7, + "instruction_following": 63.7, + "overall": 52.3 }, - "gemini-1.5-flash-8b-001": { - "reasoning": 31.1, - "vision": 18.8, - "instruction_following": 55.6, - "overall": 40.8 + "meta-llama-3.1-8b-instruct-turbo": { + "reasoning": 19.9, + "vision": 12.8, + "instruction_following": 43.4, + "overall": 29.5 }, "gemini-2.0-flash-001": { "reasoning": 55.5, @@ -1594,96 +1588,72 @@ "instruction_following": 78.6, "overall": 64.2 }, + "o3-mini-2025-01-31-low": { + "reasoning": 65.4, + "vision": 15.4, + "instruction_following": 72.0, + "overall": 64.7 + }, + "o1-mini-2024-09-12": { + "reasoning": 61.4, + "vision": 18.7, + "instruction_following": 59.1, + "overall": 57.4 + }, + "gemini-2.0-pro-exp-02-05": { + "reasoning": 59.7, + "vision": 47.7, + "instruction_following": 78.2, + "overall": 66.8 + }, "gemma-3-4b-it": { "reasoning": 29.6, "vision": 11.0, "instruction_following": 47.0, "overall": 35.7 }, - "azerogpt": { - "reasoning": 33.0, - "vision": 26.5, - "instruction_following": 45.2, - "overall": 37.7 - }, - "claude-3-opus-20240229": { - "reasoning": 43.5, - "vision": 45.7, - "instruction_following": 63.7, - "overall": 52.3 - }, "perplexity-sonar": { "reasoning": 45.5, "vision": 44.7, "instruction_following": 55.9, "overall": 49.9 }, - "grok-beta": { - "reasoning": 43.6, - "vision": 40.0, - "instruction_following": 61.7, - "overall": 51.1 - }, - "phi-3-mini-128k-instruct": { - "reasoning": 19.4, - "vision": 11.3, - "instruction_following": 29.5, - "overall": 23.2 - }, "olmo-2-1124-13b-instruct": { "reasoning": 21.4, "vision": 14.8, "instruction_following": 35.1, "overall": 26.8 }, - "gemini-2.0-flash-exp": { - "reasoning": 53.8, - "vision": 45.0, - "instruction_following": 71.3, - "overall": 60.7 - }, - "gemini-exp-1206": { - "reasoning": 60.7, - "vision": 58.2, - "instruction_following": 69.2, - "overall": 64.2 - }, - "gemini-2.0-flash-lite-preview-02-05": { - "reasoning": 49.8, - "vision": 33.5, - "instruction_following": 65.9, - "overall": 55.5 - }, - "chatgpt-4o-latest-2025-01-29": { - "reasoning": 50.4, - "vision": 47.3, - "instruction_following": 63.2, - "overall": 55.7 - }, - "qwq-32b-preview": { - "reasoning": 44.5, - "vision": 23.7, - "instruction_following": 29.7, - "overall": 36.7 - }, - "qwen2.5-coder-32b-instruct": { - "reasoning": 39.2, - "vision": 26.2, - "instruction_following": 49.6, - "overall": 42.7 - }, "gemini-2.0-flash": { "reasoning": 56.6, "vision": 47.1, "instruction_following": 76.2, "overall": 64.3 }, - "perplexity-sonar-pro": { - "reasoning": 52.9, - "vision": 73.5, - "instruction_following": 65.2, + "deepseek-v3": { + "reasoning": 53.6, + "vision": 47.6, + "instruction_following": 68.6, "overall": 59.6 }, + "gemini-2.0-flash-lite": { + "reasoning": 49.8, + "vision": 33.6, + "instruction_following": 67.4, + "overall": 56.2 + }, + "gpt-4.5-preview": { + "reasoning": 67.3, + "vision": 54.2, + "instruction_following": 68.4, + "overall": 66.8 + }, + "phi-3-mini-4k-instruct": { + "reasoning": 20.1, + "vision": 13.2, + "instruction_following": 26.3, + "overall": 22.2 + }, "gemma-2-27b-it": { "reasoning": 31.0, "vision": 30.0, @@ -1696,35 +1666,23 @@ "instruction_following": 68.4, "overall": 63.9 }, - "phi-3-small-8k-instruct": { - "reasoning": 22.8, - "vision": 15.9, - "instruction_following": 31.9, - "overall": 26.5 - }, "mistral-small-2501": { "reasoning": 36.3, "vision": 31.9, "instruction_following": 54.9, "overall": 44.0 }, - "gpt-4.5-preview": { - "reasoning": 67.3, - "vision": 54.2, - "instruction_following": 68.4, - "overall": 66.8 - }, - "claude-3-7-sonnet-thinking": { - "reasoning": 78.8, - "vision": 56.5, - "instruction_following": 75.6, - "overall": 75.8 + "phi-3-mini-128k-instruct": { + "reasoning": 19.4, + "vision": 11.3, + "instruction_following": 29.5, + "overall": 23.2 }, - "deepseek-v3": { - "reasoning": 53.6, - "vision": 47.6, - "instruction_following": 68.6, - "overall": 59.6 + "grok-beta": { + "reasoning": 43.6, + "vision": 40.0, + "instruction_following": 61.7, + "overall": 51.1 }, "gemma-2-9b-it": { "reasoning": 23.2, @@ -1732,65 +1690,101 @@ "instruction_following": 41.1, "overall": 30.8 }, - "gemini-2.0-flash-lite": { + "perplexity-sonar-pro": { + "reasoning": 52.9, + "vision": 73.5, + "instruction_following": 65.2, + "overall": 59.6 + }, + "phi-3-small-8k-instruct": { + "reasoning": 22.8, + "vision": 15.9, + "instruction_following": 31.9, + "overall": 26.5 + }, + "gemini-exp-1206": { + "reasoning": 60.7, + "vision": 58.2, + "instruction_following": 69.2, + "overall": 64.2 + }, + "qwq-32b-preview": { + "reasoning": 44.5, + "vision": 23.7, + "instruction_following": 29.7, + "overall": 36.7 + }, + "gemini-2.0-flash-lite-preview-02-05": { "reasoning": 49.8, - "vision": 33.6, - "instruction_following": 67.4, - "overall": 56.2 + "vision": 33.5, + "instruction_following": 65.9, + "overall": 55.5 }, - "phi-3-mini-4k-instruct": { - "reasoning": 20.1, - "vision": 13.2, - "instruction_following": 26.3, - "overall": 22.2 + "qwen2.5-coder-32b-instruct": { + "reasoning": 39.2, + "vision": 26.2, + "instruction_following": 49.6, + "overall": 42.7 }, - "gpt-4-turbo-2024-04-09": { - "reasoning": 47.2, - "vision": 39.6, - "instruction_following": 60.5, - "overall": 52.4 + "claude-3-7-sonnet-thinking": { + "reasoning": 78.8, + "vision": 56.5, + "instruction_following": 75.6, + "overall": 75.8 }, - "phi-3-medium-128k-instruct": { - "reasoning": 25.4, - "vision": 14.8, - "instruction_following": 39.2, - "overall": 30.6 + "gemini-2.0-flash-exp": { + "reasoning": 53.8, + "vision": 45.0, + "instruction_following": 71.3, + "overall": 60.7 }, - "chatgpt-4o-latest-0903": { - "reasoning": 49.7, - "vision": 43.2, - "instruction_following": 61.7, - "overall": 54.4 + "chatgpt-4o-latest-2025-01-29": { + "reasoning": 50.4, + "vision": 47.3, + "instruction_following": 63.2, + "overall": 55.7 }, - "gemini-1.5-flash-002": { - "reasoning": 48.4, - "vision": 31.9, - "instruction_following": 60.7, - "overall": 52.5 + "phi-3.5-moe-instruct": { + "reasoning": 31.8, + "vision": 14.2, + "instruction_following": 43.8, + "overall": 35.7 }, - "gemini-1.5-pro-002": { - "reasoning": 52.6, - "vision": 45.2, - "instruction_following": 64.9, - "overall": 57.3 + "gemini-1.5-flash-exp-0827": { + "reasoning": 39.4, + "vision": 34.6, + "instruction_following": 56.9, + "overall": 46.6 }, - "mistral-large-2407": { - "reasoning": 43.6, - "vision": 40.0, - "instruction_following": 58.3, - "overall": 49.7 + "grok-2-mini": { + "reasoning": 43.8, + "vision": 40.7, + "instruction_following": 55.6, + "overall": 48.6 }, - "o1-preview-2024-09-12": { - "reasoning": 67.1, - "vision": 53.5, - "instruction_following": 71.0, - "overall": 67.8 + "gpt-3.5-turbo-0125": { + "reasoning": 27.0, + "vision": 17.3, + "instruction_following": 50.1, + "overall": 36.2 + }, + "gemini-1.5-pro-exp-0827": { + "reasoning": 51.0, + "vision": 43.4, + "instruction_following": 64.4, + "overall": 56.2 + }, + "grok-2": { + "reasoning": 46.3, + "vision": 32.5, + "instruction_following": 59.2, + "overall": 50.8 }, - "gemini-1.5-flash-8b-exp-0827": { - "reasoning": 33.4, - "vision": 17.1, - "instruction_following": 49.4, - "overall": 39.1 + "gpt-4-turbo-2024-04-09": { + "reasoning": 47.2, + "vision": 39.6, + "instruction_following": 60.5, + "overall": 52.4 }, "gemini-1.5-pro-exp-0801": { "reasoning": 45.4, @@ -1798,41 +1792,35 @@ "instruction_following": 65.9, "overall": 53.9 }, - "claude-3-haiku-20240307": { - "reasoning": 31.5, - "vision": 23.7, - "instruction_following": 52.3, - "overall": 39.8 - }, - "gemini-exp-1114": { - "reasoning": 52.3, - "vision": 44.6, - "instruction_following": 65.2, - "overall": 57.3 + "deepseek-v2.5": { + "reasoning": 42.5, + "vision": 35.4, + "instruction_following": 56.9, + "overall": 48.2 }, - "claude-3-5-sonnet-20240620": { - "reasoning": 54.2, - "vision": 55.1, - "instruction_following": 64.9, - "overall": 58.9 + "mistral-large-2407": { + "reasoning": 43.6, + "vision": 40.0, + "instruction_following": 58.3, + "overall": 49.7 }, - "command-r-plus": { - "reasoning": 30.5, - "vision": 25.4, - "instruction_following": 45.5, - "overall": 36.6 + "gemini-1.5-flash-002": { + "reasoning": 48.4, + "vision": 31.9, + "instruction_following": 60.7, + "overall": 52.5 }, - "grok-2": { - "reasoning": 46.3, - "vision": 32.5, - "instruction_following": 59.2, - "overall": 50.8 + "deepseek-coder-v2": { + "reasoning": 44.2, + "vision": 31.3, + "instruction_following": 52.8, + "overall": 47.0 }, - "open-mistral-nemo": { - "reasoning": 23.8, - "vision": 16.6, - "instruction_following": 37.9, - "overall": 29.3 + "claude-3-sonnet-20240229": { + "reasoning": 32.5, + "vision": 34.1, + "instruction_following": 54.5, + "overall": 42.0 }, "phi-3-medium-4k-instruct": { "reasoning": 28.4, @@ -1840,29 +1828,17 @@ "instruction_following": 35.1, "overall": 30.3 }, - "phi-3.5-moe-instruct": { - "reasoning": 31.8, - "vision": 14.2, - "instruction_following": 43.8, - "overall": 35.7 - }, - "gemini-exp-1121": { - "reasoning": 54.0, - "vision": 43.2, - "instruction_following": 70.8, - "overall": 60.4 - }, - "gemini-1.5-pro-api-0514": { - "reasoning": 37.8, - "vision": 39.9, - "instruction_following": 59.7, - "overall": 47.3 + "mistral-small-2402": { + "reasoning": 26.4, + "vision": 26.5, + "instruction_following": 44.4, + "overall": 34.1 }, - "mixtral-8x22b-instruct-v0.1": { - "reasoning": 32.8, - "vision": 28.3, - "instruction_following": 43.8, - "overall": 37.2 + "gemini-exp-1114": { + "reasoning": 52.3, + "vision": 44.6, + "instruction_following": 65.2, + "overall": 57.3 }, "command-r": { "reasoning": 22.0, @@ -1870,17 +1846,35 @@ "instruction_following": 40.6, "overall": 29.4 }, + "o1-preview-2024-09-12": { + "reasoning": 67.1, + "vision": 53.5, + "instruction_following": 71.0, + "overall": 67.8 + }, + "claude-3-5-sonnet-20240620": { + "reasoning": 54.2, + "vision": 55.1, + "instruction_following": 64.9, + "overall": 58.9 + }, "phi-3.5-mini-instruct": { "reasoning": 25.8, "vision": 11.8, "instruction_following": 38.5, "overall": 30.3 }, - "qwen2.5-72b-instruct": { - "reasoning": 49.2, - "vision": 35.8, - "instruction_following": 61.0, - "overall": 53.3 + "open-mistral-nemo": { + "reasoning": 23.8, + "vision": 16.6, + "instruction_following": 37.9, + "overall": 29.3 + }, + "command-r-plus": { + "reasoning": 30.5, + "vision": 25.4, + "instruction_following": 45.5, + "overall": 36.6 }, "gemini-1.5-flash-api-0514": { "reasoning": 35.2, @@ -1888,53 +1882,47 @@ "instruction_following": 51.6, "overall": 41.8 }, - "gpt-4-0125-preview": { - "reasoning": 42.1, - "vision": 43.5, - "instruction_following": 57.4, - "overall": 48.8 - }, - "claude-3-sonnet-20240229": { - "reasoning": 32.5, - "vision": 34.1, - "instruction_following": 54.5, - "overall": 42.0 + "qwen2.5-72b-instruct": { + "reasoning": 49.2, + "vision": 35.8, + "instruction_following": 61.0, + "overall": 53.3 }, - "grok-2-mini": { - "reasoning": 43.8, - "vision": 40.7, - "instruction_following": 55.6, - "overall": 48.6 + "gemini-exp-1121": { + "reasoning": 54.0, + "vision": 43.2, + "instruction_following": 70.8, + "overall": 60.4 }, - "deepseek-v2.5": { - "reasoning": 42.5, - "vision": 35.4, - "instruction_following": 56.9, - "overall": 48.2 + "claude-3-haiku-20240307": { + "reasoning": 31.5, + "vision": 23.7, + "instruction_following": 52.3, + "overall": 39.8 }, - "gemini-1.5-pro-exp-0827": { - "reasoning": 51.0, - "vision": 43.4, - "instruction_following": 64.4, - "overall": 56.2 + "gpt-4o-2024-05-13": { + "reasoning": 48.7, + "vision": 42.8, + "instruction_following": 63.1, + "overall": 54.4 }, - "gpt-3.5-turbo-0125": { - "reasoning": 27.0, - "vision": 17.3, - "instruction_following": 50.1, - "overall": 36.2 + "chatgpt-4o-latest-0903": { + "reasoning": 49.7, + "vision": 43.2, + "instruction_following": 61.7, + "overall": 54.4 }, - "deepseek-coder-v2": { - "reasoning": 44.2, - "vision": 31.3, - "instruction_following": 52.8, - "overall": 47.0 + "gpt-4-0613": { + "reasoning": 42.1, + "vision": 45.5, + "instruction_following": 59.7, + "overall": 49.9 }, - "mistral-small-2402": { - "reasoning": 26.4, - "vision": 26.5, - "instruction_following": 44.4, - "overall": 34.1 + "gemini-1.5-pro-002": { + "reasoning": 52.6, + "vision": 45.2, + "instruction_following": 64.9, + "overall": 57.3 }, "phi-3-small-128k-instruct": { "reasoning": 28.1, @@ -1942,47 +1930,77 @@ "instruction_following": 37.8, "overall": 31.2 }, + "gemini-1.5-pro-api-0514": { + "reasoning": 37.8, + "vision": 39.9, + "instruction_following": 59.7, + "overall": 47.3 + }, + "gemini-1.5-flash-8b-exp-0827": { + "reasoning": 33.4, + "vision": 17.1, + "instruction_following": 49.4, + "overall": 39.1 + }, + "phi-3-medium-128k-instruct": { + "reasoning": 25.4, + "vision": 14.8, + "instruction_following": 39.2, + "overall": 30.6 + }, "llama-3.1-nemotron-70b-instruct": { "reasoning": 43.2, "vision": 38.7, "instruction_following": 50.4, "overall": 46.0 }, - "gpt-4-0613": { + "gpt-4-0125-preview": { "reasoning": 42.1, - "vision": 45.5, - "instruction_following": 59.7, - "overall": 49.9 + "vision": 43.5, + "instruction_following": 57.4, + "overall": 48.8 }, - "gpt-4o-2024-05-13": { - "reasoning": 48.7, - "vision": 42.8, - "instruction_following": 63.1, - "overall": 54.4 + "mixtral-8x22b-instruct-v0.1": { + "reasoning": 32.8, + "vision": 28.3, + "instruction_following": 43.8, + "overall": 37.2 }, - "gemini-1.5-flash-exp-0827": { - "reasoning": 39.4, - "vision": 34.6, - "instruction_following": 56.9, - "overall": 46.6 + "dracarys-llama-3.1-70b-instruct": { + "reasoning": 45.5, + "vision": 40.6, + "instruction_following": 62.1, + "overall": 52.3 + }, + "smaug-qwen2-72b-instruct": { + "reasoning": 40.8, + "vision": 30.1, + "instruction_following": 44.5, + "overall": 41.6 }, - "hermes-3-llama-3.1-70b": { - "reasoning": 34.8, - "vision": 37.3, - "instruction_following": 52.9, - "overall": 42.7 + "mathstral-7b-v0.1": { + "reasoning": 22.0, + "vision": 14.8, + "instruction_following": 38.3, + "overall": 28.4 }, - "openhermes-2.5-mistral-7b": { - "reasoning": 23.1, - "vision": 15.1, - "instruction_following": 36.1, - "overall": 28.1 + "deepseek-v2-lite-chat": { + "reasoning": 16.7, + "vision": 9.6, + "instruction_following": 27.5, + "overall": 20.8 }, - "mixtral-8x7b-instruct-v0.1": { - "reasoning": 22.3, - "vision": 16.3, - "instruction_following": 29.8, - "overall": 25.1 + "mistral-large-2402": { + "reasoning": 38.6, + "vision": 33.9, + "instruction_following": 51.5, + "overall": 43.8 + }, + "qwen1.5-72b-chat": { + "reasoning": 29.0, + "vision": 17.9, + "instruction_following": 39.3, + "overall": 32.6 }, "qwen2-72b-instruct": { "reasoning": 43.8, @@ -1990,17 +2008,29 @@ "instruction_following": 43.9, "overall": 43.1 }, + "mistral-7b-instruct-v0.2": { + "reasoning": 19.0, + "vision": 13.0, + "instruction_following": 30.8, + "overall": 23.6 + }, "yi-6b-chat": { "reasoning": 10.7, "vision": 10.1, "instruction_following": 17.5, "overall": 13.5 }, - "qwen1.5-0.5b-chat": { - "reasoning": 4.9, - "vision": 8.6, - "instruction_following": 12.0, - "overall": 8.2 + "gemma-1.1-7b-it": { + "reasoning": 19.8, + "vision": 12.8, + "instruction_following": 27.0, + "overall": 22.4 + }, + "meta-llama-3-70b-instruct": { + "reasoning": 36.5, + "vision": 31.2, + "instruction_following": 50.7, + "overall": 42.2 }, "vicuna-7b-v1.5": { "reasoning": 12.9, @@ -2008,35 +2038,29 @@ "instruction_following": 24.1, "overall": 17.6 }, - "starling-lm-7b-beta": { - "reasoning": 17.3, - "vision": 13.3, - "instruction_following": 21.9, - "overall": 19.0 - }, - "deepseek-v2-lite-chat": { - "reasoning": 16.7, - "vision": 9.6, - "instruction_following": 27.5, - "overall": 20.8 + "dracarys-72b-instruct": { + "reasoning": 41.7, + "vision": 32.5, + "instruction_following": 45.4, + "overall": 42.6 }, - "qwen1.5-1.8b-chat": { - "reasoning": 6.0, - "vision": 9.5, - "instruction_following": 12.3, - "overall": 9.0 + "qwen1.5-4b-chat": { + "reasoning": 12.0, + "vision": 9.4, + "instruction_following": 17.6, + "overall": 14.2 }, - "gemma-1.1-7b-it": { - "reasoning": 19.8, - "vision": 12.8, - "instruction_following": 27.0, - "overall": 22.4 + "openhermes-2.5-mistral-7b": { + "reasoning": 23.1, + "vision": 15.1, + "instruction_following": 36.1, + "overall": 28.1 }, - "qwen1.5-72b-chat": { - "reasoning": 29.0, - "vision": 17.9, - "instruction_following": 39.3, - "overall": 32.6 + "zephyr-7b-beta": { + "reasoning": 15.0, + "vision": 10.9, + "instruction_following": 26.8, + "overall": 19.8 }, "vicuna-7b-v1.5-16k": { "reasoning": 13.9, @@ -2044,41 +2068,17 @@ "instruction_following": 26.2, "overall": 18.9 }, - "chatgpt-4o-latest": { - "reasoning": 52.3, - "vision": 42.5, - "instruction_following": 63.2, - "overall": 56.3 - }, - "qwen1.5-110b-chat": { - "reasoning": 30.6, - "vision": 15.3, - "instruction_following": 36.4, - "overall": 32.0 - }, - "meta-llama-3-8b-instruct": { - "reasoning": 26.8, - "vision": 15.8, - "instruction_following": 37.7, - "overall": 30.7 - }, - "qwen1.5-4b-chat": { - "reasoning": 12.0, - "vision": 9.4, - "instruction_following": 17.6, - "overall": 14.2 - }, - "deepseek-coder-v2-lite-instruct": { - "reasoning": 28.9, - "vision": 11.4, - "instruction_following": 38.4, - "overall": 31.7 + "zephyr-7b-alpha": { + "reasoning": 14.8, + "vision": 11.6, + "instruction_following": 30.9, + "overall": 21.5 }, - "mistral-7b-instruct-v0.3": { - "reasoning": 20.3, - "vision": 15.4, - "instruction_following": 32.1, - "overall": 25.0 + "qwen1.5-0.5b-chat": { + "reasoning": 4.9, + "vision": 8.6, + "instruction_following": 12.0, + "overall": 8.2 }, "qwen2-0.5b-instruct": { "reasoning": 9.1, @@ -2086,41 +2086,23 @@ "instruction_following": 13.9, "overall": 11.1 }, - "smaug-qwen2-72b-instruct": { - "reasoning": 40.8, - "vision": 30.1, - "instruction_following": 44.5, - "overall": 41.6 - }, - "dracarys-llama-3.1-70b-instruct": { - "reasoning": 45.5, - "vision": 40.6, - "instruction_following": 62.1, - "overall": 52.3 - }, - "llama-2-7b-chat-hf": { - "reasoning": 12.9, - "vision": 10.6, - "instruction_following": 24.9, - "overall": 17.9 - }, - "dracarys-72b-instruct": { - "reasoning": 41.7, - "vision": 32.5, - "instruction_following": 45.4, - "overall": 42.6 + "mistral-7b-instruct-v0.3": { + "reasoning": 20.3, + "vision": 15.4, + "instruction_following": 32.1, + "overall": 25.0 }, - "zephyr-7b-alpha": { - "reasoning": 14.8, - "vision": 11.6, - "instruction_following": 30.9, - "overall": 21.5 + "qwen1.5-1.8b-chat": { + "reasoning": 6.0, + "vision": 9.5, + "instruction_following": 12.3, + "overall": 9.0 }, - "mathstral-7b-v0.1": { - "reasoning": 22.0, - "vision": 14.8, - "instruction_following": 38.3, - "overall": 28.4 + "mixtral-8x7b-instruct-v0.1": { + "reasoning": 22.3, + "vision": 16.3, + "instruction_following": 29.8, + "overall": 25.1 }, "qwen2-7b-instruct": { "reasoning": 24.9, @@ -2128,17 +2110,35 @@ "instruction_following": 30.5, "overall": 26.6 }, + "chatgpt-4o-latest": { + "reasoning": 52.3, + "vision": 42.5, + "instruction_following": 63.2, + "overall": 56.3 + }, + "starling-lm-7b-beta": { + "reasoning": 17.3, + "vision": 13.3, + "instruction_following": 21.9, + "overall": 19.0 + }, + "qwen1.5-110b-chat": { + "reasoning": 30.6, + "vision": 15.3, + "instruction_following": 36.4, + "overall": 32.0 + }, "qwen1.5-7b-chat": { "reasoning": 18.7, "vision": 11.0, "instruction_following": 25.3, "overall": 21.0 }, - "meta-llama-3-70b-instruct": { - "reasoning": 36.5, - "vision": 31.2, - "instruction_following": 50.7, - "overall": 42.2 + "llama-2-7b-chat-hf": { + "reasoning": 12.9, + "vision": 10.6, + "instruction_following": 24.9, + "overall": 17.9 }, "qwen2-1.5b-instruct": { "reasoning": 9.7, @@ -2146,23 +2146,23 @@ "instruction_following": 16.6, "overall": 12.6 }, - "mistral-large-2402": { - "reasoning": 38.6, - "vision": 33.9, - "instruction_following": 51.5, - "overall": 43.8 + "hermes-3-llama-3.1-70b": { + "reasoning": 34.8, + "vision": 37.3, + "instruction_following": 52.9, + "overall": 42.7 }, - "mistral-7b-instruct-v0.2": { - "reasoning": 19.0, - "vision": 13.0, - "instruction_following": 30.8, - "overall": 23.6 + "deepseek-coder-v2-lite-instruct": { + "reasoning": 28.9, + "vision": 11.4, + "instruction_following": 38.4, + "overall": 31.7 }, - "zephyr-7b-beta": { - "reasoning": 15.0, - "vision": 10.9, - "instruction_following": 26.8, - "overall": 19.8 + "meta-llama-3-8b-instruct": { + "reasoning": 26.8, + "vision": 15.8, + "instruction_following": 37.7, + "overall": 30.7 } } } \ No newline at end of file diff --git a/data/zen_models.json b/data/zen_models.json index da3e87d..a0855b5 100644 --- a/data/zen_models.json +++ b/data/zen_models.json @@ -3,7 +3,7 @@ { "id": "claude-fable-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$10.00", @@ -16,7 +16,7 @@ { "id": "claude-opus-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$5.00", @@ -29,7 +29,7 @@ { "id": "claude-opus-4-8", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$5.00", @@ -42,7 +42,7 @@ { "id": "claude-opus-4-7", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$5.00", @@ -55,7 +55,7 @@ { "id": "claude-opus-4-6", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$5.00", @@ -68,7 +68,7 @@ { "id": "claude-opus-4-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$5.00", @@ -81,13 +81,13 @@ { "id": "claude-opus-4-1", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "claude-sonnet-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$2.00", @@ -100,7 +100,7 @@ { "id": "claude-sonnet-4-6", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$3.00", @@ -113,7 +113,7 @@ { "id": "claude-sonnet-4-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$3.00", @@ -143,13 +143,13 @@ { "id": "claude-sonnet-4", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode" }, { "id": "claude-haiku-4-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.00", @@ -162,7 +162,7 @@ { "id": "gemini-3.6-flash", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.50", @@ -175,7 +175,7 @@ { "id": "gemini-3.5-flash-lite", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.30", @@ -188,7 +188,7 @@ { "id": "gemini-3.5-flash", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.50", @@ -201,7 +201,7 @@ { "id": "gemini-3.1-pro", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$2.00", @@ -231,7 +231,7 @@ { "id": "gemini-3-flash", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.50", @@ -244,7 +244,7 @@ { "id": "gpt-5.6-sol", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$5.00", @@ -274,7 +274,7 @@ { "id": "gpt-5.6-terra", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$2.00", @@ -304,7 +304,7 @@ { "id": "gpt-5.6-luna", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.20", @@ -334,7 +334,7 @@ { "id": "gpt-5.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$5.00", @@ -364,7 +364,7 @@ { "id": "gpt-5.5-pro", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$30.00", @@ -377,7 +377,7 @@ { "id": "gpt-5.4", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$2.50", @@ -407,7 +407,7 @@ { "id": "gpt-5.4-pro", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$30.00", @@ -420,7 +420,7 @@ { "id": "gpt-5.4-mini", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.75", @@ -433,7 +433,7 @@ { "id": "gpt-5.4-nano", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.20", @@ -446,7 +446,7 @@ { "id": "gpt-5.3-codex-spark", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.75", @@ -459,7 +459,7 @@ { "id": "gpt-5.3-codex", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.75", @@ -472,7 +472,7 @@ { "id": "gpt-5.2", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.75", @@ -485,7 +485,7 @@ { "id": "gpt-5.2-codex", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.75", @@ -498,7 +498,7 @@ { "id": "gpt-5.1", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.07", @@ -511,7 +511,7 @@ { "id": "gpt-5.1-codex-max", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.25", @@ -524,7 +524,7 @@ { "id": "gpt-5.1-codex", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.07", @@ -537,7 +537,7 @@ { "id": "gpt-5.1-codex-mini", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.25", @@ -550,7 +550,7 @@ { "id": "gpt-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.07", @@ -563,7 +563,7 @@ { "id": "gpt-5-codex", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.07", @@ -576,7 +576,7 @@ { "id": "gpt-5-nano", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.05", @@ -589,7 +589,7 @@ { "id": "grok-build-0.1", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.00", @@ -602,7 +602,7 @@ { "id": "grok-4.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$2.00", @@ -632,7 +632,7 @@ { "id": "deepseek-v4-pro", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.74", @@ -645,7 +645,7 @@ { "id": "deepseek-v4-flash", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.14", @@ -658,7 +658,7 @@ { "id": "glm-5.2", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.40", @@ -671,7 +671,7 @@ { "id": "glm-5.1", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.40", @@ -684,7 +684,7 @@ { "id": "glm-5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$1.00", @@ -697,7 +697,7 @@ { "id": "minimax-m3", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.30", @@ -710,7 +710,7 @@ { "id": "minimax-m2.7", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.30", @@ -723,7 +723,7 @@ { "id": "minimax-m2.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.30", @@ -736,7 +736,7 @@ { "id": "kimi-k3", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$3.00", @@ -749,7 +749,7 @@ { "id": "kimi-k2.7-code", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.95", @@ -762,7 +762,7 @@ { "id": "kimi-k2.6", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.95", @@ -775,7 +775,7 @@ { "id": "kimi-k2.5", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.60", @@ -788,7 +788,7 @@ { "id": "qwen3.6-plus", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.50", @@ -801,7 +801,7 @@ { "id": "qwen3.5-plus", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "$0.20", @@ -814,7 +814,7 @@ { "id": "big-pickle", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -827,7 +827,7 @@ { "id": "deepseek-v4-flash-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -840,7 +840,7 @@ { "id": "mimo-v2.5-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -853,7 +853,7 @@ { "id": "ling-3.0-flash-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -866,7 +866,7 @@ { "id": "nemotron-3-ultra-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -879,7 +879,7 @@ { "id": "north-mini-code-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -892,7 +892,7 @@ { "id": "laguna-s-2.1-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -907,7 +907,7 @@ { "id": "big-pickle", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -920,7 +920,7 @@ { "id": "deepseek-v4-flash-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -933,7 +933,7 @@ { "id": "mimo-v2.5-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -946,7 +946,7 @@ { "id": "ling-3.0-flash-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -959,7 +959,7 @@ { "id": "nemotron-3-ultra-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -972,7 +972,7 @@ { "id": "north-mini-code-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free", @@ -985,7 +985,7 @@ { "id": "laguna-s-2.1-free", "object": "model", - "created": 1785794599, + "created": 1785794768, "owned_by": "opencode", "pricing": { "input": "Free",