Skip to content

Add qwen3.8-max benchmark scores to model-scores.yaml #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dianlight merged 2 commits into from
Aug 3, 2026
Merged

Add qwen3.8-max benchmark scores to model-scores.yaml #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a3ba076
Add qwen3.8-max benchmark scores to model-scores.yaml
github-actions[bot] Aug 3, 2026
85b0859
Added qwen3.8-max scores; PR #24, issue closed.
github-actions[bot] Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file modified .github/scripts/resolve-model.sh
100644 → 100755
View file
Open in desktop
Empty file.
40 changes: 20 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,24 @@ Main repository form my Opencode Github Actions to share to multiple repository
## Model Recommendations by Task Type

> Automatically updated by `opencode-maintenance` workflow.
> Last updated: **2026-08-03 22:03 UTC**.
> Last updated: **2026-08-03 22:06 UTC**.
> LiveBench data: **325 models scored**.
> LiveBench snapshot: **2026_06_25**.
> Source: https://livebench.ai/table_2026_06_25.csv
> Free-first threshold: **5%**.

| Task Type | Description | Best Zen | Best Free | Best Go |
|-----------|-------------|----------|-----------|---------|
| `Plan` | Planning, architecture decisions, task decomposition | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/kimi-k3` (87.0) ⚠️ x2.0 · alt: `opencode-go/grok-4.5` (86.9) |
| `Ask` | General Q&A, explanations, analysis | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | 🏆 `opencode-go/qwen3.7-max` (76.6) |
| `Code` | Code generation, implementation, refactoring | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/big-pickle` (67.0) |
| `issue-triage` | Triage, label, categorize, route issues | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | 🏆 `opencode-go/qwen3.7-max` (76.6) |
| `issue-implementation` | Implement, fix, resolve issues | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/big-pickle` (67.0) |
| `pr-review` | Review PRs, pull requests, diffs | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/kimi-k3` (87.0) ⚠️ x2.0 · alt: `opencode-go/grok-4.5` (86.9) |
| `code-implementation` | Generate code, refactor, implement features | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/big-pickle` (67.0) |
| `frontend-design` | UI design, components, layouts, mockups | `opencode/gpt-5.6-sol` (79.1) | `opencode/mimo-v2.5-free` (54.0) | 🏆 `opencode-go/kimi-k3` (72.6) ⚠️ x2.0 · alt: `opencode-go/grok-4.5` (66.4) |
| `frontend-testing` | Playwright, Cypress, E2E, frontend tests | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/big-pickle` (67.0) |
| `api-testing` | API testing, integration tests, OpenAPI, Postman | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/big-pickle` (67.0) |
| `Plan` | Planning, architecture decisions, task decomposition | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/qwen3.8-max` (95.5) |
| `Ask` | General Q&A, explanations, analysis | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | 🏆 `opencode-go/qwen3.8-max` (93.9) |
| `Code` | Code generation, implementation, refactoring | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0) |
| `issue-triage` | Triage, label, categorize, route issues | `opencode/gpt-5.5-pro` (81.6) | `opencode/deepseek-v4-flash-free` (69.4) | 🏆 `opencode-go/qwen3.8-max` (93.9) |
| `issue-implementation` | Implement, fix, resolve issues | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0) |
| `pr-review` | Review PRs, pull requests, diffs | `opencode/gpt-5-codex` (96.1) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/qwen3.8-max` (95.5) |
| `code-implementation` | Generate code, refactor, implement features | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0) |
| `frontend-design` | UI design, components, layouts, mockups | `opencode/gpt-5.6-sol` (79.1) | `opencode/mimo-v2.5-free` (54.0) | 🏆 `opencode-go/qwen3.8-max` (88.1) |
| `frontend-testing` | Playwright, Cypress, E2E, frontend tests | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0) |
| `api-testing` | API testing, integration tests, OpenAPI, Postman | `opencode/claude-fable-5` (71.7) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0) |
| `other` | Everything else | `opencode/claude-fable-5` (83.4) | `opencode/deepseek-v4-flash-free` (66.1) | 🏆 `opencode-go/kimi-k3` (79.5) ⚠️ x2.0 · alt: `opencode-go/grok-4.5` (77.1) |

### LiveBench Score Reference
Expand Down Expand Up @@ -64,22 +64,22 @@ Main repository form my Opencode Github Actions to share to multiple repository
| `qwen3.6-plus` | Go (Paid) | ✅ LiveBench | Plan, Review | — | 69.0 | 56.1 | 77.0 | 52.5 | 67.7 |
| `qwen3.7-max` | Go (Paid) | ✅ LiveBench | Plan, Review | — | 74.1 | 55.8 | 82.4 | 58.7 | 76.6 |
| `qwen3.7-plus` | Go (Paid) | 📋 Fallback | Plan, Ask | — | 66.0 | 62.0 | 72.0 | 62.0 | 72.0 |
| `qwen3.8-max` | Go (Paid) | ❌ Missing | — | — | — | — | — | — | — |
| `qwen3.8-max` | Go (Paid) | 📋 Fallback | Plan, Review | — | 78.2 | 77.0 | 95.5 | 88.1 | 93.9 |

## Workflow Model Audit

> Audited: **2026-08-03 22:03 UTC**
> Audited: **2026-08-03 22:06 UTC**
> Workflows checked: **4**
> OpenCode steps found: **7**

| Workflow | Job | Step | Task Type | Current Model | Recommended Zen | Recommended Free | Recommended Go | Status |
|----------|-----|------|-----------|---------------|-----------------|------------------|----------------|--------|
| `opencode-issue-handler` | `process-4` | `Run opencode (Process 4 — Issue Review & Refinement)` | `issue-triage` | `opencode-go/qwen3.7-max` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5.5-pro` (81.6 (+7%)) | `opencode/deepseek-v4-flash-free` (69.4) | 🏆 `opencode-go/qwen3.7-max` (76.6 (+10%)) | ✅ |
| `opencode-issue-handler` | `process-5` | `Run opencode (Process 5 — Issue Work & PR Creation)` | `issue-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) ⚙️ | `opencode/claude-fable-5` (71.7 (+14%)) | 🏆 `opencode/big-pickle` (67.0) | `opencode-go/big-pickle` (67.0) | ❗ |
| `OpenCode Maintenance` | `Handle Checked Tasks` | `Run OpenCode for checked tasks` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) ⚙️ | `opencode/claude-fable-5` (71.7 (+14%)) | 🏆 `opencode/big-pickle` (67.0) | `opencode-go/big-pickle` (67.0) | ❗ |
| `opencode-pr-comment` | `process-2` | `Run opencode (Process 2 — Bot thread reply)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/kimi-k3` (87.0 (+19%)) ⚠️ x2.0 · alt: `opencode-go/grok-4.5` (86.9 (+19%)) | ✅ |
| `opencode-pr-comment` | `process-3` | `Run opencode (Process 3 — User-owned thread takeover)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/kimi-k3` (87.0 (+19%)) ⚠️ x2.0 · alt: `opencode-go/grok-4.5` (86.9 (+19%)) | ✅ |
| `opencode-pr-comment` | `process-6` | `Run opencode (Process 6 — PR Task Execution)` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) ⚙️ | `opencode/claude-fable-5` (71.7 (+14%)) | 🏆 `opencode/big-pickle` (67.0) | `opencode-go/big-pickle` (67.0) | ❗ |
| `opencode-pr-review` | `review` | `Run opencode (PR code review)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/kimi-k3` (87.0 (+19%)) ⚠️ x2.0 · alt: `opencode-go/grok-4.5` (86.9 (+19%)) | ✅ |
| `opencode-issue-handler` | `process-4` | `Run opencode (Process 4 — Issue Review & Refinement)` | `issue-triage` | `opencode-go/qwen3.7-max` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5.5-pro` (81.6 (+7%)) | `opencode/deepseek-v4-flash-free` (69.4) | 🏆 `opencode-go/qwen3.8-max` (93.9 (+35%)) | ❌ |
| `opencode-issue-handler` | `process-5` | `Run opencode (Process 5 — Issue Work & PR Creation)` | `issue-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) ⚙️ | `opencode/claude-fable-5` (71.7 (+14%)) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0 (+15%)) | ❌ |
| `OpenCode Maintenance` | `Handle Checked Tasks` | `Run OpenCode for checked tasks` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) ⚙️ | `opencode/claude-fable-5` (71.7 (+14%)) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0 (+15%)) | ❌ |
| `opencode-pr-comment` | `process-2` | `Run opencode (Process 2 — Bot thread reply)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/qwen3.8-max` (95.5 (+31%)) | ❌ |
| `opencode-pr-comment` | `process-3` | `Run opencode (Process 3 — User-owned thread takeover)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/qwen3.8-max` (95.5 (+31%)) | ❌ |
| `opencode-pr-comment` | `process-6` | `Run opencode (Process 6 — PR Task Execution)` | `code-implementation` | `opencode-go/glm-5.1` (`/ocf`: `opencode/nemotron-3-ultra-free`) ⚙️ | `opencode/claude-fable-5` (71.7 (+14%)) | `opencode/big-pickle` (67.0) | 🏆 `opencode-go/qwen3.8-max` (77.0 (+15%)) | ❌ |
| `opencode-pr-review` | `review` | `Run opencode (PR code review)` | `pr-review` | `opencode-go/grok-4.5` (`/ocf`: `opencode/deepseek-v4-flash-free`) ⚙️ | `opencode/gpt-5-codex` (96.1 (+11%)) | `opencode/deepseek-v4-flash-free` (73.1) | 🏆 `opencode-go/qwen3.8-max` (95.5 (+31%)) | ❌ |

_Legend: ✅ Optimal · ⚠️ Warn (free, not best) · ❗ Alert (paid when free is preferred) · ❌ Error (wrong model) · 💀 Fatal (model not set). 🏆 marks the preferred model after free-first policy (free within 5% of best Go → prefer free). ⚠️ xN marks models with elevated token consumption. ⚙️ marks steps resolved at runtime from the central config (`data/model-config.json`). Recommended Zen shows best Zen model with score difference vs current model (e.g., `model (+15%)`)._
6 changes: 6 additions & 0 deletions config/model-scores.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,12 @@ model_scores:
overall: 66.0
reasoning: 72.0
vision: 62.0
qwen3.8-max:
coding: 77.0 # BenchLM weighted coding category (#6/130, 96th pct)
instruction_following: 93.9 # BenchLM IFBench weighted (#2/33, 97th pct)
overall: 78.2 # BenchLM overall score (#6/215)
reasoning: 95.5 # BenchLM reasoning category (#1/2, 100th pct)
vision: 88.1 # BenchLM multimodal category (#2/34, 97th pct)
laguna-s-2.1-free:
coding: 52.0
instruction_following: 54.0
Expand Down
64 changes: 32 additions & 32 deletions data/audit_results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"timestamp": "2026-08-03T22:03:20.211349+00:00",
"timestamp": "2026-08-03T22:06:09.009286+00:00",
"livebench_snapshot": "2026_06_25",
"livebench_source": "https://livebench.ai/table_2026_06_25.csv",
"livebench_models": 325,
Expand All @@ -9,7 +9,7 @@
"steps_audited": 7,
"model_config_drift": {
"detected": true,
"timestamp": "2026-08-03T22:03:20.211358+00:00",
"timestamp": "2026-08-03T22:06:09.009300+00:00",
"livebench_snapshot": "2026_06_25",
"current_workflows": {
"opencode-issue-handler": {
Expand Down Expand Up @@ -61,14 +61,14 @@
"current_model_free": "opencode/deepseek-v4-flash-free",
"auto": true,
"recommended_free": "deepseek-v4-flash-free",
"recommended_go": "qwen3.7-max",
"recommended_go": "qwen3.8-max",
"recommended_free_alt": null,
"recommended_go_alt": null,
"recommended_free_multiplier": 1.0,
"recommended_go_multiplier": 1.0,
"preferred_tier": "go",
"preferred_diff": 10,
"status": "\u2705"
"preferred_diff": 35,
"status": "\u274c"
},
{
"file": ".github/workflows/opencode-issue-handler.yml",
Expand All @@ -80,14 +80,14 @@
"current_model_free": "opencode/nemotron-3-ultra-free",
"auto": true,
"recommended_free": "big-pickle",
"recommended_go": "big-pickle",
"recommended_go": "qwen3.8-max",
"recommended_free_alt": null,
"recommended_go_alt": null,
"recommended_free_multiplier": 1.0,
"recommended_go_multiplier": 1.0,
"preferred_tier": "free",
"preferred_diff": null,
"status": "\u2757"
"preferred_tier": "go",
"preferred_diff": 15,
"status": "\u274c"
},
{
"file": ".github/workflows/opencode-maintenance.yaml",
Expand All @@ -99,14 +99,14 @@
"current_model_free": "opencode/nemotron-3-ultra-free",
"auto": true,
"recommended_free": "big-pickle",
"recommended_go": "big-pickle",
"recommended_go": "qwen3.8-max",
"recommended_free_alt": null,
"recommended_go_alt": null,
"recommended_free_multiplier": 1.0,
"recommended_go_multiplier": 1.0,
"preferred_tier": "free",
"preferred_diff": null,
"status": "\u2757"
"preferred_tier": "go",
"preferred_diff": 15,
"status": "\u274c"
},
{
"file": ".github/workflows/opencode-pr-comment.yml",
Expand All @@ -118,14 +118,14 @@
"current_model_free": "opencode/deepseek-v4-flash-free",
"auto": true,
"recommended_free": "deepseek-v4-flash-free",
"recommended_go": "kimi-k3",
"recommended_go": "qwen3.8-max",
"recommended_free_alt": null,
"recommended_go_alt": "grok-4.5",
"recommended_go_alt": null,
"recommended_free_multiplier": 1.0,
"recommended_go_multiplier": 2.0,
"recommended_go_multiplier": 1.0,
"preferred_tier": "go",
"preferred_diff": 19,
"status": "\u2705"
"preferred_diff": 31,
"status": "\u274c"
},
{
"file": ".github/workflows/opencode-pr-comment.yml",
Expand All @@ -137,14 +137,14 @@
"current_model_free": "opencode/deepseek-v4-flash-free",
"auto": true,
"recommended_free": "deepseek-v4-flash-free",
"recommended_go": "kimi-k3",
"recommended_go": "qwen3.8-max",
"recommended_free_alt": null,
"recommended_go_alt": "grok-4.5",
"recommended_go_alt": null,
"recommended_free_multiplier": 1.0,
"recommended_go_multiplier": 2.0,
"recommended_go_multiplier": 1.0,
"preferred_tier": "go",
"preferred_diff": 19,
"status": "\u2705"
"preferred_diff": 31,
"status": "\u274c"
},
{
"file": ".github/workflows/opencode-pr-comment.yml",
Expand All @@ -156,14 +156,14 @@
"current_model_free": "opencode/nemotron-3-ultra-free",
"auto": true,
"recommended_free": "big-pickle",
"recommended_go": "big-pickle",
"recommended_go": "qwen3.8-max",
"recommended_free_alt": null,
"recommended_go_alt": null,
"recommended_free_multiplier": 1.0,
"recommended_go_multiplier": 1.0,
"preferred_tier": "free",
"preferred_diff": null,
"status": "\u2757"
"preferred_tier": "go",
"preferred_diff": 15,
"status": "\u274c"
},
{
"file": ".github/workflows/opencode-pr-review.yml",
Expand All @@ -175,14 +175,14 @@
"current_model_free": "opencode/deepseek-v4-flash-free",
"auto": true,
"recommended_free": "deepseek-v4-flash-free",
"recommended_go": "kimi-k3",
"recommended_go": "qwen3.8-max",
"recommended_free_alt": null,
"recommended_go_alt": "grok-4.5",
"recommended_go_alt": null,
"recommended_free_multiplier": 1.0,
"recommended_go_multiplier": 2.0,
"recommended_go_multiplier": 1.0,
"preferred_tier": "go",
"preferred_diff": 19,
"status": "\u2705"
"preferred_diff": 31,
"status": "\u274c"
}
]
}
Loading