[syntax-error-quality] 🔍 Compiler Error Message Quality: Weak diagnostics for invalid tool configuration

[github-actions]

📊 Error Message Quality Analysis

Date: 2026-07-04 | Tests: 2 | Average Score: 58/100 | Status: ⚠️ Needs Improvement

Summary: Category A (YAML syntax) errors produce acceptable messages with correct location and code context. Category B (configuration errors) messages are critically weak — they highlight the wrong line, use cryptic internal terminology, and provide no actionable guidance.

Test	Workflow	Error Type	Score	Rating
1	smoke-crush.md	YAML syntax (missing :)	72/100	Good
2	copilot-pr-analysis-base.md	Invalid tool config (string vs object)	44/100	❌ Critical

Weaknesses (top 3):

1. Wrong line highlighted (Test 2): The caret points to the innocent child key toolsets: (L4) instead of the actual bad value github: "invalid-string" (L3). Developers will look at the wrong place first.
2. Cryptic message text (Test 2): "value is not allowed in this context. map key-value is pre-defined" is internal YAML-parser language. A developer-facing message should say something like: "expected an object for github tool config, got a string — use github: toolsets:[default] format".
3. Wrong action hint (Test 1): "Correct the event or filter name" is copy-pasted from an event-validation template; it does not apply to a YAML missing-colon error. Hints should be contextual to the specific error category.

Detailed scoring per dimension

Test 1 — smoke-crush.md — YAML syntax error (id crush missing colon)

Dimension	Score	Notes
Clarity	22/25	Message is clear; exact file:line:col shown
Actionability	14/25	Hint is wrong category ("event or filter name")
Context	18/20	Multi-line code snippet with caret
Examples	5/15	No correct usage example shown
Consistency	13/15	Standard parseable format
Total	72/100	Good

Test 2 — copilot-pr-analysis-base.md — Invalid tool config (string instead of object)

Error introduced at L3: github: "invalid-string" #

Dimension	Score	Notes
Clarity	10/25	"map key-value is pre-defined" is not developer-readable
Actionability	8/25	"Fix this configuration issue" gives zero direction
Context	11/20	Caret on wrong line (L4 toolsets: instead of L3 github:)
Examples	3/15	No valid tool-config example shown
Consistency	12/15	Follows format, but message and hint are generic stubs
Total	44/100	❌ Critical

Recommended Fixes

1. Fix caret position for invalid-value errors: When a YAML key has an invalid scalar value, report the error on the value line, not on subsequent sibling lines.

2. Humanise error messages for known schema violations: For tools.* fields, the compiler knows the expected shape. Replace "map key-value is pre-defined" with a message like:

   test-2.md:3:11: error: github tool config must be an object, not a string
   → expected:
       github:
         toolsets: [default]
   

3. Make action hints context-specific: Avoid reusing hint templates across unrelated error categories. A YAML-parse error and a schema-validation error need different follow-up guidance.

Generated by 📊 Daily Syntax Error Quality Check · 19.5 AIC · ⌖ 7.48 AIC · ⊞ 6.7K · ◷

• expires on Jul 7, 2026, 12:20 AM UTC-08:00