📊 Error Message Quality Analysis
Date: 2026-04-22 | Tests: 2 | Average Score: 62.5/100 | Status: ⚠️ Needs Improvement
Summary: YAML parse errors lack examples and docs links; invalid-engine errors provide excellent fix suggestions but are missing all location context (file, line, column, source snippet). Both error types score below the 70-point threshold.
| Test |
Workflow |
Error Type |
Score |
Rating |
| 1 |
sub-issue-closer.md |
Category A — engine copilot (missing colon) |
59/100 |
Acceptable |
| 2 |
deep-report.md |
Category B — engine: claaude (typo) |
66/100 |
Acceptable |
Test 1 details — YAML syntax error (missing colon)
Introduced error (line 10): engine copilot (missing :)
Simulated output (based on FormatYAMLError + translateYAMLError source analysis):
failed to parse frontmatter:
[10:8] missing ':' after key — YAML mapping entries require 'key: value' format
9 | issues: read
> 10 | engine copilot
^
11 | strict: true
Dimension scores:
- Clarity 17/25 — message text is good;
[10:8] bracket format is non-standard
- Actionability 14/25 — names the problem but gives no "fix to:" example or docs link
- Context 16/20 — source code and line numbers shown; no file path in the error itself
- Examples 3/15 — no correct syntax shown, no documentation link
- Consistency 9/15 —
[line:col] format diverges from schema-error file:line:col: error: format
Test 2 details — invalid engine name (typo)
Introduced error (line 19): engine: claaude (should be claude)
Simulated output (based on engine_definition.go Resolve() error path):
invalid engine: claaude. Valid engines are: claude, codex, copilot, crush, gemini, opencode.
Did you mean: claude?
Example:
engine: copilot
See: https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/engines.md
Dimension scores:
- Clarity 20/25 — immediately obvious what's wrong; lists valid options
- Actionability 23/25 — "Did you mean?", concrete example, documentation link; excellent
- Context 3/20 — zero location info (no file path, line number, column, or source snippet)
- Examples 13/15 — clear example + docs link; -2 for not showing the typo corrected inline
- Consistency 7/15 — plain text with newlines; no file-anchored prefix; inconsistent with schema errors
⚠️ Weaknesses (top 3)
-
Invalid-engine error has no location context (pkg/workflow/engine_definition.go:267): The error message has great actionability but no file path, line number, or source snippet. A developer editing multiple workflows can't navigate to the exact offending line without searching.
- Suggested fix: Wrap the engine resolution error in a
console.CompilerError struct (same as schema errors) so it renders as test-2.md:19:9: error: invalid engine 'claaude' ... with a highlighted source line.
-
YAML parse error has no example or docs link (pkg/parser/yaml_error.go:TranslateYAMLMessage): The translation missing ':' after key — YAML mapping entries require 'key: value' format names the fix but never shows corrected syntax. Append a concrete example (engine: copilot) to each YAML translation entry, or add a docs URL (similar to the engine error).
- Suggested fix: Add an optional
example field to yamlErrorTranslations entries and append it when non-empty, similar to DocsEnginesURL appended in engine errors.
-
[line:col] format vs file:line:col: inconsistency: YAML errors use the non-standard [line:col] bracket notation while schema errors use IDE-parseable file:line:col: error: format. This prevents editor jump-to-error and makes errors look inconsistent.
- Suggested fix: In
FormatYAMLError, after formatting the error, convert it into a console.CompilerError with the file path (passed down from the caller), so both YAML and schema errors share the same rendered format.
References
Generated by Daily Syntax Error Quality Check · ● 9.1M · ◷
📊 Error Message Quality Analysis
Date: 2026-04-22 | Tests: 2 | Average Score: 62.5/100 | Status:⚠️ Needs Improvement
Summary: YAML parse errors lack examples and docs links; invalid-engine errors provide excellent fix suggestions but are missing all location context (file, line, column, source snippet). Both error types score below the 70-point threshold.
sub-issue-closer.mdengine copilot(missing colon)deep-report.mdengine: claaude(typo)Test 1 details — YAML syntax error (missing colon)
Introduced error (line 10):
engine copilot(missing:)Simulated output (based on
FormatYAMLError+translateYAMLErrorsource analysis):Dimension scores:
[10:8]bracket format is non-standard[line:col]format diverges from schema-errorfile:line:col: error:formatTest 2 details — invalid engine name (typo)
Introduced error (line 19):
engine: claaude(should beclaude)Simulated output (based on
engine_definition.goResolve()error path):Dimension scores:
Invalid-engine error has no location context (
pkg/workflow/engine_definition.go:267): The error message has great actionability but no file path, line number, or source snippet. A developer editing multiple workflows can't navigate to the exact offending line without searching.console.CompilerErrorstruct (same as schema errors) so it renders astest-2.md:19:9: error: invalid engine 'claaude' ...with a highlighted source line.YAML parse error has no example or docs link (
pkg/parser/yaml_error.go:TranslateYAMLMessage): The translationmissing ':' after key — YAML mapping entries require 'key: value' formatnames the fix but never shows corrected syntax. Append a concrete example (engine: copilot) to each YAML translation entry, or add a docs URL (similar to the engine error).examplefield toyamlErrorTranslationsentries and append it when non-empty, similar toDocsEnginesURLappended in engine errors.[line:col]format vsfile:line:col:inconsistency: YAML errors use the non-standard[line:col]bracket notation while schema errors use IDE-parseablefile:line:col: error:format. This prevents editor jump-to-error and makes errors look inconsistent.FormatYAMLError, after formatting the error, convert it into aconsole.CompilerErrorwith the file path (passed down from the caller), so both YAML and schema errors share the same rendered format.References