Description
The Detection Analysis Report workflow (detection-analysis) evaluates 4 misconfiguration rules against aw_info.json run summaries, but Rule 3 ("detection-enabled run failed on detection-specific steps") cannot currently distinguish a real detection-step failure from a full agent-job non-execution. In the 2026-08-21 23:28Z run (discussion #54655), 21 of 254 detection-enabled runs failed (e.g. AI Moderator 7/7, Code Scanning Fixer 2/2), and all 21 show TokenUsage: 0 and ErrorCount: 0 in their run summaries — a signature that strongly suggests the agent job never started (likely a trigger/permission issue) rather than a detection-step-specific failure. The report explicitly declines to classify these as misconfigurations because it lacks step-level logs to tell the two failure modes apart, and flags this as "worth a follow-up if step-level detection logs become available."
Expected Impact
Without step-level detection-vs-agent-job failure attribution, Rule 3 can never fire with confidence, and 21+ failing runs per day are excluded from misconfiguration detection even when the root cause (e.g. a detection step genuinely erroring) would be actionable. Adding step-level log parsing (or surfacing an existing per-step conclusion already in the GitHub Actions job data) would let the Detection Analysis Report distinguish these cases and close a real observability gap in the detection-coverage tooling.
Suggested Agent
detection-analysis workflow itself, or the shared agenticworkflows logs/log-parsing tooling it depends on for aw_info.json step-level data — could also be a "New Agent" scoped tooling change.
Estimated Effort
Medium (1-4 hours) — requires parsing per-step job conclusions (already available via the GitHub Actions API) and correlating them against the existing TokenUsage/ErrorCount heuristic.
Data Source
DeepReport Intelligence Briefing analysis run 32540163235, sourced from Detection Analysis Report — 2026-08-21 (discussion #54655).
Generated by 🔬 Deep Report · agent · 151 AIC · ⌖ 8.78 AIC · ⊞ 12.4K · ◷
Description
The Detection Analysis Report workflow (
detection-analysis) evaluates 4 misconfiguration rules againstaw_info.jsonrun summaries, but Rule 3 ("detection-enabled run failed on detection-specific steps") cannot currently distinguish a real detection-step failure from a full agent-job non-execution. In the 2026-08-21 23:28Z run (discussion #54655), 21 of 254 detection-enabled runs failed (e.g. AI Moderator 7/7, Code Scanning Fixer 2/2), and all 21 showTokenUsage: 0andErrorCount: 0in their run summaries — a signature that strongly suggests the agent job never started (likely a trigger/permission issue) rather than a detection-step-specific failure. The report explicitly declines to classify these as misconfigurations because it lacks step-level logs to tell the two failure modes apart, and flags this as "worth a follow-up if step-level detection logs become available."Expected Impact
Without step-level detection-vs-agent-job failure attribution, Rule 3 can never fire with confidence, and 21+ failing runs per day are excluded from misconfiguration detection even when the root cause (e.g. a detection step genuinely erroring) would be actionable. Adding step-level log parsing (or surfacing an existing per-step conclusion already in the GitHub Actions job data) would let the Detection Analysis Report distinguish these cases and close a real observability gap in the detection-coverage tooling.
Suggested Agent
detection-analysisworkflow itself, or the sharedagenticworkflows logs/log-parsing tooling it depends on foraw_info.jsonstep-level data — could also be a "New Agent" scoped tooling change.Estimated Effort
Medium (1-4 hours) — requires parsing per-step job conclusions (already available via the GitHub Actions API) and correlating them against the existing
TokenUsage/ErrorCountheuristic.Data Source
DeepReport Intelligence Briefing analysis run 32540163235, sourced from Detection Analysis Report — 2026-08-21 (discussion #54655).