feat: enhance gh aw audit with firewall policy analysis, diff, and report commands

[Mossaka]

Problem Statement

gh aw audit <run-id> already downloads firewall artifacts (audit.jsonl, policy-manifest.json, access.log) but doesn't leverage the policy manifest for rule attribution — the ability to show which policy rule caused each allow/deny decision. The AWF CLI already has this capability (awf logs audit with audit-enricher.ts), but gh aw audit only shows basic domain-level allow/block counts.

Additionally, there's no way to:

• Compare firewall behavior across two runs (detect regressions, new denied domains, policy drift)
• Generate comprehensive reports across multiple runs for security review

Proposed Changes (MVP-scoped)

Phase 1: Enhance gh aw audit <run-id> with automatic policy enrichment

When firewall artifacts are present in the downloaded run directory (detected automatically — no new flags), enrich the audit output with:

• Per-domain breakdown with allow/deny counts (already exists in firewall_log.go)
• Policy rule attribution — for each domain, show which rule caused the decision
  • Requires parsing policy-manifest.json and replaying ACL evaluation order (port the logic from audit-enricher.ts to Go)
  • Rules are evaluated top-to-bottom by order field; first matching rule wins
  • Domain matching: .github.com matches both github.com and *.github.com
• Denied request details with the specific rule ID, description, and reason
• Summary statistics — total requests, allowed, denied, unique domains, rule hit counts

Data sources (in priority order):

1. audit.jsonl — structured JSON lines, preferred
2. access.log — space-separated text format, fallback

Policy manifest location: policy-manifest.json in the audit artifacts directory

Output: Integrated into the existing audit markdown report (no separate flag). When policy-manifest.json is absent, fall back to current behavior (domain counts only).

Example enriched output section

### Firewall Policy Analysis

**Policy**: 12 rules, SSL Bump disabled, DLP disabled

| Rule | Action | Description | Hits |
|------|--------|-------------|------|
| allow-both-plain | ✅ allow | Allow HTTP/HTTPS to whitelisted domains | 47 |
| deny-blocked-plain | ❌ deny | Deny all other HTTP/HTTPS traffic | 3 |
| deny-unsafe-ports | ❌ deny | Deny requests to unsafe ports | 0 |

**Denied Requests (3)**

| Time | Domain | Rule | Reason |
|------|--------|------|--------|
| 14:23:01 | evil.com:443 | deny-blocked-plain | Domain not in allowlist |
| 14:23:05 | tracker.io:443 | deny-blocked-plain | Domain not in allowlist |
| 14:24:12 | evil.com:80 | deny-blocked-plain | Domain not in allowlist |

Implementation notes

• Reimplement in Go — do NOT invoke awf logs audit as a subprocess (requires Docker/sudo)
• Port enrichWithPolicyRules() and domainMatchesRule() logic from src/logs/audit-enricher.ts
• Add PolicyManifest and PolicyRule structs mirroring the TypeScript types in src/types.ts
• Parse audit.jsonl as JSON lines (each line: {"ts", "client", "host", "dest", "method", "status", "decision", "url"})
• Extend existing pkg/cli/firewall_log.go with JSONL parsing + policy manifest loading
• Follow existing patterns: console.Format*() for output, --json support for structured output

Tasks

• Add PolicyManifest, PolicyRule Go structs
• Add audit.jsonl JSONL parser (complement existing access.log parser)
• Port enrichWithPolicyRules() / domainMatchesRule() to Go
• Detect firewall artifacts automatically in AuditWorkflowRun()
• Integrate enriched policy analysis into audit markdown output
• Add --json support for enriched firewall data
• Unit tests for rule matching, JSONL parsing, policy manifest loading

---

Phase 2: gh aw audit diff <run-id-1> <run-id-2>

Compare firewall behavior across two workflow runs.

Output includes:

• New domains — domains contacted in run-2 but not run-1
• Removed domains — domains in run-1 but not run-2
• Status changes — domains that changed from allowed→denied or denied→allowed
• Request volume changes — significant increase/decrease in request counts per domain
• Anomaly flags — new denied domains, previously-denied domains now allowed

Output formats: pretty (default), markdown, json

Usage:

# Compare two runs
gh aw audit diff 12345 12346

# Markdown output for PR comments
gh aw audit diff 12345 12346 --format markdown

# JSON for CI integration
gh aw audit diff 12345 12346 --json

Example diff output

### Firewall Diff: Run #12345 → Run #12346

**New domains (2)**
- ✅ `registry.npmjs.org` (15 requests, allowed)
- ❌ `telemetry.example.com` (2 requests, denied)

**Removed domains (1)**
- `old-api.internal.com` (was allowed, 8 requests in previous run)

**Status changes (1)**
- `staging.api.com`: ✅ allowed → ❌ denied (policy change?)

**Volume changes**
- `api.github.com`: 23 → 89 requests (+287%)

Implementation notes

• Reuse Phase 1's artifact downloading and parsing
• Diff logic operates on aggregated DomainRequestStats from both runs
• Cache downloaded artifacts to avoid re-downloading when diffing against a previously audited run

Tasks

• Add audit diff subcommand with two positional run-id arguments
• Implement domain-level diff logic (new, removed, changed status, volume delta)
• Add anomaly detection (new denied, status flips)
• Pretty, markdown, and JSON formatters
• Unit tests for diff logic

---

Phase 3: gh aw audit report [--workflow <name>] [--last <N>]

Generate a comprehensive audit report across multiple runs for security review.

Output includes:

• Executive summary — total runs analyzed, overall denial rate, unique domains across all runs
• Domain inventory — all domains contacted across runs, with per-run allow/deny status
• Anomaly detection — runs with unusual patterns (spike in denials, new domains)
• Recommendations — frequently denied domains that might need allowlisting, unused allowed domains
• Per-run breakdown — summary row per run with key metrics

Usage:

# Report on last 10 runs of a workflow
gh aw audit report --workflow "agent-task" --last 10

# Report on all recent runs (default: last 20)
gh aw audit report

# JSON for dashboards
gh aw audit report --workflow "agent-task" --last 5 --json

Output: Markdown by default (suitable for security reviews, piping to files, or $GITHUB_STEP_SUMMARY).

Implementation notes

• Fetch run list via GitHub API filtered by workflow name
• Download and parse artifacts for each run (with caching)
• Aggregate stats across runs into a cross-run summary
• Follow existing addRepoFlag(), addJSONFlag() patterns

Tasks

• Add audit report subcommand with --workflow and --last flags
• Implement cross-run aggregation logic
• Anomaly detection (denial rate spikes, new domain appearances)
• Recommendation engine (frequently denied → suggest allowlist, unused allowed → suggest removal)
• Markdown and JSON formatters
• Unit tests for aggregation and anomaly detection

---

Data Format Reference

audit.jsonl (one JSON object per line):

{"ts":1761074374.646,"client":"172.30.0.20","host":"api.github.com:443","dest":"140.82.114.22:443","method":"CONNECT","status":200,"decision":"TCP_TUNNEL","url":"api.github.com:443"}

policy-manifest.json:

{
  "version": 1,
  "generatedAt": "2024-12-20T15:30:45.123Z",
  "rules": [
    {
      "id": "deny-unsafe-ports",
      "order": 1,
      "action": "deny",
      "aclName": "!Safe_ports",
      "protocol": "both",
      "domains": [],
      "description": "Deny requests to ports not in Safe_ports ACL"
    }
  ],
  "dangerousPorts": [22, 25, 109, 110, 143, 389, 465, 587],
  "dnsServers": ["8.8.8.8", "8.8.4.4"],
  "sslBumpEnabled": false,
  "dlpEnabled": false,
  "hostAccessEnabled": false,
  "allowHostPorts": null
}

Decision codes: TCP_TUNNEL/TCP_HIT/TCP_MISS = allowed, TCP_DENIED/NONE_NONE = denied

Artifact directory structure

run-{id}/
├── sandbox/
│   └── firewall/
│       ├── logs/
│       │   ├── audit.jsonl            # Structured log entries (preferred)
│       │   └── access.log             # Text format (fallback)
│       └── policy/
│           └── policy-manifest.json   # Policy rules for enrichment

[Mossaka]

Plan Review: Enhanced Audit with Policy Analysis, Diff, and Report

I've done a thorough code-level review of this plan against the current state of both gh-aw and gh-aw-firewall. Overall the plan is well-structured and the phasing makes sense. Below are accuracy checks, risks, and specific implementation feedback.

---

Accuracy Check: Current State

Mostly accurate, with one critical gap. The plan correctly describes:

• The existing analyzeFirewallLogs() flow in pkg/cli/firewall_log.go (line 306) — discovers logs in sandbox/firewall/logs/ or squid-logs-* legacy paths
• The FirewallAnalysis struct (line 115) with domain-level allow/block counts
• The access.log text format parser (parseFirewallLogLine())
• The audit report rendering pipeline (audit_report_render.go:renderFirewallAnalysis())

What the plan overlooks:

• There's already a JavaScript-based firewall log parser invoked via parseFirewallLogs() in logs_parsing_firewall.go when --parse is set. The enriched Go parser should supersede this, but the plan should note this deprecation path.
• The DomainRequestStats struct in firewall_log.go already tracks per-domain allowed/blocked counts — good foundation to build on.
• The LogAnalysis interface in log_aggregation.go that FirewallAnalysis implements — new enrichment fields need to respect this interface.

---

Critical Issue: policy-manifest.json Is NOT Currently Uploaded as an Artifact

This is the biggest gap in the plan. The issue assumes this artifact directory structure:

run-{id}/sandbox/firewall/policy/policy-manifest.json

But in reality:

1. AWF writes policy-manifest.json to ${workDir}/audit/ (e.g., /tmp/awf-<timestamp>/audit/policy-manifest.json) — see docker-manager.ts:1684
2. gh-aw's compiler only uploads /tmp/gh-aw/sandbox/firewall/logs/ — see compiler_yaml_main_job.go:353,363,373
3. There is no --audit-dir flag passed from gh-aw to AWF, so audit artifacts go to a temporary directory that's cleaned up after the run

This means policy-manifest.json is never uploaded as a GitHub Actions artifact today. Phase 1 has a hidden prerequisite:

Option A (recommended): Have gh-aw pass --audit-dir /tmp/gh-aw/sandbox/firewall/audit to AWF, then add that path to the artifact upload step in compiler_yaml_main_job.go. AWF already supports --audit-dir (see cli.ts:1328). This is a change in pkg/workflow/awf_helpers.go (BuildAWFArgs()).

Option B: Have AWF copy policy-manifest.json into the logs directory alongside access.log and audit.jsonl. Simpler but muddies the separation between logs and config.

Either way, this is a cross-repo change that must land before Phase 1 can work on real artifacts. The plan should add a "Phase 0" task or prerequisite.

---

Phase 1: Policy Enrichment — Feasibility & Feedback

The enrichment logic port is straightforward. audit-enricher.ts is only ~187 lines with clean, well-tested logic. Key porting notes:

1. Regex compilation caching is important. The TypeScript code creates new RegExp(entry, 'i') on every call. In Go, regexp.Compile() is expensive — cache compiled *regexp.Regexp objects per rule to avoid O(n×m×p) recompilation. A map[string]*regexp.Regexp keyed by pattern string is sufficient.

2. The regex detection heuristic needs careful porting. Lines 46-47 of audit-enricher.ts detect regex rules by checking rule.aclName.includes('regex') OR if domains contain metacharacters [\\^$*+?{}()[\]|]. The Go equivalent needs the same two-path check.

3. The "action consistency" invariant is subtle but critical. findMatchingRule() only returns rules whose action matches the observed outcome (line 113). This prevents misattribution — e.g., an allow rule won't be returned for a denied request even if the domain matches. Make sure this is preserved in the Go port.

4. audit.jsonl parser. The plan says "Add audit.jsonl JSONL parser (complement existing access.log parser)." The current parseFirewallLogLine() in firewall_log.go only handles the space-separated access.log format. You'll need a parallel parseAuditJSONL() that produces the same FirewallLogEntry (or a new enriched type). Consider whether to unify or keep separate.

5. Graceful degradation is well-designed. The plan correctly specifies falling back to current behavior when policy-manifest.json is absent. This keeps backward compatibility clean.

Scope assessment: Phase 1 is appropriately sized for an MVP. The core work is ~3-4 files of new Go code plus extending audit.go and audit_report_render.go.

---

Phase 2: Audit Diff — Feedback

Sound approach, but some implementation concerns:

1. Artifact caching. The plan mentions "cache downloaded artifacts to avoid re-downloading." Be specific about where: the existing run_summary.json cache in logs_models.go saves processed results but not raw artifacts. You'll need either:

   • A new cache layer for raw parsed data (preferred — avoids re-downloading AND re-parsing)
   • Or extend RunSummary with the per-domain stats needed for diffing

2. "Anomaly flags" scope creep risk. "New denied domains, previously-denied domains now allowed" is fine. But "significant increase/decrease in request counts" needs a threshold — what's "significant"? Suggest a simple percentage threshold (e.g., >100% increase) rather than statistical methods for MVP.

3. Subcommand structure. audit diff as a subcommand is clean. Make sure to follow the existing pattern where audit takes positional args — consider gh aw audit diff <run-1> <run-2> vs gh aw audit --diff <run-1> <run-2>. The former is cleaner and matches the plan.

Scope assessment: Appropriately scoped. The diff logic itself is simple once Phase 1's parsing is solid.

---

Phase 3: Audit Report — Feedback

This is the most ambitious phase and has the most risk:

1. N+1 artifact downloads. Downloading artifacts for 10-20 runs serially will be slow. Consider parallel downloads with a concurrency limit (e.g., 5). The existing downloadRunArtifacts() in logs_download.go is synchronous — you'll need goroutines.

2. "Recommendation engine" is a slippery slope. "Frequently denied → suggest allowlist" is a simple count threshold and is fine. "Unused allowed → suggest removal" requires comparing the policy manifest's allowed domains against observed traffic — doable but requires the manifest for every run. Keep this to a stretch goal.

3. Cross-run aggregation types. Consider whether to reuse FirewallAnalysis or create a new CrossRunFirewallAnalysis type. The latter is cleaner since the semantics are different (union of domains across runs, per-run breakdown).

4. --workflow filter. The GitHub API can filter runs by workflow filename, which is efficient. But the plan should note that workflow name != workflow filename — the flag should accept either and resolve via the API.

Scope assessment: Slightly ambitious for a single phase. Consider splitting into 3a (multi-run aggregation + basic report) and 3b (anomaly detection + recommendations).

---

Phase Ordering

Phase 1 → 2 → 3 is correct. Each phase genuinely builds on the previous:

• Phase 1 establishes parsing + enrichment (foundation)
• Phase 2 adds comparison (requires Phase 1's parsed data structures)
• Phase 3 adds aggregation (requires Phase 1's parsing, benefits from Phase 2's diff logic for anomaly detection)

However, add a "Phase 0" prerequisite: Getting policy-manifest.json into the uploaded artifacts. This is a cross-repo change (gh-aw compiler + possibly AWF) and should be done first.

---

Missing Pieces

1. Cross-repo artifact upload change (discussed above — the biggest gap)
2. audit.jsonl may not exist in older runs. The JSONL format was added alongside the policy manifest. The plan should be explicit about the fallback chain: audit.jsonl → access.log → no firewall data.
3. Testing strategy. The plan lists "unit tests" but doesn't mention integration testing. Consider adding test fixtures with real audit.jsonl + policy-manifest.json samples in pkg/cli/testdata/.
4. JSON output schema. Phase 1 mentions --json support for enriched data. Define the schema early — it becomes the contract for Phase 2 and 3.
5. Performance bounds. For Phase 3, what's the maximum number of runs to support? 20? 100? This affects download time and memory usage.

---

Risks

1. AWF policy manifest format changes. If PolicyManifest evolves in AWF, the Go structs need to stay in sync. Consider a shared JSON schema or version field checking (the manifest already has version: 1).
2. Rate limiting on artifact downloads. Downloading 20 runs' artifacts may hit GitHub API rate limits. Phase 3 should implement backoff.
3. Permission errors. Firewall log artifacts may be in private repos requiring specific tokens. The existing downloadRunArtifacts() handles this via gh CLI auth, but worth testing.

---

Summary

Aspect	Assessment
Overall plan quality	Good — well-structured, correct phasing
Biggest gap	policy-manifest.json not in artifacts today (needs Phase 0)
Phase 1 scope	Appropriate for MVP
Phase 2 scope	Appropriate
Phase 3 scope	Slightly ambitious — consider splitting
Enrichment port complexity	Low-medium (~2 days for a Go developer familiar with the codebase)
Key files to modify	firewall_log.go, audit.go, audit_report.go, audit_report_render.go, awf_helpers.go (in gh-aw); possibly compiler_yaml_main_job.go for artifact upload

[pelikhan]

wait for #22733

[pelikhan]

@mnkiefer take a look.

[Mossaka]

Broken down into individual issues under #22735: #22755 (policy enrichment), #22759 (audit diff), #22760 (audit report). Closing this umbrella.