[CI/CD Assessment] CI/CD Pipelines and Integration Tests Gap Assessment

[github-actions]

📊 Current CI/CD Pipeline Status

This repository has a well-structured, multi-layered CI/CD pipeline with 44 workflow files (both traditional YAML and agentic Markdown workflows). The pipeline covers static analysis, unit testing, integration testing, security scanning, and AI-powered code review. Overall health is good, but several important gaps exist — most critically around integration test coverage on PRs.

Pipeline inventory:

Category	Workflows	Runs on PRs
Build & Lint	build.yml, lint.yml	✅ Yes
Type Checking	test-integration.yml (TypeScript type check)	✅ Yes
Unit Test Coverage	test-coverage.yml	✅ Yes
Chroot Integration Tests	test-chroot.yml	✅ Yes
Examples Tests	test-examples.yml	✅ Yes
Action Setup Tests	test-action.yml	✅ Yes
Security Scanning	codeql.yml, dependency-audit.yml, container-scan.yml	✅ Yes
PR Convention	pr-title.yml	✅ Yes
AI Security Review	security-guard.md (Claude)	✅ Yes
Agentic Smoke Tests	smoke-claude.md, smoke-copilot.md, smoke-codex.md, smoke-gemini.md	✅ Yes
Agentic Build Tests	build-test-*.md (8 language variants)	✅ Yes
Docs Deploy	deploy-docs.yml	push/main only
Scheduled Security	secret-digger-*.md (hourly), security-review.md, dependency-security-monitor.md	schedule only

Recent run success rates (from ~20 latest runs):

Workflow	Success Rate	Notes
Build Verification	100%	✅ Healthy
Lint	100%	✅ Healthy
Test Coverage	100%	✅ Healthy
TypeScript Type Check	100%	✅ Healthy
Container Security Scan	100%	✅ Healthy
Examples Test	100%	✅ Healthy
Test Setup Action	100%	✅ Healthy
PR Title Check	100%	✅ Healthy
Secret Diggers	100%	✅ Healthy
CI Doctor	0% (4/4 failing)	⚠️ Broken monitor
Build Test Node.js	0% (1/1 failing)	⚠️ Agentic failures
Build Test Bun	0% (1/1 failing)	⚠️ Agentic failures
Build Test C++	0% (1/1 failing)	⚠️ Agentic failures

---

✅ Existing Quality Gates

The following checks currently run on every PR:

1. Build Verification — TypeScript compilation on Node 20 and 22 (matrix), plus build artifact verification
2. ESLint — Linting with eslint-plugin-security for security-focused code patterns
3. TypeScript Type Check — tsc --noEmit strict type validation
4. Unit Test Coverage — Jest with coverage thresholds (38% statements, 30% branches) and PR diff comments
5. CodeQL — Static analysis for javascript-typescript and actions languages with security-extended,security-and-quality queries
6. Dependency Vulnerability Audit — npm audit at --audit-level=high with SARIF upload, for both main and docs-site packages
7. Container Security Scan — Trivy CRITICAL/HIGH scan on agent and squid containers (path-filtered to containers/**)
8. PR Title Check — Conventional Commits enforcement with amannn/action-semantic-pull-request
9. Security Guard (Claude AI) — AI agent reviews iptables rules, Squid ACLs, capability dropping, and domain pattern security
10. Chroot Integration Tests — 4 test suites: languages, package managers, procfs, edge cases
11. Examples Tests — Runs basic-curl.sh, blocked-domains.sh, using-domains-file.sh, debugging.sh
12. Action Setup Tests — Validates the GitHub Action wrapper (latest, specific version, image pull, invalid version scenarios)
13. Agentic Smoke Tests — Real end-to-end firewall tests via Claude, Copilot, Codex, Gemini

---

🔍 Identified Gaps

🔴 High Priority

1. Core Integration Tests Don't Run on PRs

This is the most critical gap. The repository has 26 integration test files covering critical security and functional scenarios, but only chroot-specific tests (4 files) and example scripts run on PRs. The following security-critical test suites have no PR trigger:

• blocked-domains.test.ts — verifies the firewall actually blocks unauthorized domains
• credential-hiding.test.ts — verifies git credentials are not leaked into containers
• network-security.test.ts — verifies network isolation and iptables rules
• dns-servers.test.ts — verifies DNS restriction enforcement
• ipv6.test.ts — verifies IPv6 traffic is correctly handled
• wildcard-patterns.test.ts — verifies domain pattern validation
• exit-code-propagation.test.ts — verifies exit code passthrough
• environment-variables.test.ts — verifies env var handling
• api-proxy.test.ts, localhost-access.test.ts, git-operations.test.ts, and 10 more

A PR that breaks domain blocking, leaks credentials, or weakens network isolation would pass all current checks undetected by automated tests (relying only on AI security review).

2. Unit Test Coverage Thresholds Are Critically Low for a Security Product

Current thresholds — 38% statements, 30% branches — are far too low for a security-critical firewall. The most important modules are almost completely untested:

File	Statement Coverage	Risk
cli.ts	0%	Entry point entirely uncovered
docker-manager.ts	18%	Core container logic, 82% uncovered
host-iptables.ts	83%	Acceptable
squid-config.ts	100%	✅
logger.ts	100%	✅

cli.ts and docker-manager.ts together represent the bulk of the firewall's behavior, yet are nearly untested. A regression in container startup, cleanup, or exit code handling would not be caught by unit tests.

3. Agentic Build-Test Workflows Consistently Failing

build-test-node.md, build-test-bun.md, and build-test-cpp.md are all failing with 0% success rate in recent runs. These AI-powered workflows run on PRs but their failures are not blocking merges — they appear to be non-blocking status checks. If they're always failing, they provide no signal and may be silently masking real issues.

4. CI Doctor Monitor Is Broken

ci-doctor.md is failing 4/4 times. This workflow is meant to monitor other workflows and surface issues, but is itself broken — meaning the health monitoring layer is non-functional.

---

🟡 Medium Priority

5. Container Security Scan Is Path-Filtered — Base Image CVEs Accumulate

container-scan.yml only runs when files under containers/** change. A PR that doesn't touch container files but is deployed after a base image (ubuntu/squid:latest, ubuntu:22.04) accumulates new CVEs will not trigger a scan. The weekly scheduled scan mitigates this for main, but PRs are unprotected from base image drift.

6. No Coverage Minimum Gate on Security-Critical Modules

While coverage comparison runs on PRs and can fail if coverage regresses, there is no minimum coverage gate for specific security-critical files. A PR adding features to docker-manager.ts without tests won't be blocked as long as global coverage doesn't drop below 38%.

7. No ARM Runner Validation

Repository documentation (docs/compatibility.md) lists ubuntu-24.04-arm as "✅ Fully Supported", but no CI workflows run on ARM runners. Binary builds produce ARM64 artifacts, but correctness is not tested on actual ARM hardware in CI. Any ARM-specific behavior (iptables rules, capability dropping, seccomp) is unvalidated.

8. test-integration.yml Naming Is Misleading

The file .github/workflows/test-integration.yml is named to suggest it runs integration tests but actually contains the TypeScript type check. This creates confusion when reading the workflow list and may cause contributors to believe integration tests are running on PRs when they are not.

9. Coverage Comparison Script Uses continue-on-error: true

In test-coverage.yml, the coverage comparison step (compare-coverage.ts) runs with continue-on-error: true, meaning a bug in the comparison script silently produces no comparison. The PR comment falls back to a simple report without diff data.

---

🟢 Low Priority

10. No Performance Regression Testing

There are no benchmark tests or timing measurements tracked in CI. The firewall introduces latency (container startup time, proxy overhead) that could regress without detection. Startup time is particularly important for usability and is not measured.

11. No License Compliance Checking

No license compatibility validation runs in CI. A dependency with an incompatible license (e.g., AGPL) could be introduced without automated detection. Given the npm ecosystem, this is a realistic risk.

12. No Release Artifact Size Monitoring

The binary sizes for awf-linux-x64, awf-linux-arm64, awf-darwin-x64, awf-darwin-arm64 are not tracked across PRs. A dependency change that significantly bloats binary size would go unnoticed.

13. ESLint Config Only Covers src/ Directory

lint.yml runs npm run lint which is configured to lint src only. The scripts/ci/*.ts files (TypeScript CI scripts including test summary generation, coverage comparison, audit conversion) are not linted and could contain bugs without detection.

14. No Mutation Testing

With coverage at 38% and thresholds that don't guarantee meaningful test assertions, mutation testing (e.g., Stryker) would reveal whether existing tests actually verify behavior rather than just execute code paths.

---

📋 Actionable Recommendations

R1: Add Core Integration Test Workflow for PRs

Gap: #1 — Core integration tests missing from PR checks
Solution: Create .github/workflows/test-integration-core.yml that runs the security-critical integration test subset on every PR. Start with the highest-risk tests:

npm run test:integration -- --testPathPatterns="blocked-domains|network-security|credential-hiding|dns-servers|wildcard-patterns|exit-code-propagation" --verbose

Complexity: Low (copy pattern from test-chroot.yml)
Impact: High — closes the largest security validation gap

R2: Raise Coverage Thresholds Incrementally

Gap: #2 — Critically low coverage thresholds
Solution: Increase thresholds in jest.config.js to 60% across all metrics within 2-3 sprints. Prioritize adding unit tests for docker-manager.ts (container lifecycle, volume mounts, environment variable injection) and cli.ts (argument parsing, signal handling, cleanup flow).
Complexity: Medium (requires writing tests for complex async code)
Impact: High — surfaces regressions in core firewall behavior

R3: Investigate and Fix Agentic Build-Test Failures

Gap: #3 — Build-test workflows failing consistently
Solution: Run gh aw audit on a recent build-test-node.md failure to understand root cause. Either fix the agentic workflow prompts/configuration or mark them as experimental and remove from required PR checks until fixed.
Complexity: Low (investigation and targeted fix)
Impact: Medium — restores signal for language-specific build compatibility

R4: Fix CI Doctor

Gap: #4 — Health monitor broken
Solution: Audit ci-doctor.md workflow failures to understand why it's failing 4/4 times. Check for changes in monitored workflow names, missing tokens, or API changes.
Complexity: Low
Impact: Medium — restores pipeline health visibility

R5: Make Container Scan Run on All PRs

Gap: #5 — Path-filtered container scan
Solution: Remove the paths: filter from container-scan.yml so it runs on every PR, or add a separate "quick scan" step that checks for base image CVEs without a full rebuild. Alternatively, cache built images and scan against the cache.
Complexity: Low
Impact: Medium — catches CVE accumulation in base images

R6: Add Per-File Coverage Gates for Security-Critical Modules

Gap: #6 — No module-level coverage gates
Solution: Add perFile coverage thresholds in jest.config.js for docker-manager.ts (target: 50%) and cli.ts (target: 60%).
Complexity: Low
Impact: Medium — prevents regression in critical module coverage

R7: Rename test-integration.yml to Reflect Actual Purpose

Gap: #8 — Misleading workflow file name
Solution: Rename test-integration.yml to typecheck.yml to match its content and avoid contributor confusion. Update ci-doctor.md if it references this workflow by file name.
Complexity: Low
Impact: Low (quality-of-life)

R8: Add Scripts Linting

Gap: #13 — CI scripts not linted
Solution: Extend npm run lint to include scripts/ci/ directory, or add a separate lint step in build.yml.
Complexity: Low
Impact: Low — catches bugs in CI tooling

---

📈 Metrics Summary

Metric	Value
Total workflow files	44 (29 .md agentic + 15 traditional .yml)
Workflows triggering on PRs	~25
Unit test files	10
Integration test files	26
Integration tests running on PRs	~4 (chroot only) + examples
Integration tests not running on PRs	~22 (including all core security tests)
Statement coverage	38.39% (threshold: 38%)
Branch coverage	31.78% (threshold: 30%)
Function coverage	37.03% (threshold: 35%)
cli.ts coverage	0%
docker-manager.ts coverage	18%
Recent CI Doctor success rate	0% (broken)
Recent agentic build-test success rate	0% (3 workflows)

---

Assessment generated on 2026-02-23 based on analysis of .github/workflows/, jest.config.js, COVERAGE_SUMMARY.md, and recent workflow run data.

---

Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

Tip: Discussion creation may fail if the specified category is not announcement-capable. Consider using the "Announcements" category or another announcement-capable category in your workflow configuration.

Generated by CI/CD Pipelines and Integration Tests Gap Assessment

• expires on Mar 2, 2026, 10:29 PM UTC

[copilot-swe-agent]

The agent encountered an error and was unable to start working on this issue: The service is temporarily unavailable. Please try again later.