[Epic]: HTTP API and daemon robustness

[dennisonbertram]

Work type

Epic

Problem and outcome

Review of the HTTP API + daemon found graceful shutdown that closes SQLite stores under live SSE handlers (dropping in-flight turns), terminal events racing durable status commits, ~189 sites leaking internal error details to clients, stale hooks summary after config reload, and an over-broad streaming-timeout exemption. Outcome: shutdown never corrupts stores or drops turns, terminal frames are settlement-consistent, error responses are stable codes not internals, reload reflects reality.

Non-goals

No API shape/version changes, no auth model changes (coverage verified clean), no new endpoints, no migration of error strings to i18n.

Current architecture and evidence

Entry points: internal/server/http.go (routing, hardenHandler, auth middleware), http_conversations.go / http_runs.go (SSE), cmd/harnessd/main.go (lifecycle, store defers at 633-844, shutdown at 1182), cmd/harnessd/config_reload.go, cmd/harnessd/bind_guard.go.
Evidence: conversation SSE handlers never self-terminate by design (http_conversations.go:255-259) so Shutdown(10s) always hits deadline; run stream has waitForTerminalSSESettlement gate (http_runs.go:993) but conversation stream does not (http_conversations.go:352).

Cross-surface impact map

Config/env/defaults: none.
APIs/CLI/tools/wire formats: error bodies change to stable codes — clients matching on err.Error() strings must be audited (TUI, harnesscli, macapp).
Persistence/schema/migrations: none.
Concurrency/lifecycle/recovery: shutdown ordering: cancel handler base ctx → runner quiescence (bounded) → Close() → store teardown.
Security/auth/privacy/permissions: closes internal-detail leakage (paths, SQLite messages).
TUI/web/macOS/other clients: error-code mapping consumers updated in same slices.
Provider/model/tool catalog: none.
Deployment/observability/operations: shutdown duration metric added.
Compatibility/versioning: no wire-format break; error body enrichment additive.
Tests/evals/fixtures: acceptance-api-sse harness extended for shutdown test.
Documentation/training: runbook shutdown-order note.

Shippable child issues

• Slice D1 — Safe shutdown ordering: derive handler contexts from cancellable base ctx cancelled before store teardown; wait for runner quiescence with bounded budget; escalate to httpServer.Close() (cmd/harnessd/main.go:1182). Test: e2e — SIGTERM mid-stream completes/persists in-flight turn, no closed-handle writes. No deps. HIGHEST priority.
• Slice D2 — Conversation SSE settlement gate: apply waitForTerminalSSESettlement for terminal event types on conversation stream (internal/server/http_conversations.go:352). Test: client acting on run.completed sees fresh GET /v1/runs/{id}. No deps.
• Slice D3 — Error response hygiene wave 1: introduce error-class→code mapper; convert the highest-leak sites (conversation/checkpoint/agents handlers, rewind refusals) to log-detail-server-side + generic client message. Test: assertions on absence of absolute paths in responses. No deps.
• Slice D4 — Error response hygiene wave 2: sweep remaining sites (~189 total) mechanically; add drift test forbidding raw err.Error() in writeJSON error paths. Depends on D3's mapper.
• Slice D5 — Reload consistency: persist fresh hooks summary on reload for GET /v1/hooks (config_reload.go:157); move SIGHUP reload off signal-loop goroutine (config_reload.go:148). One PR. Test: add hook via config, SIGHUP, GET /v1/hooks reflects it.
• Slice D6 — Streaming exemption precision + bind_guard fallback: match exact method+path patterns for timeout exemption (http.go:640); log/derive-from-listener instead of hardcoded 127.0.0.1:8080 fallback (bind_guard.go:61). One PR. Tests: non-streaming POST …/wait still times out; custom-port daemon MCP calls succeed.

Dependency graph

D1 first (data-integrity). D3 → D4 sequential. D2, D5, D6 independent.

Integration contracts

Shutdown invariant: no store handle used after close; SSE clients receive terminal frame or clean close, never silence; error bodies carry machine-readable code field; hooks summary equals registered hooks at all times.

Rollout sequence

D1 ships alone first with canary observation of shutdown metrics; rest independent; backout per-PR.

Risks and observability

Risk: D1 quiescence wait delays restarts → mitigation: bounded budget + metric; signal: shutdown-duration histogram; Risk: D3/D4 break string-matching clients → mitigation: grep audit of TUI/macapp/harnesscli error handling in same PRs.

Definition of done

• All shippable child contracts satisfied
• Cross-slice integration and real user path proven
• Compatibility/migrations/rollback verified
• Security and observability acceptance met
• Docs/runbooks/indexes/logs current
• Temporary flags, duplication, and scaffolding removed or explicitly owned

Epic acknowledgement

• I will not close this epic directly from an implementation PR; every code slice will close its own contract-complete child issue.