Build the sealed holdout corpus protocol and evaluator boundary

[Punisheroot]

Context

Needle already has a frozen-corpus manifest, typed BenchmarkOracle, content
digests, focused-test preflight, campaign cost readiness, and response-quality
evaluation. The current public manifest contains four tasks. Two task IDs are
labelled holdout, but their prompts, oracle files, source needles, focused
tests, and expected evidence have already been published in Git history. They
are therefore public development fixtures and cannot support an independent
holdout claim.

The current runner boundary also does not completely prevent oracle material
from reaching a model-facing process. A real holdout package must remain
outside the public checkout and must fail closed until both independently
sealed task/oracle material and a power-analysis plan are supplied.

Expected outcome

Needle has a versioned, answer-free sealed-corpus protocol that can be fully
implemented and tested with synthetic public fixtures. Real holdout task and
oracle bytes enter only through an evaluator-owned external bundle and never
through model, worker, lifecycle, launch-payload, or ordinary log surfaces.

The protocol accepts an immutable PowerPlan produced by #7, binds its digest into the
final schedule, and checks exact per-route repetition counts without deriving
or inventing them. In the public checkout, or whenever the external sealed
bundle or validated power plan is absent, preflight truthfully returns
provider_run_ready = false. Supplying real never-published holdouts and
running a powered campaign remain a separate maintainer-owned activity.

Scope

• Introduce an answer-free corpus/campaign schema revision for calibration,
  synthetic fixtures, and externally sealed holdouts. Bind task identity,
  repository URL/SHA, route, split/evidence class, prompt and focused-command
  policy without placing evaluator answers, source needles, expected symbols,
  oracle paths, or raw oracle bytes in model-facing manifests.
• Treat the two currently public task IDs ending in -holdout as synthetic
  development fixtures that are permanently ineligible for independent
  holdout evidence. Preserve them only where useful for deterministic schema,
  evaluator, and adversarial tests; do not relabel their results as statistical
  evidence.
• Define an evaluator-only external bundle contract for real holdout task and
  oracle bytes. Bind every sealed entry to a task ID, repository SHA, schema,
  focused-test identity, and content digest. The evaluator resolves and reads
  the material only after an arm finishes; model-facing processes receive no
  filesystem handle or answer-bearing metadata.
• Add a two-stage schedule contract. Calibration observations are validated and
  supplied to Harden the paired-analysis, PowerPlan, and BCa final benchmark gate #7 to produce an immutable PowerPlan. The frozen final schedule
  then references that plan digest and records its exact per-route required
  pair counts. Every real holdout task/repetition requires FinalArm::ALL,
  calibration and holdout remain disjoint, and automatic_retries = false.
• Permit synthetic public PowerPlan and sealed-bundle fixtures for offline
  protocol tests. Synthetic values exercise the complete schema but are
  explicitly ineligible for provider evidence or economic claims.
• Extend preflight_frozen_corpus/corpus-preflight to validate source
  cleanliness, origin/SHA, safe paths, schema and digest binding, task and arm
  uniqueness, focused-test policy, campaign/cost/power-plan digests, bundle
  availability, and the non-adaptive retry policy.
• Create bounded model/worker/lifecycle launch payloads containing only the
  task prompt and approved execution metadata. They must omit oracle paths,
  oracle digests where those act as answer handles, evidence needles, expected
  files/symbols/claims, evaluator answers, and external bundle locations.
• Make every missing, stale, malformed, mutated, leaked, duplicate, or
  incompatible input fail closed with bounded diagnostics. Absence of a real
  sealed bundle or validated PowerPlan is an expected not-ready state, not a
  reason to invent data.
• Keep evaluator results, oracle version/digest, model output, infrastructure
  failure, and focused-test evidence separate. Editing an oracle or schedule
  after freeze invalidates the package and cannot retroactively validate an
  observation.
• Add offline fixture tests for valid synthetic packages, answer-bearing launch
  payloads, public-task misuse, missing/mutated external bundles, stale power
  plans, schedule mismatch, source drift, duplicate identities, malformed
  commands, and campaign/digest mismatch.

Non-goals

• Creating, publishing, or committing real holdout prompts, task identities,
  oracle bytes, needles, expected answers, or evaluator secrets.
• Producing the real power-plan count, running paid provider arms, selecting
  models/pricing, or making a statistical savings claim.
• Treating the currently published -holdout tasks as recoverable secret
  holdouts; Git history makes them permanently unsuitable for that purpose.
• Hard-coding a global holdout repetition count, deriving it inside corpus
  preflight, or accepting a count without a validated Harden the paired-analysis, PowerPlan, and BCa final benchmark gate #7 PowerPlan digest.
• Changing semantic cache/worker behavior, verifier TestPlan execution, adding
  a remote corpus service, or authorizing automatic retries or publication.

Acceptance criteria

• The answer-free schema is versioned and rejects unknown fields. Public
  manifests and model-facing launch payloads contain no oracle path, raw
  oracle bytes, source needles, expected file/symbol/claim text, evaluator
  result, or external bundle location.
• The existing public task IDs
  ripgrep-no-ignore-vcs-locate-holdout and
  ripgrep-null-data-trace-holdout are explicitly marked synthetic or
  otherwise prevented from contributing to independent holdout evidence.
• A real holdout entry is usable only through an evaluator-owned external
  bundle with matching schema, task, repository SHA, focused-test identity,
  and digest; the arm process cannot resolve or read that bundle.
• The final schedule references one immutable PowerPlan digest from Harden the paired-analysis, PowerPlan, and BCa final benchmark gate #7
  and exactly matches its per-route required pair counts. Missing, stale,
  mismatched, or synthetic-only plans return provider_run_ready = false.
• Every real holdout task/repetition schedules FinalArm::ALL; calibration
  entries remain separate, automatic retries are disabled, and a schedule
  digest detects omissions, substitutions, reordered identities, or extra
  repetitions.
• A complete synthetic fixture can exercise schema, scheduling, evaluator,
  and mutation checks offline, but is clearly marked ineligible for provider
  evidence. Without a real sealed bundle, preflight remains not ready.
• Preflight validates exact source origin/SHA and cleanliness, safe paths,
  cost/campaign/power-plan digests, focused-command policy, unique task/arm
  identities, and bundle immutability. Every violation produces a bounded
  fail-closed reason.
• Quality failure, stale hit, infrastructure failure, evaluator error,
  bundle mutation, and oracle mismatch remain explicit and block promotion;
  no failed record or missing arm is silently omitted.
• Offline adversarial tests cover public-task misuse, oracle/needle leakage,
  missing or mutated bundle, stale or mismatched power plan, schedule count
  mismatch, duplicate identity, malformed command, source drift, and
  campaign digest mismatch.

Validation

cargo test --locked -p needle-bench
cargo test --locked -p needle-bench corpus
cargo test --locked -p needle-bench final_gate
cargo run --locked -p needle-bench --bin corpus-preflight -- --workspace-root . --manifest benchmarks/corpus/router-cache/manifest.json --source-repository target/router-cache-source --output target/corpus-preflight/report.json

The checked-in public fixture must finish without provider calls and report
provider_run_ready = false when no external sealed bundle and validated real
PowerPlan are supplied. Focused tests may run only when explicitly requested
against a clean pinned source checkout.

Starting points

• crates/needle-bench/src/corpus.rs - corpus/oracle types, source validation,
  evaluator checks, cost readiness, and fail-closed preflight.
• crates/needle-bench/src/final_gate.rs - manifest, task/arm identities,
  observations, power estimation, and the current final gate.
• crates/needle-app/src/minimal_live_pilot/protocol.rs - current protocol,
  model-facing task selection, oracle loading, and quality-spec boundary.
• benchmarks/corpus/router-cache/manifest.json, campaign.json, and
  oracles/ - public development corpus that must not be represented as an
  independent holdout.
• docs/BENCHMARKING.md and PROJECT_STATUS.md - evidence and paid-run
  boundaries.

Dependencies

#7 owns the deterministic,
digest-bound PowerPlan produced from validated calibration observations. This
issue owns the answer-free corpus, external evaluator bundle, and frozen
schedule that references that plan. Both implementations can use synthetic
fixtures and may proceed in parallel; neither requires nor authorizes a live
provider run.

The future maintainer-only powered campaign owns creation and custody of fresh,
never-published holdout tasks/oracles, collection of calibration observations,
freezing the real power plan and final schedule, provider budget approval, and
any public statistical interpretation.

Ownership and risk

The benchmark maintainer owns the corpus protocol and evaluator boundary.
Oracle leakage or reuse of public answers invalidates an evidence claim;
overly-strict preflight only delays a run. Keep evaluator access physically and
logically separate, bind every transition by digest, and preserve truthful
provider_run_ready semantics.