An open research lab for finding better DeepSeek Harness configurations through reproducible experiments. Start with the official default plugins and the cheapest verified DeepSeek text model. Diagnose failures, test a small change, measure its accuracy, cost and speed, and keep it only when the evidence holds up.
I want to take part in the DSH plugin ecosystem and contribute to its ongoing evolution: useful plugins, better benchmarks, independent replications and honest negative results. This repository is my working lab and research notebook, developed with AI assistance. Contributions and challenges to the methodology are welcome.
Current result: no plugin has earned promotion. Default DSH remains the incumbent. This is early research, not a claim to have built the most intelligent harness.
Which model-plus-harness configuration performs best for a specified workload, cost limit and latency target? Can we improve it repeatedly without overfitting the tasks used to develop it? When is a plugin better than a simpler configuration change—or giving the default agent the same additional inference budget?
The ambition is broad capability across coding, reasoning, investigations and session recovery. Current measurements cover small authored fixtures with local evidence. Real repository benchmarks, live research and genuine multi-session memory evaluation still need work.
flowchart LR
A[Shipped default] --> H[Calibrate difficulty on development tasks]
H --> B[Audit failures and costs]
B --> C[One structural or guidance change]
C --> D[Free checks and cheap screening]
D --> E[Paired development tests]
E --> F[Fresh confirmation and replication]
F --> G[Keep, replace or remove components]
G --> A
- Require default accuracy below 50% overall and per claimed domain on a separate, audited development set before expanding experiments. Follow the baseline calibration gate; the CLI does not enforce it.
- Let observed failures drive candidate development.
- Allow additions, removals and replacements. More plugins are not inherently better.
- Separate structural changes from guidance changes; test interactions before merging.
- Keep useful accuracy/cost/speed tradeoffs, with uncertainty, rather than declaring an unqualified single winner. Routing between profiles also needs its own test.
- Treat agent self-feedback as a claim to check against traces, not an explanation that becomes true because the model said it.
- Preserve failed runs and verifier mistakes. Freeze candidates before confirmation.
See the experimental protocol, research landscape and roadmap. HarnessCompass, HarnessLens and critiques of harness evolution inform the questions; this repository does not claim to reproduce their results.
The September external MBPP/BBH/HotpotQA suite also exposed a ceiling problem: default passed all 75 coding and all 75 reasoning attempts (25 tasks repeated three times in each domain). Those results are observations from a limited suite, not a broad capability verdict. Retain easy tasks for regression checks; future main evaluations must first pass difficulty calibration on separate development data.
| Component | Status |
|---|---|
| Real DSH headless runtime, pinned dependencies and native plugin patches | Working |
| Docker isolation and a metered DeepSeek-only gateway | Working |
| Persistent global, campaign, run and request-count limits | Working; estimates under pinned prices, not an account-wide billing guarantee |
| Paired randomized experiments and exact graders | Working; verifier corrections are recorded |
| Separate edit manifests, static task-marker lint and frozen guidance adapters | Working; semantic review remains necessary |
| Bounded blind/hindsight feedback packets and quote-anchor checking | Working preparation; feedback collection and causal review are not automated |
| Portfolio analysis and conservative promotion audit | Working; no automatic promotion |
| Autonomous continuous plugin invention, independent holdout service, ecosystem leaderboard | Not implemented |
Local candidates are verify-first and core-tools. Other arms change shipped PTC/pruning settings or thinking effort. Model-setting changes are reported separately from plugin gains.
The public evidence archive contains five paid development campaigns, 84 attempted task runs, per-call accounting, reports, verifier errata, source snapshots where available, and generated workspace artifacts.
| Experiment | Observation | Interpretation |
|---|---|---|
| 12-task diagnostic screen | Default, thinking-off and verify-first: 12/12; PTC: 11/12 | Narrow, ceiling-scoring screen; no promotion |
| Reduced tool exposure | First main request about 41KB → 19KB; 26 → 6 schemas | Less input did not translate into comparable bill savings because caching and output costs matter |
| Final corrected-contract smoke, 3 tasks | Default: 3/3; low and disabled thinking: 2/3 each | Both cheaper settings hit the repair response limit; far too small for a general claim |
The lab also caught its own grader mistakes: numeric equality and underspecified JSON output formats. Original results remain available with explicit errata. Current suites are diagnostic-v2 and transfer-v3. Public generated seed splits are not secret, independently stewarded holdouts. Never combine different suite versions into a leaderboard without a justified comparison design.
Requires Python 3.9+; no API key, Docker or paid generation is needed here.
git clone https://github.com/Milbaxter/dsh-evolution-lab.git
cd dsh-evolution-lab
python3 -m unittest discover -s tests -vThe initial host was an 8GB Apple-silicon Mac using Colima with 2 CPUs/2GB RAM. For macOS with Colima and Docker CLI installed:
colima start --profile deepseek-lab --cpu 2 --memory 2 --disk 16 --vm-type vz --mount-type virtiofs
export DSH_LAB_DOCKER_CONTEXT=colima-deepseek-lab
docker --context "$DSH_LAB_DOCKER_CONTEXT" build -f containers/Dockerfile.agent -t deepseek-lab-agent:0.1 .
docker --context "$DSH_LAB_DOCKER_CONTEXT" build -f containers/Dockerfile.gateway -t deepseek-lab-gateway:0.1 .
docker --context "$DSH_LAB_DOCKER_CONTEXT" pull python:3.12-slim
python3 scripts/check_graders.py
python3 -m lab.run --mock --suite transfer --variants default core-toolsFor an existing Docker Engine on Linux, set DSH_LAB_DOCKER_CONTEXT=default and
use that context for the same build commands. This portability option is included;
initial paid evidence was collected on the Mac, not validated across platforms.
Mock runs boot the real harness but return a fixed response: their task scores are
not model-performance evidence.
For a deliberately small paid run after free checks:
python3 scripts/set_key.py # hidden input; stores the key outside this repo
python3 scripts/check_balance.py # free credit lookup
python3 -m lab.run --stage smoke --suite transfer --variants default verify-first --campaign-cap 0.12Current shared response limit: 16,384 output tokens, including reasoning, per model call. The maintainer selected this budget; earlier reports used4,096 and must retain their original labels.
Model and tariff snapshot: deepseek-v4-flash, checked 2026-09-07. Verify the
official pricing when resuming;
the gateway refuses prices older than seven days. The initial cumulative lab
ceiling is USD20. A top-up never resets the ledger or silently raises that ceiling.
Every auxiliary request and retry must pass through the same gateway.
Keep state/, logs/, reports/ and credentials local. The public evidence archive
is a reviewed export, not a reason to blindly commit future raw sessions. On a
Mac VM, access live SQLite accounting through lab.run.live_accounting from Linux
while the gateway is running. See the protocol before any larger campaign.
Open an issue or a discussion, or send a pull request. The most useful early contributions are:
- External task packs with strong, independently checked graders.
- Reproductions of our baseline and negative results.
- A candidate tied to a specific observed failure, with predicted costs and risks.
- True restart/memory scenarios, matched-budget controls, and better attribution.
- Interoperability with existing DSH evaluation projects instead of duplicate infrastructure.
Read CONTRIBUTING.md. A community result should say what it beats, on which tasks, under which constraints, and with what uncertainty. The goal is shared progress toward better harnesses—not a preferred plugin winning by design.
MIT licensed; third-party notices are in NOTICE.md.