ConfigPilot

A B2B commercial decision engine where the LLM is never trusted with the number that matters.

An OEM category manager watches a competitor quietly restructure their laptop lineup: new names, new bundles, same silicon. Some configuration starts moving at stable price. Which of our SKUs is exposed, and what can we actually do about it without breaking margin or promising inventory we don't have?

ConfigPilot answers that, and the interesting part is what it refuses to let the model touch.

The brief

AI Fund (Andrew Ng's venture studio) ran this as a 48-hour build challenge for an Engineer in Residence role, in partnership with an OEM PC team. The ask, verbatim:

Build a system that ingests synthetic competitor market data plus synthetic own-portfolio data, detects a competitor portfolio restructure or configuration traction signal, maps it against the own portfolio at the spec level, and outputs a recommendation that respects cost, margin, and inventory constraints.

Four requirements:

1. Configuration-level portfolio mapping that survives competitor renaming or restructuring without relying on name matching.
2. Traction detection that identifies specific configuration attributes driving sell-out velocity gains at stable price.
3. Constrained recommendations that visibly respond to own-portfolio cost, margin, and inventory data.
4. At least one depth area.

All data here is synthetic. No vendor internals. Everything below is measured against that ask, and the honest-limits section says where it falls short.

The idea: AI proposes, engine verifies, human approves

Most LLM product demos put the model in the decision path and bolt on a disclaimer. That inverts once real money is attached to the output. The model should be free to read messy data, argue, draft, and propose, and structurally incapable of writing the fields that decide anything.

ConfigPilot draws that line in three places, and each one is tested adversarially rather than asserted in a doc.

1. The model is not shown the economics it would need to cheat.

orchestrator.py builds the analyst prompt from audited "atoms" (competitor context and public signal) and withholds unit cost and target margin entirely. When the model proposes a price, the evaluator recomputes the margin floor (unit_cost / (1 - target_margin)) from raw economics the model never saw. A sub-floor price cannot be argued past it, because the argument and the check don't share an input.

test_ai_proposing_a_sub_floor_price_is_vetoed_by_the_engine   engine/tests/test_orchestrator.py

2. The model cannot write an identity field, even by lying about provenance.

Critical axes (CPU, RAM, storage, display, GPU, form factor, segment) are never AI-inferred. They stay MISSING until a real source fills them, and missing critical specs downgrade automated price/inventory actions into sales enablement rather than guessing. The enforcement isn't a prompt instruction: _fields_from_provider downgrades any critical field not marked parsed/missing, so a fabricated value mislabeled verbatim with 0.99 confidence still lands as MISSING.

test_ai.py:154-171   feeds the model's output '13" FAKE' / provenance=verbatim / confidence=0.99
                     asserts it resolves to MISSING

3. Proposals fail closed.

_validate_proposal rejects unknown SKUs, null-price price_adjust, and unsupported action types before anything downstream sees them. Two of those guards close bugs that existed and were caught by tests, not hypotheticals.

The result: a BYOK model can be wrong, adversarial, or hallucinating and the worst it achieves is a visible veto with a reason attached. That boundary, not the dashboard, is the thing I'd want reviewed.

What it does

synthetic competitor + own-portfolio data
  -> normalize (regex parsers, provenance-tagged: verbatim / parsed / missing / inferred)
  -> resolve competitor SKUs into clusters by spec fingerprint, names excluded from the key
  -> classify movement: spec traction / discount-driven / stockout artifact / insufficient data
  -> map clusters onto own SKUs by weighted spec similarity
  -> recommend under hard gates: margin floor, inventory, lead time, missing-data safety
  -> evaluate, hash, and hand to a human

The recommendation stage is the part that most visibly works: it computes the obvious action, then applies gates that change the answer. Same competitor signal, different own-portfolio economics, different action. Price match at 20% target margin, reposition at 30% because the floor blocks the counter, cost-structure review when no substitute exists.

test_price_adjust_holds_across_a_margin_band   0.18 / 0.20 / 0.22 -> PRICE_ADJUST
                                               0.30              -> REPOSITION

Every run emits an AuditRecord with a SHA-256 content hash over sorted canonical JSON. Hashes exclude timestamps and all model narration, so runs are byte-reproducible and BYOK text can't perturb them.

What this repo demonstrates

For anyone evaluating the work:

• LLM guardrails and AI safety engineering. A human-in-the-loop trust boundary where schema-constrained structured output is structurally barred from writing identity fields, prices, or margins. Untrusted text is delimiter-wrapped, prompt injection is the default assumption, and provider errors and key-shaped strings are redacted before reaching a client.
• Deterministic verification of AI output. Fail-closed proposal validation, independent margin-floor recomputation, and byte-reproducible SHA-256 audit records over canonical JSON.
• Entity resolution. Spec-fingerprint matching that survives renames, with a RapidFuzz name-matching baseline computed alongside as a control, plus a live endpoint that matches a pasted spec the seed never saw.
• Eval harness design. Production invariants versus labeled ground-truth verification modes, mutation testing, golden determinism tests, and a contract test that fails if any engine module imports FastAPI, an LLM SDK, or an HTTP client.
• Full-stack B2B SaaS. Python 3.12, FastAPI, Pydantic, DuckDB, Polars on the backend. React, TypeScript, Vite, Tailwind on the cockpit. Deployed on Vercel as two projects. BYOK provider layer speaks OpenAI-compatible and Anthropic formats.

What's real vs. what's scaffolding

I audited this repo after the fact, harder than the deadline allowed during it. Publishing the audit alongside the code, because a system's limits are more useful than its claims, and because I'd rather you read my assessment than run your own and find I'd oversold it.

Real mechanisms. These respond to data and are pinned by tests that can fail:

• The verification boundary above. All three rails, all three adversarially tested.
• The constrained recommendation engine. Real arithmetic, gates that flip the action.
• Determinism and hashing. Byte-identical runs, canonical JSON, narration excluded.
• Fail-closed proposal validation.
• test_mutation.py, which mutates engine outputs and proves the checks bite.

Heuristics wearing more confident clothes:

• Traction attribution. The reported "RAM drove 56% of the gain" is the configured similarity weights renormalized, 0.15/(0.15+0.12). It does not vary with the observed magnitudes. A control sibling gates whether attribution runs, but doesn't estimate it. Real difference-in-differences needs repeated observations per SKU; the corpus has two periods and mostly one observation per SKU, so it isn't estimable here. The docstring used to claim DiD. It was wrong and I fixed it.
• Traction detection. velocity >= 0.15 over two scalars. No significance test, no small-N handling. min_prior_units gates on units sold, which is not sample size.
• Rename resistance. Exact match on an 11-field normalized tuple with the name left out. That's definitionally rename-proof but it isn't entity resolution. It does nothing clever, it just declines to look at the name. The RapidFuzz baseline it beats is a strawman: in the seed, names encode the specs, so a tokenizing baseline would win without any fingerprint.
• Normalization. Handles real variance ("16GB (1x16GB) DDR5-5200", "13th Gen Intel(R) Core(TM) i7-1355U") but has known silent failures: "2x8GB" parses to 8, 1TB vs 1000GB splits a cluster, Core Ultra generations collapse. The bad ones are bad because they're marked parsed rather than missing, so the guardrails don't catch them.
• The eval's independence. It imports nothing from recommend, but shares config and analytics. Against an external model's proposal it's a real check. Against the engine's own output it's a consistency check: break a shared threshold and the engine is wrong while the eval scores 1.0. The hand-authored labels in VERIFICATION mode are the load-bearing signal.

Scaffolding. Demonstrates a shape, doesn't do the work:

• Model routing. A cost table of hardcoded token estimates. Nothing imports it outside a display panel, and every live call uses one model. It shows what tiered routing would save, computed from constants. It is not measured savings.
• Warehouse, imports, and governance panels. Real code paths, but they write to a local filesystem and are inert on the read-only serverless deploy.

The data. 19 competitor SKUs, 21 observations, 2 periods, 10 hand-authored scenarios with hand-authored ground-truth labels written by me, in the same commits as the rules. There is no held-out set. Every threshold sits mid-plateau: sweep any one and all 10 scenarios stay byte-identical across a wide band, meaning the seed corpus cannot falsify the thresholds it ships to validate. The scenarios prove the mechanisms are wired end-to-end. They do not prove the detector is right.

What I'd do differently

Build a generator, not fixtures. This is the root cause of most of the above. Hundreds of SKUs with knobs for rename rate, unit-format chaos, missing-field rate, and a planted traction effect of known size. Then rename resistance is a precision/recall curve against the name baseline instead of six hand-picked pairs, attribution is "I planted +18% RAM-driven and recovered 16.9%" instead of a constant, and the thresholds get chosen by where the curve actually breaks.

One depth area, deep. The challenge asked for at least one. I shipped six, and the thinnest one (routing) is a net negative: it invites exactly the scrutiny the strong parts would have survived. Those six cost me the hours that would have made attribution real.

Write the docs last, and write them honest. The original TDD claimed a difference-in-differences estimator that doesn't exist. My docstrings were more accurate than my documents, which is a tell: the code knew what it was, and the write-up got polished past it. Nothing here would have been weaker for saying "this is a heuristic, and here's the interface where the estimator goes."

Run it

make demo                   # API + cockpit, end to end
cd engine && uv run pytest  # 260 tests: 259 pass, 1 skips without a key
uv run configpilot eval-all # all 10 scenarios vs. ground truth

Deterministic and offline by default, no key needed. To exercise the live proposal and veto loop, set CONFIGPILOT_AI_* (see .env.example). It's OpenAI-compatible or Anthropic-format, BYOK, server-side only. Without a key the "AI proposal" is the engine's own recommendation replayed through the verification path, which exercises the plumbing but cannot demonstrate a veto. The veto tests monkeypatch a hostile model to prove it, and one live test skips without a key.

Stack: Python 3.12, Pydantic, FastAPI, DuckDB, Polars, React, Vite, TypeScript, Tailwind. Roughly 16k LOC engine, 15k LOC cockpit, 260 tests across 28 files.

Docs: PRD | TDD | What I built | What I reused | What broke | AI implementation

---

Built by Mathew Stevens. Most of my work is trading and wallet infrastructure: SolPulse, Agentic, and TxShield, a pre-sign transaction-safety SDK. Those are the same problem as this one wearing different clothes. Something automated wants to act, and the job is deciding what it's allowed to do unsupervised. ConfigPilot is that boundary applied to a commercial decision instead of a signature.