predict-first

Say what will happen. Then measure. Then edit.

A skill for Claude Code and Hermes Agent, plus a plain checklist for humans. It is for every change whose success is a number, not a passing test: trading strategies, machine-learning models, simulations, solvers, performance work, data pipelines.

The problem

You change something in your bot, your model, or your simulation and want to know whether it got better. So you run it and look at the numbers. The catch: you only decide afterwards what you expected. If the numbers look good, you keep the change. If they look bad, you find a reason why that run does not count. Changes that only looked good by accident creep in one by one, and the code gets worse while you believe you are improving it.

Backtests are the worst case. A filter that "fixes" last year's crash looks brilliant on last year's data. That is not evidence. That is memory.

What the skill does

It makes the agent follow a fixed order before it touches your code:

1. Write down what should happen, with numbers. Not "it gets better" but "max drawdown falls below 12 %, hit rate stays above 55 %, and on the 2022 data it will probably do nothing." The last part is mandatory: every experiment needs one prediction against the change and one result that would prove the idea wrong.
2. Try it without editing your code. The agent writes a small script next to your project that loads your real program and swaps out the one function or parameter for the duration of the run. Your repository stays untouched. Old and new run on the same data and the same seeds.
3. Compare, line by line. Prediction next to result, in a table. Did the idea deliver what it promised? Predictions cannot be reworded after the run.
4. Only then make the change. Or drop the idea, and know why.

This is the trick science uses to stop lying to itself: decide beforehand what counts as success. Whoever forms the expectation after the result is always right. Whoever writes it down first can fail, and that is exactly what makes the result worth something.

Example: a volatility filter that fails out of sample

examples/trading/ has a small moving-average strategy on synthetic prices. The developer added a rule, "stay in cash when volatility is high", because it made the crash go away on the data they were looking at. Run the experiment:

cd examples/trading
python predict_first_vol_filter.py

predict-first: volatility filter for strategy.signal

  baseline, in-sample    return  +79.0%   sharpe  1.05   max dd  24.7%   trades 33
  filter,   in-sample    return  +56.3%   sharpe  1.08   max dd  13.7%   trades 31
  baseline, out-of-sample return +262.8%   sharpe  1.79   max dd  19.7%   trades 16
  filter,   out-of-sample return   -2.6%   sharpe -0.08   max dd   8.4%   trades 8

ID  Prediction                                                      Measured  Result
P1  in-sample Sharpe gain >= 0.3                                       +0.02  FAILS
P2  in-sample max drawdown shrinks >= 20 %                              +44%  holds
P3  out-of-sample Sharpe gain < half of in-sample gain (warning)     -75.50x  holds
P4  out-of-sample Sharpe not worse than baseline (falsifier)           -1.87  FAILS

VERDICT: falsified out of sample, do NOT adopt

The drawdown really did halve on the development data. That is what the developer saw. Everything else was wrong, and the predictions written before the run are what made that visible.

The second example, examples/simulation/, is a physics integrator with an energy leak. There the proposed fix holds every prediction, including the counter-prediction that it does not fix phase accuracy, and the verdict is "adopt". Both examples are pure Python, no dependencies, and run in under a second. The CI checks that they still reach their verdicts.

Install

Claude Code, skills CLI

npx skills add Flowbudget/predict-first

Claude Code, plugin marketplace

/plugin marketplace add Flowbudget/predict-first
/plugin install predict-first@flowbudget

Hermes Agent

hermes skills install https://github.com/ghraw/Flowbudget/predict-first/main/skills/predict-first/SKILL.md

Any agent that reads SKILL.md

Copy skills/predict-first/ into your agent's skills directory (~/.claude/skills/ for Claude Code, ~/.hermes/skills/ for Hermes). The skill follows the agentskills.io format.

After installing, ask the agent something like "does this fix work?" or "will this filter improve the strategy?" and it will run the protocol instead of editing first.

Without an agent

CHECKLIST.md is the same protocol as a list you can print. skills/predict-first/templates/predict_first_template.py is a starting point for your own experiment scripts, and templates/report.md is the verdict table for commit messages and pull requests.

What it is not

• Not a backtesting library, not a trading signal, not investment advice.
• Not a replacement for tests. If a unit test can decide the question, write the test.
• Not a statistics course. It does not choose your metrics. It makes you commit to them before you look.

Roadmap

• A hook that blocks edits to repository files while an experiment is open, so the rule is enforced rather than recommended.
• More examples: a machine-learning training change, a performance optimisation.

License

MIT