THE MODEL — Backtesting & Validation OS Brain
THE MODEL — Backtesting & Validation OS Brain
For finding out whether a strategy works or whether you fitted it. One orchestrator plus ten specialists that split the data properly, measure degradation out of sample, and count how many variations you tried before you liked one.
Every backtest works. That is the problem. The validation script does the part people skip: it splits the series into in-sample and out-of-sample, reports the performance gap between them, and asks how many parameter combinations you searched. If you tried forty variations and kept the best, the best one looks good for reasons that have nothing to do with the market. It reports an expectation adjusted for that search, and the adjusted number is usually far worse. People do not enjoy running it, which is exactly why it is worth shipping.
backtest.py
Splits a return series into in-sample and out-of-sample windows, reports the degradation between them, and adjusts the result for how many parameter combinations were searched. Flags any strategy whose out-of-sample performance collapses. Every OS Brain in this drop carries its own executable component, so the judgement calls stay with you and the arithmetic stops being a matter of opinion. Runs anywhere Python runs, and the specialists still work on their own if you never open it.
Say it in plain words — it works out the rest
Inside · orchestrator + 10 specialists
Where it refuses to skip ahead
- No result without an out-of-sample window — an in-sample backtest describes the past, it does not test anything.
- No claim without the search count — the best of forty attempts is not the same finding as the first of one.
- Costs and slippage go in before the result — strategies that only work gross of costs do not work.
- A held-back set is used once — reusing it turns it into another in-sample window and it stops being evidence.