Inside BitBank · Engineering an Autonomous AI Research Lab · 6 of 10

Honest Backtesting: The Discipline of Trying to Be Wrong

2026-07-24 · 13 min read

A backtest should be an adversary, not a sales chart. Its job is to expose leakage, fragile assumptions, hidden turnover, and regime dependence before money or trust is at risk.

A backtest is a simulation of decisions under historical information. It is not a time machine, and it is not proof. Its quality depends on how faithfully it limits the strategy to what could have been known and traded.

The One-Row Error

The most damaging errors can look trivial. Calculate a signal with a candle's final close, then fill at that same close, and the strategy has traded at a price known only after the decision. Correctly shifting execution to the next available event can transform exceptional performance into ordinary performance.

Every signal therefore needs an information timestamp and every fill needs an execution timestamp. Tests should encode their ordering.

Why Random Splits Fail

Financial rows are dependent. Adjacent periods share volatility, trend, and overlapping labels. Randomly distributing them across training and test allows near-neighbours from the future to teach the model about the past.

Chronological evaluation preserves order. Rolling or expanding training windows simulate repeated historical decisions: train on what was available, freeze the candidate, test on the next unseen block, then move forward.

Purging and Embargoes

A 24-hour forward-return label near the training boundary reaches into the next day. If the test block begins immediately, training contains outcomes from the test period. A purge gap removes overlapping examples. An embargo can also leave separation after a test block when repeated folds or portfolio effects require it.

Model the Decision, Not Just the Prediction

Forecast metrics such as error, rank correlation, or classification accuracy are useful diagnostics. They are not trading results. The simulator must translate forecasts into positions and account for when positions change.

Fees, bid-ask spread, slippage, funding, borrow, minimum sizes, latency, partial fills, and unavailable markets can all alter the decision. Not every study needs a full exchange emulator, but every study should model the costs that could plausibly create or erase its edge.

Stress the Assumption That Pays

If profit depends on fast turnover, widen costs and delay fills. If it depends on shorting, stress borrow and funding. If it depends on a few crashes, remove or perturb those episodes. If it depends on cross-sectional ranking, reconstruct the point-in-time universe.

A candidate need not survive absurd conditions. The purpose is to locate its breaking point and decide whether the margin is credible.

Compare Fold by Fold

Two strategies experience the same market in each test fold. Comparing their difference within each fold removes some regime noise and answers the deployment question directly: did the candidate improve on the incumbent under identical conditions?

Report the distribution, not only the mean. Positive-fold count, median, worst fold, drawdown, turnover, and the share of gains concentrated in a few periods tell us whether the average describes a repeatable effect.

The Holdout Must Be Allowed to Say No

A final recent holdout should remain untouched during search. It is not another dataset to optimize after the first result disappoints. If the holdout vetoes a candidate, the correct output can be no deployment.

This happened in our Solana research. Older folds made parts of the sweep look promising, but recent evidence did not support promotion. Preserving that negative decision is more valuable than finding a narrative that rescues the model.

Beware the Backtest Overfitting Budget

Every feature, parameter, restart, and researcher choice consumes information from the data. Even if the final model is simple, the process that selected it may be extremely complex. Experiment tracking should count the search, keep locked tests, and demand replication when a winner emerges from a large population.

Reproducibility Is a Result

A run should identify its code revision, data manifest, configuration, random seeds, environment, model artifact, and output metrics. The simulator should be deterministic where possible. When GPU kernels or sampling introduce nondeterminism, multiple seeds quantify it.

Backtests Need Tests

We test accounting identities, position transitions, fee application, drawdown calculation, timestamp order, and known toy scenarios. A constant price with no trades should not make money. A single controlled trade should produce the manually calculated result. Changing future data must not change an earlier signal.

What a Pass Means

Passing validation means narrower evidence: under this recorded protocol, the candidate improved on the incumbent and survived specified challenges. It does not guarantee the market will repeat. That modest interpretation keeps the system ready to monitor, limit, and reverse its decisions.

Validation determines whether an idea deserves consideration. Risk engineering determines how much consequence that idea is allowed to have. Chapter seven examines uncertainty, abstention, exposure, drawdown controls, correlation, and why a forecast is incomplete without a risk policy.