When the Holdout Says No: Why We Did Not Deploy Our SOL Sweep
2026-07-12
Not every optimization round should end with a new model. After improving BTC and ETH, we applied the same 70-fold protocol to SOL. The deployed adaptive selector had a weak baseline of -1.011% per 504-hour fold, so there appeared to be plenty of room for improvement. A grid of bounded-context trend, long/cash, long/short, volatility-target, and exposure variants produced several attractive development results—but none passed the recent holdout.
The Tempting Result
The leading conservative development configuration averaged about +0.77% over the older 46 folds with a 3.93% worst fold drawdown. Higher-exposure variants exceeded +2% on that same development segment. If we had ranked only those rows, the sweep would have looked like a clear success.
The Veto
On the untouched recent 24 folds, the conservative development winner averaged -0.24%. Every leading long/cash configuration was negative. The strongest recent long/short variant was still around -0.07%. Because our promotion rule requires positive performance in both older and recent regimes, SOL kept its incumbent and no strategy change was deployed.
Best Practice: Define the Veto Before Seeing the Result
A holdout is useful only if it can stop a launch. Lowering the threshold after seeing a disappointing result turns evaluation into another training step. We predeclared that a pair-specific override must beat the same incumbent, remain positive in the recent holdout, respect a drawdown cap, and survive higher costs. SOL failed the recent-regime requirement, so headline development P&L could not compensate.
Negative Results Reduce Future Overfitting
Publishing a rejection prevents repeated searches over the same idea and makes the true number of attempted experiments visible. This matters because selecting the maximum from hundreds of configurations creates its own optimism. The next SOL round should introduce genuinely new information or a different causal hypothesis, not progressively tune the same EMA grid against the holdout.
Backtests do not guarantee future results, and a rejected model can later become useful under a new protocol. For now, the evidence supports no SOL deployment. Knowing when not to ship is one of the most valuable outputs of a disciplined ML system.