Backtest Holdout Leakage: Track Evaluation Reuse

Backtest holdout leakage: Treat an evaluation that influenced later rule changes as development evidence for that next candidate.

A strategy-search discussion questioned whether repeated holdout feedback remained independent. The reviewed Reddit post was published October 11, 2026. Its author results were not independently reproduced, and the available comment tree remains incomplete. The exercise below is our original research proposal, not a replication of the post.

A holdout is protected by how it is used. Its file name does not make it untouched. If someone reads its failures and adjusts filters, those failures have influenced the next candidate. The new candidate needs another evaluation boundary.

Backtest holdout leakage: exercise: label each access

Imagine a researcher selects one of twenty variants on development data. The first holdout run shows a weak morning session. They remove morning trades and rerun it. Record both accesses, the rule change and the person or process making the decision. The second result is development evidence for the revised rule, even if no numeric optimizer touched the file.

Before the next final evaluation, write down the selected configuration, data identity, execution assumptions and success criteria. Store a digest of the specification. If the final result is disappointing, preserve it rather than changing the rules inside the same reported experiment.

Audit the process boundary

Chronological splitting helps organize time but does not prevent manual feedback. A notebook, dashboard export or printed summary can consume a holdout. Access logging should therefore cover summaries as well as raw rows. Absence of a log is not proof that nobody looked.

Test that final results cannot enter ranking functions, altered inputs fail the frozen specification, and repeated accesses are recorded without replacing earlier events. Keep exploratory analysis in a clearly marked development area. A clean boundary can make a result easier to interpret, but it does not guarantee that the selected strategy will work in a new market regime.

Additional validation

Record who or what accessed the final summary and when. A scheduled job that writes a report can expose information before someone opens the underlying dataset. Separate technical reruns that reproduce an unchanged calculation from decisions that modify a strategy. Both deserve a record, but only the latter changes the research specification under evaluation.

Evidence and workspace use

Primary reference: The Probability of Backtest Overfitting provides background for this experiment, not validation of the Reddit strategy. Treat invented numbers above as arithmetic fixtures. Preserve the dataset, configuration and timestamp conventions with the output so another researcher can inspect the same calculation.

Find the offline workflow through Tradewink Advisory Research. Choose a synthetic sample before importing offline JSON. Start in an advisory research workspace: save a small input fixture, write the expected result by hand, then compare the implemented output. Expand the dataset only after the fixture balances. Keep unresolved assumptions visible in the report and retain adverse outcomes. Synthetic research output does not establish live execution behavior. For sizing background, read position sizing; for broader context, see risk management.

Continue the research

Trading Bot Winning Streaks: Audit the Evidence and Backtest vs Forward Test: Trace the First Divergence develop adjacent checks. Use a small synthetic fixture first; these educational diagnostics do not validate a trading edge or change account settings.