Backtest Selection Bias: Record Discarded Strategies

Backtest selection bias: Retain the search history that produced a selected strategy, including unsuccessful candidates.

An overfitting discussion highlighted extensive candidate and exit-rule selection. The reviewed Reddit post was published September 17, 2026. Its author results were not independently reproduced, and the available comment tree remains incomplete. The exercise below is our original research proposal, not a replication of the post.

The selected strategy inherits the history of the search that produced it. A promising result cannot be interpreted as though its entry rules appeared before anyone looked at the data. Keep failed candidates because they reveal the selection process, not because they deserve equal deployment attention.

Backtest selection bias: exercise: reconstruct a search ledger

Imagine ten entry families and five exits tested on one development interval. That is fifty declared configurations before considering sizing or asset filters. Give every configuration a stable ID, input hash and timestamp. Record the metric available when the decision was made and the reason each candidate was retained or rejected.

Choose one configuration using a rule written before the comparison. Freeze its feature construction, costs and execution assumptions. Evaluate it on a subsequent interval without editing the rules. A disappointing result remains part of the experiment. Changing an exit afterward creates another candidate rather than rewriting the old record.

Separate alerts from executions

If historical reconstructed alerts differ from stored live alerts, compare the information available at each decision. Do not silently substitute the reconstruction for the original signal stream. A replay of identical stored inputs helps locate a discrepancy, but matching output does not establish profitability.

Useful tests reject duplicate trial identities, prevent a final evaluation from entering candidate ranking, and detect a changed dataset under an unchanged label. Publish search scope beside the selected performance summary. Unlogged human decisions still limit the audit, so an incomplete ledger should say which choices were reconstructed from memory. A large trade count cannot erase repeated selection on the same history.

Additional validation

Create one ledger row for a human filter decision as well as for an automated trial. The analyst who discards a setup after inspecting its losses participates in the search process. Preserve the earlier candidate and its result. This record will not reconstruct forgotten choices perfectly, but it prevents later reports from shrinking the visible search history.

Evidence and workspace use

Primary reference: The Probability of Backtest Overfitting provides background for this experiment, not validation of the Reddit strategy. Treat invented numbers above as arithmetic fixtures. Preserve the dataset, configuration and timestamp conventions with the output so another researcher can inspect the same calculation.

Find the offline workflow through Tradewink Advisory Research. Choose a synthetic sample before importing offline JSON. Start in an advisory research workspace: save a small input fixture, write the expected result by hand, then compare the implemented output. Expand the dataset only after the fixture balances. Keep unresolved assumptions visible in the report and retain adverse outcomes. Synthetic research output does not establish live execution behavior. For sizing background, read position sizing; for broader context, see risk management.

Continue the research

Block Bootstrap Backtests: Test Block-Length Sensitivity and Cross-Asset Backtests: Reconcile Units and Cash develop adjacent checks. Use a small synthetic fixture first; these educational diagnostics do not validate a trading edge or change account settings.