Adaptive Backtest Overfitting: Log Research Decisions

Adaptive backtest overfitting: Record changes to datasets, translations and selection rules as well as numerical parameters.

An adaptive backtest needs a decision ledger broader than its parameter grid. Changing the universe, translating a rule differently, selecting a timeframe, or redefining a regime can influence the final winner. If only numerical settings are logged, much of the research process disappears from the evidence.

The Reddit discussion described many correlated strategy tests. Author outcomes and background remain unverified, and some replies were inaccessible. The workflow here is original; it neither reproduces those tests nor assigns them a statistical correction.

Adaptive backtest overfitting: define a research decision

A decision is a choice that can change the eligible hypotheses or their evaluation. Examples include replacing missing input data, switching from daily to hourly signals, correcting an ambiguous exit, adding a sector exclusion, and narrowing the performance window. Record a correction as a correction, but preserve the prior artifact and the reason it was rejected.

Not every run is an independent hypothesis. Repeating the same frozen specification with the same inputs is a reproducibility check. Nearby parameters can produce nearly identical positions. Conversely, a single manual choice after studying a loss cluster can introduce substantial selection. Raw counts alone do not describe this structure.

A synthetic search history

Start with an invented study containing three rule families and two timeframes: six declared candidates. Suppose one fails because required data is missing, two are neutral, and three look promising. Then a researcher studies the promising group and invents four regime filters. Log those four as new decisions tied to the inspected results, rather than replacing the initial six with four final rows.

If another tool supplied a candidate whose prior search history is unavailable, record that upstream history as unknown. One imported file is not evidence of one upstream trial. Preserve the acquisition time, source identity, available search metadata, and the subsequent local changes separately.

Make the ledger executable

For each entry save the parent decision, frozen rule text, code and config identity, data snapshot, availability cutoff, common evaluation calendar, and status. Include failures and rejected variants. Save metric units and portfolio basis so a later analyst cannot confuse standalone full-capital curves with a shared account.

Bailey and colleagues examine selection overfitting. Use that methodological concern to design evidence collection, not to invent an effective-trial count from missing records. A correction method requires its own assumptions and inputs; a ledger makes those assumptions reviewable but cannot recover undocumented experiments.

Reserve a genuinely later question

Freeze a selector and evaluation plan after development, then test it on subsequently reserved observations. Record when that interval was accessed. A later policy change consumes the information it used and becomes another development step. Report the complete adaptation schedule if the system repeatedly optimizes, rather than evaluating only the final setting as if it existed throughout history.

At Tradewink Advisory Research, begin with the offline study ledger guide. Create the six-candidate synthetic history and add the four subsequent filters with parent links. Verify that attempts to overwrite a frozen artifact fail and that missing outcomes stay visible. This evaluates research discipline; it does not establish an edge or authorize automatic strategy promotion.

Continue the research

Arbitrage Execution Risk: Audit the Unhedged Leg and Paper vs Live Trading: Define Broker Event Contracts develop adjacent checks. Use a small synthetic fixture first; these educational diagnostics do not validate a trading edge or change account settings.