Survivorship Bias Tests: Match the Backtest Windows

Survivorship bias tests: Hold calendar windows and execution assumptions constant when comparing historical and surviving universes.

A large retail-strategy comparison raised survivor-universe and unmatched-period concerns. The reviewed Reddit post was published October 10, 2026. Its author results were not independently reproduced, and the available comment tree remains incomplete. The exercise below is our original research proposal, not a replication of the post.

A comparison of current survivors with a historical universe is useful only if other important assumptions align. If one sample includes a different market period, the observed performance gap mixes universe construction with calendar effects. An audit should identify that ambiguity before assigning a cause.

Survivorship bias tests: exercise: build a matched comparison

Create a fictional historical universe with three securities, one of which later disappears. Construct two tests on the same twelve dates: all securities eligible at each historical cutoff, and only the later survivors. Keep warm-up, capital, entry rule, exits and costs identical. Report missing and delisted observations instead of deleting inconvenient rows.

Then run the survivor set on a different twelve-date interval. Mark that result as a separate experiment. It cannot estimate the effect of survivorship on the original dates because the market path also changed.

Distinguish screens from evidence

A fast accelerator may be useful for ranking rough candidates, but its assumptions must be visible. Same-close gross screening is not interchangeable with next-open net execution. A candidate that survives a screen should pass an executable accounting model before stronger claims appear in the report.

Useful tests require equal evaluation starts after warm-up, ensure future membership changes cannot alter earlier eligibility, and settle a disappearing instrument under an explicit policy. Compare random or simple controls under the same executable assumptions. A large number of variants is search scope, not independent confirmation. Keep the discarded variants in the research ledger and reserve a final interval from further tuning.

Additional validation

A matched-window table should show the count of eligible instruments at each date, not merely a final universe size. Add a test security that disappears halfway through the fixture. Inspect the final cash bridge and the last available mark. A policy that silently removes its position can improve the survivor comparison without producing a real economic settlement.

Evidence and workspace use

Primary reference: The Probability of Backtest Overfitting provides background for this experiment, not validation of the Reddit strategy. Treat invented numbers above as arithmetic fixtures. Preserve the dataset, configuration and timestamp conventions with the output so another researcher can inspect the same calculation.

Find the offline workflow through Tradewink Advisory Research. Choose a synthetic sample before importing offline JSON. Start in an advisory research workspace: save a small input fixture, write the expected result by hand, then compare the implemented output. Expand the dataset only after the fixture balances. Keep unresolved assumptions visible in the report and retain adverse outcomes. Synthetic research output does not establish live execution behavior. For sizing background, read position sizing; for broader context, see risk management.

Continue the research

Backtest Holdout Leakage: Track Evaluation Reuse and Trading Bot Winning Streaks: Audit the Evidence develop adjacent checks. Use a small synthetic fixture first; these educational diagnostics do not validate a trading edge or change account settings.