Backtest Reproducibility: Build an Audit Checklist

Backtest reproducibility: Save inputs, settings and accounting rules so a claimed failure can be checked independently.

A backtest diagnostic tool discussion raised reproducibility and resampling objections. The reviewed Reddit post was published September 7, 2026. Its author results were not independently reproduced, and the available comment tree remains incomplete. The exercise below is our original research proposal, not a replication of the post.

A report that claims to break a strategy should expose enough inputs to explain why. A chart can identify an interesting question while hiding a different trade definition, an undisclosed filter or an accounting mismatch. Audit the calculation before debating its visual presentation.

Backtest reproducibility: exercise: rebuild one diagnostic

Create six fictional closed trades with unique IDs and dollar outcomes of 20, 20, 20, minus 10, minus 10 and minus 10. Their total is $30. Permuting their order leaves that total unchanged but changes the path of cumulative profit. If a shuffle implementation changes the sum without a stated sizing or compounding rule, investigate the implementation first.

Now give the first three trades overlapping timestamps and a shared exposure limit. Resampling them independently may imply capital that the original account could not deploy. An equity-row experiment preserves the account's realized allocation decisions, although it still cannot create unseen market conditions.

Request an experiment packet

The packet should include the input hash, timestamp convention, source-close IDs, cost schedule, random seed, resampling unit, replacement rule and exact statistic. Include an example small enough to calculate by hand. Provide the formula for profit factor and explicitly state how zero losses are represented.

Validate that duplicated IDs fail, the same inputs reproduce the same result, and changing the seed changes sampled paths without modifying source observations. Reconcile fees in dollars rather than adding an unexplained percentage haircut at the end. A vendor's refusal to disclose strategy rules need not prevent these basic accounting checks, but it limits independent validation of signal generation.

Additional validation

Add a checksum for the six example outcomes and require it to survive every pure permutation. Save the sampled index sequence alongside each path so a surprising tail can be traced back to its source observations. If the tool cannot export that sequence, state the resulting audit limitation instead of describing the chart as independently reproducible.

Evidence and workspace use

Primary reference: The Probability of Backtest Overfitting provides background for this experiment, not validation of the Reddit strategy. Treat invented numbers above as arithmetic fixtures. Preserve the dataset, configuration and timestamp conventions with the output so another researcher can inspect the same calculation.

Find the offline workflow through Tradewink Advisory Research. Choose a synthetic sample before importing offline JSON. Start in an advisory research workspace: save a small input fixture, write the expected result by hand, then compare the implemented output. Expand the dataset only after the fixture balances. Keep unresolved assumptions visible in the report and retain adverse outcomes. Synthetic research output does not establish live execution behavior. For sizing background, read position sizing; for broader context, see risk management.

Continue the research

Backtest Selection Bias: Record Discarded Strategies and Block Bootstrap Backtests: Test Block-Length Sensitivity develop adjacent checks. Use a small synthetic fixture first; these educational diagnostics do not validate a trading edge or change account settings.