Walk-Forward Testing: Log Every Holdout Access

Walk-forward testing: Keep a dated access log so an evaluation window remains distinguishable from development data.

A walk-forward discussion described repeated tuning across annual slices. The useful distinction was between testing a frozen rule on later observations and revising that rule after seeing those observations. Calendar labels alone cannot establish that a test remained untouched.

The source discussion is a starting question, not verified strategy evidence. Accessible comments were reviewed incompletely; some replies remained unavailable. Author results are unverified. The exercise below is original educational reasoning with hypothetical inputs, not a reproduction of those results.

Walk-forward testing: walk-forward access log: record decisions first

Before another run, write down the candidate family, eligible training dates, selection metric, trading costs and next evaluation interval. Store the chosen configuration before opening the next interval's results. Record failed runs and rejected variants as well as successful ones. Otherwise the final report loses the research choices that helped produce its apparent winner.

An access log should distinguish reading inputs, computing a result and using that result to change a rule. A final test that fails is still consumed once its outcome informs development. Renaming the next experiment does not make the same observations fresh again. This is a recordkeeping problem as much as a modeling problem.

Worked example: three windows, two meanings

Imagine training on January through June, selecting in July and evaluating in August. Freeze the selected rule before August is scored. September can then be a new prospective window under a previously stated update schedule. The example describes an experimental sequence, not a recommended window length.

Now imagine reviewing August losses and changing the entry filter, then reporting the revised August score as untouched performance. The dates are chronological, but the experiment is retrospective. Keep that score as development evidence. Preserve the original August result and the reason for the revision so a reviewer can follow the sequence.

Test the boundary with an intentional failure

Use a tiny synthetic dataset and a deliberately poor candidate. Run development, lock the selection and consume the final interval. Attempt to evaluate a revised candidate on the same final interval. The workflow should clearly reject the untouched-test claim or mark the second evaluation as reused evidence. A failed candidate should leave a durable record rather than disappearing.

Next inspect what preprocessing saw. A chronological split can still leak if a scaler, feature selector or threshold was fitted on all rows before splitting. Rebuild each transformation using only its permitted training inputs. Save those fitted objects with the selection record so the test can be reconstructed.

Read the resulting report honestly

Keep development, validation and prospective performance in separate sections. Concatenating genuinely unseen evaluations can describe a specified update process; averaging development and final scores obscures their different roles. A useful report names the latest data that influenced a decision and the first data that evaluated it.

The access log does not eliminate uncertainty or prove an edge. It makes the claimed experimental boundary inspectable and exposes when further evaluation needs genuinely new observations.

Reference and first useful step

Research selection depends on the number of attempted configurations. The Probability of Backtest Overfitting.

Open Advisory Research to explore a local sample report and inspect its assumptions before comparing your own saved evidence. Choose a synthetic sample before importing offline JSON. Keep outputs advisory; this exercise does not place trades. For background, read backtesting fundamentals and paper trading.

Continue the research

Backtest History: Measure Market Regime Coverage and Backtest Slippage: Build a Daily-Bar Cost Frontier develop adjacent checks. Use a small synthetic fixture first; these educational diagnostics do not validate a trading edge or change account settings.