The numbers behind what our AI can't do

We tested whether daily-bar direction (higher or lower in H trading days?) can be predicted meaningfully better than chance: 40 large-cap US tickers, 3-5 years of daily bars, a walk-forward split (first 70% of dates train, last 30% test), and an L2-regularized logistic regression over 11 technical features.

The base rate of up-days in the sample was ~54%, so these results sit within about two percentage points of guessing “up” every time. Our standing engineering prior, verbatim: “Out-of-sample directional accuracy sits at ~50-52%. Near-100% is not achievable. Any candidate implying otherwise is mis-specified.”

Source: docs/analysis/directional-accuracy-ceiling-2026-06.md (2026-06-06); docs/research/algo-improvement-research-prompt.md, Priors (calibrated 2026-07-09).

The intuitive rescue — “only predict when the model is very confident” — was tested directly. In-sample it works spectacularly: a blended confidence signal reached 74.7% accuracy at 1% coverage. Out-of-sample it inverts (H=10):

At the highest thresholds the model is worse than chance: it learned to be confident precisely where it overfit the training period. The source document attributes the in-sample lift to regime snooping, small-sample noise at 1% coverage, and survivorship bias in the universe. This is why our confidence scores are documented as ranking and filtering tools — not probabilities of profit.

Source: docs/analysis/directional-accuracy-ceiling-2026-06.md (2026-06-06).

We built five candidate ranking overlays (extra indicators, chart patterns, candlestick patterns, volume spread analysis, cross-asset catch-up), wired them behind default-off flags, and ran a counterfactual backtest: 40 large-cap tickers, 3 years of daily bars, 27,320 scored candidates, 10-day forward returns, 2,000 bootstrap resamples.

Every confidence interval spans zero; every p-value exceeds 0.44. All five overlays remain off in production. The standing rule this produced: do not backtest an overlay on daily bars and claim an edge — validate via live shadow-attribution instead.

Source: docs/analysis/wired-flag-calibration-2026-06.md (generated 2026-06-05); docs/research/algo-improvement-research-prompt.md, Priors.

Frontier language models are trained on data that includes how tickers actually performed. “Backtesting” an LLM-driven strategy over dates before the model's knowledge cutoff is therefore not a valid out-of-sample test — the model may simply be remembering the answer. There is no patch for this; it is a property of how the models are built. Our standing prior, verbatim: “LLM weight lookahead is unfixable. Frontier models know how tickers performed. LLMs stay a conviction multiplier on rule-based output, never the primary signal.”

• LLMs never originate signals. Rule-based screeners and strategy engines produce candidates; LLM output can only scale conviction on an existing rule-based setup.
• Point-in-time context in simulations. Every backtest is required to pass an as-of timestamp to the trade-reflection store and vector memory, so a simulated system cannot consult lessons from trades that close in its “future.”
• Single-model debate is labeled as such. A one-call bull/bear “debate” shares the model's bias on both sides and is not treated as independent disagreement.

Source: docs/research/algo-improvement-research-prompt.md, Priors; docs/analysis/llm-multi-agent-trading-failure-modes-2026-04-23.md (2026-04-23, 24-source review incl. Look-Ahead-Bench arXiv 2309.17322 and Federal Reserve working paper 2025090 on correlated model errors).

Our research loop maintains a list of verified negatives: ideas that were proposed, researched, and rejected. Rebuilding one requires new evidence, not a fresh citation of the original paper.

1. SPRT sequential promotion gate — Refuted by its own arithmetic; the claimed "decision in 12-18 trades" is impossible at the stated parameters.
2. Scale-out / partial exits — Expectancy-negative; improves win-rate optics only.
3. Equity-curve / loss-streak risk throttling — Reduces Sharpe and return/drawdown in testing.
4. Loser time-stops on mean reversion — Pure churn, no edge.
5. Book-level volatility targeting (as a drawdown reducer) — The drawdown-reduction claim has a peer-reviewed refutation (Bongaerts et al., Financial Analysts Journal, 2020). We do ship the exposure-governor variant of vol targeting as a sizing control, flag-gated off by default — it is the drawdown-reduction claim that was killed.
6. Pre-earnings straddle — 3-8% round-trip spread cost against a 1.9% gross in-sample edge.
7. Month-end London 4pm fix FX flow — Decayed after the 2015 fix-window reform.
8. Pre-FOMC drift; COT on index futures; DXY-to-crypto lead-lag; broken-wing butterfly — Decayed, nil, null, and no published out-of-sample science, respectively.

Source: docs/research/algo-improvement-research-prompt.md, Verified negatives (calibrated 2026-07-09).