# Tradewink Accuracy Methodology — What Our AI Can't Do

**Published:** July 2026
**Maintained by:** Tradewink Research
**Canonical page:** https://www.tradewink.com/ai-limitations/methodology

This document publishes the negative results from Tradewink's own research program:
the measured ceiling on directional accuracy, the failure of confidence gating
out-of-sample, the null results from our overlay backtests, the strategies we killed
on evidence, and the one bias in LLM trading we consider unfixable. Every number
below comes from a dated internal research document in the Tradewink repository,
cited inline. Nothing here is a hypothetical illustration.

Why publish this? Because a signals vendor that only shows you wins is asking you
to trust marketing. We would rather show you the boundary of what the system can
know — and explain what we optimize instead.

---

## 1. Out-of-sample directional accuracy sits at ~50-52%

**Source:** `docs/analysis/directional-accuracy-ceiling-2026-06.md` (2026-06-06);
reaffirmed as a standing prior in `docs/research/algo-improvement-research-prompt.md`
(§3, calibrated 2026-07-09).

We tested whether daily-bar direction (will this stock be higher or lower in H
trading days?) can be predicted meaningfully better than chance. Universe: 40
large-cap US tickers, 3-5 years of daily OHLCV. A walk-forward split (first 70% of
dates train, last 30% test, standardization fit on train only) with an
L2-regularized logistic regression over 11 technical features.

Unconditional out-of-sample accuracy:

| Horizon | OOS n | Accuracy |
|---------|-------|----------|
| 5 days  | 12,610 | 52.70% |
| 10 days | 12,550 | 52.25% |
| 20 days | 12,430 | 51.72% |

The base rate of up-days in the sample was ~54%, so these results are within about
two percentage points of guessing "up" every time. Our standing engineering prior,
verbatim from the research prompt that governs our improvement loop:

> Out-of-sample directional accuracy sits at ~50-52%. Near-100% is not achievable.
> Any candidate implying otherwise is mis-specified.

## 2. Confidence gating fails out-of-sample

**Source:** `docs/analysis/directional-accuracy-ceiling-2026-06.md` (2026-06-06).

The intuitive rescue — "only predict when the model is very confident" — was tested
directly. In-sample it works spectacularly: a blended confidence signal reached
74.7% accuracy at 1% coverage. Out-of-sample it inverts:

| Calibrated gate (H=10) | Fired | Coverage | Accuracy |
|------------------------|-------|----------|----------|
| P >= 0.55 | 6,518 | 51.9% | 51.64% |
| P >= 0.60 | 733 | 5.8% | 50.48% |
| P >= 0.65 | 45 | 0.4% | **37.78%** |
| P >= 0.70 | 5 | 0.0% | **20.00%** |

At the highest confidence thresholds the model is *worse than chance* — it learned
to be confident precisely where it overfit the training period. The in-sample lift
is explained in the source document by regime snooping, small-sample noise at 1%
coverage, and survivorship bias in the 40-ticker universe.

This is why Tradewink's confidence scores are documented as ranking/filtering
tools, not probabilities of profit (see the "Confidence scores" section of
https://www.tradewink.com/ai-limitations).

## 3. Daily-bar overlay backtests: a clean null over 27,320 candidates

**Source:** `docs/analysis/wired-flag-calibration-2026-06.md` (generated 2026-06-05);
summarized as a standing prior in `docs/research/algo-improvement-research-prompt.md` (§3).

We built five candidate ranking overlays (extra indicators, chart patterns,
candlestick patterns, volume spread analysis, cross-asset catch-up), wired them
behind default-off flags, and ran a counterfactual backtest: 40 large-cap tickers,
3 years of daily bars, 27,320 scored candidates, 10-day forward returns, 2,000
bootstrap resamples.

| Flag | n | Top-K lift | 95% CI | p-value | Verdict |
|------|---|-----------|--------|---------|---------|
| extra_indicators | 27,320 | -0.00036 | [-0.00101, +0.00035] | 0.8350 | not significant |
| chart_patterns | 27,320 | -0.00059 | [-0.00131, +0.00035] | 0.8820 | not significant |
| candlestick | 27,320 | -0.00001 | [-0.00110, +0.00103] | 0.5105 | not significant |
| vsa | 27,320 | +0.00006 | [-0.00095, +0.00111] | 0.4460 | not significant |
| catch_up | 27,320 | +0.00011 | [-0.00050, +0.00057] | 0.4475 | not significant |

Every confidence interval spans zero; every p-value exceeds 0.44. All five overlays
remain off in production. The standing rule this produced:

> Daily-bar backtests of ranking/scoring overlays return a clean null (5 flags,
> 27,320 candidates, p > 0.44). Do not backtest an overlay on daily bars and claim
> an edge.

## 4. LLM weight lookahead is unfixable — and how we contain it

**Source:** `docs/research/algo-improvement-research-prompt.md` (§3);
`docs/analysis/llm-multi-agent-trading-failure-modes-2026-04-23.md` (2026-04-23,
24-source research review incl. Look-Ahead-Bench arXiv 2309.17322, an NBER
lookahead covariance test, and Federal Reserve working paper 2025090 on correlated
model errors).

Frontier language models are trained on data that includes how tickers actually
performed. "Backtesting" an LLM-driven strategy over dates before the model's
knowledge cutoff is therefore not a valid out-of-sample test — the model may simply
be remembering the answer. There is no patch for this; it is a property of how the
models are built. Our standing prior, verbatim:

> LLM weight lookahead is unfixable. Frontier models know how tickers performed.
> LLMs stay a conviction multiplier on rule-based output, never the primary signal.

Containment measures in the Tradewink codebase:

- **LLMs never originate signals.** Rule-based screeners and strategy engines
  produce candidates; LLM output can only scale conviction on an existing
  rule-based setup.
- **Point-in-time context in simulations.** Any backtest or walk-forward run must
  pass an `as_of_ts` timestamp to the trade-reflection store and vector memory so
  the simulated system cannot consult lessons from trades that close in its
  "future." Live trading has no such restriction because live time only moves
  forward.
- **Single-model debate is labeled as such.** A one-call bull/bear "debate" shares
  the model's bias on both sides and is not treated as independent disagreement.

## 5. Strategies we killed on evidence

**Source:** `docs/research/algo-improvement-research-prompt.md` (§6 "Verified
negatives — do not rebuild", calibrated 2026-07-09).

Our research loop maintains a list of ideas that were proposed, researched, and
rejected. Rebuilding one requires new evidence, not a fresh citation of the
original paper. The current list:

1. **SPRT sequential promotion gate** — refuted by its own arithmetic; the claimed
   "decision in 12-18 trades" is impossible at the stated parameters.
2. **Scale-out / partial exits** — expectancy-negative; improves win-rate optics only.
3. **Equity-curve / loss-streak risk throttling** — reduces Sharpe and
   return/drawdown in testing.
4. **Loser time-stops on mean reversion** — pure churn, no edge.
5. **Book-level volatility targeting** (as a drawdown reducer) — the drawdown-reduction
   claim has a peer-reviewed refutation (Bongaerts et al., Financial Analysts Journal,
   2020). We do ship the exposure-governor variant of vol targeting as a sizing
   control, flag-gated off by default; it is the drawdown-reduction claim that was killed.
6. **Pre-earnings straddle** — 3-8% round-trip spread cost against a 1.9% gross
   in-sample edge.
7. **Month-end London 4pm fix FX flow** — decayed after the 2015 fix-window reform.
8. **Pre-FOMC drift; COT positioning on index futures; DXY-to-crypto lead-lag;
   broken-wing butterfly** — decayed, nil, null, and no published out-of-sample
   science, respectively.

## What we optimize instead

**Source:** `docs/analysis/directional-accuracy-ceiling-2026-06.md`
("Achievable Accuracy Targets").

Since directional hit-rate on daily bars is bounded near chance, the dimensions
where measurable improvement is possible — and where our engineering effort goes —
are:

| Dimension | What it measures |
|-----------|------------------|
| **Calibration** | Does a stated 65% probability win about 65% of the time? (Brier score, reliability diagrams) |
| **Integrity** | Are outcome labels in the database correct? (win/loss signs, stale positions) |
| **Expectancy** | Is average P&L per signal positive? A 50% hit rate can be profitable at 2:1 reward-to-risk. |
| **Execution** | Does the fill match the signal price? (slippage, post-fill drift) |

Discipline — sizing, stops, risk limits, honest bookkeeping — is the product.
Prediction is not.

---

## Scope and honesty notes

- The studies above are research-harness results on free daily OHLCV data
  (40 large-cap tickers, survivorship-biased by construction, as each source
  document discloses). They measure the *ceiling* of directional prediction, not
  Tradewink's live trading performance.
- These findings are why Tradewink ships trading disabled by default, in paper
  mode, with confirmation required.
- Nothing in this document is investment advice. Trading involves risk of loss.
  Past performance — and past *backtests* especially — do not guarantee future
  results.

*This file is republished from Tradewink's internal research corpus. If a number
here disagrees with the cited source document, the source document wins and this
file has a bug — please report it to support@tradewink.com.*