Confidence Alone Doesn't Guarantee Accuracy

No, a model cannot reliably achieve high directional accuracy by only predicting when it is confident. Our analysis showed that while raising a confidence threshold can improve in-sample accuracy on the remaining subset, this effect is often an artifact of overfitting and does not translate to out-of-sample performance. In fact, high-confidence predictions can sometimes be worse than chance.

What went wrong

We tested a recurring claim: that abstaining from predictions on low-confidence cases would recover a significant accuracy edge. We applied this to daily bars across 40 large-cap US tickers over three to five years, using free open-high-low-close data. Our baseline directional accuracy on daily bars was already hovering around 50 to 54 percent, which is at or near chance. When we implemented selective prediction by raising the confidence threshold, we observed an in-sample lift on the reduced set of predictions. However, this lift was illusory. Out-of-sample, these high-confidence buckets frequently performed worse than chance. The model had learned to be confident precisely in the areas where it had overfit the historical data.

This experience corroborated an earlier real-data backtest of default-off ranking flags within our Tradewink system. In that instance, every top-decile confidence interval spanned zero, indicating no reliable predictive power even at the highest confidence levels.

Why it happens

The core issue is overfitting. Machine learning models, especially those trained on historical financial data, are prone to learning noise and spurious correlations. When a model is trained to output a confidence score alongside its prediction, it can learn to associate certain patterns with high confidence. If these patterns are specific to the training data and do not generalize, the model will become "confidently wrong" on unseen data. Raising the confidence threshold effectively filters out predictions where the model is less certain, but if the model's certainty is misplaced due to overfitting, you are simply filtering for the most confidently incorrect predictions. This is an "artifact" of the training process, not a genuine improvement in predictive capability. The model isn't learning to be more accurate; it's learning to be more certain about its mistakes on the training set.

What we changed

Our product consequence from this investigation was clear: confidence scores are documented as a self-assessment of criteria match, explicitly not a probability of profit. We shifted our focus from trying to "filter" for accuracy via confidence to understanding the true nature of the confidence signal. Instead of treating confidence as a gatekeeper for profitable trades, we now use it as an indicator of how well a given data point matches the patterns the model has learned. This means confidence scores are not used to decide whether to trade, but rather to inform other aspects of our system, such as risk management or feature engineering. We also implemented more rigorous out-of-sample testing protocols, ensuring that any perceived gains are validated on data the model has never seen during training. This includes testing different confidence thresholds on separate validation sets before deploying to production.

How to check your own system

Before you trust a confidence threshold to filter your trading signals, run these checks:

1. Baseline Accuracy: Establish your model's raw directional accuracy on daily bars (or your chosen bar interval) without any confidence filtering. This should be at or near chance for a non-predictive model.
2. In-Sample Lift: Apply various confidence thresholds to your in-sample data. Observe if accuracy increases on the subset of predictions that meet the threshold. Note this as an "in-sample lift."
3. Out-of-Sample Validation: Crucially, apply the same confidence thresholds to a completely separate, unseen out-of-sample dataset. Compare the accuracy of the high-confidence predictions in this out-of-sample set against chance.
4. Confidence Interval Check: For your highest confidence predictions (e.g., top 10% by confidence score), examine if the confidence interval for the predicted move spans zero. If it does, the model is not confident enough to rule out a move in either direction, even at its peak certainty.
5. Compare to Baseline: If the out-of-sample accuracy of your high-confidence predictions is not significantly better than your baseline accuracy (and ideally, significantly better than chance), then your confidence threshold is not providing a true edge. If it's worse, you are likely confidently overfitting.

This systematic approach will reveal whether your confidence metric is a genuine signal of predictive power or a byproduct of an overfitted model.

Disclaimer

This article describes engineering decisions in a trading system. It is not investment advice. Trading involves substantial risk of loss and is not suitable for all investors. Past performance does not guarantee future results. Always do your own research and consider your financial situation before trading.