Neural networks for price prediction: why they overfit so easily

Train a neural network on five years of 5-minute Nasdaq bars and ask it to call the direction of the next bar. With enough layers and enough training, it will get there: 90%, 95%, sometimes higher, on the data it was trained on. Show it next month's bars and the number collapses toward a coin flip.

That gap is not a bug in the code or a shortage of GPU time. It is the predictable result of pointing a very flexible function at data that is mostly noise, and it is the most common way people waste a year on neural network trading.

What a neural network is actually doing

A neural network is a function approximator. You give it inputs (past returns, volume, indicator values) and a target (the next bar's direction), and it adjusts millions of internal weights until its output matches the target on your training data.

Nothing in that process asks whether the pattern will exist tomorrow. The optimizer has one job: reduce the error on the examples in front of it. If the only way to cut that error further is to memorize the odd sequence of bars on one Tuesday in March 2021, it memorizes that Tuesday. It has no concept of "this was a coincidence."

Three properties of market data that break it

Neural networks work extremely well on images and language. Financial price series have three characteristics that image and language data do not.

The third point is the one people underestimate. Five years of 5-minute bars sounds like a lot: roughly 100,000 rows. But consecutive bars are highly correlated, they come from one instrument, and they share a handful of market regimes. The number of genuinely independent observations is orders of magnitude smaller than the row count, while the model has millions of parameters. That ratio is the definition of a setup that fits noise.

You do not have 100,000 examples. You have a few dozen market regimes, sampled 100,000 times.

Why the accuracy number lies

Almost every impressive result you will be shown is measured on data the model has already seen, or on data contaminated by it. Three mechanisms do most of the damage.

The third is the quiet killer, because it feels like diligence. Every architecture you discard on the basis of test performance spends a little of that data's independence. Run enough trials and you will find a configuration that looks excellent on any dataset, including pure random numbers. That is the core argument of the backtest-overfitting literature, and it hits neural networks hardest because they are so easy to run hundreds of times. The general version is in overfitting and curve fitting, and the partial defense is in in-sample versus out-of-sample.

What doing it right would actually require

Quantitative firms do use machine learning, so the honest position is not "it never works." It is that the requirements are steep and almost nobody at home meets them.

• Purged, walk-forward validation. Test only on data that comes strictly after the training period, with a gap so that labels built from future bars cannot leak backward. Explained in walk-forward testing.
• A trial budget you count. If you tested 300 configurations, the best score must be discounted for that. Most people neither count nor discount.
• Cross-sectional breadth. Hundreds of instruments, not one index, so the model sees partly independent examples of the same phenomenon.
• Modest targets. Predicting a small edge in the distribution of outcomes, not the next bar's direction.
• Costs inside the objective. A model that is 52% accurate on a move smaller than the spread plus commissions has found nothing tradable.

That last one ends more projects than any statistical issue. Slight predictive accuracy is common. Accuracy that survives commissions and slippage is rare. More in machine learning in trading.

Why retail "AI predicts price" claims fail

When a product says it uses AI to forecast price, there are only a few possibilities, and most are bad.

1. It is an in-sample result. Impressive, meaningless, and the most common case.
2. It is a real but tiny edge, quoted before costs. Correct arithmetic, wrong conclusion.
3. It is a rule-based system with AI in the marketing copy. Very common, and not necessarily bad, but the label is doing no work.
4. It works and is not for sale. A genuine, capacity-limited edge gets traded, not licensed for $99 a month.

The alternative is not "no math", it is fewer parameters

Note the geometry rather than any hit rate. With the target at three times the stop, break-even arrives at a 25% win rate, and that is the point. The system does not predict the next move. It takes a small structural advantage and lets a 3 to 1 payoff do the work over hundreds of trades. A model that claims to call direction correctly most of the time is claiming something categorically different, and much less believable. The comparison is in AI versus rule-based trading, and the wider picture in the AI trading guide.

Frequently asked questions

Can a neural network predict price at all?

It can find genuine statistical structure, but usually a small amount, and rarely enough to survive transaction costs on one instrument. Institutions extract value from machine learning by applying it across hundreds of markets with strict validation and very low costs, not by forecasting the next bar of one index. Treat any claim of reliable directional prediction as an in-sample result until it is shown on data the model never saw.

Why does more data not fix overfitting here?

Because more rows are not more information when the rows are correlated and come from the same regimes. Ten years of one index gives more bars but not many more independent market environments, and the model still has millions of parameters to spend. Length also makes non-stationarity worse, since the oldest data came from conditions that no longer exist. Breadth across instruments helps far more than length on one.

Is deep learning worse than simpler models for this?

For most retail applications, yes, and for a specific reason: capacity. A model with more parameters can fit more noise, so with a low signal-to-noise ratio it will. Simpler models with few parameters are more constrained, which makes their failures more visible and their successes more likely to hold up. That is the same reason a rule-based strategy with a handful of conditions is easier to trust than one with forty tuned thresholds.

What single check exposes an overfit model fastest?

Run it on a time period that was never touched during development, in strict chronological order, with realistic commissions and slippage applied. If performance drops sharply, the model learned the training data rather than the market. Do this once and honestly, without going back to adjust anything afterward, because the moment you tune on that period it stops being out-of-sample and becomes part of the training set.

Educational content, not investment advice. Trading futures involves substantial risk of loss and is not suitable for every investor. Any reference to Algomyth results is hypothetical, simulated over historical data, and shown in full on the performance page; past performance does not guarantee future results.