Overfitting is not a model failing to learn. It is a model that learned exactly what it was shown, and nothing more.
A model with overfitting in machine learning has adjusted its parameters so precisely to the training data that it picks up patterns specific to that dataset rather than patterns that hold more broadly. Parameters are internal values adjusted during training to make predictions. The number of parameters is determined by how the model is designed. When applied to new data, its predictions become unreliable. The model looked excellent during training. The problem only becomes visible when you test it on data it has never seen.
A model that overfits will score well on every metric you check during training, meaning its error rates look low and its accuracy looks high. The problem is that training metrics only measure performance on data the model has already seen.
Why the Training Process Produces Overfitting
Machine learning models are trained by minimizing loss, a measure of how wrong the model’s predictions are on the training data, meaning how far off its predictions are from the correct answers. During training, the model adjusts its parameters to reduce that error as much as possible.
That process has no way to distinguish between patterns that will hold on new data and patterns that exist only in the training set. The model does not make that distinction. It adjusts whatever reduces error right now.
Given enough flexibility, meaning enough parameters relative to the amount of training data, a model will learn both kinds of patterns. A simple model, one with few parameters, might miss real patterns entirely. A complex model, one with many parameters, might learn everything, including the noise.
This is not a flaw in the training process. It is what the training process is designed to do. Minimizing error on training data and learning patterns that generalize are not the same objective, even though they often move together. Overfitting is what happens when those two objectives come apart.
What Overfitting in Machine Learning Looks Like
A healthcare company built a model to predict which patients would be readmitted to the hospital within thirty days of discharge. Trained on two years of records, it performed well. When tested on new patients, accuracy dropped noticeably.
The investigation turned up something odd. The model had learned patterns tied to how a specific group of physicians documented certain conditions during that two-year window. That documentation practice had since changed. The model had not learned to predict readmission. It had learned to recognize a habit that no longer existed.
A retail bank ran into a version of the same problem. Their default prediction model trained well, then degraded when evaluated on applications from the following year. The training data covered a period of unusual economic stability, and the patterns the model had relied on weren’t reliable outside that window. When conditions shifted, the predictions did too.
Both models learned something true about their training data. Neither learned something that transferred. That’s what overfitting actually produces. Not a model that fails obviously, but one that looks right until you look closely at what it learned.
When to Address Overfitting and When Not To
The clearest signal is a gap between training performance and performance on new data. That gap tells you the model has learned something it shouldn’t have, and there are a few ways to address it.
Regularization adds a penalty, an additional term in the calculation that increases as parameter values grow, that discourages the model from relying too heavily on any single feature, meaning any input variable it uses to make predictions. The model can still learn that a feature is predictive, but it cannot assign that feature an outsized weight unless the pattern holds consistently across many examples. In the loan default case, that would have reduced the influence of patterns tied to a specific economic window and pushed the model toward features with more stable behavior.
Cross-validation tests the model across multiple splits of the training data into separate portions rather than a single held-out set. Training and evaluating across different portions gives a more reliable picture of how the model will perform on data it hasn’t seen, and often surfaces overfitting before it reaches deployment.
More data helps too, for a straightforward reason. A coincidence that appears in five hundred examples is less likely to survive in five thousand.
If a model performs poorly on both training data and new data, the problem is different. This is called high bias, meaning the model is too simple to capture the real patterns in the data. Adding constraints will make that worse. The fix is more flexibility, not less. Poor performance on both points to bias. Strong training performance with poor test performance points to overfitting. They are different problems and they require different solutions.
Overfitting and the Broader ML Problem
Every model is trying to solve the same underlying problem: learn enough to generalize, but not so much that it memorizes. That balance shapes model selection, data collection, and how you measure whether training actually worked. Overfitting in machine learning is what happens when that balance tips too far. The tools for addressing it, regularization, cross-validation, and data collection, only make sense once you understand why the training process produces it in the first place.
This post is part of a series on why machine learning models fail in production and how to diagnose them. For more information:
What Regularization Does and Why Your Model Needs It
What Is Cross-Validation in Machine Learning
Bias vs. Variance: Why Your ML Model Can’t Have It All
