Overfitting is not a mistake the model makes. It is what happens when a model does exactly what it is told. Regularization in machine learning is not just a way to prevent overfitting. It changes what the model is allowed to learn in the first place. Instead of only minimizing error on the training data, regularization restricts the kinds of parameter values the model can use, which changes the solutions it can find.
In the previous post on how machine learning works, we saw how a model adjusts its parameters to minimize loss on the training data. That process is effective, but it does not distinguish between patterns that reflect the real world and patterns that exist only in the training data. Regularization addresses this by placing constraints on the optimization process so that solutions that assign very large weights to individual features are less likely to be chosen.
Regularization in Machine Learning
Regularization in machine learning changes the objective the model is optimizing. Instead of only minimizing loss on the training data, the model is trained to minimize a combination of the loss and a penalty on the size of its parameters, an additional term added to the objective that increases as parameter values increase.
Regularization changes which parameter values the model will consider acceptable, especially discouraging very large weights. The model is still trying to fit the training data, but it is also being guided toward solutions that are less extreme and more stable. As a result, the model tends to rely more on patterns that appear consistently across the data and less on patterns that depend on specific details of the training set.
Why Optimization Creates Overfitting
Training works by minimizing loss on the training data, but that process has a limitation. Given enough parameters and enough passes through the data, meaning repeated iterations over the same training examples, the optimization process will find parameter settings that make accurate predictions on the training examples by learning patterns that are not true outside of that dataset. This is not a failure of the training process, but a consequence of the objective.
The process does not distinguish between patterns that generalize and patterns that are specific to the training data. This is called overfitting in machine learning. The model has not learned the underlying pattern. It has learned the training data, including noise, coincidences, and quirks that do not hold more generally.
A Simple Regularization Example
In a customer proof of concept predicting whether a song would become a hit, a model trained on features like tempo, danceability, and release month produced strong results on the training set. When tested on new songs, it performed poorly. A disproportionate number of hit songs in the training data happened to be released in July, and the model learned that July releases were associated with success and assigned a large weight to that feature. That relationship held within the training data, but it did not hold more broadly.
Google Flu Trends showed a similar issue at a larger scale. The model used search query data, meaning the terms people were typing into Google, to predict flu outbreaks and initially performed well. These searches acted as a proxy for how many people were getting sick. Over time, it began to overestimate flu rates because it had learned that certain search terms were associated with flu cases in the training data, even when those terms were not actually caused by people being sick.
In both cases, the optimization process produced solutions that fit the training data closely but did not generalize well to new situations.
What Regularization Actually Does
Regularization in machine learning adds a penalty to the loss function, an additional term in the objective that discourages large parameter values. Instead of minimizing loss alone, the model minimizes a combination of the loss and this penalty, which increases as parameter values increase. This pushes the model toward solutions that balance fit with simplicity. The model can still learn that release month has some relationship to the outcome, but it will not assign an extremely large weight to that feature unless the pattern is consistent across many examples.
In the song example, regularization reduces the influence of July releases and shifts the model toward features that show more consistent behavior across the data. In the flu example, regularization would have reduced the influence of search terms that appeared predictive in the training data but were not consistently related to actual flu cases.
The tradeoff is worth understanding. Regularization will almost always make training performance slightly worse. The model is being prevented from fitting the training data as closely as it could, but that is the point. A small loss in training accuracy is the price of a model that holds up on new data.
L1 vs L2 Regularization
There are two common types of regularization in machine learning. L1 regularization, often called Lasso, pushes less useful parameter weights toward zero and can remove features from the model entirely. L2 regularization, often called Ridge, reduces the size of all parameter weights but typically keeps them in the model. In the song example, L1 might remove the July release feature from the model entirely if it is not consistently predictive. L2 would keep it but reduce how much it influences the final prediction. L1 is often used when many features may not be relevant. L2 is more appropriate when most features contribute something, but none should have an outsized influence.
Regularization is not limited to regression models. It applies to any model that learns parameters by optimizing a loss function, including neural networks, logistic regression, and support vector machines.
When to Use Regularization
Regularization is useful when a model performs well on training data but poorly on new data. This difference suggests that the model has learned patterns that do not generalize. Regularization helps reduce this gap by limiting how strongly the model can rely on any single pattern.
It is also useful when working with many features, especially when some features may reflect how the data was collected rather than meaningful relationships in the underlying problem.
When Not to Use Regularization
Regularization is not always appropriate. If the model is too simple and cannot capture patterns in the training data, adding regularization will make the problem worse by further limiting how much the model can adjust its weights. In that case, the issue is bias, and the model needs more flexibility, not further limits on its weights.
Regularization and Generalization
Regularization in machine learning helps manage the gap between training performance and performance on new data. It does this by changing the optimization objective so that the model is less likely to rely on patterns that are specific to the training data.
Every supervised machine learning model relies on the assumption that patterns in the training data reflect something about the real world. Regularization is one way to protect that assumption, but it is not the only one. How you collect data and how you split it for evaluation both shape what the model is exposed to. The model you choose determines how much flexibility it has to fit those patterns in the first place.
