Picture this. You’re tutoring a student for an upcoming math test. You help them solve dozens of practice problems over several days, and by the end, they’re getting every problem right. You feel confident they’ve mastered the material.
Test day arrives, and they completely bomb it. New problems that used the same concepts in slightly different ways were a total disaster. They memorized your practice problems but never actually learned the underlying math.
This is one of the most common mistakes I see in machine learning, and one of the most expensive. It’s why you can’t judge a model on data it has already seen. I’ve watched teams celebrate 97% accuracy only to deploy models that barely work, because they never tested on genuinely new data.
We’ve covered supervised vs. unsupervised learning and classification vs. regression. Now we’re hitting something that seems obvious but somehow trips up even experienced teams: the absolute necessity of testing your models on data they’ve never encountered.
The short version: Training data is the part of your dataset a model learns from. Testing data is a separate part the model never sees during training, held back to check whether it works on examples it hasn’t seen. A model that scores well on training data and poorly on testing data has memorized its examples without learning the pattern.
The Memorization Trap
Machine learning models are basically very sophisticated pattern memorizers. When I say they “learn,” I’m being generous. They’re really just adjusting thousands of internal parameters until they get good at predicting the examples you’ve shown them. No consciousness, no understanding, just mathematical optimization finding the best settings for your specific dataset.
Here’s the problem. Give a complex enough model enough time, and it’ll memorize your training data perfectly while learning absolutely nothing useful about the real world.
Think of it like learning to drive. Practicing in an empty parking lot (training) might make you feel confident, but the real test is navigating rush hour traffic you’ve never seen before (testing).
I saw this happen with a public health department that built what they thought was an incredible system for predicting disease outbreaks. 97% accuracy! They were ready to revolutionize epidemic response. But when I dug into their process, I found they’d accidentally included the reporting hospital’s ID number as an input feature. Instead of learning disease patterns, the model was literally just memorizing which specific hospitals had reported outbreaks in their historical data.
For new outbreaks in different regions, it was useless. After fixing this and testing properly, their accuracy dropped to 76%. Still decent, but a far cry from the 97% they thought they had.
How to Split Training and Testing Data in Machine Learning
The basic approach is straightforward:
- 70-80% for training
- 20-30% for testing
In practice: Import your data, randomly shuffle it, take the first 80% for training, save the last 20% for testing. Train your model only on the 80%. Then, and only then, see how it performs on that untouched 20%.
But the devil’s in the details. Random splitting works most of the time, but not always. If you’re working with time-based data like stock prices or sensor readings, you need to train on earlier data and test on later data. Otherwise you’re basically letting your model see the future, which works great until you try to actually predict the future.
Here’s the golden rule: your testing data stays completely locked away until the very end. Don’t peek. Don’t use it to choose features. Don’t use it to tune parameters. I’ve seen teams gradually corrupt their test sets by repeatedly checking performance and adjusting their approach. It’s like studying the actual exam questions. Sure, you’ll do great on the test, but you haven’t actually learned anything.
[The ML Advocate Monthly]
New ML Advocate posts plus the best machine learning and AI reads, once a month. No hype, no doom.
The Three-Way Split
That raises a practical problem. If you can’t look at test data during development, you need another way to tell whether your changes are helping.
Enter validation data:
- 60-70% training
- 15-20% validation
- 15-20% testing
Validation and testing data both hold back examples the model hasn’t trained on. The difference is when you look at them and what you do with the results.
| Training data | Validation data | Testing data | |
|---|---|---|---|
| What it’s for | Teaching the model | Tuning and comparing versions of the model | A final check on new data |
| When you use it | Throughout training | Over and over during development | Once, at the very end |
| Typical share | 60 to 70% | 15 to 20% | 15 to 20% |
| Think of it as | Homework | Practice tests | The real exam |
Think of validation as practice tests. You study (train), take a practice test (validate), adjust your approach, repeat. The real exam (test) stays hidden until you’re done.
This approach proved crucial on a water quality monitoring project where the stakes were high. The team iterated for weeks, getting 92% accuracy on their validation set for predicting contamination events. When they finally ran the untouched test set, they got 91%. Close enough that everyone trusted it would work in production, and it did.
When You Don’t Have Enough Data
Sometimes you’re working with limited data and can’t afford to set aside big chunks for validation and testing. Cross-validation helps here.
You split your training data into 5 or 10 pieces, train multiple models where each uses a different piece for validation, then average the results. It’s like taking multiple practice tests with different question sets.
This technique particularly shines in archaeological research where data is precious. A colleague’s project identifying pottery styles across ancient civilizations got a more reliable performance estimate from cross-validation than from a single split, which is meaningful when you’re trying to understand cultural connections across millennia.
Real Examples
House price prediction is a perfect beginner example. You’re predicting prices using square footage, bedrooms, and neighborhood. Train on houses sold in 2020-2022, test on houses from 2023. If your model only works on the old houses, it’s useless for today’s market, because housing markets shift constantly.
Music preservation efforts have gotten interesting results with this approach. Researchers trained models to identify and catalog endangered folk songs from field recordings using thousands of ethnomusicologist-labeled samples. The system learned to recognize specific musical patterns, vocal techniques, and instrumental signatures. After training, it could process new archive recordings and help preserve cultural heritage. The key was testing on recordings from completely different regions and ethnic groups than the training data.
I’ve seen agricultural optimization projects tackle this well, like crop yield prediction through soil analysis and weather patterns. Teams train on farms across the Midwest, then test on farms in completely different climates and soil types. No memorizing specific fields, just recognizing agricultural patterns that transfer across regions.
Mistakes I’ve Seen
Data leakage is sneaky. A linguistics research team I know normalized all their dialect data before splitting it. It sounds innocent, but the normalization process used statistics from the entire dataset, meaning information about the test set influenced the training data preprocessing. Performance looked amazing until they fixed it.
Time travel is another classic. There’s a climate modeling system that performed beautifully when researchers randomly selected test years from historical weather data. In production, it failed completely. Turns out random splitting had let the model use future climate information to predict past patterns. Once they properly tested forward in time only, reality hit hard.
Validation set overfitting happens when you iterate too many times on the same validation data. You start optimizing for that specific set’s quirks rather than general patterns. Sometimes you need to refresh your validation data if you have enough examples.
What Makes a Model “Good Enough”?
There’s no magic accuracy number that makes a model deployable. Context is everything.
Archaeological dating might need 95%+ accuracy because miscategorizing artifacts could rewrite history. Urban planning prediction might be valuable at 70% because the stakes are lower. Sometimes a 75% accurate model that explains its reasoning beats a 90% accurate black box.
The better question to ask is “Is my model good enough for this specific problem, with these specific consequences for being wrong?”
When Your Model Disappoints
Finding out your model doesn’t work as hoped is normal. I’ve been there more times than I care to admit.
New learners often panic when their test accuracy is lower than training accuracy. This is normal! A gap of 5 to 10 points is usually fine. It’s when training is 95% and testing is 60% that you know you’re in trouble.
Start with your data. Performance problems often trace back to data quality issues, not algorithm choice. Then look at where your model makes mistakes, and check for patterns or specific types of examples it struggles with.
If it performs poorly on both training and validation data, it’s probably too simple (underfitting). If it crushes training data but fails validation, it’s memorizing rather than learning (overfitting).
Machine learning is iterative. Most successful projects go through multiple rounds of “this doesn’t work, let me try something else.”
Common Questions
What’s the difference between training and testing data?
Training data is what the model learns from. Testing data is held back until the end to measure how well the model handles new examples. If you test on data the model already saw, all you’re measuring is its memory.
What’s the difference between validation and testing data?
You use validation data over and over while building the model, to tune settings and compare versions. You use testing data once, at the very end, as the final check. Once you start making changes based on test results, that test set stops being a fair measure.
What’s a good train-test split?
Most projects start with 80/20, or 70/15/15 when they also need validation data. With very large datasets, the test share can be smaller, since even 5% can mean thousands of examples. With small datasets, use cross-validation instead.
Why is my test accuracy lower than my training accuracy?
A small gap, around 5 to 10 points, is normal. A large gap usually means the model is overfitting: it learned the quirks of its training data instead of patterns that carry over to new data.
The Bigger Picture
Training vs testing might seem like a technical detail, but it’s really about the fundamental challenge of building systems that work in the real world, not just on the data you happen to have.
Our mental models so far:
- Supervised vs unsupervised: Do you have labeled examples?
- Classification vs regression: Predicting categories or numbers?
- Prediction vs inference: Do you need accurate answers or explanations?
- Training vs testing: Does it actually generalize?
These frameworks help you think systematically about any ML project, regardless of which specific algorithms you end up using.
Next up we’ll tackle bias vs variance: how models can fail in two completely different ways, and why finding the right balance beats chasing accuracy alone.
Ever had a model that looked amazing in development but flopped in production? What went wrong? I’m always curious about where teams hit these walls because the patterns tend to repeat across different domains. Tell us at hello@mladvocate.com.
