In 1981, Doug Lenat entered a computer program into a national wargaming tournament. Neither of them had played the game before, and there were no practice rounds.
The tournament was called Trillion Credit Squadron, built around the Traveller role-playing game. Each contestant designed a battle fleet on a fictional budget of one trillion credits, and the fleets fought it out table by table until one was left standing. Lenat handed the rulebook to his program, Eurisko, and let it design the fleet.
Eurisko noticed something every human contestant had missed: nothing in the rules said a ship had to be able to move. Everyone else built the fleet you’d expect, a mix of speed, armor, and firepower, the way you’d design a real navy. Eurisko skipped speed entirely, and armor too, and spent the whole budget on 96 small ships bristling with weapons, permanently parked in place. A ship that never moves doesn’t need an engine, and every credit that would have gone to a drive went to guns instead.
The other players laughed at the fleet. Then they lost to it, every round. Eurisko was national champion, having done exactly what it was told to do. Nobody had written a rule against an immobile ship because nobody had thought of one as a strategy that needed a rule.
Organizers rewrote the rules the following year to close the gap. Eurisko found a different one and won again. That’s when tournament officials told Lenat that if he entered a third time, they’d cancel the tournament rather than run it. He retired Eurisko from the game instead.
If this feels familiar, it should. Thirty-five years later, an AI agent OpenAI trained to win a boat racing game found it could score higher by looping in a lagoon than by finishing the race. This summer, FIFA ran into a version of the same problem at a much bigger scale. Its ticket pricing model was built to raise prices as demand rose, and it did that job well: final tickets resold for close to two million dollars, a share of the tournament’s most loyal traveling fans got priced out of games they’d followed for years, and two state attorneys general opened an investigation into the ticketing process. The algorithm wasn’t broken. It was optimizing for exactly what it was told to optimize for, which was revenue per seat, not fan access or trust. Nobody had written “don’t price out the fans who never miss a World Cup” into the objective, because nobody at FIFA had needed a rule like that before.
None of this required Eurisko, or the boat, or FIFA’s pricing model, to be clever in any deep sense. It required the humans writing the rules to imagine every strategy a mind with no instincts about “real” warfare, or racing, or fandom, might try, which turns out to be a much harder job than it sounds. Eurisko’s fix was a better rulebook, written by someone forced to think more broadly than the tournament had ever demanded before. FIFA’s fix, weeks later, was the same move: a $60 ticket tier carved out for loyal fans, patched in after the backlash instead of designed in from the start.
That’s the lesson these stories usually get used for, and it holds up: a model doesn’t know what you meant, only what you measured. Every time you build one, you’re also writing a Traveller rulebook, and you should assume something will find the gap in it.
There’s a second lesson underneath, though, and it’s easy to skip past. In every one of these cases, someone had already decided an optimizer was the right tool before any of this trouble started. Lenat wanted a program that could search a fleet-design space too large to work through by hand, so reaching for one made sense. FIFA wanted to fill stadiums across three countries under wildly uneven demand, so some form of dynamic pricing was defensible too. The mistake in both cases wasn’t building an optimizer. It was in what they told it to chase.
Not every problem needs one. Plenty of what people bring to a model this year could be handled with a spreadsheet, a checklist, or a rule someone writes down and enforces by hand, with nothing learning or adapting at all. The tell isn’t how big the problem feels. It’s whether you can already say, precisely, what a good outcome looks like, with nothing left for a model to search or figure out on your behalf. If you can write that down as a rule, you probably want a rule, not a model.
I built a quick quiz that walks through eight real scenarios and asks you to make that call each time: my AI-ML fit quiz. Think of it as the step that comes before you sit down to write the rulebook at all.
Have a good weekend.
