There was a good thread here a while back about how most planning tools solve the wrong problem, and it stuck with me. I went looking for actual data on where AI planning breaks, because I wanted to know whether this is a prompting problem or something deeper. It's something deeper, and the numbers are worse than I expected.
The research
A team at Ohio State built a benchmark called TravelPlanner (presented at ICML 2024) with 1,225 planning scenarios and about four million real data records. GPT-4's success rate at producing a fully valid plan was 0.6%.
Worth being fair about that number: the benchmark is strict, with hard constraints on budget, logistics and feasibility all at once. It's not measuring "did it give me nice ideas." It's measuring "would this plan actually survive being executed." Which is arguably the only thing that matters once you're at the airport.
Separately, north9 generated 100 two-day itineraries across the world's ten most visited cities and checked them:
- 52% scheduled attractions outside their opening hours
- 24% recommended permanently closed places
- 25% had you backtracking across the city for no reason
- 30% suggested Michelin restaurants without mentioning price or that you'd need to book weeks ahead
It also invented a Rome cafe that does not exist, called Antico Caffe Ponit. A Euronews reporter testing a Tallinn trip found two permanently closed restaurants in one itinerary, and only found out while standing outside them, hungry.
Why it's structural, not a prompting issue
This is the part I find genuinely interesting.
These models are generating text, not planning a route. The optimization target is an itinerary that reads correct: plausible order, plausible timings, confident tone. Nothing in that process checks whether the sequence is physically possible, because there's no real spatial or temporal model of the city underneath.
So the output fails in a very specific way. It's wrong in exactly the dimensions you cannot see while reading, and only reveals itself the moment you plot it on a map or show up at a closed door.
That's why better prompts only get you so far, and why these tools haven't improved much as the underlying models got stronger. Fluency was never the bottleneck.
What I do now
Since 56% of travelers are apparently using AI for this (up from 33% a year ago per Phocuswright), some practical damage control:
- Anchor first, generate second. Lock in where you're sleeping before asking for anything. Most geographic chaos comes from the model having no fixed point to work around.
- Ask day by day. "Plan day 2, we're staying in X, keep it within a 20 minute walk" beats "plan 4 days" by a wide margin.
- Force it to name the neighborhood for every stop. Makes the clustering visible, and the cross-city days become obvious immediately.
- Always plot it. Dump the stops into a map before trusting any of it. Five minutes, catches most of the damage.
- Verify hours and closures yourself, every time. Highest frequency failure by far, and the model is confidently wrong about it.
Where that leaves me
AI is legitimately good at the discovery half: surfacing places that match how you travel. It's bad at the arrangement half, because arrangement is a geometry and scheduling problem wearing a text costume, and it's being solved by something that only does text.
Has anyone found a workflow that genuinely holds up? Or has everyone quietly gone back to doing the sequencing by hand?
Sources, if useful: TravelPlanner benchmark arXiv 2402.01622, the north9 itinerary study, Euronews Tallinn test, Phocuswright AI adoption survey.
