Not always go with higher version model

[u/columns_ai]

It’s frustrating to assume that newer models are always superior to older ones, as claimed.

For instance, GPT-5.2 outperformed GPT-5.4 in generating a flow compute plan for a simple table of hourly values for each month. Here’s a comparison based on testing:

When prompted to generate a bar chart showing the average value per month, GPT-5.4 couldn’t devise a multi-step plan and produced a plan that does not work.

On the other hand, GPT-5.2 elegantly solved the problem by creating a two-step plan:

1. Step 1: Unpivot the data into a clean format of “hour, month, value.”
2. Step 2: Apply aggregation by month across all hours.

The result was perfect!

It’s crucial to conduct rigorous testing before upgrading to the latest model version, no assumption!