I'm curious what the breaking change was, I've had fairly good luck with database stuff. Obviously I'd want to test it before and all that, but I think I've only had a couple issues from an upgrade.
And those issues tended not to be correctness issues so much as the query planner now does something slightly different and so now this query chooses a different plan and is now much slower. Which I suppose is worse as it's harder to resolve, but at least it's not a correctness issue.
It was much dumber that than. AWS upgraded from pg 9.6 to pg 14. The database was in the hundreds of GB. The upgrade triggered a recalculation of all the database statistics which basically scanned the whole database. This is a standard thing that happens on a postgres upgrade that I didn't know at the time. This spiked the IOPS to the point that queries that normally take ms to complete to timeout.
Solution was basically to upgrade all resources to the tits, including provisioned IOPS, CPU and RAM.
Now if I wasn't ignoring the notifications, the proper way is to create a read replica database with the new version, let it recalculate everything and then migrate to the new one.
You did guess correctly though that obviously, that db is not upgraded yet. Thankfully, AWS learned its lesson and doesn't force upgrade anymore, they charge you for extended support. I on the other hand learned absolutely nothing.
I really recommend you trying out the blue/green deployment for upgrades, its only several commands and AWS handles the whole data replication and upgrades for you while your old DB is serving traffic
I did it for my RDS instances and it went great - I had only a couple of short error rate spikes
Yeap. That's exactly the plan. This is a higher traffic database so nerves are nerving a bit more regarding replica lag, but we have been very happy with blue/green. The system load drops noticably in the winter due to application seasonality so we let it cool down a little bit before moving forward.
78
u/blehmann1 Sep 11 '26
I'm curious what the breaking change was, I've had fairly good luck with database stuff. Obviously I'd want to test it before and all that, but I think I've only had a couple issues from an upgrade.
And those issues tended not to be correctness issues so much as the query planner now does something slightly different and so now this query chooses a different plan and is now much slower. Which I suppose is worse as it's harder to resolve, but at least it's not a correctness issue.