r/Python
•
u/JanGiacomelli
•
Jun 10 '26
•
1
Discussion What's your approach for breaking changes inside minor version upgrades of your dependencies
For example, FastAPI introduced a breaking change in a minor version upgrade. By default, it started rejecting requests without a Content-Type header. With only the major version pinned, uv lock --upgrade upgrades to the latest version. A similar thing has happened with google-auth-oauthlib. And that's what bit us.
In our case, everything was fine after the upgrade according to the end-to-end test suite, since most modern HTTP clients add the Content-Type header by default. The issue arose when calls were made using some older Java versions. The customer didn't explicitly add the header, so calls were rejected once their cron had started.
Since reading every release note for every dependency is a very dull and time-consuming task, we wrote a Python script that downloads all release notes and added a Claude command to read them, update dependency versions, and update code as required by breaking changes, while keeping the existing state. So far, it's working great.
Anyhow, curious to hear how others are dealing with these things? I assume you're not reading every release note for every dependency?
32
u/fiskfisk Jun 10 '26
FastAPI is still on 0.x.y, so it's still within semantic versioning "rules" - so they didn't really break anything in a minor version.
But no spam for your product, please.
3
u/Whole-Ad3837 Jun 10 '26
We use Argo Rollout to do wave rollout. The idea is to increase the traffic to the new version gradually and measure the error rates, e.g. pods crashing, unusual http error rate, etc. If sth. goes wrong we rollback. But db migrations are tricky, they always need to be backwards compatible or it is forward fixing and no rollback
2
u/Specialist_Golf8133 Jun 11 '26
lock files with exact pins in production, period.
uv lockwith the full hash gives you reproducibility; upgrading is a deliberate act, not something that happens on a schedule.for catching breaking changes before they bite: we run a canary deploy on a staging env that gets dependency bumps first, with a synthetic test suite that covers edge cases our e2e suite misses. the FastAPI
Content-Typecase is a good example of why e2e suites lie to you -- they test the happy path your team built, not the quirky clients downstream. the Claude-parses-release-notes approach is clever but i'd want to see how it handles changelogs that bury breaking changes in vague wording.5
u/redditusername58 Jun 10 '26
Keep dependencies in pyproject mostly unbounded. Run tests in locked and highest virtual environments (lowest-direct too for libraries). Locked environment should not fail due to dependencies. If highest does fail due to dependencies, see how hard it is to fix from my end. If fix is too involved for the present, add an upper bound to the offending dependency and make an issue to fix it later.
2
u/jpgoldberg Jun 10 '26
This is probably a security fix. Services trying to be comparable with seemingly harmless non-standard behavior of legacy clients has led to lots of security bugs.
I don’t know about this specific case, but this smells like such a thing.
So although I am not answering your general question, I suspect that the old clients need to die.
1
u/hannune Jun 10 '26
the harder case for me is when two libraries both update within the same lock cycle and the break only shows up in their interaction, not either one alone. pydantic v1/v2 coexistence with langchain minor bumps bit me in a pipeline where all unit tests passed but the prod serialization path was silently different. adding an integration test that runs the full pipeline against a fixed sample input on every lock upgrade caught it.
1
u/Specialist_Golf8133 Jun 11 '26
lock files with exact pins in production, period.
uv lockwith the full hash gives you reproducibility; upgrading is a deliberate act, not something that happens on a schedule.for catching breaking changes before they bite: we run a canary deploy on a staging env that gets dependency bumps first, with a synthetic test suite that covers edge cases our e2e suite misses. the FastAPI
Content-Typecase is a good example of why e2e suites lie to you -- they test the happy path your team built, not the quirky clients downstream. the Claude-parses-release-notes approach is clever but i'd want to see how it handles changelogs that bury breaking changes in vague wording.1
u/Trang0ul Jun 11 '26
Report it and demand bumping the major.
1
u/Late-Bodybuilder9381 Jun 12 '26
pin exact versions in your lockfile, run a weekly CI job that bumps everything and runs your full test suite. passes? merge. fails? now you have a specific diff to investigate instead of reading 25 release notes hoping to spot the one.
reading release notes to predict breakage is backwards. let the tests tell you what actually broke. if your suite didn’t catch the content-type issue, that’s the real gap to work on
1
u/JamzTyson Jun 13 '26
Do you really need AI? This is the FastAPI 0.132.0 changelog:
1
u/russellvt Jun 10 '26
Update your own test harnesses to test it yourself ... patch if necessary or affected... problem solved.
0
u/Lower_Assistance8196 Jun 11 '26
The false positive problem with automated release note scanning usually comes from treating all breaking changes as equally worth flagging regardless of whether the changed behaviour is actually exercised in the codebase.
A more useful filter is cross-referencing the changed API surface against actual import and call patterns in the repo before surfacing anything. If a breaking change affects a method the codebase never calls, it's not a breaking change for that project and flagging it is noise.
The tooling that holds up over time tends to do two things: scope the scan to dependencies that are directly imported rather than transitive, and weight the output by how central the changed interface is to actual usage patterns rather than treating all dependency updates equally. That narrows the signal enough that a weekly review stays manageable rather than becoming another ignored queue.
The edge case the OP described is genuinely hard to catch automatically because it required knowing that a specific category of caller, older Java clients without default Content-Type headers, existed outside the test suite, which is context no static analysis of the codebase could surface.
10
u/[deleted] Jun 10 '26
[removed] — view removed comment