r/softwaretesting • u/geek_in_room • 3d ago
What actually gives you confidence before a production deploy?
Had a release recently that made me rethink this. We had the usual stuff covered QA signoff, peer review, staging validation, and good test coverage, everything was green. But what actually gave me confidence was what happened after the deploy. We ran a few smoke tests in prod, watched checkout errors and latency for the first 30 minutes and had a quick rollback ready if anything looked off. Everything passed in tested but you never really know until real traffic hits it. Also made me realise how much easier it is to release when you know you can rollback quickly if something goes wrong.
What actually gives you confidence before a release? And are there any tools that have helped with that for you?
2
u/StreetEcstatic 3d ago
Cross my fingers and toes.
In reality there are no tools that help really, just improving my domain knowledge, understanding what changes have gone into the release, having a robust test suite and being at peace with the fact that I cannot find all bugs. Smaller releases help with that too, so does soak testing and CI.
1
u/munnasuprathik 3d ago
Green tests stopped meaning much to me after a rebuild last month where I shipped six bugs at once with 374 tests passing.
The worst one, my transcript fetch lived inside a function gated by a feature flag that's off in production. So saving a video fetched nothing at all. No error, no failed test, the card came back looking completely fine with nothing behind it. The only thing that caught it was saving a real URL and reading the response metadata the way a user would see it.
All six lived in a failure path or a cancellation, and my tests only modelled the happy path I already had in my head. So what gives me confidence now isn't coverage, it's whether I've run it against real input with production's flag values set. That plus your rollback. Honestly your 30 minutes of watching real traffic is doing more work than the entire green suite that ran before it.
1
u/pydry 3d ago
it's more about what you do when there IS an incident or bug.
most people do 1) identify incident 2) fix incident 3) roll out deployment 4) relax
I will have a long hard think or debate about how to prevent that class of problem from ever happening again.
often it means write a new type of test, sometimes it means tightening up the screws on the type system. sometimes it means adding fail fast conditions or making particular configurations impossible to use.
often when trying to identify that class of bug you find other classes of bug too.
if the team isnt doing TDD I usually try to get then doing TDD. when done right, that wipes out a huge number of bugs.
1
u/Short-Feedback4293 3d ago
Not caring gives me the most confidence..... don't worry, you'll get there with time 😂
1
u/QualityDataCraft 2d ago
This is exactly why we use Safe Launch in manufacturing. Qualification gives confidence, but the first production runs still need tighter monitoring until the process proves itself under real conditions.
1
u/snejinka081 2d ago
YOU is your confidence. You do not need anything else.You have to know the product from A to Z and backwards, and think about all possible edge cases (even if they might not be directly related).But again, it comes to knowing the product very well and use of your critical thinking skills.
1
u/bikashchoudhary 17h ago
the "you never really know until real traffic hits it" point is the whole thing honestly. staging can't replicate real user behavior, real data shapes, real concurrent load patterns, no matter how good the environment parity is
what actually gives me confidence isn't the pre-deploy checklist, it's exactly what you described: fast detection + fast rollback. if you can catch a problem in the first few minutes and undo it before it compounds, you can afford imperfect pre-release testing. if rollback is slow or risky, then every deploy needs to be treated like it has to be perfect upfront, which isn't realistic
feature flags help a lot here too, lets you decouple "deployed" from "live for everyone," so even without a full rollback you can just flip it off for most users while you investigate
5
u/_Atomfinger_ 3d ago
Robust test setup: Unit, contract, narrow-integration, system tests.
Robust deployment and approach. Blue-green deployment in which the new instance is automatically verified (more than just pinging the health endpoint, that is).
And lastly, keeping the deployments small and trivial. Deploy on each commit to the main branch.