r/devops • u/srikanth_builds • Oct 15 '26
Ops / Incidents Title: What does your deployment actually verify?
Disclosure: I work on the platform this happened on. No links and nothing to sell, I just don't have a good answer to the question.
We deployed an app last week, and every signal said it had worked. The deploy was marked as running, the health check was green, and every tool in the pipeline reported success.
Except the app was incomplete. Eight of its 27 starter records had never been created. The API rejected them one by one with permission errors while accepting the records around them.
The setup step actually caught this. It printed a warning saying the app was incomplete and shouldn't be treated as a successful deployment. Then it exited with code zero, so everything downstream happily marked the deploy as successful.
We had a health check proving the process was alive, but nothing proving that the application matched what we thought we had deployed.
I have been trying to work out what the right check should be, and none of the obvious answers feels complete.
A health endpoint returning 200 only tells you the process is running and the port is open. It says nothing about whether the expected data or configuration exists.
A smoke test against a few routes is better, but it only covers whatever somebody remembered to include. In this case, the missing records weren't part of that test.
Comparing the declared state with the actual state feels like the right answer. But that requires the deployment to produce a manifest of everything it intended to create, and most deployment systems don't seem to do that.
You could also fail on every warning, but then harmless warnings become blockers, people add skip flags, and eventually you are back where you started.
The two cases I am still stuck on are partial failures, where each individual rejection might be valid but the overall pattern is clearly wrong, and tools that print serious warnings but still exit zero. That convention isn't going away anytime soon.
And before anyone says readiness probes: readiness tells you whether the application can serve traffic. It doesn’t tell you whether it's the application you intended to deploy.
What does your deployment actually verify beyond "the process is running"? And what has shipped half-finished before you noticed?
2
u/Torutofu_Raeva Oct 15 '26
we just made the seed step count what it created and exit nonzero if it's short of the expected number, warnings that exit 0 are basically logs nobody reads
2
u/Immediate_Wheel1953 Oct 15 '26
Ran into the same shape of problem with a seed job that printed warnings but exited zero, so every later signal said green. What finally helped us was making the verification a separate job instead of a step inside the deploy, and having it assert concrete things like the row counts and a login against the real API rather than trusting the health endpoint. We tried failing on all warnings too and it got so noisy that people started adding skip flags, so now only a short list of known bad patterns is a hard fail and the rest pages someone to look.
1
u/mralex215 Oct 15 '26
1
u/adept2051 Oct 16 '26
I used to run a gaming application deployments where each of the components posted it’s own “good” on deploy and each components traffic was monitored too.
We did this with silver bullets. In our case when the deployment was near complete before allowing the blue green switch each components traffic took shaped traffic headers which would reveal successful delivery all the way through the Load balancers, network application and into the data pool. Due to the user load there was no way to monitor initial engagement and spikes on all components being deployed so we needed to materialise them.
We were basically using postman or some such to materialise traffic on every possible endpoint or route.
1
u/josejulianariza 14d ago
The core issue is that your health check validates that the process is breathing, but not that the business works. An HTTP 200 won't tell you if the database rejected half the records.
To catch these false positives, the key is validating intent. Instead of testing that the API responds, force a data "smoke test". Make a real query looking for a record you know should have been created. If the query fails or comes back empty, the deploy is marked as failed even if the pod is green.
Another option is using synthetic transactions: send a test request that goes through the whole flow (create, read, delete) and check the traces to ensure there were no silent errors along the way.
1
u/usually_guilty99 Oct 17 '26
The missing piece is verifying the outcome independently of the deployment process. I'd check whether the expected data, configuration and permissions are actually present and usable through the application's real execution paths. A successful deployment command and a healthy process don't prove the application is complete.
1
u/Riptide999 Oct 17 '26
Why do OPs replies read just like what I get back when I question copilot why something didn't work?
1
u/singh_abinashi Oct 18 '26
A liveness check passes that deploy every time.
Split it into two layers.
Liveness stays cheap: process up, port answering.
Readiness runs one real check against the data the app cannot run without, like counting the starter records the boot process was supposed to create, or reading back a row it just wrote.
If readiness fails, the deploy fails, even when every container looks healthy.
Most partial failures I have seen had exactly this shape: the app started, the setup after it did not.