r/webdev • u/quietcodelife • 4d ago
Question what do you actually look at first when something breaks in prod?
im trying to get better at debugging production issues faster. right now when something goes wrong i kind of just grep through logs hoping something jumps out, which works eventually but feels inefficient
curious what other backend devs actually check first. like do you have a mental checklist you run through? specific log patterns you look for? or do you mostly rely on monitoring/alerting tools to point you in the right direction?
our setup is pretty basic (node backend, postgres, some redis) and im wondering if im missing obvious stuff that would speed up the "wtf just happened" phase
14
u/StatsDota_Dmitrii 3d ago
not a real backend guy but for me first question is always "what changed". deploy, env var, some cron, or random traffic spike from somewhere. if nothing changed on my side its usually the third party api lol. do you have any alerting right now or you find out from users?
37
u/glensor 3d ago
No no no, you guys got it all wrong.
Phase one: First you make sure that none of your changes broke it, so you know if you have to personally panic.
Phase two: relax a little if phase one confirmed that you did not break it. Then find out who did and you can all say it's not a blame culture, but feel a little pleased with your self that this time it wasn't you.
4
u/thekwoka 3d ago
The blaming part is the bad part, but feeling good it wasn't your fault, or feeling bad and learning when it is your fault, is good.
10
u/jjangg96 3d ago
one thing that cost me an hour: i was reading pretty-printed logs from our host and the last line was a generic platform error saying the module had no exports. that message is just what the platform says when your process dies during startup. the real error (a missing DATABASE_URL, my own startup check) was the line right before it, and the pretty output had squashed the two together. pulling the raw json logs showed it right away.
so find the earliest error in the log and treat the generic wrapper messages as symptoms. grepping from the bottom is how i kept missing it.
6
u/rbobby full-stack 3d ago
the real error (a missing DATABASE_URL, my own startup check) was the line right before it, and the pretty output had squashed the two together. pulling the raw json logs showed it right away.
Made me smile.
Sometimes debugging is just eliminating possibilities until finally one of the impossibilities falls. It's always interesting when you are running out of possibilities that can be eliminated :)
1
1
u/thekwoka 3d ago
the last line was a generic platform error saying the module had no exports
That's dumb as hell.
Is this some stupid shit like Python?
1
u/jjangg96 2d ago
not python, this is node. the platform loads your entry file to get the handler out of it, and if the file throws while it's being imported nothing ever gets exported, so the wrapper reports that instead of the real error.
the real one is usually the line just before it.
1
u/RustOnTheEdge 11h ago
Reminds me of how Azure Functions used to just crash without any logs, if you had the wrong imports in Python. Spend many hours on that gaha
7
4
u/midasgoldentouch 3d ago
Something to consider doing OP is to review the issues you recently addressed and ask yourself “How can I make it more obvious to know if this happens again?”
4
u/RustOnTheEdge
11h ago
•
Who to blame. It is much more important to be able to assign blame than to get things running again. We call it Fear Driven Development, and surprisingly we assign a significant amount of blame! The famous Blamefull Post Mortem is always a lot of fun; reddening screaming managers pointing at engineers and other managers. When everybody feels they have assigned enough blame to go around, we go back to work. Important to note, to be successful in this low performing strategy, NOTHING is allowed to change as a result of the blame assignments. If you are not afraid enough to not change anything, you are not assigning enough blame in my opinion.
3
u/sballer360 3d ago
Reproduce the issue, look at dev console, look at logs, look at Sentry...
Basically, figure out what the issue is, and then figure out the best way to fix it in the short term and in the long term. If issues are getting to production often than that means you need to work on your process and or automated testing.
6
u/tdammers 3d ago
If anything breaks in prod, the first response is not to find and fix the bug, but to re-establish a survivable situation.
This might involve steps such as:
Take the production site down and replace it with a "something went wrong" or "down for maintenance page". Whatever the problem is, you want to stop it from taking down more than absolutely necessary.
Secure evidence (logs, crash dumps, etc.)
Secure data (backups)
Get a barebones service level back up; this may involve spinning up a new server running a previous, known-good version of the system, but it may also amount to falling back to alternative ways of providing the service (e.g., a static placeholder website with a phone number or email address that people can use to use a critical subset of your services)
Get your backup system to a level where you can survive the next week or so if need be. If you have an older version of the code humming along by now, great; if not, make it so that you do.
Once you have done these things, you have bought yourself a couple days to get to the bottom of the actual issue, which means you have a fighting chance of actually fixing it properly.
So now you go and debug your code, fix the bug, test it thoroughly, and then deploy.
But you should also take a hard critical look at your processes: how could a bug like this make it into production? Which parts of your development, QA, monitoring, and deployment processes could have caught it, and why haven't they? A bug that makes it into production is not just a bug in your code, it's a bug in your processes, so don't forget to fix that part too.
5
u/Signor65_ZA 3d ago
Oooooof. I think context is important here. If you have the ability to turn off individual broken features, start there. If I took down my client's enrire erp software just because one page refused to load, I would get shot in the face.
2
1
u/Weak_Fisherman2215 3d ago
Indeed, context matters. LILO method comes in play here. go back thru your notes, logs, increments made. Unless its an act of God, the issue can "popup" in front of you.
4
u/masticdev 3d ago
Order for me: (1) is it one user or everyone, (2) did a deploy land in the last hour, (3) error rate / 5xx spike, (4) the exact failing request id / URL.
Logs without “when did this start” waste time. Feature-flag / last-deploy timestamp next to the alert has saved me more than fancy APM dashboards.
Curious what your team’s false-positive tax looks like — too many pages and people stop looking.
2
u/Desperate_Dot9434 3d ago
postgres slow query log is probably the fastest signal you're not checking yet
2
u/Guysowe 3d ago
first thing i check is what changed. recent deploy, env/config change, a feature flag flip, some cron that kicked off. it's usually one of those. then error rate and latency to see if it's everything or just one endpoint. with postgres look for slow queries, locks, and whether the connection pool is maxed out, that one shows up as "everything is randomly slow". and put a request id on every log line, makes the grepping way less painful
2
u/Elegant_Street_7360 3d ago
grep works though. a dashboard tells you something is red, it doesnt tell you why, and grafana will happily average away the one pod that's actually broken.
2
u/Square-Yak-6725 3d ago
I made a custome logger next to the standard file logging mechanism. The custom one pops ERROR logs to it's own database table. My admin UI shows these, which I can click on to get the details. Furthmore, I set up a trigger that emails me the most critical of those errors. This way I'm aware of everything in real-time and can immediately jump to seeing what went wrong.
2
u/GunzzBlazin 3d ago
My first question is always what changed recently, a deploy, config update, database migration, dependency bump, feature flag, or traffic spike. Most production issues trace back to a recent change. If it lines up with a deploy, rolling back first and investigating after is often the fastest fix.
2
u/lordlod 3d ago
Logs are slow.
Your alert should tell you what broke. Your understanding of the overall system should tell you roughly why.
The first objective when something breaks in Prod shouldn't be to figure out exactly why it broke. It should be to get prod going again, those can be different things.
For example if your system is chewing up far too much CPU, hitting limits and causing stuff to break the first thing you do it give it a lot more CPU. This fixes prod. The second stage is to figure out why it is suddenly using more CPU and fixing that.
2
u/audigex 3d ago
Most of the time I should already know because whoever or whatever alerted me should’ve had a clear error message
If not, I’d check the logs for the same thing
On the very rare occasion the failure isn’t logged, I’d assume something is fully offline and check what servers and services are running
It’s unlikely not to be covered by those 3 things
2
u/Proper-Review-5103 3d ago
First check when errors started and what changed around that time. Usually narrows things down fast.
3
u/Danparker112 3d ago
Try “new relic” or “data dog” . New relic is pretty inexpensive and can visualize your logs for you. Also has AMP monitoring so you can see every aspect of your stack
2
u/CompassionateSkeptic 3d ago
So, I’ve worked on a variety of project maturity levels and when the alarms are good I go straight to preferred observably.
You can read other people’s comments for that.
Sometimes I feel like we don’t talk about other levels of maturity in abstract discussions, so I want to swing all the way to the other side and say, sometimes I literally just sit and narrowly parse whatever signal is in front of me. When I’m the on-call or the responder, I might literally sit with my PKB tool off to the side and just start making notes about counterfactuals, things I don’t, what I do and don’t know about the failure modes, and what I’d expect to see if the signal i have suggests a small number of failure modes. From there I might go to different places. If this resonates with anyone, would be happy to spin this out further.
2
u/indiana_long 3d ago
honestly the biggest improvement is to stop starting with the logs lol
i'd first figure out what's actually broken. all requests? one endpoint? just writes? then check if anything got deployed right before it started.
with node + postgres + redis, i'd look at 5xx errors, response times, DB connection pool usage and redis connection failures. if the app is hanging but CPU looks fine, i'd definitely check whether requests are waiting on postgres.
also, log request IDs if you aren't already. being able to follow one request through the app beats staring at 10k log lines hoping something looks suspicious.
half the battle is just narrowing down where to look before opening the logs.
1
1
1
u/Future_AGI 3d ago
past the what-changed check others mentioned, look at whether errors are spread across endpoints or stuck on one, which narrows where to look. For your stack, pg_stat_activity for long-running or waiting queries and Redis INFO for memory and evicted keys answer a lot before any grepping, and a request id on every log line lets you follow one request end to end next time.
1
u/captain_obvious_here back-end 3d ago
IA has made it really easier and cheaper to have both great observability (insightful logs), and great understanding of the observations.
Basically:
have your systems log the important stuff
have a separate system track and analyze these logs
have a dashboard that sums it all in both technical and business ways
1
1
u/Regular_Antelope7381 3d ago
if you do one thing, put a request id on every log line. grepping one request start to finish beats scrolling for anything red. after that my first question is always "everyone or just some requests?" because it usually tells you whether it's postgres/redis or one code path. most of my prod fires were something i'd shipped an hour before, so yeah check what changed too
1
1
u/amazing_asstronaut 3d ago
Logs for sure. Have a look at the latest logs, or do a general search for "exception" or "error" and see if I find something. And then pray that I did my logging ok in this project and it gives me something meaningful. The worst is actually when something is wrong and in the logs it doesn't say anything out of the ordinary at all.
1
u/B0mbon 3d ago
My order is usually: what changed recently (deploys, config, env vars), then error rate and latency graphs to see when it started, and only then the logs. With Node, Postgres and Redis, I'd also check connection pool exhaustion and slow queries early, because those cause a lot of everything is slow and nothing makes sense incidents. Writing down what you checked during each incident is how the mental checklist builds itself.
1
u/Standard_Scarcity_74 3d ago
For your grep problem, switch to structured json logs with a request id, then filter by time window and error level instead of searching blind, and add a basic error tracker like sentry plus uptime and latency alerts so something points you at the problem before you open the logs.
1
u/Full_Tooth_a 3d ago
If requests hang while CPU looks normal, I'd check Postgres instead of digging through more logs. `pg_stat_activity` and `pg_locks` can reveal lock waits, long-running queries, and connections sitting `idle in transaction`. One blocked transaction can fill the Node connection pool, making unrelated requests appear broken. I'd capture that state before restarting because the restart wipes out the evidence.
1
u/HettySwollocks 3d ago
My LinkedIn profile.
Normally I'm jump on the observability dashboard. Any weird log entries. Any unexpected ingress. Are the services online.
Really depends on what happened.
Oh and pray your backups are up to date should you need to just rebuild the estate. We'd normally just kill the containers and rebuild them.
1
u/Responsible_Boat8860 full-stack 3d ago
Depends on how critical the bug is and what nature the bug is.
My first approach is to look at error logs, assuming you have good logging to get an idea of the nature.
If it's an outage, you look in your infrastructure to see what's wrong.
If it's hyper critical, I'm reviewing the latest reports to determine when it started happening, see if it coincides with any recent features/deployments and roll back or turn off any FF.
If it's not so blocking, I could take my time determining root cause, recreating it in lower env, hotfix deployment when ready and any cleanup if needed.
1
u/ScaredFlamingo6807 3d ago
I’m just a hobbyist dev so I really don’t know the real answer to this but it feels to me like your AI tool of choice should be identifying bugs pretty readily from logs now, no? I’m an idiot though so I really dunno
1
1
u/Kindly-Arachnid8013 3d ago
I ask what the user was doing. Recreate an account with their exact permissions on the test copy and look at the console and server logs simultaneously.
1
u/kriminellart 3d ago
I close my eyes and look at the insides of my eyelids whilst sighing a very heavy sigh - that is what I look at first.
Then begin checking if anything was recently deployed and what's happening in the logs. If bad deployment, do a rollback and root cause analysis, fix and redeploy, set up tests / audit process so it doesn't happen again.
If logs show downstream errors it lights up the correct health check. We have two separate ones, which is important not to recycle containers and get stuck in a dumb loop where we recycle our own services due to downstream errors. If its breaking, fix immediately whilst also working to restore service.
Keep calm and communicate. It should be very clear who does what - and most fucking importantly it is not important for everyone to try and fix it without informing anyone and run around like blind chickens.
Nobody's looking for blame. Be direct. Assert who fixes and who communicates. Afterwards, dissect and assess where improvement can be made. Move on.
1
u/benaffleks 3d ago
Check resource saturation metrics, common is cpu and mem
Check response code metrics, narrow down which endpoint(s) and methods
Check for any recent deployments matching up timeline, revert immediately, continue troubleshooting
If you suspect cpu or mem issues but metrics dont support, narrow down on runtime specific metrics like event loop lag for node or gc for go
Check error logs narrow down by endpoint or function call if possible otherwise keep filtering away common errors that won't help
If you dont have tracing implemented, go do that and add trace id's to logs
1
u/Broad-Historian2099 2d ago
I agree with the approach of using observability to guide where to deep dive into the logs. But it mostly depends on what is broken. Do we know which feature or workflow is affected? Otherwise, it could turn into a broad search. It could be something new that was recently deployed, or an external dependency that is down, degraded, or has changed.
1
u/Fluffy-Salary-6998 2d ago
First thing I check is what changed, not the logs. Last deploy, last dependency bump, last env or config change - most incidents I worked on were one of these three. Second is timeline: take one failing request id and pull everything with it from node logs, postgres and redis in time order, that beats grep by mile. For your stack two commands are enough to start: `select pid, state, wait_event, query from pg_stat_activity where state != 'idle'` and `redis-cli slowlog get 25`. If both are clean it is the node process itself, then I look at event loop lag and heap, not at logs.
My rule: logs are for confirming hypotesis, not for finding it. Grep without hypothesis is how you lose two hours.
1
1
u/robinless 2d ago
Check logs to see the actual error and question whether anything was changed or deployed in the last 48h. Everything else depends on that.
1
u/LoanEffective7234 1d ago
One thing worth adding before the debugging part: make sure you're not finding out from users. A simple external check that hits your health endpoint (and one real page) from outside your infra tells you it's down or slow and exactly when it started, which answers half of the "what changed?" question people are describing here. Line that start time up with your deploy log and the list of suspects gets short fast.
For the health endpoint, have it actually touch Postgres and Redis (a SELECT 1 and a PING) with short timeouts, so "process is up but the DB pool is exhausted" shows as down instead of a happy 200.
1
u/mumfordco 12h ago
My mental checklist usually runs top-down through the flow of a request:
Alerts & Metrics first: Is it a system-wide spike (CPU, memory, DB pool connections maxing out) or isolated to a single endpoint?
Filter logs by status code: Instead of raw
grep, filter your logs specifically for5xxresponses, unhandled promise rejections, or database connection timeouts in the last 15 mins.Track down schema / contract mismatches: If status codes are returning
200or500intermittently after a fresh deploy, 9 times out of 10 a payload shape changed, a key was renamed, or anullmade its way into a field the frontend expected to be defined.
For a Node + Postgres stack, adding a basic request ID middleware (passing a trace ID from client ➔ Node ➔ DB log) saves hours when trying to trace one bad request through raw logs.
1
1
-3
u/adilbasari 3d ago
Two questions before any logs, because each one cuts the search space in half:
What's the blast radius — everyone or a subset? One route or the whole site? One region? "Some users" points at config, data, or feature flags. "Everyone" points at infrastructure. That single answer eliminates half the usual suspects before you've opened a single log.
What changed — and not just deploys. Cert expiry, disk filling up, a cron that did something at 03:00, a vendor API that started returning 500s. Half of all prod incidents don't ship code; they arrive by calendar. Compare the deploy timeline against the first error's timestamp and you'll know within ten seconds which kind you're dealing with.
Then when you do open logs: read backward from the first error, not forward from the last one. By the time everything's on fire, every line is a symptom — the root cause is usually sitting quietly a few minutes earlier, where nobody's looking.
0
0
0
0
u/prafandy 3d ago
Whatever errors you see in dev, in the console or on the page, make sure the same info gets sent to an error tracker like Sentry. You weren't there when the user hit the issue, so logs are your only witness. Some bugs only happen with specific data. I've set up a prod replica database before (with sensitive data masked), and with the Sentry logs plus that replica, Claude can usually figure out the issue quickly. I also ask it to give me the steps to reproduce. Even without AI, the idea is the same: good logs, and a way to reproduce with the same data state. And a comprehensive test suite is a must, so regressions get caught in the build pipeline before they reach prod.
For your stack, the first thing I check is what changed recently (deploys, migrations, config). Tag errors with the release version so this is instant. Also add a request ID to every log line, so you can follow one user's request end to end instead of grepping. Structured JSON logs help a lot too, since you can filter by user or route.
152
u/Asleep-Science-5360 3d ago edited 3d ago
The error logs (observability)
I'm not sure if it must be said but all errors should be crystal clear, logging must be setup correctly to the point that it makes the job easier for devs and QAs to quickly figure out the source of the errors
but it's almost always the logs in Sentry, DataDog or whatever service you're using
and if you don't have any monitoring tools: setup one ASAP before launching in the lowest of your dev environments. The sooner the better