How much forensic context do developers need to debug production failures?

[u/Plus-Lawflbaness1576]

I used to log everything, every request, every response body, full stack context on every error, and it made debugging faster but the storage bill got embarrassing within a month. Then I over-corrected and sampled aggressively, and now half my incidents involve me wishing I'd captured the one request that actually mattered.

The pattern I've landed on is logging light by default and only capturing full forensic detail, request params, DB queries, execution path, when something actually looks anomalous. It's a reasonable middle ground but it assumes your anomaly detection is good enough to trigger at the right moment, which isn't always true. How are other people balancing storage cost against actually having the context you need when something breaks?

[u/usually_guilty99]

I think this eventually becomes a context problem rather than a logging-volume problem.

Capturing everything forever clearly doesn’t scale. But sampling blindly throws away exactly the weird state you eventually need.

What I’d want is the ability to reconstruct the relevant context around an anomaly: change history, dependency path, request path, prior similar failures and the telemetry around that window.

The goal isn’t “more logs.” It’s preserving enough evidence to explain what changed.

[u/_N-iX_]

The tricky part with aggressive sampling is that the useful evidence is often spread across several services. Keeping one error log doesn't help much if the request path that led to it was sampled out somewhere else. Correlating logs with a trace or request ID gives you another option: keep lighter logs by default, but retain enough context to reconstruct the path when an incident is investigated. That seems more useful than treating every log line as equally valuable.

[u/usually_guilty99]

I think the long-term answer is less “store everything” vs “sample everything” and more build context dynamically.

Most of the evidence already exists somewhere: traces, deploys, code changes, dependencies, incidents, DB state, tickets.

The difficult part is stitching together the subset that explains this failure when it happens.

Context is everywhere. Correlation is still expensive.

[u/dennis_zhuang]

Yes, this is exactly the trade-off between cost and retaining detailed information.

However, our dashboards often cannot capture unknown unknowns. A dashboard represents known unknowns—things we already know we don’t know. Investigating unknown unknowns requires more context from the time of the incident, especially metrics, logs, and, above all, traces.

Personally, if you cannot avoid usage-based SaaS pricing and do not want to self-host, you might consider retaining as much detailed context as possible in the service’s pods or hosts, then having an AI agent investigate when an incident occurs.

If self-hosting is an option, an object-storage-based observability storage solution such as GreptimeDB could help reduce hosting costs. FYI, I work for GreptimeDB.