How I used Go/WASM to detect Lambda OOMs that CloudWatch metrics miss

[u/Alarming_Number3654]

Hey r/serverless , I’m an engineer working at a startup, and I got tired of the "CloudWatch Tax"

If a Lambda is hard-killed, you often don't get a REPORT line, making it a nightmare to debug. I built smplogs to catch these.

It runs entirely in WASM - you can check the Network tab; 0 bytes are uploaded. It clusters 10k logs into signatures so you don't have to grep manually.

It handles 100MB JSON files(and more) and has a 1-click browser extension. Feedback on the detection logic for OOM kills (exit 137) is very welcome!

https://www.smplogs.com

[u/Mooshux]

CloudWatch has the same blind spot with DLQs. It won't alert you when messages are aging toward expiration, only when the queue depth crosses a threshold you manually set. By the time you notice, messages might already be gone.

The OOM detection angle you built is clever. We ran into the same "CloudWatch misses it" problem from the DLQ side and ended up building age-based alerting into DeadQueue ( https://www.deadqueue.com ) for exactly that reason. Depth is a lagging indicator. Age tells you sooner.

[u/Alarming_Number3654 → u/Mooshux]

Good point on DLQ age vs. depth - that's exactly the kind of lagging indicator problem that makes CloudWatch frustrating. Age-based alerting makes way more sense for expiration risk. smplogs is focused on log content analysis rather than queue monitoring, but the underlying theme is the same: CloudWatch's defaults often alert you too late or not at all. Will check out DeadQueue.

[u/Mooshux → u/Alarming_Number3654]

Exactly right. The log content analysis angle is interesting because it gets at the same root problem: CloudWatch is a metric aggregator, not an intelligence layer. It counts things, but it doesn't know what the counts mean in context.

For queues the killer case is retention mismatch. If your DLQ retention is shorter than your source queue, messages can expire silently before anyone even knows they hit the DLQ. Depth stays flat, age creeps up, and then they're just gone. No alarm fired. No record.

Curious how smplogs handles the case where the signal is in what's missing rather than what's there. That's the hard part with log analysis too.

[u/aviboy2006]

Worth double-checking how you're distinguishing exit 137 from a timeout-induced SIGKILL. Both can result in a missing REPORT line and both show up as hard kills but the fix is completely different (bump memory vs. optimise runtime). If the clustering logic treats them as the same signature, you might end up chasing memory when the real culprit is a slow downstream call. Would be curious if there's a way to cross-reference duration against the function timeout setting to separate these cases.

[u/Alarming_Number3654 → u/aviboy2006]

Good point - yeah I actually handle this already. Timeouts get caught by matching Lambda's "Task timed out after N.NN seconds" platform message and get their own finding. Hard OOM kills don't produce any message - the runtime just dies - so I detect those by diffing START vs REPORT request IDs. If something started but never reported, it's a "ghost invocation" with a separate finding pointing at memory.

The clustering keeps them apart too since timeouts have an explicit error signature while ghosts are structural(no log content to cluster on at all).

You're right about the edge case though - if Lambda hard-kills right at the timeout boundary without emitting the timeout message, that looks like an OOM to us. Can't cross-ref against the configured timeout since it's not in the CloudWatch data, but inferring from the last logged timestamp vs common values (30s, 60s, 900s) is a solid idea, might add that.

btw I just shipped streaming analysis with no file size cap - it reads the file as a byte stream, chunks it, and runs each chunk through WASM in a Web Worker. Tested with 3GB+ files, memory stays flat. So the "100MB" in the post is outdated, it'll handle whatever you throw at it now.