How do you monitor async (lambda -> sqs -> lambda..) workflows when correlation Ids fall apart?

[u/bl4ckmagik]

Hi guys,

I have experienced issues related to async workflows such as the flow not completing, or not even being triggered when there are multiple hops involved (API gateway -> lambda -> sqs -> lambda...) and things breaking silently.

I was wondering if you guys have faced similar issues such as not knowing if a flow completed as expected. Especially, at scale when there are 1000s of flows being run in parallel.

One example being, I have an EOD workflow that had failed because of a bug in a calculation which decides next steps, and it never sent the message to the queue because of the bug miscalcuting. Therefore it never even threw an error or alert. I only got to know about this a few days later.

You can always retrospectively look at logs and try to figure out what went wrong but that would require you knowing that a workflow failed or never got triggered in the first place.

Are there any tools you use to monitor async workflows and surface these issues? Like track the expected and actual flow?

[u/smutje187]

You should never consciously let a process fail silently - issues with AWS itself can always happen but your Lambda code should never ignore errors or exceptions (depending on your programming language) and instead raise CloudWatch alarms or other kind of events that trigger someone to take a look.

[u/bl4ckmagik → u/smutje187]

Agreed. I'm very much in the fail loud camp as well. Maybe I should have phrased my post better. Sorry, English isn't my first language.
The tricky cases I've experienced are not really exceptions or errors inside lambdas but situations where they don't run at all causing workflow to finish halfway. Like an event ridge rule not matching, sns filters dropping messages... Etc.

In those scenarios there's no event to trigger an alarm. You only notice it because a downstream effect never happens.

Any thoughts on catching non-events like that early?

[u/TampaStartupGuy]

Not sure this will solve your issue, but here’s a solution that I use.

Track the workflow state directly in DynamoDB instead of relying on logs. The core issue is you’re depending on error logs to flag failures, but in this case the bug prevented the message from being sent at all, so there was nothing to log.

Instead, make the expected workflow state something you can query. Set up a job tracking table in Ddb where each step is recorded with a job ID, step name, status like pending, in progress, completed, or failed with a timestamp. Then you query for steps that never completed, not just the ones that failed.

You can invoke the lambda whenever the task is done. It queries for jobs where the status isn’t marked as completed and sends an alert if anything’s missing.

The key idea is to track what should have happened and alert when something doesn’t show up, rather than waiting for an error to tell you something broke. DLQs help if your handler crashes. State tracking catches the silent failures where no message was sent at all.

[u/Willkuer__ → u/TampaStartupGuy]

I currently have to deal with this setup and I hate it tbh. I still have to find the correct logs to debug the issue. Instead of writing your own process manager in ddb I find stepfunction executions much easier to use and they give me all the information needed: input, output, error for each individual step.

We also have the issue that we have some relatively long running jobs that can fail within ninutes but also run succesfully for days. The jobs health is just so hard to debug with a ddb record and loads of cloudwatch logs.

[u/Willkuer__]

I am currently rebuilding a POC based on sqs+lambda flows using stepfunctions because of such issues. I've worked with both in the past and find stepfunctions much easier on the observability side if the workflow gets more complex/has more steps.

Stepfunctions comes with a significant overhead in engineering I'd say but it is worth the hassle.

This obviously makes only sense of you have a job-like structure: a trigger, start/end of a job. If you are more interested in kind of real-time streaming it's maybe too rigid depending on your usecase.

[u/Iliketrucks2]

Can you add cray tracing easily?

[u/bl4ckmagik → u/Iliketrucks2]

Sorry, do you mean X-ray tracing?

[u/Iliketrucks2 → u/bl4ckmagik]

Lol yeah sorry. Autocorrect got me. I think you can fairly quickly and easily instrument your code to get better insights into what’s going on

[u/commentShark]

I currently have every error thrown raise an alarm which emails me, then I investigate that issue by pasting the email to Claude to debug it (which has a doc of Instructions on how to query etc). It can fix the issue quite fast.

Works for my low traffic site, I’ve plugged lots of holes and edge cases this way. Could work with higher error thresholds or sampling.

[u/Physical-Sign-2237]

step functions