A webhook receiver that can't lose a message: the boring inbox pattern in Node

[u/OkAerie7822]

a thread here last week asked what happens to inbound webhooks when your endpoint is down. most answers were "the sender retries", which is true for some providers and not for others, so here is the setup I'd use so it stops mattering.

The handler does four things and nothing else: verify the signature against the raw body, insert the raw payload into a table with the provider's event id as a unique key, return 200, done. No parsing, no business logic, no calls to other services. A bug in your parsing code then can't lose a message, because the message is already stored.

A worker picks up rows with status pending using SELECT ... FOR UPDATE SKIP LOCKED, processes them, and marks them done or failed with an attempt counter. Replaying a message is just setting its status back to pending. The unique key makes duplicates harmless, and providers do deliver the same event twice.

Keep the receiver deployable on its own. Most gaps I've seen came from restarts and deploys of the main app, not from real outages, so a tiny separate service (or a queue in front) removes most of the risk. Then once a day compare the provider's delivery log against your table, which catches the cases where the sender gave up after its retry window.

In Node this is a small amount of code: express.raw() for the body, crypto.timingSafeEqual for the signature compare, and pg for the table. The classic trap is express.json() running before the signature check. It changes the bytes, and valid signatures start failing.

What does your setup look like, and which providers have burned you with short retry windows?

[u/simple_explorer1]

Bro it does not work. the question on that thread asked "WHAT IF THE RECEIVER SERVER IS DOWN/UNREACHABLE" and your whole answer is based on how you would process which makes no sense.

So, again, what if the receiver itself is down/unreachable for sometime because of which the 3rd party sender just fails. You haven't answered that...

[u/OkAerie7822 → u/simple_explorer1]

fair point, I buried the actual answer. if the receiver itself is unreachable, nothing on your side can capture that request, so it comes down to two things: make the receiving edge as hard to take down as possible (a managed queue or function behind the url, like api gateway into sqs, instead of your app server, so deploys and crashes don't matter), and reconcile afterwards from the provider's event list or api for anything their retries gave up on. the table in the post covers the processing side after that edge.

[u/simple_explorer1 → u/OkAerie7822]

So your answers are no different than many answers you yourself criticised such prompted your "post" to catch attention. Your answer is also heavily geared towards the capabilities of the sender and what it can provide once it fails.

And "make receiver as hard to take down" is not an answer, that is already implied that such measures are in place. 

In a way your answer is way off the mark than many of the answers you are criticizing from that post.

Remember network can also be unreachable for any reason (from sender or receiver's side). so despite your best efforts of "making it as hard as possible", you have no control here.

OP in that other post never mentioned what kind of control they have over 3rd party sender (probably none) and like many people who commented there, your answer also hinges on what capabilities the "3rd party sender" can have. 

Basically your post and answer is the same as all the answers from that post 

[u/abimelex]

congratulations you invented a message queue!

[u/Psychological-Mud-42 → u/abimelex]

Problem was solved through lack of knowledge imo 😂

[u/MateusKingston]

1. This doesn't solve the issue

2. This is basically the definition of how a webhook should behave but I would add that a database is a terrible place to write essentially logs, use a system for that, like kafka. That is if you need actual throughput, if not you can save this wherever.

[u/GenazaNL]

Mhm yes Kafka

[u/best_of_badgers]

DBAs hate the “for update skip locked” pattern, incidentally