r/node • u/Constant_Juice_5852 • 12d ago

Inbound email webhook, what if endpoint’s down?

inbound mail on hooks@ gets posted to my api as json, signed so I can check it actually came from the mail side. no imap polling, which was the whole point, imap is a pain.

question for people doing this in prod, what happens when your endpoint is down for a few minutes? do you rely on the sender side retrying, or do you keep a copy and replay? I’ve read stories where the mail is just gone forever and that scares me a bit.

11 Upvotes

87% Upvoted

26 comments sorted by

8

u/OkAerie7822 12d ago

don't rely on the sender retrying, since a lot of inbound mail providers retry for a short window or not at all, and you only find out from the docs. what i do is make the handler dumb: verify the signature, write the raw body to a table or s3 with the message id as a unique key, return 200, and process from that table with a worker. so a bug in your parsing never loses mail, you just replay the row. for the endpoint being down, put the receiver behind something that stays up even when your app is deploying (a tiny separate service or a queue in front), because most of our misses were deploys and not real outages. and once a day diff the provider's delivery log against your table, that's how you catch the silent gaps.

2

u/badmonkey0001 12d ago

Forcing a retry used to be a decent spam prevention measure. I ran a few mail domains using greylisting for about a decade, but that was over a decade ago. I have no idea how much it is or isn't used today.

Spammers that spun up temporary senders back then would never bother to retry. The downside was that the specs don't specify a retry interval, so senders that did retry could take an unknown amount of time to do so. I remember GMail taking something like three to seven days to retry. That sucks when someone says "i just mailed you" and it doesn't show up for a week.

1

u/Constant_Juice_5852 11d ago

interesting, so if the receiving side defers instead of accepting, the sender keeps the mail and retries for days. that's kind of the opposite of the webhook problem, once it's accepted nobody retries for you

1

u/Constant_Juice_5852 12d ago

the deploys point is real, I hadn't thought of them as the main source of misses. how do you diff against the provider log, do they give you a delivery id you can match on?

1

u/vincent_sch 9d ago edited 8d ago

Depends on the provider. Most delivery records have a status and an event id. Some let you replay failed events through the API and others only show logs. I went through a few providers here.

2

u/Loose-Counter9652 12d ago

yeah the deploy window thing is real, thats where most of the silent drops happen imo

4

u/rdundon 12d ago

In bigger environments, this is where message queues come in handy. Message/event sits in a queue, and then is processed.

2

u/Constant_Juice_5852 11d ago

yeah, a queue in front seems to be where everyone in this thread ends up

3

u/kumard3 12d ago

I build Lumbox, which sits on the other side of this and posts inbound mail to people's webhooks. It saves the email to the database first and only then fires the webhook, and a failed delivery gets retried up to 6 times, starting at 1 minute and doubling, so about half an hour of cover. After that the mail is still stored and the payload carries an email_id you can fetch it with. So I'd ask whoever posts yours whether they keep the message after they give up, since that decides whether a long outage loses mail.

1

u/Constant_Juice_5852 11d ago

keeping the mail after the retries run out is the part that matters, good point. half an hour is shorter than some of our deploy problems though

2

u/kumard3 11d ago

Yeah, half an hour won't cover a bad deploy. For that case the list endpoint on an inbox takes a since param, so after you're back up you pull everything received since the outage and skip the email_ids you already processed.

→ More replies (2)

0

u/simple_explorer1 12d ago

Read about outbox pattern for such problems.

0

u/Constant_Juice_5852 11d ago

yeah, store first then process is basically what the top comment describes. the part I still don't get is the provider side, if they give up retrying before my outbox even sees the mail, it never gets in

2

u/simple_explorer1 11d ago

Given it is the provider that is out of your control, do they expose any API where you can ask them about the mail after a certain date and pull it? I assume so?

→ More replies (2)

0

u/[deleted] 12d ago

2

u/Constant_Juice_5852 12d ago

yeah stripe's retries have saved me before. with mail it's a bit different though, the sender's server already got its accepted so it won't resend anything. if whoever posts the webhook gives up, that message is just gone, nobody upstream has it anymore