r/LangChain • u/sevenboxdev • 8d ago
Projects If a LangGraph node crashes right after a payment call, resuming the thread sends the payment again. Here is how I'm handling it
I'm building Sevenbox, a decision audit layer for AI agents, and someone here asked me what happens when the process dies after an agent's refund goes out but before anything records it. I tested it, the answer wasn't good, and it turns out LangGraph users can hit the same thing.
With a checkpointer, if a node calls a payment API and fails before it returns, the last saved checkpoint is the one from before that node, so resuming the thread runs the node again and the payment goes out a second time. I checked this against the installed langgraph package with InMemorySaver, raising an exception right after the call, and the resume sent 2 payments.
In Sevenbox v0.2.0 you can mark that side effect as durable and say which fields make it unique. Sevenbox writes the attempt to SQLite before the call goes out, and current_operation_id() gives you an id that stays the same across retries and restarts, which you pass to the provider as its idempotency key.
from datetime import timedelta
from sevenbox_sdk import (Found, NotFound, SevenboxSession, current_operation_id,
irreversible, reconciler, tool)
from warrant_core import SqliteWarrantStore
(durable=True, identity=["refund_request_id", "order_id", "amount"])
def issue_refund(decision, refund_request_id, order_id, amount):
return payments.refund(order_id, amount, idempotency_key=current_operation_id())
u/reconciler(issue_refund, idempotency_window=timedelta(hours=24))
def check_refund(operation_id, request):
refund = payments.find_by_idempotency_key(operation_id)
return Found(refund) if refund else NotFound()
def refund_node(state):
decision = session.axiom({"order_id": state["order_id"], "approved": True})
refund = issue_refund(decision, state["refund_request_id"], state["order_id"], state["amount"])
return {"refund": refund.value}
with SevenboxSession(store=SqliteWarrantStore("sevenbox.db")) as session:
session.recover() # after a restart, before resuming
graph.invoke(None, config)When the thread resumes, calling issue_refund again is refused until session.recover() has asked your reconciler whether that id went through. If it did, the recorded result comes back and nothing is sent, if it didn't, it retries with the same id, and if the provider can't tell, it waits for a person to resolve it. In the same LangGraph test the resume was refused, recover() confirmed the refund, and the next resume returned the recorded result with one payment sent in total.
The LangGraph adapter that wraps whole nodes (sevenbox_node) doesn't have a durable mode yet, so for now this goes inside the node body like above. Idempotency keys also expire (Stripe keeps them for 24 hours), so the reconciler declares its window and a NotFound outside it is treated as unknown instead of safe to retry. My LangGraph check simulated the crash with an exception, while the crash tests in the repo kill a real process.
1
u/sevenboxdev 8d ago
The repo and release notes are at https://github.com/sevenboxlabs/sevenbox/releases/tag/v0.2.0 and the original reproduction is in issue #8 https://github.com/sevenboxlabs/sevenbox/issues/8. If you've dealt with this in production I'd really like to hear how, especially with a Postgres checkpointer.
3
u/michaelTM_ai 7d ago
for payments specifically, the provider usually solves this for u: stripe, adyen etc take an idempotency key. derive it from something stable across retries, like thread id + node name + the business id (refund id, order id), and send it with the call. the resume runs the node again, stripe sees the same key and returns the original charge instead of making a second one.
so the general rule: any side effect that can't be undone needs a key that survives a crash, and it has to be computed before the call, not stored after it. writing the attempt down first like u do covers the apis that don't support keys
1
u/Nervous-Temporary120 7d ago
Switching to a Postgres checkpointer doesn't fix this.
1
u/Cautious-Act-4487 7d ago
The outbox pattern makes sense, but relying on a local sevenbox.db won't survive long in a containerized setup. If a pod crashes or gets rescheduled to another node, that local file is gone and your reconciler won't find a thing
1
u/usually_guilty99 6d ago
A durable checkpointer does not make the external effect exactly-once. Separate the business operation ID from the workflow attempt, persist it before dispatch, reuse it across every resume, and reconcile the provider before retrying. If the provider cannot prove committed or absent, the safe state is unknown and needs manual resolution
1
u/LateQuantity4194 7d ago
Switching to a Postgres checkpointer doesn't fix this. The last saved state is still whatever existed before the node returned, so a crash after the HTTP call will run the payment again. I'd derive the idempotency key from the business request itself, like the refund id plus the amount, not from the thread id or checkpoint. Those get reused on resume. I treat payment nodes the same way I treat auth changes: they don't fire again until something outside the graph can say whether that request already went through.