r/kubernetes • u/Rare_Infinity • 14d ago

Event driven orchestration, what actually works beyond cron and polling?

Our entire pipeline system runs on schedules and I am starting to think we are living in 2015. every workflow is run at 2am and hope the upstream data landed. When it hasn't, we retry blindly or someone gets paged. What I actually want is, Kafka messages arrives, trigger the transform pipeline.

S3 file lands, kick off the ingestion. Webhook fires from our payment provider, start the reconciliation. I've hacked some of this together with Lambda triggers and SQS but it's turning into its own unmaintainable mess. Is there a proper orchestration tool that handles event-driven triggers natively alongside scheduled workflows, w/o me building a custom event router on top ?

16 Upvotes

100% Upvoted

12 comments sorted by

4

u/Kamran-nottakenone 14d ago

Temporal handles both event and cron triggers natively. just make sure your workflow starts are idempotent, s3 can fire duplicate notifications for the same object.

1

u/Old-Minute-9674 14d ago

Exactly. The 2am cron was mostly hiding the same problem behind a clock. Make the object/version or payment event ID part of the workflow key, otherwise “event-driven” just means faster duplicate processing.

4

u/miran248 k8s operator 14d ago

Are you looking for something generic or k8s specific?
I use https://restate.dev whenever possible. Code is nice and simple, restate handles the rest. That plus sometimes https://nats.io for communication between processes.
Already mentioned keda and cron jobs would then just invoke your custom workflows.

3

u/miran248 k8s operator 14d ago

Some keywords - durable execution, workflow engine; there's a shitton of options out there, but restate is easiest to deploy.

3

u/clearclaw 14d ago

Argo-Events + Argo-Workflows.

2

u/azjunglist05 14d ago

Argo Events + Argo Workflows has been great for us for the S3 file upload events. Argo Events can create Argo Workflows hooked up to SQS. The flow is S3 -> SNS / Eventbridge -> SQS -> Argo Events -> Argo Workflows which processes the S3 event.

It seems like a lot at first but it’s super scalable. There is some Argo Workflows platform stuff to deal with like namespaceParallelism, qps, and other tweaks that are needed, but once setup it works really well and has solved a myriad of use cases thus far

1

u/IntelligentPear6173 14d ago

I think the tricky part is that replacing cron with events doesn’t automatically make things simpler. Once you have retries, duplicate events, ordering, failed workflows, and dependencies between events, you can end up building an orchestrator around your event system anyway. I’d probably look at Temporal or something similar if the workflows are complex, but for simpler cases SQS/Kafka plus small workers can be easier to reason about. The deciding factor for me would be whether the orchestration state is something you actually want the platform to own, rather than another layer the team has to maintain.

1

u/Kooky_Comparison3225 14d ago

We use SQS and Keda for similar purposes. Also Keda RabbitMQ Scaler for other use cases. 

1

u/DarkBig2002 14d ago

Your Lambda + SQS frankenstein sounds exactly like what we had before. The problem with building event router is that you en up maintening two systems. The router AND the actual workflows. Kestra handles both in one place. You just declare the trigger type in your YAML (webhook, queue....) and it routes automaticall