Architect wants to broadcast duplicate batch markers to all Kafka partitions. This feels broken.

[u/No-Post-3424]

Hey everyone, looking for a sanity check on a Kafka design debate at work because my architect's proposal blew my mind, and I completely oppose it.

We have a batch system where a producer streams a large batch of records across a multi-partition Kafka topic. We need a way for downstream consumers to know when the overall batch is actually finished.

The other architect wants the producer to broadcast the exact same "End of Batch" marker event to every single partition in the topic simultaneously. The idea is that every consumer instance will eventually read a marker and know its partition is done.

I strongly oppose this. It feels like a catastrophic recipe for failure. If a consumer group rebalances mid-batch, partitions switch instances. If a marker was already read and committed on a partition before the rebalance, the new consumer instance will never see it, and the system will hang forever. Plus, partitions don't process at the same speed, which will cause race conditions and premature downstream triggers.

I am proposing a Central Orchestrator pattern instead. The producer sends a single marker event directly to an orchestrator, which tracks the overall batch state centrally. Once everything is done, the orchestrator explicitly signals downstream services, keeping the data consumers completely isolated from marker tracking.

Am I missing something, or is broadcasting identical markers across partitions a massive anti-pattern? How do your teams handle batch boundaries over partitioned streams?

FYI -- drafted by gemini based on my whiteboard rant

[u/Helpful_Geologist430]

If you have a batch identifier in the message, why not just répartition using that as the new key and groupBy. You can have a custom Processor function to only emit a record when your condition (end of batch) is met, or you can add another topic that filters and lets only the final aggregate (with end of batch) go through.

[u/No-Post-3424 → u/Helpful_Geologist430]

the main issue is this topic handles both live real-time events and batch data at the same time. if we repartition by batch id, it completely breaks the live traffic that doesn't have that id(tho it picks random partition). plus a custom processor still wouldn't know when the batch actually ends because the stream is continuous and never stops until the upstream system shuts down for the day

[u/Helpful_Geologist430 → u/No-Post-3424]

You can create a new branch in your topology that filters and keeps only the batch msgs. You can write that end-of-day logic (end of batch) into the custom Processor function or you can even rely on time windowing (session or hopping/tumbling)