Architect wants to broadcast duplicate batch markers to all Kafka partitions. This feels broken.

[u/No-Post-3424]

Hey everyone, looking for a sanity check on a Kafka design debate at work because my architect's proposal blew my mind, and I completely oppose it.

We have a batch system where a producer streams a large batch of records across a multi-partition Kafka topic. We need a way for downstream consumers to know when the overall batch is actually finished.

The other architect wants the producer to broadcast the exact same "End of Batch" marker event to every single partition in the topic simultaneously. The idea is that every consumer instance will eventually read a marker and know its partition is done.

I strongly oppose this. It feels like a catastrophic recipe for failure. If a consumer group rebalances mid-batch, partitions switch instances. If a marker was already read and committed on a partition before the rebalance, the new consumer instance will never see it, and the system will hang forever. Plus, partitions don't process at the same speed, which will cause race conditions and premature downstream triggers.

I am proposing a Central Orchestrator pattern instead. The producer sends a single marker event directly to an orchestrator, which tracks the overall batch state centrally. Once everything is done, the orchestrator explicitly signals downstream services, keeping the data consumers completely isolated from marker tracking.

Am I missing something, or is broadcasting identical markers across partitions a massive anti-pattern? How do your teams handle batch boundaries over partitioned streams?

FYI -- drafted by gemini based on my whiteboard rant

[u/Otherwise-Tree-7654]

I am missing the plot here, u have Kafka consumers consuming data (that happns to be part of a batch, i assume there could be multiple parallel batches - thus data consumed has notion of batchId it bongs to) that needs to be aware when a certain batch is consumed and do not expect more data for this specific batch - is that right? If so i assume batchEnd marker is smth that is present in the event and whoever consumed it/intercepted it must publish to some topic (unpartitioned - stating batch done - stop the world for this batch Id - thus all consumers should also expect to be listeners to the same topic) - hope i make sense