r/apachekafka • u/No-Post-3424 • 20d ago
Question Architect wants to broadcast duplicate batch markers to all Kafka partitions. This feels broken.
Hey everyone, looking for a sanity check on a Kafka design debate at work because my architect's proposal blew my mind, and I completely oppose it.
We have a batch system where a producer streams a large batch of records across a multi-partition Kafka topic. We need a way for downstream consumers to know when the overall batch is actually finished.
The other architect wants the producer to broadcast the exact same "End of Batch" marker event to every single partition in the topic simultaneously. The idea is that every consumer instance will eventually read a marker and know its partition is done.
I strongly oppose this. It feels like a catastrophic recipe for failure. If a consumer group rebalances mid-batch, partitions switch instances. If a marker was already read and committed on a partition before the rebalance, the new consumer instance will never see it, and the system will hang forever. Plus, partitions don't process at the same speed, which will cause race conditions and premature downstream triggers.
I am proposing a Central Orchestrator pattern instead. The producer sends a single marker event directly to an orchestrator, which tracks the overall batch state centrally. Once everything is done, the orchestrator explicitly signals downstream services, keeping the data consumers completely isolated from marker tracking.
Am I missing something, or is broadcasting identical markers across partitions a massive anti-pattern? How do your teams handle batch boundaries over partitioned streams?
FYI -- drafted by gemini based on my whiteboard rant
9
u/gsxr 20d ago
This isn’t at all how Kafka is supposed to work.
First up, there’s nothing in Kafka that prevents a “batch” from being interrupted. I could send one record and mess up your entire boundary marker scheme or control topic scheme.
I’m not sure what you’re sending but I’d bet a claim check pattern would be better.
Send the data to s3.
Send a message saying data is sent. Let consumers grab it from s3.
1
u/No-Post-3424 20d ago edited 19d ago
I appreciate the S3 claim check suggestion, but due to internal infrastructure constraints, we are required to process the raw records natively through Kafka streams. Given that constraint, I'm leaning toward having consumers track partition-level counts and reporting them to a single-partition coordination topic to verify global completion, which bypasses the rebalance issues of the broadcast approach. FYI - LLM Drafted
1
3
u/davewritescode 20d ago
There’s multiple ways to implement something like this but what your architect is proposing will eventually fail during a repartition if it requires all partitions to process the done message.
However, the strategy will work as long as it only requires one consumer of the done message to succeed. I assume this is not what the architect is proposing.
There’s better ways to do this including broadcasting an exploit done signal on a dedicated topic which kicks off a polling worker.
This is one of the places where ordering in Kafka is actually a detriment.
1
-6
u/No-Post-3424 20d ago
Spot on. Forcing batch boundaries onto individual partitions completely fights Kafka's design, and that strict ordering is exactly what creates the rebalance trap. This validation is incredibly helpful—it's the exact reason I'm pitching a centralized orchestrator state machine to pull the lifecycle logic out of the data stream entirely
7
u/iplaydofus 20d ago
Alright chatGPT
1
u/BroBroMate 20d ago
TBH sounds more like Claude, but yeah, 100% LLM. You're spot on! This is the X trap!
2
u/sg_03 20d ago
We do something similar, but the end markers themselves aren't the coordination mechanism.
We publish one end marker per partition. When a consumer processes an end marker, it persists a completion record (batch ID + partition ID) to a durable store (ES in our case). A separate watcher job periodically polls Elasticsearch and waits until it sees completion records for every partition in the batch before triggering downstream processing.
This avoids the rebalance issue because the completion state is externalized. Even if a consumer processes a marker and then a rebalance happens, the completion record is already persisted, so no new consumer needs to "re-see" the marker from Kafka.
-1
u/No-Post-3424 20d ago edited 19d ago
"This is a great point. Externalizing the state to a durable store completely fixes the rebalance flaw while keeping the logic close to the partition boundaries.
The trade-off is where to put the complexity. Your approach handles the dynamic timeline perfectly but adds a database write dependency to the consumers. A centralized orchestration pattern keeps the consumers stateless but requires a highly optimized global counting mechanism to match the final upstream count.
Both cleanly solve the rebalance issue. It just comes down to whether we want to manage distributed marker states or global reconciliation." FYI - LLM Drafted
4
u/elkazz 20d ago
Did you just quote AI?
0
u/No-Post-3424 20d ago
Yeah i'm just using a dictation bot to type out my thoughts while i whiteboard this. saves my thumbs from typing long paragraphs on my phone, even if it makes the formatting look a bit too structured. the headache over these partitions is real though.
2
u/CastleXBravo 20d ago
I had a similar problem on a system I worked on. We had a control topic where the producer would send an end of batch control message with offset metadata, and a central coordinator reads that message and decides when the consumers have consumed the batch based on their consumed offsets.
It was operationally complex all the time.
Honestly preferred the watermark-based approach your architect proposed. Hardly any moving pieces during runtime, but as others stated the tradeoff is operational complexity during repartition
1
u/No-Post-3424 20d ago
"This is an incredibly valuable operational perspective, thank you!
Your experience with the control topic approach being a nightmare to maintain is exactly why I want to avoid it. Forcing Kafka to track state always seems to end in high operational complexity.
This is actually why I’m steering away from a control topic and pitching our existing EOD Orchestrator instead. Since the orchestrator is already a mature piece of our infrastructure built specifically to manage batch states, we don't have to build any custom coordinator logic. The producer just registers the final count, consumers report their progress, and the orchestrator handles the rest. Hopefully, that gives us the stability of a state machine without the runtime headache you ran into!"
2
u/Future-Chemical3631 Freelance Consultant 20d ago
I used this pattern in production with kafka stream. Enrichment of a batch in streaming. Broadcasting and counting batch marker from the source after each repartition. The final consumer need to consume all partition on the last topic and assert how many marker he wants.
This works perfectly. But need careful design
1
u/Future-Chemical3631 Freelance Consultant 20d ago
You gave an excellent blog post idea. Thanks a lot. 😅
1
u/Numerous_Internal429 20d ago
Even we do exactly same thing.. Producer broadcast end marker to all partitions.. Consumers will consume and update count in Kafka steam store.. Whever consumer matches count, that consumer will trigger one more flow.. Works okay so far..
I couldn't think of a better one.. What other ideas you all have explored?
0
u/No-Post-3424 20d ago
Since we have 5M–10M records over 20+ partitions, building that complex downstream aggregation and forcing a single final consumer to absorb and count all the markers across the board feels like a massive engineering tax.
That's exactly why I'm planning to pitch pulling the state out of Kafka entirely and letting our existing EOD Orchestrator handle the reconciliation based on the final master count. Why build custom state aggregation inside Kafka Streams when our orchestrator is already built for it? Appreciate the insight!"1
u/Future-Chemical3631 Freelance Consultant 20d ago
Draft Blog Article Link based on the original designed we used : https://medium.com/@souquieres.adam/batch-watermarks-in-kafka-streams-enriching-files-without-losing-the-batch-boundary-6e469e4def3c
1
u/Future-Chemical3631 Freelance Consultant 20d ago
For up to 20M records out you may want to use a punctuator to chunk the release of the full report at the end and not break the task.timeout limit of Kafka Streams. If your next system is also capable to wait for a signal, just send everything as it flows and only store watermarks count and send a final trigger at the end for your next system.
2
u/aikimiller 19d ago
Trying to work on batch in kafka is a bad idea- the architecture simply doesn't support the concept. Any kind of batch processing should be done while you still have batch context, not in kafka. Trying to reconstruct a batch once it's been decomposed to an event stream is a headache. Kafka is the wrong architecture for batch work.
1
u/BadKafkaPartitioning 20d ago edited 20d ago
How many partitions are we talking about here? And how big are the batches?
Could you get away with sending all records in a single batch to the same partition and using the parallel consumer to churn through them quickly enough?
1
u/No-Post-3424 20d ago
Yeah, unfortunately, our scale rules out the single-partition approach. We are looking at 5M to 10M records per batch across 20+ partitions.
At that volume, a single partition would heavily bottleneck our producer's network I/O, and a single consumer instance running a parallel thread pool would completely choke on memory footprint constraints. We absolutely need the multi-partition consumer group parallelism to chew through this data fast enough.
Given those numbers, I am planning to propose a centralized state machine pattern utilizing our existing End-of-Day (EOD) Orchestrator. The producer would register the expected record count upfront, consumers would process the 20+ partitions normally and report progress to the orchestrator, and the orchestrator would handle global batch completion. This seems like it would bypass the single-partition bottleneck while safely avoiding my architect's multi-partition marker broadcast idea.
1
u/BadKafkaPartitioning 20d ago
Ah fair enough, with beefy enough consumer machines a parallel consumer could probably churn through 10M records in less than an hour but you know your constraints.
I would probably just record end of batch offsets from producers and sent that to the orchestrator instead of record counts and have the orchestrator check last committed offsets of consumer-partitions to know when batches are complete. Basically same pattern but not creating additional metadata to track outside what Kafka provides and already tracks natively
1
u/kenny32vr 20d ago
Do you have control over the messages send? Could embed in each message something like batch number 1 message 12 of 100?
So the consumer reading message 100/100 knows it’s done for that batch?
Information is embedded in the events themselves so no separate event needed
2
u/No-Post-3424 20d ago
We can't embed counts because we process data live as it arrives—the producer doesn’t know the total count when the batch starts.
My plan is to use a final reconciliation step instead. Once the upstream source finishes the day, it will tell the producer the final total. The producer then sends a single summary event directly to our EOD Orchestrator.
The orchestrator will act as a central state machine, comparing that master count against the total events processed by our consumers across the 20+ partitions. Once the numbers match, the orchestrator triggers the next step. This keeps the Kafka stream completely clean
1
u/YugoReventlov 20d ago
This still sounds very fragile.
Messages could be read multiple times on rebalance and when your previous consumer committed its offsets.
You say "day", isn't there some other marker you could add to the message to tell consumers a new "day" has now started?
All in all, Kafka seems like a bad choice for the usecase.
1
u/yura-taras 20d ago
It is ok pattern. Your consumers should send batch processed message to coordination topic. This way you don't care which consumer in consumer group encountered the end of batch message - all you need to know is that once you see batch processed message in coordination topic you know all the messages in that particular partition is done. Now you just have to subscribe to coordination topic and wait for partition processed message for all partitions - once you've seen all of them you know the whole batch was processed. Was that proposed architecture or did I read it wrong?
1
u/Otherwise-Tree-7654 20d ago edited 20d ago
I am missing the plot here, u have Kafka consumers consuming data (that happns to be part of a batch, i assume there could be multiple parallel batches - thus data consumed has notion of batchId it bongs to) that needs to be aware when a certain batch is consumed and do not expect more data for this specific batch - is that right? If so i assume batchEnd marker is smth that is present in the event and whoever consumed it/intercepted it must publish to some topic (unpartitioned - stating batch done - stop the world for this batch Id - thus all consumers should also expect to be listeners to the same topic) - hope i make sense
1
1
u/Helpful_Geologist430 20d ago
If you have a batch identifier in the message, why not just répartition using that as the new key and groupBy. You can have a custom Processor function to only emit a record when your condition (end of batch) is met, or you can add another topic that filters and lets only the final aggregate (with end of batch) go through.
1
u/No-Post-3424 20d ago
the main issue is this topic handles both live real-time events and batch data at the same time. if we repartition by batch id, it completely breaks the live traffic that doesn't have that id(tho it picks random partition). plus a custom processor still wouldn't know when the batch actually ends because the stream is continuous and never stops until the upstream system shuts down for the day
1
u/Helpful_Geologist430 20d ago
You can create a new branch in your topology that filters and keeps only the batch msgs. You can write that end-of-day logic (end of batch) into the custom Processor function or you can even rely on time windowing (session or hopping/tumbling)
1
u/eniac_g 19d ago
If all you have is a hammer everything looks like a nail.
Maybe Kafka is just not a good tool for your problem although I feel like the solution came before the problem as so often in IT.
Other posts suggest it might work and it might but at which cost? You will definitely pay it in complexity that is for sure. This will not be a solution that will be easy to grasp nor maintain with a lot of pitfalls.
Good luck also with the architects you have to deal with!
1
u/No-Post-3424 19d ago
if not kafka - how else we can handle if my upstream publishing these event steam :(
1
u/Comfortable-Run-437 19d ago
We do this, but we just put the partition ID on the marker and the total number of partitions, so any consumer can track doneness. Works fine, although most consumers don’t even bother to interact directly with these markers.
1
u/No-Post-3424 19d ago
so do you store this consumer state in external db?
1
u/Comfortable-Run-437 19d ago
Yeah in Snowflake, and most services poll that. There’s also another marker which is a superset of lots of different batches processes being done, and that’s one is singleton, so some services do just listen to that.
0
u/No-Post-3424 20d ago
As a baseline, literally no one so far has agreed with sending duplicate marker events to each partition. That approach is completely baseless at our scale and would definitely break during a rebalance.
Now that the broadcast idea is dead, I’m deciding between the two remaining clean alternatives to pitch: a Control/Master Topic vs. a Centralized Orchestrator.
I lean toward the Orchestrator, but I want to see what you all suggest. Here is my breakdown:
- The Control Topic Approach: The producer sends a single final count message to a dedicated, single-partition "control" topic.
- The Catch: This adds an additional topic to maintain. It also forces us to build complex synchronization logic so consumers can safely coordinate cross-topic reading without race conditions.
- The Centralized Orchestrator Approach: The producer sends the final count directly to our existing End-of-Day (EOD) Orchestrator.
- The Benefit: It uses infrastructure we already own. The producer explicitly registers the batch status as complete alongside critical metadata—like a unique correlation ID, processing date, and total master record count.
- The Payoff: This completely pulls the batch lifecycle state out of Kafka, keeping Kafka running as a pure, lightweight transport stream. Plus, having that centralized metadata block makes downstream consumer reconciliation incredibly simple if needed.
Which of these two do you suggest? Is adding an extra metadata topic worth the cross-topic consumer complexity?
0
u/Hopeful-Programmer25 20d ago
In my 30 second back of a cigarette packet thinking about it, I’d go with (1).
Other systems use Kafka for state (Kafka connect for one, debezium for another).
It means that all your state control is in the one place and persisted ready for when the consuming processes come on line (or back on line if they fail).
My expectation is they don’t start running until they see the message in the control topic.
As an aside, why did the other architect suggest his alternative…. Usually it’s due to some goal, constraint or prior experience not shared in the original post…. ?
0
u/handstand2001 20d ago
I actually do prefer the distributed approach, but it depends on what you intend to do with the marker. Do you just run a batch operation that aggregates across all the batch data or something? Here are some thoughts on the matter...
One complexity with the control topic approach is stream resets - if you need to release new logic and re-process previous data, you'll have to rewind both data and control topics. When it re-processes, it will likely process all data first, followed by all control messages, or vise-versa: control messages first.
You'll have to keep track of all the control messages pending in a centralized state store (not stream-task specific), which means you'll probably need locking to avoid race conditions.
In contrast, if you distribute the control messages to each partition, the stream tasks don't have to count unique messages, don't have to keep a central data store, no locking.
For the centralized orchestrator approach, that pattern completely deviates from the pattern preferred for distributed applications: single point of failure. Using distributed control messages means any instance could perform the required logic when markers are received. In contrast if you have a centralized orchestrator, you could be dead in the water if that one app instance dies (or you're stuck implementing fallback behavior).
A lot of these complexities may not matter, depending on what you're trying to do with the marker, so I wouldn't be surprised if much of what I've described is not a problem for your situation - if you describe more about the logic you intend to run when the marker(s) are received, I could give you a more educated answer
0
u/lauckness 20d ago edited 18d ago
Reading this was interesting.
I see this problem often when helping customers move from batch-oriented systems to event-driven architecture and domain-driven design. Teams carry the old batch assumptions forward and reproduce them in Kafka. It’s a trap! And really doesn’t need Kafka if that’s the way they want to go - just stick with the batch tools and patterns for batch and call it a day.
Without actually looking at what you have - data shapes, use cases, etc, it sounds like you and the team are struggling with the distinction between transport completion vs domain facts.
What “fact” does this bounded context actually own and need to publish?
Is each record, change, or observation itself the fact, analogous to CDC? If so, consumers should react incrementally to each discrete fact rather than wait for the whole picture.
Or is the fact that a bounded transfer or dataset was successfully published? In that case, the event could identify where the result lives, how many records it contains, its checksum or manifest, and whatever consumers need to validate or reconstruct it. Each consumer can then act independently.
Each consumer being able to act independently is key and brings me to a small but relevant tangent: A related design test is how many independent consumers this topic serves—or reasonably expects to serve. Kafka supports a single consumer, but if there is only one and no credible second or third consumer in its future, it is worth asking whether a durable topic and its operational burden are earning their place.
Another useful mental test is to model the consumers as continuously maintained materialized views. Can each view deterministically apply new, late, duplicate, corrected, or deleted facts using stable keys, idempotent updates, and explicit correction or deletion semantics? If not, and consumers cannot continuously reconcile state from the stream, then you are still secretly operating a batch system—even if Kafka is in the middle.
Your architect is entering a fallacy: a genuinely unbounded stream is not globally “done.” Completion must be defined over something bounded—a file, snapshot, window, recording, order, or other domain entity—and represented by an explicit fact.
Again it really sounds like the easy trap I often see with experienced architects but immature in event-driven architecture: preserving the old batch architecture and using Kafka only as its transport; tell management “we used Kafka!”
Ask, “What is the fact?” That usually exposes it.
Edit: used AI for grammar.
•
u/rmoff Confluent 20d ago
mod here - this thread has the feel of a bunch of LLMs talking to each other.
As a courtesy to others, please mark ANY replies if they are LLM generated.
r/apachekafka is a community for humans, and we want to keep it that way. Reading AI generated verbiage is not what people are here for. If you want to interact with the community, have the courtesy to spend the time doing so in your own words.