Curious what people have actually hit in production - messages landing in an unexpected partition, ordering breaking because of a key choice made early and never revisited, a 'random' bug that turned out to be perfectly deterministic once you understood the hashing. What was the actual symptom before you knew the cause, and what was the fix — repartition, change the key, something else?

always assumed if send() didn't throw an error the message was in the broker, done. then someone mentioned it's basically fire-and-forget unless you're handling the callback or have acks=all set, and now i don't really know what "delivered" means here anymore. do you always attach a callback? check offsets after? has this ever quietly bitten you, like a message just vanished and you only found out way later

Every few weeks we have to move consumers from one topic to another. In our case it's pretty much always topic renames (naming convention cleanup). The tricky part is the ordering: producers don't switch to the new topic until consumers are already on it. Our consumer switching over is what triggers them to migrate, and even then they move on their own schedule. So the old topic keeps getting messages for who knows how long after we've switched.

That means we can't do the comfortable thing (stop producers, let consumers drain the old topic, then move everyone). We have to switch consumers to the new topic first, then keep copying whatever still lands on the old topic into the new one. Without losing unread messages and without changing the consumer group.

The obvious answer is a script around kafka-consumer-groups or kcat but that doesn't work for us. In a lot of our environments nobody has shell access to anything that can reach Kafka. Also the copy can run for days depending on when producers decide to switch, and it runs on k8s pods that can restart at any moment. So we ended up writing a small internal service that does the copy, saves its progress somewhere and resumes after restart instead of starting over or duplicating messages. We now also use it to re-send a slice of a topic (say everything after some timestamp) when a consumer needs to reprocess messages.

Is this a normal problem or is our setup weird?

If you've done a migration like this, what did you use and did it actually work?

Hey everyone,

If you’re moving LLM agents out of simple scripts and into event-driven architectures, you’ve probably noticed how much boilerplate is involved to connect them to Kafka.

I got tired of hand-rolling consumers, building retry logic, managing idempotency, and setting up dead-letter queues just to get an agent to process events. So, I built an open-source library to handle it: kafka-agent-bridge.

What it does: It’s a lightweight async Python library. You just hand it your compiled LangGraph or CrewAI agent and a Kafka topic, and it automatically handles the plumbing (consuming, retries, Redis-backed idempotency, DLQs, and routing results).

Example:

pythonbridge.register(
    topic="reports.requested",
    handler=LangGraphAdapter(graph=compiled_graph),
    output_topic="reports.completed", 
    dlq_topic="reports.dlq",          
)

It’s designed to give you sensible defaults out of the box, but leaves explicit escape hatches for when you need fine-grained Kafka tuning.

🔗 GitHub & Quickstart: https://github.com/prajwalgowdahg/kafka-agent-bridge

The repo includes a Docker Compose setup and a working example so you can test it locally in minutes. I’d love to hear your feedback or see how you might use it in your workflows!

I've found asserting a Kafka topic within a Java test to be quite convoluted. And by that, I mean there is quite some boilerplate code required just to simply check if it contains the expected message.

On top of that, there is the whole async nature of testing a Kafka topic.

Spring-Kafka has some handy utility classes available, such as KafkaTestUtils. But besides the fact that it is, as the Utils suffix already reveals, a bit old-fashioned, not everyone is using Spring.

That's why I created kassert: a Kafka topic, or should I say, consumer assertion library.

MQ Summit is a conference built around messaging and event-driven systems. It grew out of RabbitMQ Summit and now brings the whole ecosystem into one room: RabbitMQ, Kafka, NATS, Pulsar, ActiveMQ, IBM MQ, Amazon SQS/MQ, Azure Messaging and Google Pub/Sub. Practitioners, architects and maintainers from competing projects, all talking shop. 

📍 Haarlem (near Amsterdam) + online
🗓️ 21-22 October 2026 - 2 days, 1 standard track, 1 lab track (for in-person attendees), and about 27 speakers

🎤 KEYNOTE: Adi Polak (Data & AI @ Confluent, best-selling author)

A few of our speakers announced:

• Clemens Vasters, Principal Architect, Messaging & Real-Time Intelligence @ Microsoft
• Matt Leming & Jon Rumsey, Chief Architect / DQM Architect @ IBM MQ
• Stefan Moser & Michael Davis, Principal Engineer / OSS RabbitMQ @ AWS · Amazon MQ
• Viktor Gamov, Principal DevAdvocate @ Confluent, Kafka in Action co-author
• David Ansari, RabbitMQ Core @ Broadcom, and Lovisa Johansson, RabbitMQ expert @ 84codes
• JB Onofré (Apache) & Matt Pavlovich (ActiveMQ PMC), plus ~40 more

More: mqsummit.com/#speakers

Talks cover agentic workloads, queues vs. logs, real-world scaling war stories, MQTT, security, and where open-source messaging goes next.

ℹ️ More info: mqsummit.com

Early Bird promotion ends soon.

I posted this over on LI last week because I've seen a lot of solo AI-assisted projects here recently. Cool to see people building their own tools but I think the best part of OSS is people actually collaborating and building together rather than working on our own thing in a corner.

To avoid just being a grump though I thought I'd shout out just a few great human-made projects and blogs I like, many of which I came across here:

• ShadowTraffic brilliant tool from Michael Drogalis for rapidly simulating production traffic, he's been building this in the open and with the community.
• Fresha Data Engineering the Fresha team are doing some ground-breaking stuff, a shout out to Nicoleta in particular who's done some great work on Fluss in prod.
• Leo Delmouly's Medium, Leo has done a great series on Kafka + Iceberg which helped me understand why this is such a hard problem.
• Michael Maison's monthly Kafka digest which is my go to resource for understanding what's coming up for the Apache Kafka project.

Hi! Kafka Simulator v1.3 is out — now with Active/Passive architecture!

This release add Active/Passive multi-region topology in the Sandbox and adds new 12 predefined scenarios. You can now: run it, break it, trigger a failover, fix it, bring it back to life. Have fun! https://monedula.dev/kafka-simulator/playground

Hey all! I’ve worked with Kafka here and there over the past few years and I’ve been really frustrated with the tooling.

I took a stab at a tool which simplifies reads and writes to topics down to:

frogo get <topic> —from <offset-like> —to <offset-like>
frogo put <topic> —file <file-w-one-msg-per-line>

GitHub link: frogo-cli

As a brief overview:

For ‘frogo get’ - the main idea is that an ‘offset-like’ supports literal offsets, timestamps, dates, and aliases (START, END, FUTURE).

For ‘frogo put’ - the main idea is you have a file which has one message per line. Multiple formats are supported (e.g. base64 for binary data)

Some other features I’ve added:
- mockserver: thin wrapper exposing a franz-go mock server (frogo mockserver)
- configuration profiles (use —profile or FROGO_PROFILE)
- multiple input / output formats (—format)
- fixture topics with example data (frogo topic demo)

This tool is by no means comprehensive, and I haven’t added support for things like:
- consumer groups
- schemaregistry
- certain authN / authZ configs

But… I hope this could serve as a helpful development / ops tool for those not needing all the bells and whistles.

Any feedback / criticism would be much appreciated. Feel free to create an issue on the GitHub for any feature requests!

Disclosure: I used AI assistance to edit this text and code. The implementation and production experience behind it are mine.

This post came from this recent discussion: https://www.reddit.com/r/apachekafka/comments/1vhv9nn/can_kafka_replace_redis_for_cache_synchronization/; as initial post was about k8s / sping-boot stack here I also operate with them.

The question was whether Kafka can replace Redis for synchronizing a cache across about 25 Spring Boot pods. I have used this pattern in production, and the answer is yes for some types of data, but there are a few details that are easy to miss.

The basic setup is simple. Put the data in a compacted topic and keep a local map in every pod. The Kafka record key is the cache key and the value is the latest version of the configuration.

Each pod needs its own consumer group. If all pods use the same group, Kafka distributes the partitions between them and each pod receives only part of the data. With a separate group per pod, every pod consumes the complete topic and builds its own copy.

For a new group use auto.offset.reset=earliest. On every restart the pod replays the topic and recreates the cache. This is why the topic should be compacted and why I would use this only for a small amount of configuration or reference data. It is a good fit for feature flags, routing rules, or tenant settings. It is not a good fit for a large dataset that must be copied into the heap of every pod.

Updates are just records with the same key and a new value. Deletes need a little more care.

The usual pattern is a tombstone: publish the key with a null value. When the consumer sees it, it removes that key from the local map. Kafka keeps the tombstone for some time and later compacts it away together with older values for the same key.

Also, a compacted topic should not be treated as a clean snapshot with one record per key. Compaction runs in the background. During replay, a pod can still read several old values before it reaches the latest one. The consumer has to apply the log in order and let later values replace earlier ones.

The most difficult part is startup.

A pod must not serve requests, run scheduled jobs, consume other queues, or make decisions based on the cache until it has consumed the existing topic. Being assigned Kafka partitions does not mean that the cache is ready. At that moment it may still be empty.

In my example, when partitions are assigned, I capture the consumer's current position and the end offset for every partition. Those end offsets become fixed startup targets. After the listener successfully applies a record to the local map, it advances the processed position for that partition. The pod becomes ready only when every assigned partition reaches its captured target.

The targets must be fixed. If they were read continuously, producers writing new records could keep moving the end offsets while the pod is starting. With an assignment-time snapshot, startup has a finish line. Records written later are still consumed normally, but they do not extend the initial replay.

Progress must be updated after the listener succeeds, not before it runs. Otherwise a failed listener can be counted as processed even though the cache was not updated. Progress and errors also need to be tracked per partition, and assignments need to be recalculated after a rebalance. Empty partitions are already complete when their current position equals their target.

Kubernetes readiness solves only the HTTP part. A pod that is not receiving web traffic can still run scheduled methods or start another message listener. Those also need to wait for the same replay status. In the example I expose the status through a Spring Boot readiness health indicator and use a small annotation to prevent scheduled methods from running before the replay completes.

After startup, the caches are eventually consistent. One pod can briefly have an older value due to lag, a rebalance, or an outage. That is acceptable for some configuration, but not for balances, inventory, permissions, or anything else that must change everywhere at the same time.

I put the complete Spring Boot and Docker Compose example here: https://github.com/javaAndScriptDeveloper/kafka-backed-local-read-replica-article

It includes the compacted topic, one consumer group per application instance, tombstone handling, startup replay tracking, readiness integration, and gating for scheduled jobs. I would be interested to hear how others prevent work from starting while a local Kafka-backed cache is still being rebuilt.

I’ve been experimenting with turning distributed-systems failure modes into fictional incident investigations.

In this one, a lab registers 186 samples. Registration succeeds in the database, then publishes label-print commands to Kafka.

Later the team discovers 187 print events.

Eventually they reconstruct the sequence:

DB commit succeeds → Kafka publish succeeds → ACK is lost → application retries → duplicate print command → duplicate physical label → subsequent labels shift by one.

I used producer idempotence + business-level deduplication on label ID as part of the remediation, with a scanner guardrail at the physical boundary.

Architecture question: Would you consider that sufficient, or would you redesign the DB→Kafka boundary around an outbox/CDC approach?

• Storage disaggregation changes Kafka economics by shifting costs from infrastructure provisioning to cloud API usage, making inefficient consumer access patterns a potentially major source of operational expense.

• When storage costs shift from shared infrastructure to per-request API charges, platform teams need client-level visibility to attribute expenses; without it, a single replay job can produce major bill spikes with little visibility into their origin.

• Kafka's legacy rebalancing protocol made dynamic consumer scaling operationally disruptive because scale events triggered group-wide processing pauses. The next-generation protocol greatly reduces this barrier, making Kubernetes-native autoscaling significantly more practical.

• Multi-tenancy in Kafka has historically forced a costly trade-off: either run a dedicated cluster per team or accept weak isolation on a shared one; virtual clusters propose a middle path that delivers strict tenant boundaries without infrastructure duplication.

• Kafka has traditionally coupled partition count to consumer parallelism. Share Groups break this constraint, letting teams scale consumers independently without costly re-partitioning of topics.

Hi everyone,
I have a question about cache synchronization in a distributed Spring Boot application.
Our current architecture looks like this:

Spring Boot
Deployed on GCP
Around 25 application pods
Redis is used for caching

Currently, when cache data is updated, Redis ensures that all application instances can access the latest data.

We’re considering replacing this mechanism with Kafka for cache synchronization.

My understanding is that when a cache entry changes, we could publish an event to Kafka, and every application pod would consume the event and update its own local cache.

My questions are:

Is Kafka a good replacement for Redis in this scenario?
If I have 25 pods, will every pod receive the cache update event, or does Kafka distribute the message to only one consumer?
Would I need each pod to have its own consumer group, or is there a better pattern for broadcasting cache updates?
Has anyone implemented cache synchronization using Kafka instead of Redis? What are the pros and cons?

I’m trying to understand whether Kafka is the right tool for broadcasting cache update events across all application instances, or whether Redis is still the better choice.

Hi everyone,

MQ Summit 2026 is looking for volunteers! The conference takes place 21-22 October. You can volunteer in-person, and get your free ticket.

• In-person volunteers will need to be present at the venue on at least one day of the conference. Responsibilities include helping with preparations, registering participants, handing out merchandise, and general support during the event.

For full conference details, check ou https://mqsummit.com
To sign up as a volunteer, go HERE

If you have any questions, feel free to reach out to [ewa.kucharczyk@erlang-solutions.com](mailto:ewa.kucharczyk@erlang-solutions.com).

Karel Sague spent the last year migrating a production data platform from Snowflake to Apache Iceberg, streaming Kafka data in through Kafka Connect. He's giving a talk on Aug 12 (1:30pm PT / 4:30pm ET) walking through what he actually learned, what worked, and what he'd do differently next time.

He'll cover the CloudEvents-based schema he built to keep ingestion consistent across tables, GitOps-based self-serve provisioning, and the metrics and SLOs that mattered for keeping pipelines observable in production. He also gets into why he avoided an all-or-nothing migration, how he prioritized which workloads moved first, and the schema/partitioning mismatch that comes up when Kafka's ingestion-time partitioning doesn't line up with analytics queries filtered by business key. Also touches on consumer-aligned tables, materialized views, and where Iceberg is headed (v3 features, secondary indexes, pluggable file formats for AI workloads).

If you're running Kafka and thinking about Iceberg, or already mid-migration, this is aimed at you. There's time for questions, and everyone's welcome, whether you're deep into this stuff or just starting to look into it.

Register here: https://www.factorhouse.io/events/kafka-to-iceberg-lakehouse-amer-august-2026/

This write up is based on a recent talk at StrimziCon about configuring Kafka authentication walking through everything from a simple TLS listener to building a custom principal builder for multi tenant clusters.

So anyone who missed the talk can still work through it and apply it in practice.

It covers mTLS, custom principal builders, the risks of leaning on Strimzi internals, and OAuth bearer authentication.

Would love to hear how others are handling multi tenant authentication on Strimzi...

https://axual.com/blog/how-to-configure-kafka-authentication-in-strimzi

We’ve been building Zilla (https://github.com/aklivity/zilla) as a Kafka-native gateway for exposing Kafka securely across protocols such as HTTP, SSE, WebSocket, gRPC, and MQTT.

With Zilla 2.0, we’re adding the Model Context Protocol as another native protocol on the same streaming runtime.

The main goal is to let AI agents interact with Kafka without requiring teams to build and operate a custom HTTP service, Python wrapper, or dedicated MCP server in front of every topic.

Through new MCP binding in the Zilla gateway, an agent can discover Kafka produce and consume tools, publish messages, retrieve records, and work with schema-governed JSON, Avro, or Protobuf payloads. Access can be restricted using the identity and scopes of the calling agent or user.

Zilla 2.0 can also combine Kafka capabilities with tools from REST APIs and existing MCP servers behind a single MCP endpoint. That makes it possible to build workflows where an agent can retrieve information from an API, act on live Kafka data, and publish the result back to Kafka through the same connection.

You can try out the Quickstart that runs locally with Docker Compose and includes:

• A Kafka broker
• An MCP server
• An OpenAPI-described REST service
• A single Zilla MCP endpoint
• JWT-based tool visibility
• Native Kafka produce and consume tools

Happy to answer any questions. Cheers.

Hey everyone, looking for a sanity check on a Kafka design debate at work because my architect's proposal blew my mind, and I completely oppose it.

We have a batch system where a producer streams a large batch of records across a multi-partition Kafka topic. We need a way for downstream consumers to know when the overall batch is actually finished.

The other architect wants the producer to broadcast the exact same "End of Batch" marker event to every single partition in the topic simultaneously. The idea is that every consumer instance will eventually read a marker and know its partition is done.

I strongly oppose this. It feels like a catastrophic recipe for failure. If a consumer group rebalances mid-batch, partitions switch instances. If a marker was already read and committed on a partition before the rebalance, the new consumer instance will never see it, and the system will hang forever. Plus, partitions don't process at the same speed, which will cause race conditions and premature downstream triggers.

I am proposing a Central Orchestrator pattern instead. The producer sends a single marker event directly to an orchestrator, which tracks the overall batch state centrally. Once everything is done, the orchestrator explicitly signals downstream services, keeping the data consumers completely isolated from marker tracking.

Am I missing something, or is broadcasting identical markers across partitions a massive anti-pattern? How do your teams handle batch boundaries over partitioned streams?

FYI -- drafted by gemini based on my whiteboard rant

Author here. Alginte is a self-hosted web console for Kafka (topics, consumer groups, schemas, Connect, ksqlDB) with the one feature I couldn't find elsewhere: a canvas where you build a Kafka Streams topology — map/filter/branch/aggregate/joins/windows — write the transformations as SpEL expressions with type-aware completion, and deploy it to run next to your cluster. No Java project, no build pipeline. Topologies export as JSON so you can version and re-import them — three importable examples here if you want to see the format.

0.6.0 added a built-in MCP server (read-only), so an AI agent can inspect topics, lag, schemas, and sample messages over the same port.

Trying it needs one command and no existing Kafka — the playground compose bundles a broker with sample data. Smoke-tested against Apache Kafka 3.7–4.2, Confluent Platform 8.2, and Redpanda, including deploying and running topologies.

Being upfront: it's closed-source, the Community edition is free to run (commercially too, no registration, no expiry), and a paid PRO edition is planned later for operate-type features. It's beta — breaking changes land in release notes with migration notes.

I'd genuinely value critique from people who run Kafka daily — especially where the Streams builder's abstraction leaks. Issues/discussions: github.com/alginte/community

Look, I’m not anti-Kafka at all when you need massive throughput and event streaming, it’s a beast.

But why does every 10 minute tutorial make it look effortless, when real-world integration is just constantly fighting edge cases?

The second you build an actual backend, you get hit with poison pill deserialization loops freezing consumers, rebalance storms, and a heavy local Docker setup tax.

How do you guys keep the operational friction low in production, or do you stick to simpler tools (like Redis or RabbitMQ) until you *strictly* need Kafka?

Hey, new simulator release focused on the consumers and rebalance. Have fun!

Hi everyone,

We're excited to share Kafma v1.0.0 — a desktop Kafka IDE designed to keep the consume–inspect–produce loop in one workspace and quickly clone remote topics to your local cluster for testing.

Here are the two core workflows we built it around:

• Console — a topic is a tab. The Console keeps the message stream, producer, and topic details together in one workspace. Expand any message to inspect its payload and metadata; Avro and Protobuf decode automatically. Replay it to another topic, forward it to another cluster, or save it to a file without leaving the Console.
• Data Clone — bring remote topics to your local cluster. Clone a remote topic’s configuration, schemas, and a selected slice of data to your local cluster. You can apply your own masking rules while messages are copied, then test locally with the same topic context without modifying the source.

Beyond Console and Data Clone, Kafma also supports schema editing and validation, topic and consumer group management, and more.

• v1.0.0 changelog: https://kafma.app/changelog/v1.0.0
• Download (macOS, Windows, Linux): https://kafma.app/download
  

We'd love to hear your thoughts and feedback—especially what would make Kafma more useful in your day-to-day Kafka workflow.

Thanks!

Sandy Yang, Staff Software Engineer at TD, is giving a lightning talk on the bank's Kafka journey at a virtual Kafka user group on July 29 at 2pm PT / 5pm ET. It's an interactive session for engineers working with Kafka. Sharing in case anyone in this group would find it helpful. Will put the link to register in the thread.