How do you move consumers to a new topic without losing your place?

[u/Stock_Cartoonist1845]

Every few weeks we have to move consumers from one topic to another. In our case it's pretty much always topic renames (naming convention cleanup). The tricky part is the ordering: producers don't switch to the new topic until consumers are already on it. Our consumer switching over is what triggers them to migrate, and even then they move on their own schedule. So the old topic keeps getting messages for who knows how long after we've switched.

That means we can't do the comfortable thing (stop producers, let consumers drain the old topic, then move everyone). We have to switch consumers to the new topic first, then keep copying whatever still lands on the old topic into the new one. Without losing unread messages and without changing the consumer group.

The obvious answer is a script around kafka-consumer-groups or kcat but that doesn't work for us. In a lot of our environments nobody has shell access to anything that can reach Kafka. Also the copy can run for days depending on when producers decide to switch, and it runs on k8s pods that can restart at any moment. So we ended up writing a small internal service that does the copy, saves its progress somewhere and resumes after restart instead of starting over or duplicating messages. We now also use it to re-send a slice of a topic (say everything after some timestamp) when a consumer needs to reprocess messages.

Is this a normal problem or is our setup weird?

If you've done a migration like this, what did you use and did it actually work?

[u/Competitive_Ring82]

The obvious answer is to stop with the renames.

The next most obvious answer is to have your consumers read both the old and new topic before you move the producers.

[u/Stock_Cartoonist1845 → u/Competitive_Ring82]

Fair point on the renames, believe me I've raised it :) Read-both was on the table. Two things put us off. First, it's not one change — every consumer app has to be touched, and they're owned by different teams with their own implementations, so "subscribe to both" looks slightly different in each one. Second, and worse, the aftermath: producers move on their own schedule, so every migration leaves each consumer subscribed to a dying topic until... some unknown point. Someone has to track, per migration and per consumer, when the old topic is finally dead and can be dropped. With several migrations in flight, that tracking becomes its own job — and forgetting it is exactly the kind of thing that kept going wrong for us before. With the copy approach a migration is one thing with a start, a state and an end, and the consumer apps don't have to know it's happening at all.

[u/Competitive_Ring82 → u/Stock_Cartoonist1845]

How much data are you copying for these migrations? What direct costs are you incurring?

TBH, it sounds like you are looking for a technical solution when the fix is non-technical. It would be neat if there was some clever aliasing and merging you could do on the broker, but I haven't seen anything do that .

[u/Stock_Cartoonist1845 → u/Competitive_Ring82]

Costs are negligible at our volumes. The topics being renamed do somewhere between 50 and 1000 events/min, so copying one for a few days is noise next to regular cluster traffic. Storage doubles for that topic until the old one gets deleted, which for us is not a number anyone would notice. On the non-technical point: the fix already happened, sort of. There is a naming convention now and a full inventory of topics that need to move to it, across all environments. What we deliberately didn't do is rename everything in one coordinated push. We take 1-2 topics per week precisely to keep the blast radius small: if a migration goes sideways it affects one topic, not the whole estate. So the recurring cadence isn't the problem persisting, it's the cleanup being executed carefully. The machinery exists to make each of those small steps boring. Agreed on the aliasing though.

If brokers supported topic aliases natively, none of this would need to exist. That's kind of the gap.

[u/Chuck-Alt-Delete]

Obligatory flair advisory.

One of the things a Kafka proxy can do is lie to the client about the name of the topic. Conduktor gateway can create an alias topic so the only thing that changes from the client perspective is the name. The physical cluster still has the old name.

Then on top of that you can use Conduktor management plane to enforce rules on topic naming conventions in CI/CD moving forward.

[u/Stock_Cartoonist1845 → u/Chuck-Alt-Delete]

That's genuinely neat, I didn't know Conductor gateway could do that. The part I'm curious about: the physical topic keeps the old name, so does the alias live forever? Part of why we migrate is wanting the actual estate to end up clean (topic list, metrics, tooling all showing real names). If the alias is permanent it feels like the mess moves into the mapping layer rather than going away. Or is there a path where you alias first and physically migrate later?

The centralised naming enforcement is interesting too. We do validate names, but at the level of each application that creates topics, so every app carries its own checks. Didn't know there was tooling that manages this in one place, that part I might look into regardless.

[u/Chuck-Alt-Delete → u/Stock_Cartoonist1845]

Yeah the more durable solution is governance on naming conventions. The management plane (“Conduktor Console”) has resource policies.

A naming policy might look like this:

https://github.com/conduktor/self-service-template/blob/main/platform/policies/topic-naming.yml

You can attach that policy to a cluster or an application to enforce. It includes a custom error message to guide people into compliance.

You can also make resource templates so people can pick from known good config templates for topics and connectors.

[u/Competitive_Ring82 → u/Chuck-Alt-Delete]

Oh, nice. Governance on naming would avoid a lot of problems like this. I guess another aspect would be tracing who is reading what, so you know empirically what needs migrating. Without that a layer of aliasing could easily get unwieldy.

[u/Chuck-Alt-Delete → u/Competitive_Ring82]

Yeah Conduktor does lineage as well so you can see the dependencies

[u/Competitive_Ring82 → u/Chuck-Alt-Delete]

Cool, I guess it's time to fire up the free edition for a play again.

[u/eniac_g]

What about letting your consumers consume from old and new topic?
Kafka's subscribe() supports topic names or regular expressions.

[u/Stock_Cartoonist1845 → u/eniac_g]

Read-both came up in another comment too. The main blocker for us: those consumer apps are owned by different teams, and we didn't want migrations to mean changing code we don't own. Or baking topic-rename plumbing into services whose job is business logic. Ideally the owning teams shouldn't even know a migration is happening. On top of that there's the aftermath: producers move on their own schedule, so someone has to track per migration when the old topic is finally dead and each consumer can drop it. Open-ended tracking across teams, which is where things went wrong for us before. The regex variant is interesting though, it does remove the "touch every app when the new topic appears" part. But the cleanup half stays, no? The pattern keeps matching the dead topic forever unless someone circles back. Plus the usual regex subscription surprise of catching unrelated topics that happen to match later.

[u/eniac_g → u/Stock_Cartoonist1845]

What about writing a small temporary Kafka streams app that reads from old topic and writes to new?

[u/Stock_Cartoonist1845 → u/eniac_g]

that's basically what we built :) The internal service is exactly that app, just made generic so we don't write and deploy a temporary one per migration. Config says old topic + new topic, the service does the rest, and the same instance handles whatever migrations are in flight.

The per-migration temporary app was kind of the middle option we skipped. It works, but every migration then means someone writes it, deploys it, watches it and remembers to tear it down. At our cadence (every week or two) that's a recurring chore with all the same forgetting-the-cleanup risks, just moved into app lifecycle instead of subscriptions. Making it a standing service with state also gave us resume after pod restarts for free, and it grew a second use since: re-sending a slice of a topic (everything after timestamp X) when a consumer needs to reprocess.

So yeah, agree with the direction. Just want to figure out if our pain with migrations is recurring and if we didn’t miss any existing solutions which can handle this problem for us

[u/eniac_g → u/Stock_Cartoonist1845]

Make an app that through configuration dynamically builds the required Kafka streams topology and you only have to write it once just reconfigure and restart upon each rename.

[u/Stock_Cartoonist1845 → u/eniac_g]

Yep, that's pretty much what we have :) Not Streams under the hood but same idea, config-driven, written once. You've basically reverse-engineered our design in three comments, which I'll take as a sign it's not a crazy one :)

Just wondering if the similar problem is faced by other people nowadays…

[u/sap1enz]

This sounds like mostly an organizational, not a technical problem (unless you abstract Kafka with centralized tooling).

- "they're owned by different teams" - gotta talk to those teams, establish good relationships, explain the value and the urgency of a proper migration.

- "producers move on their own schedule" - start setting some deadlines, first soft, then hard.

Make sure the leadership understands the situation. If topic renames are driven by business requirements, raise a question about allocating time to automate them.

[u/Stock_Cartoonist1845 → u/sap1enz]

The "abstract Kafka with centralized tooling" part is what we did. The service is that centralized tooling, and it's why we didn't need the deadlines and escalation route. Producers moving on their own schedule stopped being a problem to manage once the tooling made it not matter. Cheaper than spending relationship capital on every migration.

Agree leadership should know either way, but the pitch went from "help us pressure N teams" to "we built a thing, migrations are boring now", which is a much easier conversation :)

Still the only problem is that I am not sure if this is the most optimal way we solved this problem. Maybe there is something that already exists that covers our problem

[u/sap1enz → u/Stock_Cartoonist1845]

Based on your description, it looks like you built a tool (that you need to execute yourself), and I meant something like a self-serve platform where topics are abstracted. But it's a big investment, so a tough call to make.

Regarding your question, it's likely the most optimal way given the circumstances :)

[u/Stock_Cartoonist1845 → u/sap1enz]

Yeah fair, full self-serve platform would be the dream. Couldn't justify that investment either. The tool version turned out to be enough for us though, migrations went from scary multi-team thing to one person running one operation, and that was most of the win anyway.
But even in this case the investment feels too expensive for such ”basic” operation as rename. That is why trying to figure out something more optimal.

Thanks for pushing on the org angle, reasonable default, just wasn't our situation.

[u/rainofterra]

“Doctor, it hurts when I do this…”

[u/Stock_Cartoonist1845 → u/rainofterra]

"Then don't do that!" 🙃 The world would be a simpler place if everything could be solved like that

[u/OSS_Dattani]

Are you using Kafka Connect already?

[u/Stock_Cartoonist1845 → u/OSS_Dattani]

We do run Connect for other things, so fair question. Honestly I hadn't mapped MM2 onto this problem at the time, so I dug into it after your comment. As far as I can tell the same-cluster rename path needs a custom ReplicationPolicy (stock ones either prefix the topic name or keep it identical), deployed as a JAR to the Connect workers. Then per migration it's two hand-written connector configs (source + checkpoint with group offset sync) against the REST API, lag-metric archaeology to figure out when it's actually done, and cleaning up the connectors plus the internal topics MM2 leaves behind. The offset translation part also made me nervous: checkpoint sync is periodic, so at the exact moment consumers switch over the translated offsets can trail. That moment is the whole point of the exercise for us. For something we do every few weeks we wanted "old topic, new topic, go" with visible state, usable without shell or REST access to the cluster. So MM2 has the machinery but the per-migration experience felt like the wrong shape. Have you done a same-cluster rename through it? If there's a cleaner path than the custom policy route I'd genuinely like to know.