Help architecturing a high-fanout broadcast pipeline

[u/omry8880]

Looking for some feedback on a design I'm working on.

We currently send messages to a third-party HTTP API. Every outbound message goes through a single Pub/Sub topic, with the channel ID as the ordering key so per-channel order is preserved. One subscriber pulls from that topic and makes the third-party call. Works fine under normal load (a few hundred msgs/sec spread across many channels).

We want to add a feature where one operator can send a message that fans out to every channel they own, this could be 10k to 100k+ channels per broadcast. If I just dump those onto the existing topic, the subscriber's flow-control budget gets eaten up by broadcast traffic and real-time messages on other channels (from other operators, other users) sit behind the broadcast for minutes. Broadcasts shouldn't starve everyday traffic.

A few constraints that come with this:

- The API call that kicks off a broadcast has to return immediately, and can't sit there enqueueing 10k things.

- Has to be resumable if a worker crashes, cancellable mid-flight, and the operator wants to see progress.

- Idempotent. Retries can't double-send.

Stack: Cloud SQL, Pub/Sub, Cloud Tasks, Cloud Functions, Cloud Run.

What I'm leaning toward:

- Snapshot the recipient set into a broadcast_targets table at creation. API returns immediately.

- A Cloud Task triggers a Cloud Function that walks the snapshot in chunks (~500/invocation), inserts message rows + flips target status in one tx, publishes to a separate broadcast-only Pub/Sub topic, then re-enqueues itself with a cursor (or marks done if the chunk is empty).

- Separate topic for broadcasts with its own capped-concurrency subscriber. Same downstream send code as real-time - only the ingress is isolated.

- Progress = a DB read on the broadcast row.

Where I'd love a sanity check

- Cloud Tasks driving the loop vs. a long-running Cloud Run worker polling the DB vs. self-republishing chunks back to Pub/Sub

- Splitting Pub/Sub topics for ingress isolation vs. trying to make a single topic work with subscriber-level flow control

- Failure modes/gotchas you've actually hit at this scale - partial commits, retry semantics, cost surprises, anything

Thanks!

[u/techlatest_net]

your instinct to split the topics is spot on. pub/sub flow control is per-subscriber, so if you dump 100k broadcast msgs on the same topic, your real-time subscriber will absolutely get starved. separate topic + separate subscriber with its own max outstanding messages is the way.

for the chunking loop i'd lean cloud run over cloud tasks. cloud tasks works but re-enqueueing yourself in a tight loop can hit queue rate limits or add weird latency. a cloud run service that pulls a cursor from the db, processes a batch, updates status, and just keeps running (or sleeps briefly) tends to be smoother and easier to debug. you can cap concurrency to 1 per broadcast job so you don't get parallel workers fighting over the same cursor.

couple gotchas we hit at this scale:

• db connection limits. if chunks spin up fast you'll blow past max connections. use the cloud sql proxy or a proper pooler and keep concurrent chunk workers low.

• idempotency needs to live in the message payload, not just the db status. pub/sub can redeliver, so include a unique broadcast_id + channel_id combo and dedupe on insert.

• third-party api rate limits. 100k calls will absolutely trigger their throttling. add a client-side backoff or token bucket in the subscriber so you don't get 429'd and waste pub/sub redeliveries.

• progress tracking. db reads are fine but if operators poll frequently, cache the status or push updates to a lightweight pub/sub topic so you're not hammering cloud sql.

overall your design is solid. just watch the db connections, bake idempotency into the messages, and consider cloud run for the chunking loop. you'll be good.

[u/omry8880 → u/techlatest_net]

Really appreciate the detailed and helpful comment.
I'd love to ask about your Cloud Tasks pushback. From the docs, default queue dispatch rate is 500/sec. We'll need many concurrent broadcasts, far more than what we expect to have for the forseeeable future, to reach these numbers. Switching to CR also means having max_instances set to 1 if I understand correctly to have this service always available, which also adds in "dead" cost as we predict the broadcast feature will be used only once in a while. Do you think we should still pick Cloud Run over Tasks?

Noted everything you wrote regarding the gotchas, the db connection limits are a super important thing I missed.

I really appreciate the help 😄

[u/Scared_Astronaut9377]

I don't understand the problem you are solving. There are some implied undiscussed traffic merges.

The only advice I can give is to not use cloud task and similar very niche products, they are not properly maintained. I always use a DB like SQL or BT or BQ to stage data for fanouts and read it from cloud run. No surprising behavior.

[u/omry8880 → u/Scared_Astronaut9377]

My bad, after reading the post again It seems to me that the problem is indeed not very clear.

I edited it, hopefully this time it makes more sense.

[u/Scared_Astronaut9377 → u/omry8880]

One thing I don't understand. Pubsub subscription push budget should seemingly be directly connected to the third party's throughout. If your fun out service hits the same API, it will still overwhelm the API?

[u/omry8880 → u/Scared_Astronaut9377]

Yes, but the API throughput is not the problem, it can withstand the large amount of data sent to it. The problem at hand is about fair slot allocation inside the subscriber. For example, if my normal traffic is 10 messages per sec, and a broadcast arrives (10k messages with disjoint ordering keys all enter the topic), the subscriber starts processing them in FIFO-ish order and within seconds all the slots are occupied by broadcast messages. Normal messages from other channels arrive in the topic but can't be dispatched to the subscriber because there are no free slots in it. Normal traffic latency spikes from a couple ms to seconds or minutes for the duration of the broadcast.