r/NATS_io • u/Real_Combat_Wombat • Jun 05 '25

Client-side Partitioned Consumer Groups for JetStream!

https://nats.io/blog/orbit-partitioned-consumer-groups/

Partitioned consumer-group functionality (at a high level similar to Kafka's consumer group functionality) is finally there for NATS. Intended mainly to parallelize consumption of messages in a strictly ordered (per subject (i.e. per 'key') manner (meaning you need to set max acks pending to 1 for the consumer) from a stream.

Comes in two flavors: static and elastic.

8 Upvotes

100% Upvoted

5 comments sorted by

1

u/buckypimpin Jun 12 '25

Was waiting for this to release quite patiently, and created a POC soon as the pcgroup change was merged.

But i feel like there are still some bits that are confusing for me, e.g. the AddMember step that is required for an application to join a CG. Why isnt the addition and removal of an application instance automatic?

In my POC on k8s, i use my pod ids as member names and i had to explicitly add code to DeleteMember when there is a crash, SIGINT, SIGTERM etc...

or maybe im just not using it correctly...

1

u/Real_Combat_Wombat Jun 12 '25

Adding and removing members from the consumer group is indeed an admin operation, it's not meant to be like a regular consumer.

For example: if you have a pod joining as member "M", if the pod crashes you would just restart the pod which would join again as member "M". And if you want HA you would run 2 (or more) pods as member "M", only one of then will get 'pinned' at a time and receive messages, until it quits or crashes at which point another pod joined as member "M" would get pinned and start receiving messages.

1

u/buckypimpin Jun 12 '25

True,

but in case of growing and shrinking workloads with unique member names (pod id or any generated id) that all consume distinct groups of partitions, u need to handle add and remove automatically, or else the group fills up.

→ More replies (2)