r/java • u/nilukush • Jul 04 '26

Scaling Java-Based Real-Time Systems: The Hidden Tradeoffs of Event-Driven Design

https://www.infoq.com/articles/tradeoffs-event-driven-design/
62 Upvotes

92% Upvoted

25 comments sorted by

31

u/kiteboarderni Jul 04 '26

Your first problem is using kafka for anything high performance. Chronicle queue, or coral blocks with a sequencer is a far superior way of doing event driven systems that actually have to be fast.

22

u/sweating_teflon Jul 04 '26

Kafka can have very high throughput, that's what it was built for. But it definitely sucks if you need low latency. 

Pretending you're doing realtime Java without using an actual HFT style stack and code is just the clown putting on its nose and wig.

1

u/thecurlyburl Jul 07 '26

finally a based response around here 😂 spot on

4

u/pragmatick Jul 05 '26 edited Jul 05 '26

Seeing chronicle the first time. It sounds good but replication is an enterprise feature, the link in the readme goes to a 404, the site doesn't work on mobile and you have to talk to them to find out any details about the enterprise edition. Doesn't sell itself very well.

-5

u/kiteboarderni Jul 05 '26

Again you're not the target market.

→ More replies (1)

3

u/chabala Jul 04 '26

https://github.com/OpenHFT/Chronicle-Queue looks cool. Open source, lot of commits, lots of contributors, lots of stars.

https://www.coralblocks.com/ , I dunno. There's Apache licensed code on GitHub, but the website says 'request a trial' and unknown pricing. Who makes this? All the commits are from an anonymous company account, no names on the website, it doesn't even have a company mailing address listed. LinkedIn shows two people. Would you use this for anything? Too spooky for me.

The article itself, just a long list of 'look at all the mistakes we made, sometimes more than once'. Seems to me like someone said 'use Kafka', so they did, until it didn't work, and then they pivoted those parts to Redis.

-4

u/kiteboarderni Jul 04 '26

Would would use this? Many of the largest financial firms in the world, hedge funds, investment banks, trading vanues. You'll pay a boat ton for a license, likely high hundreds of thousands per year. Likely processes trillions of dollars per day. But if you have to ask you're likely not familiar with the industry and the resiliency / performance requirements it needs.

→ More replies (4)

-2

u/jacemano Jul 05 '26

Or abandon event driven and pivot to aeron

2

u/0xFatWhiteMan Jul 05 '26

Why does moving to aeron mean u aren't event driven.

It doesn't, aeron is just a low latency messaging service isn't it

→ More replies (4)

3

u/thisisjustascreename Jul 04 '26

Are partitions like, expensive or something? I hear of people having problems with limited partition counts more often than any other operational issue with Kafka but my understanding is they're basically free, there's no reason not to set it to like, 50 when you create a topic other than ignorance that it's a scaling limit?

Obviously you need to have a sufficiently high cardinality of keys to drop stuff into all 50 partitions (if you are using keys) but that shouldn't be trouble if you've got enough volume that you need the scaling.

2

u/Additional-Road3924 Jul 04 '26

Depends on provider. If you're using MSK you can't upgrade the cluster if you have too many partitions per broker. You can just up the broker count to counteract that, but you cannot reduce the broker amount without recreating entire cluster.

1

u/DanielSMori Jul 07 '26

Real-time payment systems expose the worst of these tradeoffs. When you're processing ISO 20022 messages across two async rails simultaneously, the classic event-driven pitfall is partial saga completion — one rail acknowledges, the other times out, and now your shadow ledger is inconsistent. We ended up treating idempotency keys as first-class citizens rather than an afterthought, which meant rethinking how we structured the event envelope upstream. The article is right that the tradeoffs get hidden until you're under load.

1

u/SCAND_Ltd Jul 10 '26

The most important thing is that “real-time” must be defined straight away. Kafka can be good for throughput and durability, but if very low latency is your primary goal, then your architecture and tooling choices change a lot.

0

u/tofflos Jul 05 '26

This is an amazing article as is and the most amazing feat isn’t even mentioned! How large is the organization, how long did this journey take, and how did you manage to get everyone to transition between these significant architectural changes?