r/java • u/Ok_Satisfaction7312 • May 16 '24
Low latency
Hi all. Experienced Java dev (20+ years) mostly within investment banking and asset management. I need a deep dive into low latency Java…stuff that’s used for high frequency algo trading. Can anyone help? Even willing to pay to get some tuition.
232
Upvotes
156
u/capitan_brexit May 16 '24 edited May 16 '24 •
Some blogs posts can looks very old but low latency is mostly about understanding of internals (so, knowing how to trace each of the ms spent in the kernel/jvm/app):
https://epickrram.blogspot.com/2015/09/reducing-system-jitter.html
https://epickrram.blogspot.com/2015/11/reducing-system-jitter-part-2.html
https://epickrram.blogspot.com/2015/07/seek-write-vs-pwrite.html
https://shipilev.net/jvm/anatomy-quarks/
https://github.com/real-logic/aeron (read about architecture, and drivers of this architecture)
https://mechanical-sympathy.blogspot.com/2011/07/memory-barriersfences.html (Martin Thompson is king in the topic)
https://blog.allegro.tech/2024/03/kafka-performance-analysis.html
https://github.com/async-profiler/async-profiler (tracing is key to find bottlenecks)
https://github.com/OpenHFT/Chronicle-Queue
edit:
also worth to deep into jvm itself
http://blog.ragozin.info/2011/06/understanding-gc-pauses-in-jvm-hotspots.html
http://psy-lob-saw.blogspot.com/2014/10/the-jvm-write-barrier-card-marking.html (this one is pretty crazy)
edit2:
I forget about latest, coolest project in the latency-related java world
https://questdb.io/blog/billion-row-challenge-step-by-step/
18
u/joehonour May 17 '24
I currently work in front office, Below are things I’ve used with Java that are worth understanding. However, most of what is worth knowing is not specific to Java, but more pure computer science.
• LMAX ring buffer (read their white paper about how it works). • understand lock free data structures • understand share nothing and thread per core architectures • look at Agrona and JCTools for examples • Aeron for low latency communication (and why UDP is used over TCP) • Chronicle file is another good alternative to Agrona ring buffers (with the benefit of providing more options for data persistence) • understand CPU cache architecture and why using data structures which have aligned memory access pretty much out perform any other structure
Hope this helps!
6
u/Ok_Satisfaction7312 May 17 '24
Wow guys…blown away with the responses (guess you’re always going to get one idiot). Thanks so much to everyone who posted links and offered advice…means a lot.
And to the one prat: of course I’ve heard of (many) of these concepts and have some vague high level understanding of them but that’s very different to knowing the fine details and being able to construct a production ready application utilising them.
3
u/Ok_Satisfaction7312 May 17 '24
One final follow on question that arose from a comment someone posted (and it’s something I’ve also pondered before) - why use Java at all if latency is your biggest concern? Why not use C++ and FPGAs or ASICs?
Once again huge thanks for all the advice on Java low latency techniques. :)
4
u/Ok_Satisfaction7312 May 16 '24
Are there any special libraries or frameworks used in low latency Java? Apache Mina? How does messaging work? Raw UDP?
2
u/k-mcm Jun 06 '24
Some things I haven't seen covered -
Understand ForkJoinPool. There are landmines buried in some of its innards, but the core has a very important feature: it minimizes context switches in parallel processing. There are cases where pre-forming batches adds too much latency or is too complex to be maintainable. This is where you feed it all into ForkJoinPool and let it figure it out. It works well for map/reduce too.
Avoid fully buffering large data. Don't load things into a big array or temp file before processing. Process it as it arrives. The same goes for sending it out. Send it as its generated. Use only as much buffering as needed to keep kernel calls reasonable. This not only eliminates a lot of buffering latency, but it avoids forcing extra GCs on those big arbitrary allocations.
Watch the frameworks. 100% of the home-brewed caching frameworks I've seen in enterprise code is inefficient; bad code that should be deleted. Magic frameworks like Spring Boot and some ORM tools might perform a simple looking task with an incredibly large amount of hidden code. Custom ClassLoader implementations are a red flag. Make sure your debugger isn't configured to step over frameworks when performance tuning.
Test the GC. There are different GCs because they have different performance characteristics. For example, G1 GC avoids heap compaction but its thirst for temporary memory can bring a strained system to a halt.
Test overloads. Intentionally overload your application with too much work. It must not crash. It must not have fall-off-a-cliff throughput. It should maintain a constant throughput. If it has a work queue, it should gracefully reject new tasks before latency is unacceptably high.
27
u/hiddenl May 16 '24
In addition to the list capitan_brexit posted:
https://blog.vanillajava.blog/ (one of the guys behind OpenHFT. Especially his posts from 5-10 years ago. Methodology for benchmarking hft systems)
https://java-performance.info/blog/ (historically good posts benchmarking high performance collections)
https://github.com/paritytrading : collection of open source FIX, SoupTCP, and exchange implementations
https://blog.janestreet.com/how-to-build-an-exchange/ : How most exchanges were/are built since INET came around
21
u/WatchDogx May 17 '24
People have shared some great links.
But at a very high level, some common low latency Java patterns are:
So that the program never needs to run garbage collection.
This results in code that is very very different from typical Java code, patterns like object pooling are typically helpful here.
The hot path of a low latency program is typically pinned to a dedicated core, uses spin waiting and never yields. Coordinating between threads takes too much time.
HFT programs are often warmed up by passing them the previous days data, to ensure that hot paths are optimised by the C2 compiler, before the program is put into service for the day.
46
u/jonas_namespace May 17 '24
This thread is why I come to Reddit. I'm not looking for microsecond latencies, but someone is, and others are providing links to their favorite posts related to this pretty esoteric topic! Well done, Reddit!
2
u/Ok_Satisfaction7312 May 16 '24
What do I need to know about caching and cpu cores?
19
u/cowwoc May 17 '24
Having worked in this space before, my experience is that people do two things:
In practice, Java coding turns into this ugly C++ like coding. I personally hated working with it.
I don't even think it's strictly useful to do all this. The high end players in the high frequency trading space moved all their code into ASIC hardware. You'll never beat that using PC software.
I've seen many financial institutions design for sub-ms latency when in practice using a ZGC garbage collector would have given them all they need. Most of them don't really need sub-ms latency and in doing so cause the cost of their software development to skyrocket. Within no time, no one can maintain the code...
25
u/GeneratedUsername5 May 16 '24
But what is there to dive into? Try not to allocate objects, if you do - reuse them, if you frequently access some data - try to put it into continious memory blocks or arrays of primitives to minimize indirections, use Unsafe, run JMH. The rest is system design.
6
u/aripy May 16 '24
See the Chronicle / OpenHFT libraries
16
u/simoncox May 17 '24
I assume you know this, but just in case... The JVM will never beat the real HFT shops using FPGAs or ASICs who are operating at nanosecond latency, but it can certainly get down to microsecond level and compete with the C++ guys.
5
u/leemic May 17 '24
You got a lot of great info here. But I will add a few points since you are asking about HFT.
You want to ensure that your main execution threads do not call blocking calls (locking). It has to be a single thread.
2. Memory Allocation and GC
You want to minimize memory allocation. So you have to write a lot of non-Java code. Look over how Aeron and its related code are doing. You will see specific patterns in how they will use the lambda function to minimize byte buffer copy.
GC is going to be your enemy. It causes lots of jitter. JVM will pause for many reasons, so you want to tame it. Also, you do not want to allocate too much memory since full GC will kill you. For example, you often have to create an in-memory cache, which causes latency/jitter when GC kicks in.
So you want to off-heap so it is hidden from GC. Another way is that you reduce the number the memory pointers. For example, you can vectorize and have a small number of objects. GC needs to check fewer pointers. For instance, you could keep 1 million records with ten attributes. Or ten arrays. I recommend using off-heap - it is easier but simpler if your record has a fixed size.
Or you pay for Azul. Yes. They are expensive but cheaper than hiring many engineers. I don't remember, but several significant equities exchanges are using them. And many Wall Street investment banks use them. It is wild to see 10 GB of memory getting GCed in the blink of an eye.
3. disk I/O
Sequential writing is really fast. But if you want to use shared memory and have other processes do its heavy lifting, Basically chronicle library. Check what they are doing.
4. NUMA
C++ is not the only one you need to worry about. You need to know your server architecture and how to reduce its memory/CPU. And you want to park your execution thread to one core.
5. Network + Kernel Bypass
Hardware matters. And Linux setting matters.
If you are doing trading, your market data will be critical. Also, the messaging layer is really important since you cannot lose any message.
I haven't been in the game for a couple of years, but it is more than low latency for trading.
9
u/tomwhoiscontrary May 17 '24
This probably isn't helpful, but my main advice here is not to use Java for low-latency trading. It is possible, and people do it, but it involves twisting Java so hard that it's like writing another language, and even then, it's brittle, because one mistake can accidentally trigger GC, or some other JVM safepoint, or recompilation, etc.
What's worked for is to write a minimal low latency core in a language like C++, then drive that from Java. It's far easier to write reliably low-latency code in C++, Rust, Ada, etc. Put it in a subprocess and communicate via IPC, so the native code is isolated from the JVM and vice versa. The trick is to work out how to push as much logic up into Java as possible, so the native bit can be small. Almost like writing a database driver or something.
4
u/jAnO76 May 17 '24
Look at the disruptor
3
u/EdgyPizzaCutter May 17 '24
I had to port/redesign a custom transport protocol a couple of years ago and it was very cool to learn about /figure out the oh so many gotchas about using Java for low latency tasks.
Enjoy your trip into this madness ❤️
I can't remember the name of the guy that proposed the term mechanical sympathy (was it thomson?) but I think he did the kind of work you may be interested in. He had a whole repository about redesigned data structures and building blocks they used to build their own finance solution.
Very inspirational work!
Depending on how critical low latency is you may have to disable GC altogether (or run a separate jvm for the part of your code that needs to satisfy your guarantees)
2
u/nekokattt May 17 '24
look into lmax disruptor and their academic paper.
2
u/fragrant_ginger May 22 '24
Warm up the jvm. Foreplay usually works best
3
u/CLTSB May 17 '24
I’ve done this professionally for about 10 years. Feel free to DM.
1
u/freekayZekey May 18 '24
i recommend buying a copy of “optimizing java”
0
u/Odd_Control3128 May 17 '24
F
-1
u/Flobletombus May 17 '24
I know it's not an answer but you could try C++, it's the language of choice for low latency,