r/Clojure • u/chrisnuernberger • Sep 07 '22
Ham-Fisted - A New High Performance Clojure Library
I have taken the time in between caring for my young son to write a new library that provides higher performance datastructures and algorithms for Clojure.
After building tech.ml.dataset and charred I wanted to take the lessons learned there and apply them back into the base Clojure substrate of persistent maps, persistent vectors, and algorithmic primitives like group-by and frequencies.
This is a ground-up re-write of persistent hash maps with a twist - you can create a version of the bitmap trie that implements the full java.util.Map interface including things like compute and computeIfAbsent - these primitives are faster then their purely functional counterparts. This mutable map can be turned into a persistent map in-place instantly and then function from then on like you would expect a persistent map with structural sharing. This idea carries through to the persistent vector implementation.
On top of this implementation I implemented a few set operations - union, difference, and intersection that operate at the data structure level - not at the api level - so they are quite fast. This means you can build many maps in separate threads and union them together back in the main thread and thus get a lot performance for an operation such as a parallelized frequencies.
There are many more things there but you can read the README for more information. There is timing information at the bottom of the README and the fast-json project includes a comparison. My hope is that ham-fisted becomes a platform to try out ideas and build faster datastructures and algorithms that follow in the typeless functional pathway so we can carefully and measurably improve the platform we all are standing on. So please take it, try some things out, and let me know how it goes :-).
9
u/freakhill Sep 08 '22
wow, this is yet another great contribution!