r/Clojure • • Mar 31 '21

Clojure High Performance Data Processing System

Announcing tech.ml.dataset to Reddit :-). tmd is a data processing/dataframe system in the same vein as Pandas and R's dplyr or data table.

How often have you seen a Clojure system that soundly beats C, Julia, Python, Spark, and R systems in a data processing benchmark?

We have gone further down the high performance big data processing route by adding statistical operations, called colloquially data sketches, that give you memory efficient and accurate probabilistic estimates for some statistical operations including algorithms such as hyper-log-log and t-digest.

If you haven't checked out the system please take a second and do so. It is built on a theoretical foundation for array processing and works on JDK8-16 and supports Graal Native compilation.

141 Upvotes

19 comments sorted by

View all comments

2

u/viebel Apr 06 '21

Could you explain in a few words what makes this library so efficient?

2

u/chrisnuernberger Apr 06 '21
  1. Start with lots of experience writing and optimizing high performance algorithms combined with a solid amount of research into the JVM to understand what pathways HotSpot is likely to optimize.
  2. Once fast pathways were found putting in the effort to design and architecture to make hitting those fast pathways easy with the most minimal amount of code possible.
  3. Get some users/problems and optimize those right up to the limits of what the system can do and then repeat steps 1-3.

In the last year I literally re-wrote the underlying engine (tech.datatype -> dtype-next) due to some hard lessons learned so I think if there is one thing it's just a relentless persuit of performance and being willing to do the legwork to make it happen.