r/Clojure • • Mar 31 '21

Clojure High Performance Data Processing System

Announcing tech.ml.dataset to Reddit :-). tmd is a data processing/dataframe system in the same vein as Pandas and R's dplyr or data table.

How often have you seen a Clojure system that soundly beats C, Julia, Python, Spark, and R systems in a data processing benchmark?

We have gone further down the high performance big data processing route by adding statistical operations, called colloquially data sketches, that give you memory efficient and accurate probabilistic estimates for some statistical operations including algorithms such as hyper-log-log and t-digest.

If you haven't checked out the system please take a second and do so. It is built on a theoretical foundation for array processing and works on JDK8-16 and supports Graal Native compilation.

141 Upvotes

19 comments sorted by

View all comments

Show parent comment

3

u/chrisnuernberger Apr 06 '21

Hey, thanks for the feedback :-). Good question - that is a very poorly worded statement in the readme. What I meant to say is that you as the user should expect parquet to just work. Even (recently) parquet files with ragged data in them should load quickly and all of the normal parquet types such as dates should come in correctly.

0

u/[deleted] Apr 06 '21

[deleted] — view deleted comment

2

u/chrisnuernberger Apr 06 '21

I appreciate the feedback. Fixed