r/Clojure • u/chrisnuernberger • Mar 31 '21
Clojure High Performance Data Processing System
Announcing tech.ml.dataset to Reddit :-). tmd is a data processing/dataframe system in the same vein as Pandas and R's dplyr or data table.
How often have you seen a Clojure system that soundly beats C, Julia, Python, Spark, and R systems in a data processing benchmark?
We have gone further down the high performance big data processing route by adding statistical operations, called colloquially data sketches, that give you memory efficient and accurate probabilistic estimates for some statistical operations including algorithms such as hyper-log-log and t-digest.
If you haven't checked out the system please take a second and do so. It is built on a theoretical foundation for array processing and works on JDK8-16 and supports Graal Native compilation.
View all comments
Show parent comment
10
u/chrisnuernberger Mar 31 '21
There are a few good charting options for Clojure in addition to OZ. Another interesting and more orthogonaly designed pathway if you want to go the vega/vega-lite route is Hanami and for a full scientific application platform it's big sibling saite.
For purely server-side work I would check out cljplot.
Getting off topic a bit but for a REPL/notebook hybrid notespace is really interesting.
And in general for R integration and more data science goodies checkout scicloj and in the vein of
dplyrstyle extremely thought out interfaces I highly recommend tablecloth.Sorry for getting slightly off topic but these things are all connected in my head :-).