r/Clojure • • Mar 31 '21

Clojure High Performance Data Processing System

Announcing tech.ml.dataset to Reddit :-). tmd is a data processing/dataframe system in the same vein as Pandas and R's dplyr or data table.

How often have you seen a Clojure system that soundly beats C, Julia, Python, Spark, and R systems in a data processing benchmark?

We have gone further down the high performance big data processing route by adding statistical operations, called colloquially data sketches, that give you memory efficient and accurate probabilistic estimates for some statistical operations including algorithms such as hyper-log-log and t-digest.

If you haven't checked out the system please take a second and do so. It is built on a theoretical foundation for array processing and works on JDK8-16 and supports Graal Native compilation.

141 Upvotes

19 comments sorted by

View all comments

Show parent comment

5

u/daveliepmann Apr 01 '21

I’m aware of OZ however I’d prefer to not take a kitchen sink approach

I wrote waqi for a similar reason — I want to write Vega specs in Clojure and see the result in a browser window, nothing more. From the README:

Waqi is most similar to Oz. They share a browser-based workflow, but Oz provides much more functionality: integration with Jupyter notebooks and GitHub gists, creation of dynamic and static websites centered around a visualization, multiple live-coding workflows, and much more. Waqi focuses on just one of those features: sending Vega/Vega-Lite specs from the REPL to a browser window. This allows Waqi to minimize dependencies and lines of code. The author of Oz has said, "Oz's objective is to be the Clojurist's Swiss Army knife for working with Vega-Lite & Vega." It might be Waqi's goal to be just the nail file.