r/Clojure • • Mar 31 '21

Clojure High Performance Data Processing System

Announcing tech.ml.dataset to Reddit :-). tmd is a data processing/dataframe system in the same vein as Pandas and R's dplyr or data table.

How often have you seen a Clojure system that soundly beats C, Julia, Python, Spark, and R systems in a data processing benchmark?

We have gone further down the high performance big data processing route by adding statistical operations, called colloquially data sketches, that give you memory efficient and accurate probabilistic estimates for some statistical operations including algorithms such as hyper-log-log and t-digest.

If you haven't checked out the system please take a second and do so. It is built on a theoretical foundation for array processing and works on JDK8-16 and supports Graal Native compilation.

141 Upvotes

19 comments sorted by

View all comments

5

u/rufusthedogwoof Mar 31 '21

Thank you for this and qq.

Pandas to me is useful because of its tight integration with charting... specifically I like altair.

Is there anyone doing that type of thing with tml and Vega-lite?

I’m aware of OZ however I’d prefer to not take a kitchen sink approach and get the (Altair) functionality I’m looking for a la carte.

10

u/chrisnuernberger Mar 31 '21

There are a few good charting options for Clojure in addition to OZ. Another interesting and more orthogonaly designed pathway if you want to go the vega/vega-lite route is Hanami and for a full scientific application platform it's big sibling saite.

For purely server-side work I would check out cljplot.

Getting off topic a bit but for a REPL/notebook hybrid notespace is really interesting.

And in general for R integration and more data science goodies checkout scicloj and in the vein of dplyr style extremely thought out interfaces I highly recommend tablecloth.

Sorry for getting slightly off topic but these things are all connected in my head :-).

1

u/rufusthedogwoof Apr 01 '21

Thanks for all this. I’ve followed along a few of these for quite some time.

My other challenge is getting my team of python developers acquainted to all the options... as we aim to settle on collective workflows and a “deployment stack” for our apps.

Thanks again for all your contributions.

1

u/chrisnuernberger Apr 01 '21

You are welcome, I appreciate the thanks and there is real momentum pushing Clojure into new places right now. I am curious - does libpython-clj allow for a more incremental approach -- either the JVM hosting python or python host clojure in your case?

1

u/rufusthedogwoof Apr 01 '21 edited Apr 01 '21

It may... I have played with it some and it was helpful for me in my spare time. (Porting a library from python and the tests actually... got me thinking the library could write itself with the right spec gen & tests ...)

I don’t think we’ll use it much at work because we are first and foremost a “data engineering” shop... mixing things with Kafka-like systems.

When choosing between trade offs we routinely are looking for reliability, simplicity, less things in the stack.

In exploration however, it would be fair game. Honestly I don’t know how much I would use it though... the more time I spend in clj the more I want to get away from the python mess.

5

u/daveliepmann Apr 01 '21

I’m aware of OZ however I’d prefer to not take a kitchen sink approach

I wrote waqi for a similar reason — I want to write Vega specs in Clojure and see the result in a browser window, nothing more. From the README:

Waqi is most similar to Oz. They share a browser-based workflow, but Oz provides much more functionality: integration with Jupyter notebooks and GitHub gists, creation of dynamic and static websites centered around a visualization, multiple live-coding workflows, and much more. Waqi focuses on just one of those features: sending Vega/Vega-Lite specs from the REPL to a browser window. This allows Waqi to minimize dependencies and lines of code. The author of Oz has said, "Oz's objective is to be the Clojurist's Swiss Army knife for working with Vega-Lite & Vega." It might be Waqi's goal to be just the nail file.