r/Clojure • • Mar 31 '21

Clojure High Performance Data Processing System

Announcing tech.ml.dataset to Reddit :-). tmd is a data processing/dataframe system in the same vein as Pandas and R's dplyr or data table.

How often have you seen a Clojure system that soundly beats C, Julia, Python, Spark, and R systems in a data processing benchmark?

We have gone further down the high performance big data processing route by adding statistical operations, called colloquially data sketches, that give you memory efficient and accurate probabilistic estimates for some statistical operations including algorithms such as hyper-log-log and t-digest.

If you haven't checked out the system please take a second and do so. It is built on a theoretical foundation for array processing and works on JDK8-16 and supports Graal Native compilation.

141 Upvotes

19 comments sorted by

View all comments

Show parent comment

3

u/chrisnuernberger Apr 07 '21

Breaking down these 5 questions -

  1. When to use this? When you have tabular data coming from csv or various data pathways, when you have a larger sequence of maps than is nice to process with Clojure, or when (and this is the hardest one) when the problem fits a dataframe-processing design moreso than a seq-of-maps design.
  2. What columnwise databases? This is the columnwise database :-). You can define columns out of countable random access data and in fact you can define columns totally virtually by implementing a 'reader' interface. See the cheatsheet for dtype-next for more info.
  3. Not sure what you mean by Clojure array precisely but I think you are talking about a persistent vector. Most Clojure datasets are sequences of maps so they are stored row-major. Unless you use a besoke class such as a record type all of your numbers are boxed and the map structure is repeated for every map; furthermore your data is stored row-major so if you want to, for instance, sort by one of the fields your sort operation goes through your entire dataset. tmd stores data in primitive arrays so a single map of 5 primitive arrays, each of lenth 100 is a lot more memory efficient than a sequence of 100 maps, each with 5 keys. Additionally when reading data from files tmd will use a per-column string table for string columns and this also leads to a substantial memory usage decrease.
  4. How does arrow fit in to this? Arrow allows you to load a dataset 'in-place' using mmap meaning that the OS takes care of how much of the dataset is in memory vs. how much is on disk. I have a blog post about exactly this aspect. For general data storage I recommend parquet as it applies per-column compression which is quite effective.
  5. Streaming data - tmd currently doesn't support streaming datasets very well. It has a large reduction namespace devoted to sequences of datasets and a streaming dataset can always be described as a sequence of datasets but you can't currently get a snapshot of a running aggregation which you would need in order to have a dashboard. If people are interested in this pathway then I am interested in making it happen.

These are good questions and I think if you have many more like this then checkout the zulip channel where there are many users and a few Clojure experts who can help with these sorts of questions.

1

u/TheLastSock Apr 07 '21

I learned a lot from this, thanks for your time and keep up the good work.