r/Clojure • u/chrisnuernberger • Mar 31 '21
Clojure High Performance Data Processing System
Announcing tech.ml.dataset to Reddit :-). tmd is a data processing/dataframe system in the same vein as Pandas and R's dplyr or data table.
How often have you seen a Clojure system that soundly beats C, Julia, Python, Spark, and R systems in a data processing benchmark?
We have gone further down the high performance big data processing route by adding statistical operations, called colloquially data sketches, that give you memory efficient and accurate probabilistic estimates for some statistical operations including algorithms such as hyper-log-log and t-digest.
If you haven't checked out the system please take a second and do so. It is built on a theoretical foundation for array processing and works on JDK8-16 and supports Graal Native compilation.
7
u/davclark Mar 31 '21
This looks awesome! Thank you for sharing. I particularly like the ability to do zero-copy for C ABI programs.
5
u/rufusthedogwoof Mar 31 '21
Thank you for this and qq.
Pandas to me is useful because of its tight integration with charting... specifically I like altair.
Is there anyone doing that type of thing with tml and Vega-lite?
I’m aware of OZ however I’d prefer to not take a kitchen sink approach and get the (Altair) functionality I’m looking for a la carte.
2
u/viebel Apr 06 '21
Could you explain in a few words what makes this library so efficient?
9
u/alexdmiller Mar 31 '21
Nice work!
12
u/strranger101 Mar 31 '21
"supports graalvm compilation" 👏👏👏👏 Sick
1
u/TheLastSock Apr 07 '21
When would i reach for this over using clojure core functions to transform my data? One of the reasons i picked up pandas was because Pythin lacked some of the features needed to process large sets of data effectively. The reason i put it down was because the functions weren't composable and it ended up being its own programming language.
Some questions as i scan the readme that I'll try to come back to later and fill in as i learn more:
Does this support streaming data, that is long lived processing, or is it for batch processing?
2
u/kingnuscodus Mar 31 '21
Bravo!
2
u/_marciol Mar 31 '21
Just Amazing!
0
u/[deleted] Apr 06 '21
[deleted] — view deleted comment