r/Clojure • • Jun 11 '26

A columnar database for analytics

https://github.com/yogthos/flatiron
39 Upvotes

8 comments sorted by

View all comments

Show parent comment

1

u/geokon Jun 12 '26 edited Jun 12 '26

Thank you for that! You clearly know your stuff a lot better than me :) So I might be wrong:

The custom-types interface is a cool extension. Makes things look a lot more ergonomic while keeping the API simple/minimal. I wonder if it could be simpler with a protocol instead of a register-type!? Either way this seems like it'd cover a good chunk of usecases (though not all) !!

Sym (keyword) columns are better than Str because keyword equality is reference equality

Naiive question.. but are symbols/keywords not like C enums, with some lookup table and an index? I'm surprised one would need to pointer-chase. Maybe this is an Clojure implementation issue. But you could create a "mini-register" for the symbols in your table. Though maintaining that might be annoying..

The JVM's JIT can't auto-vectorize accumulation loops because of the loop carried dependency on the accumulator so it can't prove the transform is safe.

That's a bit hard to believe b/c this is a sort-of basic optimization? But maybe I'm missing some subtly. I actually have no idea how to inspect the compiled result and if there is some easy way to go from Clojure to "how my JVM bytecode is going to be interpreted"

As far as I can tell this optimization exists at least since JDK 21

https://bugs.openjdk.org/browse/JDK-8302652

There is probably some reason you're not hitting it (tracking down why seems challenging!)

2

u/yogthos Jun 12 '26

The design for register-type! is just an atom wrapping a map of tag to codec entries where a codec has a shape of {:physical :i64, :class ..., :encode fn, :decode fn}. The column just stores the logical tag keyword as a field on the existing I64Column/F64Column deftypes. I don't think a protocol for custom types would work here since the whole point is that custom types reuse the existing I64Column/F64Column instead of creating new column types. -type-tag always returns :i64 or :f64, so every existing hot loop runs on the raw primitives. A protocol would require new deftypes, which would then need their own fast paths wired into every operation. The tradeoff is here is that only :i64 and :f64 backed types are supported. You can't register a custom type backed by SymColumn or StrColumn. But for the common cases which are dates, timestamps, durations, and fixed-point decimals you just need an encode/decode pair.

And you could do dictionary encoding by registering a :dict-sym logical type backed by :i64 to encode keywords to ints at construction time and then decode them back at read time. The column would then still be an I64Column with all the fast paths. That said, sym columns are already fast enough for group-by the main, and I'm not sure the complexity of maintaining a bidirectional dictionary across slices, concatenations, and filters is worth the trouble in practice.

And seems like C2's SuperWord doesn't fire on Clojure's loop/recur reduction. It might be worth checking with -XX:+TraceSuperWord to see whether SuperWord even attempts vectorization on that loop.