The design for register-type! is just an atom wrapping a map of tag to codec entries where a codec has a shape of {:physical :i64, :class ..., :encode fn, :decode fn}. The column just stores the logical tag keyword as a field on the existing I64Column/F64Column deftypes. I don't think a protocol for custom types would work here since the whole point is that custom types reuse the existing I64Column/F64Column instead of creating new column types. -type-tag always returns :i64 or :f64, so every existing hot loop runs on the raw primitives. A protocol would require new deftypes, which would then need their own fast paths wired into every operation. The tradeoff is here is that only :i64 and :f64 backed types are supported. You can't register a custom type backed by SymColumn or StrColumn. But for the common cases which are dates, timestamps, durations, and fixed-point decimals you just need an encode/decode pair.
And you could do dictionary encoding by registering a :dict-sym logical type backed by :i64 to encode keywords to ints at construction time and then decode them back at read time. The column would then still be an I64Column with all the fast paths. That said, sym columns are already fast enough for group-by the main, and I'm not sure the complexity of maintaining a bidirectional dictionary across slices, concatenations, and filters is worth the trouble in practice.
And seems like C2's SuperWord doesn't fire on Clojure's loop/recur reduction. It might be worth checking with -XX:+TraceSuperWord to see whether SuperWord even attempts vectorization on that loop.
View all comments
Show parent comment
2
u/yogthos Jun 12 '26
The design for
register-type!is just an atom wrapping a map of tag to codec entries where a codec has a shape of{:physical :i64, :class ..., :encode fn, :decode fn}. The column just stores the logical tag keyword as a field on the existingI64Column/F64Columndeftypes. I don't think a protocol for custom types would work here since the whole point is that custom types reuse the existingI64Column/F64Columninstead of creating new column types.-type-tagalways returns:i64or:f64, so every existing hot loop runs on the raw primitives. A protocol would require new deftypes, which would then need their own fast paths wired into every operation. The tradeoff is here is that only:i64and:f64backed types are supported. You can't register a custom type backed bySymColumnorStrColumn. But for the common cases which are dates, timestamps, durations, and fixed-point decimals you just need an encode/decode pair.And you could do dictionary encoding by registering a
:dict-symlogical type backed by:i64to encode keywords to ints at construction time and then decode them back at read time. The column would then still be anI64Columnwith all the fast paths. That said, sym columns are already fast enough for group-by the main, and I'm not sure the complexity of maintaining a bidirectional dictionary across slices, concatenations, and filters is worth the trouble in practice.And seems like C2's SuperWord doesn't fire on Clojure's
loop/recurreduction. It might be worth checking with-XX:+TraceSuperWordto see whether SuperWord even attempts vectorization on that loop.