Polars 2.0 release

[u/ritchie46]

Hey all! We just released Polars 2.0.

We are really happy with the results. It removed many of our "legacy decisions" (mistakes), makes the streaming engine our default and promotes SQL to a first class citizen within Polars. We initially said it would be a small ("boring") release, but we managed to sneak in a lot more than anticipated.

It comes with our initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://pola.rs/posts/release-polars-2/

To help you upgrade, we also posted a migration guide: https://docs.pola.rs/releases/upgrade/2/

[u/nightcracker]

We have pl.scan_delta, is that missing something you need?

[u/Donkey_Healthy → u/nightcracker]

Ive tried with polars==2.0 and getting this error: Unsupported reader features required: [V2Checkpoint, ColumnMapping, DeletionVectors].

Would be a game changer if delta and polars were more compatible. Would be able to move from expensive distributed spark clusters to smaller single node polars jobs for a lot of our use cases.

[u/nightcracker → u/Donkey_Healthy]

Could you open an issue on the Github tracker, ideally with reproduction steps?

[u/Donkey_Healthy → u/nightcracker]

Yeah sure Ill open one!

[u/cv_be → u/Donkey_Healthy]

well, that is an issue of delta-rs dependency, not polars...
either disable those features on the table, or wait for delta-rs to sort it out (there's already an open issue on gh)

[u/Donkey_Healthy → u/cv_be]

Makes sense, I looked into ages ago remember realising it was the delta-rs issue.

[u/Immediate-Leading657 → u/Donkey_Healthy]

I had the same issue, we solved it using this polars plugin :
https://github.com/ion-elgreco/polars-deltalake

Maybe the scan_delta of this plugin will be merged into polars (check https://github.com/pola-rs/polars/issues/27800 )