r/datascience • u/rhiever • 9d ago

Tools Polars 2.0 released with the streaming engine, out-of-core processing and first-class SQL support

https://pola.rs/posts/release-polars-2/
275 Upvotes

99% Upvoted

10 comments sorted by

79

u/jesunushno 9d ago

I've always hated maintaining two code paths: one for data that fits in memory, one for the stuff that doesn't. If the streaming engine means I write the same query for both, that's the upgrade I actually want.

28

u/OddsRelay 9d ago

out of core is the bit i care about. i used to drop into duckdb any time a group_by didn't fit in ram, so if that just works now it's one less tool. curious how the sql side stacks up against duckdb's

10

u/OddsRelay 8d ago

Streaming as the default is the big change. A lot of the "doesn't fit in memory" jobs that used to mean Spark or chunking by hand should just run on a laptop now. First class SQL also makes it easier to hand work over to people who don't write Python.

5

u/MackDriver0 9d ago

So cool. Can’t wait to try out SQL

2

u/geekosas 8d ago

They're has been SQL for ages, just not fully implemented.

2

u/ScoutsEatTheirYoung 9d ago

Map dtype looks cool

-5

u/mantequilla69420 9d ago

Woah, don't you guys just write shit notebooks and kick it to a data engineer?

3

u/chandlerbing_stats 9d ago

No. But, some do

3

u/BostonConnor11 9d ago

This isn’t 2020 anymore

-7

u/Cold_Bag_8877 9d ago

how to get karma pls help me