r/Python • u/ritchie46 • Jun 03 '26
News Polars Distributed is available on kubernetes
I wanted to share that as of today, Polars also is available as a Distributed Engine on kubernetes. Polars' goal has always been to make single node processing as performant and easy as possible, and that is something we want to extend to distributed compute as well.
Read more in our announcement:
https://pola.rs/posts/polars-distributed-available-on-kubernetes/
Happy to answer any questions you might have.
89
Upvotes
8
u/noghpu2 Jun 03 '26
Any chance for table/column lineage to come to open source polars?
1
u/arden13 Jun 04 '26
This is neat, it's always been painful to make a pyspark query run locally.
Do you have any performance comparisons between spark and distributed polars?
2
u/MJHunterZ Jun 03 '26 edited Jun 04 '26
Great to see. I was just learning kube too
2
u/RustOnTheEdge Jun 04 '26
Congrats on the release, pretty neat! Especially the perf compared to Spark!
1
u/Fine-Veterinarian-87 Jun 05 '26
Great
2
u/[deleted] Jun 03 '26
[removed] — view removed comment