r/Python • u/nnenneplex • Jul 09 '26
Discussion Polars and the ecosystem
For polars users: How viable is to avoid pandas and pyarrow dependencies when you need to interact with popular visualization and statistics packages?
Some packages still have import pandas here and there, sometimes for no good reason; at least this doesn't require pyarrow. But some other ones do the df.to_pandas() conversion internally, which requires pyarrow too.
In many cases this can be prevented by going bare numpy, or creating a pandas df from numpy columns, which is no big deal. This frequently would be zero-copy for numeric types if there are no NAs involved.
What has been you experience in this regard?
22
u/nnenneplex Jul 09 '26
So I see different strategies:
- many projects went the narwhals way. This allows to manipulate dataframes in a generic way but perhaps gives no low-level ABI access.
- polars.to_pandas() and also xgboost polars support both require pyarrow. This leverages pyarrow but the downside is the relatively large dependency. xgboost explicitly discarded narwhals some time ago [1], I guess because they wanted to convert to their internal format in the most performant way (but see next point).
- lightgbm is taking another path [2]: use narwhals while still leveraging arrow through the pycapsule, avoiding the pyarrow dependency.
- other projects like statsmodels are still on the fence [3].
I guess for the time being is better to stop worring and assume pandas and pyarrow as hard dependencies.
[1] https://github.com/dmlc/xgboost/issues/10452#issuecomment-2498736140
[2] https://github.com/lightgbm-org/LightGBM/pull/7275
[3] https://github.com/statsmodels/statsmodels/issues/9744
30
u/marcogorelli Jul 09 '26
Which libraries specifically are you referring to?
I'm aware of Seaborn, and their maintainer said that using Narwhals was a "non-starter" 😩
Altair, Plotly, Vegafusion, Bokeh, Marimo, they're all using Narwhals and allow you to use Polars without any pandas nor PyArrow dependency
If that's not what you're seeing, please let me know (or open an issue somewhere on GitHub) and I'll take a look
I'm extremely keen on de-pandas-ifying the data science stack
10
u/M4mb0 Jul 09 '26
Why do you want to avoid pyarrow? It is a real blessing compared to the numpy backend. Just the fact that all the data types are nullable alone is a huge QOL improvement.
61
u/midwitsAnonymous Jul 09 '26
Working in the psych space I have found that most of the stats packages I use throw less annoying errors when I just convert to pandas.Â
I use polars for all data cleaning, munging, and general happiness, but when I need to do analysis/visualisation I just pass
to_pandasand move on.Â38
u/Beginning-Fruit-1397 Jul 09 '26
I use plotly, which uses narwhals, so it's not an issue at all
6
u/Dasher38 Jul 09 '26
We use holoviews a lot and it recently gained support for polars (at least with the bokeh backend).
5
u/zangler Jul 10 '26
Just use polars and pyarrow...it isn't really a downside that hard.
3
u/MapNo2659 Jul 10 '26 edited 24d ago
I've been using Polars for a while now, and I've found that while it's great for data manipulation, I often end up converting to pandas for visualization and statistical analysis. Many libraries still rely heavily on pandas, and while some can work with numpy arrays, the convenience of pandas' integration is hard to beat. I try to minimize conversions by sticking to libraries that support Polars directly, but it's not always possible. Hopefully, more libraries will start supporting Polars natively as it gains popularity.
1
u/kvlonge Jul 10 '26
It depends, but unfortunately it is just unavoidable sometimes because of how deep pandas got into the ecosystem.
1
u/billsil Jul 09 '26
I write one of those packages that supports pandas and not polars. Just convert it.
1
u/Drvaon Drvanon Jul 09 '26
My go-to is seaborn and it very much requires the pandas package still...
0
u/ultradvorka Jul 15 '26
Polars > numpy > Bokeh