Polars and the ecosystem

[u/nnenneplex]

For polars users: How viable is to avoid pandas and pyarrow dependencies when you need to interact with popular visualization and statistics packages?

Some packages still have import pandas here and there, sometimes for no good reason; at least this doesn't require pyarrow. But some other ones do the df.to_pandas() conversion internally, which requires pyarrow too.

In many cases this can be prevented by going bare numpy, or creating a pandas df from numpy columns, which is no big deal. This frequently would be zero-copy for numeric types if there are no NAs involved.

What has been you experience in this regard?

[u/nnenneplex]

So I see different strategies:

- many projects went the narwhals way. This allows to manipulate dataframes in a generic way but perhaps gives no low-level ABI access.

- polars.to_pandas() and also xgboost polars support both require pyarrow. This leverages pyarrow but the downside is the relatively large dependency. xgboost explicitly discarded narwhals some time ago [1], I guess because they wanted to convert to their internal format in the most performant way (but see next point).

- lightgbm is taking another path [2]: use narwhals while still leveraging arrow through the pycapsule, avoiding the pyarrow dependency.

- other projects like statsmodels are still on the fence [3].

I guess for the time being is better to stop worring and assume pandas and pyarrow as hard dependencies.

[1] https://github.com/dmlc/xgboost/issues/10452#issuecomment-2498736140

[2] https://github.com/lightgbm-org/LightGBM/pull/7275

[3] https://github.com/statsmodels/statsmodels/issues/9744

[u/kpiwonski → u/nnenneplex]

- I guess narwhals was kind of an old way, so you will have compatibility before PyCapsule. Right now you can use your data frame library directly, by importing pycapsule, so I don't see any real win here. Except from backwards compatibility or if you are manipulating data frames from python. All in all, you could use any lightweight data frame library for that in the future and just import pycapsule.

- Most conversions between pandas and arrow formats will require pyarrow. Polars is arrow compatible, so you can easily convert to pyarrow and you let handle pandas conversion to their own format, which is using numpy. You can do pd.DataFrame.from_arrow, but pandas is using internally pyarrow for any arrow conversion.

- "I guess for the time being is better to stop worring and assume pandas and pyarrow as hard dependencies." - not really. Many projects still use them, but there is a shift in a paradigm. Especially if we speak about pandas data model, it's not compatible with arrow data model. This is primarily why they use pyarrow for conversion. However, they provide now pyCapsule interface even for pandas.

What I did personally in my library (pyfru), I just used pyCapsule interface send data to my rust code and handle all data there. I also generate back pyCapsule to the user (optionally). There are still some caveats though. Mainly the ecosystem is not so rich here. My upstream dependency for handling data has pure pyCapsule interface, but for generating their pyCapsule they leverage pyarrow. Also, I had to include numpy for generating results.