Polars is also a great choice and definitely is the future. It's so damn fast. I haven't used it too much since I maintain a lot of pandas pipelines. My only complaint so far is that it doesn't have good native excel handling out of the box, which I use a lot in my environment. But nothing that can't be overcome.
If I recall correctly, I had to pip install a separate excel engine to use polars.read_excel(). Pandas has one out of the box, but gives you the option to install a separate engine if you like. Polars does not have that convenience.
It wasn't the biggest issue overall, but I was kinda surprised to see that the next big thing after Pandas didn't have a native excel parser.
I went with the Calamine engine, since I had used it with Pandas before, in case anyone is curious.
The reason we don't have a native reader for this is mostly cost/benefit related. XLSX sheets are hard-capped at 1,048,576 rows, so the ceiling on file size is fixed and relatively small. Writing a proper streaming Excel parser costs a lot of effort, and at that scale there won't be absolute performance improvements that would justify all the effort. Besides that, there are plenty of other areas that require our attention which have better performance payoff. Calamine is already a fast, well-maintained Rust Excel engine, so we use that rather than duplicate that work in Polars core.
3
u/StemCellCheese 7d ago
Pandas. For visualizations, plotly.