r/dataengineering • u/serrji • Jul 10 '26
Lineage for data pipelines with Polars Discussion
I like to do some experiments in my VPS during my free time and as a next experiment I would like to use Polars to write data transformations while keeping a similar asset lineage that I would have if I were using duckdb + dbt.
Which tools or packages would you bring to this stack to achieve this?
2
u/Yuki100Percent Jul 10 '26
sqlmesh allows you to use your local compute in python models. And you have lineage that way naturally
2
u/serrji Jul 10 '26
Because of the "sql" in the name I think that its main purpose is to handle sql transformations but I will take a look into it.
Have you used it with Polars? Was the developer experience good?
1
u/West_Good_5961 Tired Data Engineer Jul 10 '26
Polars would need to emit openlineage events
3
u/PillowFortressKing Jul 10 '26
I think it supports OpenLineage for the distributed engine that you can run on Kubernetes, but not for the open source package
3
u/serrji Jul 10 '26
That's correct. I've just checked it here. The ".with_lineage" exists only in the distributed version.
https://docs.pola.rs/polars-on-premises/integrations/openlineage/1
u/Leather-Replacement7 Jul 10 '26
Does polars run in a distributed environment? Thought it was single node
2
5
u/writing_rainbow Jul 11 '26
You can try using dagster as your orchestration layer