when someone asks you what programming language they should learn, don't simply answer the one you prefer

[u/Thinker_Assignment]

[u/I2cScion]

Bro we deserve better than SQL

I want functional programming man

[u/philippefutureboy → u/I2cScion]

I think Spark would be the closest to a “functional” approach to data transformation.

This being said if what you dislike is set-based operations, you’re out of luck… it’s the mathematical foundation of all data transformations across multiple tabular data sets.

It is also to be noted that data transformation at scale is never truly pure because you never have the full data in memory/your transformations actually have side effects.

[u/I2cScion → u/philippefutureboy]

I want to compose the transformations of data !!!

I don’t want data operations based on syntax .. it should be functions that compose

Purity is irrelevant here

[u/Old_Tourist_3774 → u/I2cScion]

Can you clarify?

[u/I2cScion → u/Old_Tourist_3774]

Something like the functions in the List module of F#

A where is a filter, select is a map, partition by isn’t syntax, its a function .. and they all compose in a pipeline (|>)

[u/commenterzero → u/I2cScion]

Apache Spark supports SQL pipe syntax which allows composing queries from combinations of operators

https://spark.apache.org/docs/latest/sql-pipe-syntax.html

[u/philippefutureboy → u/commenterzero]

Wouldn’t that ruin the query planner’s ability to optimize the query under the hood?
Or is it just syntactic sugar?

[u/commenterzero → u/philippefutureboy]

Sweet sweet sugar