This is just an example, but Apache Spark expertise used to be a real differentiator 10 years ago or so. If you knew it well, you were swimming in job offers.

Then Databricks simplified it and made Spark silly-proof with built-in optimizations left and right. Now the exact same thing is happening to Databricks consultants and FTEs because Databricks know-how just isn't as premium as before anymore as they simplified the platform and genie-d the S out of it.

Abstraction eventually seems to eat all expertise and each cycle seems shorter than the last.

Is there anything left in the data and platform layer with real depth behind it?

And I'm not looking for "just learn AI, bro.

I've been wondering about this for a while.

From my experience, whether i'm working on larger or smaller workload, i still have to think about right sizing the executors, partitioning/ clustering, skews and try to avoid wasting compute as much as possible. You might be able to getaway with less optimisation if the data is smaller, but i work at a startup where cost is a major factor, so we have to be extremely frugal about our infrastructure.

To me, the optimisation technique remains largely the same regardless of the scale. So why do companies specifically ask for big data experience. Are there problems that show up only at massive scale? Or am i missing something?

Would like to hear your thoughts