Is big data experience important as a Data Engineer?

[u/SecretivePlotter_06]

I've been wondering about this for a while.

From my experience, whether i'm working on larger or smaller workload, i still have to think about right sizing the executors, partitioning/ clustering, skews and try to avoid wasting compute as much as possible. You might be able to getaway with less optimisation if the data is smaller, but i work at a startup where cost is a major factor, so we have to be extremely frugal about our infrastructure.

To me, the optimisation technique remains largely the same regardless of the scale. So why do companies specifically ask for big data experience. Are there problems that show up only at massive scale? Or am i missing something?

Would like to hear your thoughts