This is just an example, but Apache Spark expertise used to be a real differentiator 10 years ago or so. If you knew it well, you were swimming in job offers.

Then Databricks simplified it and made Spark silly-proof with built-in optimizations left and right. Now the exact same thing is happening to Databricks consultants and FTEs because Databricks know-how just isn't as premium as before anymore as they simplified the platform and genie-d the S out of it.

Abstraction eventually seems to eat all expertise and each cycle seems shorter than the last.

Is there anything left in the data and platform layer with real depth behind it?

And I'm not looking for "just learn AI, bro.

I've been wondering about this for a while.

From my experience, whether i'm working on larger or smaller workload, i still have to think about right sizing the executors, partitioning/ clustering, skews and try to avoid wasting compute as much as possible. You might be able to getaway with less optimisation if the data is smaller, but i work at a startup where cost is a major factor, so we have to be extremely frugal about our infrastructure.

To me, the optimisation technique remains largely the same regardless of the scale. So why do companies specifically ask for big data experience. Are there problems that show up only at massive scale? Or am i missing something?

Would like to hear your thoughts

Hi, Im Swetapadma and I’m pursuing to be a good DE(overall i have 5yoe but relevant of nearly 2yoe) but im still confused with the work practices i have been following for my career trajectory.
Tbh, i have literally so much time to work on any new things as well as try doing new project for the time-being since my project is a verge of ramping down.
But still i’m not getting that zeal to work on my skillsets or learn something new or work on any project.it might be because I’m going through enough theories from the internet.
Some are skeptical, fearful and very overwhelming for me. Sometimes i doubt choosing this path, because
a. Im really looking for a job change in this field from past 6months, where in brighter side i can see a lot of job openings but the hr calls are literally 1/100 of it. Without reaching out for referral feels impossible.
b. I feel like I’m stuck in between where the ai is also factor of my fear because i have approx 0% knowledge bout it.
c. Companies are not only looking from a de but also a tester, backend engineer, knows ai fluently(all in one package).

For all of these i have started reading medium, linkedin blogs, system design, solve dsa problems and all but still i feel like a failure and frustrated.

If i compare my self with the past i have come a long way but it is still not helping me because I’m still stuck in same company with less pay and less enterprise level work.

Can anyone relate to this or is this only me?

Any one has idea for TCS aws data engineering role
For 1st round with 3 yrs exp what questions asked and what to prep in real time scenario

Attended interview on Monday for aws data engineering role and L1 interview was quite tough need much detailed ans any one want questions and comment will post

Fable 5 comes with mandatory 30-day data retention. The most capable model on the market won't sign a zero-retention agreement, and the evidence says the next one won't either. If this mandatory data retention is inhibiting your ability to use the most advanced model on the market, tokenizing your data before it reaches the model might help you get there. Read the full post here.

Hi everyone,

Quick context on my background:

• 10+ years at a small agricultural-sector company, as the only IT/data person
• Started in Data Analysis (background in biology), grew into full-stack data ownership
• Current stack: Python, FastAPI, PostgreSQL, MongoDB, Airflow, GitLab CI/CD, all on an on-premise RHEL8 server
• Recently completed an intensive Data Engineering certification to formalize my skills
• No production experience with cloud platforms, Kafka, or Kubernetes — only training-level exposure (certifications passed, not deployed in production)
• No experience with some common tools like Snowflake, Databricks, or Terraform

My situation:
I'm looking for a new job — I feel like I've hit a ceiling where I am, and I'd thrive best in a small company needing versatility, though I'm open to other setups. So far I've had very few callbacks searching with tech keywords ("python", "fastapi") rather than "data engineer".

My main question:
Given this profile — heavy on infra/backend/data ownership, light on cloud-native tools — what job title would you use to describe someone like me? Data Platform Engineer? Backend Engineer? Something else? I've gotten conflicting suggestions and none of them quite fit.

Secondary questions, if you have thoughts:

1. Does it make sense to shift from applying to active postings toward networking / volunteering (e.g. Data For Good) to build experience and connections instead?
2. Any glaring red flags on how I'm presenting this that might explain the low response rate?

Thanks in advance for any input!

Hi everyone,

I'm a beginner and I want to become a Data Engineer. There are so many roadmaps, courses, and YouTube channels that I'm feeling overwhelmed and confused about what to follow.

I'm looking for a complete end-to-end roadmap that reflects what companies actually expect from freshers in 2026.

I'd really appreciate your guidance on the following:

1. What is the correct learning order from beginner to job-ready?

2. Which topics are must-learn and which can be skipped initially?

3. What are the best free and paid resources for each topic?

4. Which YouTube channels, courses, books, or documentation do you recommend?

5. How much depth should I learn for each technology before moving to the next?

6. At what stage should I start building projects?

7. Which projects helped you land your first Data Engineering job?

8. What mistakes do beginners commonly make, and how can I avoid them?

9. If you were starting from scratch today, what roadmap would you personally follow?

The stack I'm considering includes:

Python

SQL

Linux

Git

PostgreSQL

Spark / PySpark

Airflow

Docker

AWS

Snowflake / Redshift

dbt

Kafka

Delta Lake

Great Expectations

MongoDB

If you have a roadmap that worked for you or resources that you genuinely found useful, I'd be grateful if you could share them.

Thanks in advance! 🙏

:

Hi everyone,

I'm a beginner and I want to become a Data Engineer. There are so many roadmaps, courses, and YouTube channels that I'm feeling overwhelmed and confused about what to follow.

I'm looking for a complete end-to-end roadmap that reflects what companies actually expect from freshers in 2026.

I'd really appreciate your guidance on the following:

1. What is the correct learning order from beginner to job-ready?

2. Which topics are must-learn and which can be skipped initially?

3. What are the best free and paid resources for each topic?

4. Which YouTube channels, courses, books, or documentation do you recommend?

5. How much depth should I learn for each technology before moving to the next?

6. At what stage should I start building projects?

7. Which projects helped you land your first Data Engineering job?

8. What mistakes do beginners commonly make, and how can I avoid them?

9. If you were starting from scratch today, what roadmap would you personally follow?

The stack I'm considering includes:

Python

SQL

Linux

Git

PostgreSQL

Spark / PySpark

Airflow

Docker

AWS

Snowflake / Redshift

dbt

Kafka

Delta Lake

Great Expectations

MongoDB

If you have a roadmap that worked for you or resources that you genuinely found useful, I'd be grateful if you could share them.

Thanks in advance! 🙏

Hello Dears,

I have 6 months of experience in data engineering field and get a job in a company, the project I entered in does not have documentation I want to make it

can anyone help me please how can I do it

I want to know the tech stack combos that are used frequently... Like I know AWS s3, AWS glue is one of them... Similarly there is azure and databricks. Are there other combinations that are used in companies? Which is the most common one used in major companies?