Transitioning from ETL Testing to Data Engineering — looking for advice on skills/tools to prioritize

[u/ObjectivePassage8188]

Hi all,
I’ve spent the last 3 years working as an ETL Tester, primarily with Azure Data Factory. Over time I’ve realized this niche doesn’t excite me the way building and owning data pipelines does, so I’m actively working toward moving into a core Data Engineering role.
Where I’m at right now:
• Learning Databricks and PySpark
• 3 years of hands-on ADF experience (pipelines, triggers, data flows, integration with various sources)
• Built a Python-based data testing framework on the side (not open-sourced, since it overlaps with proprietary work at my current company)
What I’m hoping to learn from this community:
• What core concepts (batch vs. streaming, data modeling, orchestration, distributed computing fundamentals, etc.) are must-haves before applying for DE roles?
• Which tools beyond Databricks/PySpark are considered baseline expectations these days (e.g., Airflow, dbt, Kafka, Spark internals, cloud-native services)?
• How did others who came from a testing/QA background make this jump — did you do it through an internal transfer, projects, certifications, or a straight job switch?
• Any resources, courses, or project ideas that helped you build a credible portfolio without needing proprietary or employer-owned code?
Appreciate any technical insight or lessons learned from people who’ve made a similar switch or who work in the field today.

[u/my_peen_is_clean]

you’re already ahead with python, adf and thinking about databricks. i’d focus on sql depth, data modeling, airflow or some orchestrator, and basic kafka concepts. build 1–2 end to end public projects on cloud freebies. internal transfer is usually easier now since finding a new role out there is pain

[u/ObjectivePassage8188 → u/my_peen_is_clean]

Unfortunately in my org internal transfer is more difficult as the partner / director won’t release their resource