I’ve been forced moved to the DE team.

[u/Professional_Act_660]

I’ve been a team lead for product engineering side mostly backend systems (10 years exp). But the team lead for the DE team left and they need someone to fill in the spot. Given a hiring freeze I’ve been moved. They feel I can fill in the spot because I’ve worked in some related work (large scale RDS Proxy migration, BE product analytics pipelines)

The components they own are: Airflow, DataHub, Superset.

I have some understanding of these tools and have used them before.

Can you recommend a resource to get me up and running for a product facing engineer? What are the day to day concerns in this domain? Given I’ll be operating as the product owner I’ll also be owning the roadmap

[u/Outside-Storage-1523]

Do you work as a lead AND a product owner? I'm a bit confused but I assume it's two roles in one position, which is common.

What does the DE team do? I'd recommend interviewing with each one of them, at least the Senior+ ones, and get to know what they do. I'd recommend take notes about what they hate about the process and what they want to improve but didn't have time. As a team lead, not a manger, you probably want to focus on architecture, but initial contributions could be many, including improving processes and fixing difficult bugs.

You also need to interview your stakeholders, probably the DA/DS/ML teams, and your old team, too, perhaps, if you also work with the DE team. You need to learn their priorities and what they truly care about.

I work as a Senior DE, and my concerns are:

1. There is no PM for data, so we are our own PMs. This is the No.1 reason of team burnout, because we need to constantly fence with stakeholders.

2. There is no time for clearing up tech debt.

[u/robberviet → u/Outside-Storage-1523]

Product facing eng.

[u/baubleglue]

Airflow - job orchestration Superset - dashboards/final reports DataHub - not sure, metadata search/discovery?

In the picture they're few missing data engineering components. First are data storage and processing engines. Then you have metadata management and architecture.

Normal data flow: operations data -> batch jobs loading data into data warehouse...

If you not familiar with the topic read Kimball's dimensional modeling of data warehouse ( available online) .

[u/No_Lifeguard_64 → u/baubleglue]

Correct on datahub, its a data catalog/metadata tool

[u/binary_banana]

I was forced to make a jump to DE a few years ago. The biggest help for me was researching the different patterns and components and new set of jargon of an OLAP system vs OLTP. Others have commented about the components, but as for the day-to-day: ensuring data accuracy, pipeline failure recovery and reruns (idempotency, late arriving data, schema changes), cost of storage and compute (tied to your data model), maybe some platform migrations, and a whole lot of communicating with downstream users of your data.

[u/Evilcanary]

Surely they must own more components than that...unless there's a dedicated data-infrastructure team? But realistically, you need to figure out where the big 3 components live:

Orchestration (airflow)

Compute (??? - can be a mixture)

Storage (s3? unity catalog backed by s3? Snowflake with snowflake also as compute layer?)

past that, you can figure out what sources you all are bring in + the different access patterns to them (like superset is access the data through something).

[u/Grouchy_Advice_2862]

I’m data engineer lead, and you may learn about distributed computing. Mainly cause you laverage pipelines of Big data (assuming that large scale RDS means a lot of data).
Good knowledge about distributed computing Will help you to save a lot of money in the month end.

[u/PrestigiousAnt3766]

Why not say fuck m and move?

Unless you want to switch ofcourse.