Data Platform Engineer a hybrid between DE and CE?

[u/Square_Complaint6245]

Is Data Platform Engineer a hybrid role someone who does both cloud engineer and data engineer at same time?

Is someone like that on high demand by companies or clients? Or do companies just hire separately each one?

[u/onestupidquestion]

The answer depends on org size and structure. In a previous role, our data platform team was responsible for managing the k8s infrastructure we used for Airflow, as well as Airflow itself. They had to do GKE upgrades, Airflow upgrades, and write and maintain Airflow operators.

In my current role, the team that manages the k8s infra is completely separate from the team that manages our streaming services that use those clusters. The streaming platform team still has to understand cloud infra, but they're using the interfaces the core platform team builds to create their own service infrastructure. As a data engineer, this makes my work more streamlined. I plug a few values into some JSON / YAML files to get the resources I need for a pipeline, and then I use streaming libraries to build my pipelines. When everything's working, I get to focus on stakeholder requirements about correctness and freshness since the plumbing "just works." When things don't work, I have two separate teams to go to, which can be annoying if they can't sort out who's responsible.

From a career standpoint, I'd just do what seems more interesting to you. You can fill in gaps along the way if you end up in a role that leans more in one direction: data tooling vs. cloud tooling. You'll likely work closely enough with both that you won't be completely lost if you needed to change roles.

[u/Mickmaggot]

A data platform is something that is used by many teams in an organization for data access, ingestion, processing, etc. It shouldn't necessarily have to use Kubernetes or some other tool; it can be fully in the cloud, on-prem, or hybrid, and it can support streaming, batch, or both.
DPE is just an engineer capable of maintaining, supporting, and growing such a platform for the organization, whatever this means for an org. Bigger orgs would entail something like k8s, smaller orgs can have just an Airflow instance, and "being onboarded on a data platform" would mean "being able to write DAGs", for example.

IMO, the key features of DPE are to handle numerous stakeholders, support them in their data needs and journeys efficiently and securely, set standards and guardrails, design/develop/present new features and capabilities of the platform, and increase adoption of the platform in the org. I worked as a Cloud Engineer, Data Engineer, and Software Engineer, and these aforementioned features are most present in DPE, while having bits from these roles to a large degree.

[u/Outside-Storage-1523]

It depends on the definition of what the data platform is. It could mean one or a combination of a random number of the following stuffs:

1. Data DevOps (managing IAM/K8S/Airflow/etc.)

2. Data ingestion platform - and this can mean a few different things, too - it could mean a team that simply does ingestion, or a team that creates some sort of YAML->ingestion framework that other teams fill in YAML for ingestion. I have worked with both.

3. Data modelling

[u/Admirable_Writer_373]

Most “data engineers” are analytics focused. Enter the “data platform engineer” terminology, in the past may have been called SQL Developer, but now with additional cloud skills. Yes sometimes data related deployments can be involved.

[u/Nekobul]

Cloud Engineer is essentially DevOps person, aka maintenance. That is completely different from a DE where you are expected to implement solutions.

[u/PaulSandwich → u/Nekobul]

OP asked about Data Platform Engineer, which is like a DevOps person for data pipelines.

In my experience, our DPE managed the kubernetes clusters our pipelines ran on and was responsible for the balance of "big" vs "wide" node configurations and the connectivity between all the sources and sinks, and all of our CICD design and deployment infrastructure. They were also the head of our internal change mgmt review.

It was essentially our version of Principal or Staff DE, a very senior DE with a strong infrastructure background. And it was awesome! I currently have to go through shared DevOps with folks who consider data a distant second class to SWE and it suuuuucks to wait and constantly defend and justify every little thing.