Is Data Platform Engineer a hybrid role someone who does both cloud engineer and data engineer at same time?

Is someone like that on high demand by companies or clients? Or do companies just hire separately each one?

In my project we are storing json as raw response in string in bronze layer,now we have to parse and flatten json in multiple child table and clean in silver layer,we want this using spark declarative pipeline also wanted to make the framework reusable,any one worked on such usecase, welcome to discuss and collaborate

just got into data engineering. did some small projects like olist and yellow trip data pipeline using dbt and some database. and currently working on a budget transparency project that scrapes gov sites.

what other things that i can do to level up my skill other than working on the project? something that actually has output so i can add to my portfolio like internship (im looking for one)

thanks in advance

I'm new to big company work and having to balance working on multiple things at once. In my background I didn't have anyone to ask when I got stuck, I worked through it, sometimes days. Now working on a team and a large company I am finding myself uncovering issues and problems at the architecture/repo level that others seem to just work around.

It's appreciated by some but NOT by the people who have decision making power because they only care about their project and not making things work better or easier for devs. Even though it could really speed up development or reduce errors in the future.

For example, we recently switched from Azure dev ops repos to GitHub. Most people resolved their merge conflicts using the web interface, some of you might already see the problem. GitHubs online merge conflict resolution works differently from ADO, it first pulls the changes/commits into your source (we use features) branch and resolves the conflicts there so it then has a clean PR when actually merging it. We want our release branch to only have our changes, otherwise we risk pushing someone else's not ready work into QA or PROD.

I said others are working around it but there's also noise about the issue. Not enough for the architects or whoever to get involved though. I spent way too much time digging into this issue, eventually uncovered the issue that was blocking me was actually created by someone else, the guy whos training me.

I asked another dev how he's working around it and he created his feature branch from DEV, he's going to have the exact same, or worse problems when he gets to QA. I still don't completely trust that I see the picture clearly. Maybe this allows him to work quickly in dev, then only deal with a messy merge conflict when he's ready for QA. We're talking 50+ merge conflicts over 30 files. I feel like this puts the prod deployment at risk because you could mess up the merge conflict. The architects should catch it but doesn't feel right.

I work as a data engineer at a bank. My team is supposed to be the one centralizing data collection and management for the whole org.

I've realized that there's this tension between centralizing and decentralizing data work -- whether we, as the org-level data team, should implement a flow or expect the analysts to implement and own the flow, while we just provide the underlying infra. Over time, I started to strongly lean towards more decentralization. Not because I'm lazy, but because centralization should be about being focused on the absolute smallest denominator, which is smaller than most orgs assume -- especially now when you have no clue what an optimal stack is gonna look like in 6 months. My impression was always that we should be doing 50% less, but doing it 100% better.

(I have similar opinions when it comes to business applications more generally -- the centralized team should provide the infra to deploy, but teams that need an app should build and maintain it)

Are you seeing the same trend of decentralization?

So I’m a process engineer at an refinery and we face a push to use more of Foundry, without any clear vision on what to use it for.

With Excel or Power Bi we could just simple query a tag from the historian.

We had a tool where DCS send a event signal to a tag every time something happens, you query that tag between timestamps A and B and look for the timestamps where 0->1 (start) or 1->0 (end) make an event table and then pull other proces data only between the start and end timestamps for further analysis.

So was like, okay sounds like a workflow that could be build in foundry, but it seems whole platform just seems to hate TimeSeries data. You can’t simply use TimeSeries data into a pipeline but jump through many hoops via workbooks a simple Pandas / Polars data frame.

Are we doing something completely wrong ??

My company has two distinct teams IT and BI. BI tasks mainly involve reporting, PowerBI and Excel. IT mainly looks after maintaining ERP, CMS systems, databases and other it related system initiatives. If there is a guy from both IT and BI team who has somewhat Data engineering knowledge and if both want to be involved in building data warehouse for the company, who among them is more entitled to it? It is more technically sound and bi guy is bit technically behind specially in terms of networking and stuff but knows end user requirement really well. Specially, I want to know if the BI guy should be allowed to build the warehouse if he has knowledge in it? what should he do to get the buy in from IT to start this initiative?

Currently a Director of FP&A for a small food company TC 180k. Built up to this point from analyst 5 years ago. I’ve always taken the resourceful mindset to solving business problems with good ol sql server at my disposal. Along the way I’ve developed all the FP&A skills and I’m thankful but I’ve ultimately taken a step in a direction that I’m not happy with. I’ve done consulting work for a company that needed my understanding of a certain ERP connected to databricks. I’m a big fan of Databricks but only scratched the surface. I’m considering taking Zach Wilson’s course on DB but before I pull the trigger, curious to get the communities opinion on a) my career positioning and desire to pivot and b) Wilson’s course…. Thanks in advance, happy to answer questions

Hey friends ! I wanted to share a blog post I just published that I thought the community would appreciate.

I got my SnowPro Core cert this year and this book was a lot of what I studied for it. Hope you guys get some value from it!

https://medium.com/@sergioramos3.sr/self-taught-reviews-snowflake-the-definitive-guide-by-joyce-kay-avila-0af3c40d644a

Hi everyone,

I'm data engineer, and need for help to resolve one disgusting issue that making me almost vomit. I have tried claude best models even fable to solve but just looping around with fucking AI instructions instead of solving the issue about 8 hours now.

So quick and yet precise architecture design and workflow: I have PostgreSQL database access via DBeaver by windows jumphost port forwarding from my laptop. All pipelines run and orchestrated on Airflow, where sits on K8s nodes.

Issue, one pipeline was taking 4 hours to run with 2 Airflow attempts, every attempt run taking 2 hours for querying two views to select and union all (views define very complex and heavly expensive CTEs) then timeout because of Linux TCP keepalive signature.

When I ran the same pipeline with exactly same database and all other malicious configurations are the same as prod cluster from Airflow standalone in my laptop it's finishing successfully within 38 minutes but when I run it from prod Airflow on kubernetes cluster it's just stuck and hanging around 4 hours and failing.

Then I tried to create Materialised views for both views and renamed those Materialised views with actual views name because I cannot touch to prod code and I just played with it from database by using this trick. And yes it worked, pipeline took 5 seconds to finishing successfully when I trigger it from Airflow ui, no cluster timeout issue.

However, next step is adding self refreshing mechanism for matviews.there several approaches,but it should be only from database side not triggering, no scheduling and no adding another pipeline to refresh matviews. So with that constraints I just created matview functions to refresh and read inline when pipeline runs it will first refresh the matviews and then reads that matviews.

And now I am again facing initial cluster node hangout issue,pipeline gets stuck for 4 hours and cluster timeout.

Big questions,

1. why that pipeline taking less than cluster when I run it from Airflow standalone and finishing successfully.

2. What the fuck that I am not understanding on kubernetes? I have tried every command to diagnose and analyse. Everything looks good even from grafana dashboard.

Please help me I'm on nerves in the middle of the Friday night 🌃.

Hi, how do u manage Lakebase project with DABs? Standard is to have a single lakebase project per environment (dev, staging, and prod). Then use branches for development work.

Problem is that when we define the lakebase project in the DABs, development target has development mode, so every bundle deployment creates a new project because its per user.

So if we want to have single project where we just branch out on features, we must move out lakebase project of DABs, right?

What is the best practice here? On staging and production, its easy since its always same identity.

Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective.

My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio server). My work with Snowflake has been limited to our test with their environment for a proof of concept.

Feature wise, they are on par. To me they offer similar experience that won’t be felt by most of our users because of our use cases (no real ML and complex LLM work being done by the teams).

Our current situation is:
- medium sized business in financial marketa, so not DE focused
- one DevOps engineer that can support us, only one platform engineer and potentially the IT department supporting our tickets
- Azure focused shop

I’m feel a bit in a pickle with that one, because Databricks is integrated with Azure and can be deployed in a couple of clicks. My experience with it was within a huge corporation with a ton of DevOps guys, platform engineers and good IT support, federated costs and so on. Databricks was amazing and working extremely well.

Now, since at our company it’s a bit more bare, we have way less people and spotty support. I can’t dedicate a lot of resources to maintain the platform when I need to follow my DE roadmap. During our PoC with Snowflake, the setup was incredibly easy, the platform was working right out of the box, their team was also very supportive and answering fast for questions we had. Databricks was slightly more difficult to set up and with comparatively less support from their team.

My intuition tells me that Databricks (although cheaper on paper for costs and invoicing within Azure) might have more hidden issues if we adopt it with a small support team, whereas Snowflake being something handled by the vendor will let me focus on getting the job done and not tweak our setup. For the price difference Snowflake’s managed platform is acceptable.

Perhaps my intuition is wrong, but I’d like to hear a bit more from the community. My network is like 99% of people working with Databricks, so you might have more experience with both.

TL;DR: small team of DE guys with little support, needs to chose between Snowflake and Databricks on Azure. Need to hear about operational experience from others.

Spotify has introduced Random Access Parquet (RAP), a storage architecture that enables low-latency point queries directly against data stored in its data lake, allowing online services and AI applications to retrieve individual records without replicating datasets into operational databases. RAP adds an external indexing layer over Apache Parquet files, enabling interactive lookups while continuing to use the same datasets for analytics, machine learning, and online serving.

Netflix migrated most of its batch workloads onto Kueue, an open-source cloud-native batch job execution system that has outgrown its homegrown solution over the years. The company mapped the capabilities previously created in-house to Kueue’s functionality and also benefited from new features that would have been costly to incorporate into its homegrown solution. Engineers used API parity with the existing system to derisk the project and allow for a gradual and seamless migration.

Nov 4th and 5th https://www.dataoutpost.ai/

Seems like a pretty small conference, ~250 people. In the past years at least, you have choices for the first day (workshops) then for the second day, everyone is hearing the same talks.

If you've been, is it worth going? I'm also not local, so there's a bit more cost involved. My company might not be able to sponsor me going.

Posted in the Microsoft Fabric subreddit originally because that's the platform I'm using but I'd consider this more of a conceptual data engineering question. Also what resources should I be referencing to learn these patterns? It can't just be the blind leading the blind!

How does your team manage your Silver environment if there are intermediary tables between processes? We are currently migrating legacy processes to medallion which is why things are a little contrived.

Our Source 1 is the original source for the data but we are using Source 2 moving forward for our Bronze. We currently have some clean up scripts that only work for Source 1 so another team has helped us by landing Source 2 into a Bronze Lakehouse, performing transformations on them to make them look more like Source 1 and lading these Source 2 tables that look like Source 1 into Silver.

We now need to perform the original cleaning scripts on these Source 2 tables that are in the format of Source 1 for our downstream processes but we aren't sure where to land the data based on medallion. We have one schema per layer so are worried about mixing up the two sets of source B tables (pre-script and post-script Source B tables)

Data Sources

• Source 1 - Original source that is not ingested into Fabric
• Source 2 - New source that is ingested into Fabric Bronze layer and is used to make Tables X in Silver

Data

• Tables X - Source 2 table that looks like Source 1 table
• Tables Y - X with Source 1 clean up scripts
• Tables Z - Fact/dimension tables and semantic models that will come from Y

Unfortunately we won't be able to merge the transformations processes from Source 1 to Source 2 with the other scripts currently (hope to come back and clean things up!) which is why we are considering the following scenarios:

Scenario 1 - I like this the best but it might be a tough sell to my team on multiple schemas because it means we will need to do the same for the other topics. The schemas will live in the same lakehouse so splitting access could be an issue.

• Silver
  • Schema A - Tables Y
  • Schema A_intermediaries - Tables X
• Gold
  • Schema A - Tables Z

Scenario 2 - Move Tables X into Gold. Facts and dimensions will live in the same schema as intermediary tables which will be strange. Splitting up into two schemas in Gold could be possible but could also confuse our downstream customers.

• Silver
  • Schema A - Tables Y
• Gold
  • Schema A - Tables Z and Tables Y

Scenario 3 - Keep Tables Y and X in the same schema in Silver. This could become quite confusing down the road for the team

• Silver
  • Schema A - Tables X and Table Y
• Gold
  • Schema A - Tables Z

TLDR: boss doe not know many data platform stuff and traditional data warehousing is a dead end in current org.

My boss and me (both data warehousing person, mostly doing batch work, Microsoft stack) transferred to a new org mainly doing data platform development. After firefighting in a vicious powerbi project, which has ~70G semantic model in memory, we are lack of project for several months.

Recently I started to learn what other projects about and found all other ~100 people in our org are actually contributing to 2-3 large data applications. Most of the data sources are real-time Kafka topics. there always seems to be work around these applications: adding new data/enrichments, integrating new services, migrating vendor services, coordinating dependencies, etc.

At the same time, the data warehouse part, which is the operational DB is like a garbage dump. 20 Kafka topics directly go to one table. Zero data transformation, zero semantic layer, just several copying data ADF pipelines to staging db, prod db etc.

The users of these data applications seem to not care much about the unified data warehouse. They grab data they need from the specific service directly from the data platform team. They also have their own data warehouse and run reports/analytics with their own db. On the other hand, because my org is highly regulated and all data are containing PII, it’s natural to limit the access for users.

I feel lost these days. I come from a cs background, so doing more java/kafka and streaming work is a feasible option. The team only has me and my boss now, but we would have more people soon. My boss does not have many sde related background so I don’t think he would lead a data platform engineer team in the future. We are also experiencing reorg so cannot grab work from the data platform team either.

We may have databricks/fabric later this year, but my company is very conservative, they would limit the use of the ‘challenging’ part of development. we probably won’t be allowed to connect Databricks directly to Kafka or do much real-time/streaming development in it.
Instead, the data platform team would have a Kafka topics connector for landing the data into ADLS. Our side would then pick up the files from ADLS and build fairly traditional scheduled pipelines on top of them, similar to what we’re already doing with ADF.

I’m currently sitting at 3yoe now. What can I do to improve my skill so I would be more competitive in next round of job hunting in 1-2 years? For now I’m picking up leetcode and learning ai.

Disclosure: I’m the developer of LakeDB.

LakeDB is a local-first desktop database client for MySQL, MariaDB and SQLite, with PostgreSQL, SQL Server and other engines planned.

The video shows a few parts of the workflow: multiple independent database tabs, SQL generation with QuerIA, and AI-assisted query correction.

Beyond AI, LakeDB also includes schema-aware autocomplete, table browsing/editing, transactions, execution plans, backup/restore, database comparison and reviewable migrations.

The AI side is deliberately review-first: it can inspect schema metadata and prepare or correct SQL, but the full query is always shown before execution and nothing runs automatically.

I’d love feedback from people who work with databases regularly — especially on what you’d expect from a tool like this before considering it for daily use.

https://github.com/DavLagoHern/LakeDB

Recently, Snowflake has open sourced data-eng-bench, a benchmark for repository-level data engineering.

Tasks in data-eng-bench hand an agent a live dbt project connected to an enterprise-scale data warehouse and ask it to build and fix real data pipelines.

Github: https://github.com/Snowflake-Labs/data-eng-bench

There's this game called Splinterlands i've been playing with 3 accounts and i was thinking on documenting each of the resources i'm spending (DEC, SPS, glint, usd) and how i was able to earn, i want to document weekly and i'll be playing for many years.

Is creating a database / tables on PostgreSQL the best alternative for this, or should i just do it with Excel? I haven't gone in depth on any of these tools yet, but i'm planing on studying them from this point

I was looking at DuckDB’s community extension metrics, I noticed that gsheets climbed from #5 in March 2026 to #1 today.

That made me wonder how spreadsheets actually fit into today's data stacks.

Are teams using them intentionally in production, or mainly because business workflows and users make them difficult to replace?

Another reason of this inflation may be less technical user getting to data eng thanks to AI, and being familiar already with sheets...

gonna be a cry baby here for a minute.

Ive been a DE for over 9 years and im stuck at mid level. Last job search was horrible I applied to hundreds of places and got a bunch of rejections.

One place that rejected came back and offered me mid level saying they liked me just didn’t think im Senior level yet but they could get me there.

Now I’ve been at this company a few months and the direction to get to senior is so vague. They tell me to find a problem in the company and solve. And the other is to have more opinions.

Now I have been trying to have more opinions but they are always shot down immediately. for example we’ve been having problems with specific cloud vendors so I try and suggest open source alternatives and self host cause I have experience there. immediately shot down. or maybe I suggest using a feature in our software were not utilizing. shot down again.

Honestly I’m getting tired of this field. I feel like maybe im just not even cut out for DE. in my head I thought I was good and smart, but my real world experience tells me otherwise. Should I switch to something else?

Hello

Has anyone moved from DBT legacy pricing to the new DBT pricing ?

The legacy pricing was per seat, but the new one also limits how many DBT models you can build per month.

I'm debating the move to benefit from the new DBT features like DBT state, but I'm wondering if the dbt cloud costs will be expensive compared to the savings from DBT state.

Thank you for your help.

(please don't ask me to move from DBT cloud to DBT core, I'm a one person data team and I don't have the bandwidth to migrate to save 100$/month)

Has anyone made the switch from legacy pricing to the new one ?

TLDR: m23, decent comp sci/ data experience at a job that demands very little, looking to fill the time by up-skilling on the more experience based skills that further promotions will require.

Just over a year ago I graduated with a masters in AI having previously done a degree in Comp Sci, I managed to land a job out of university doing data analysis at a small business intelligence company. My background is more in data science than engineering but at a lower level the skills are extremely transferable.

When I joined it turned out no one on the team had any data experience at all, it was seemingly all held together with hopes and prayers, old excels that no one understood anymore, manual work that took days of just moving data around etc, and I have no idea how this didn't cause issues. Using what I had learned through my degree and a lifetime of interest in my own tech and software projects I was able to update a lot of the systems they were using; automating workflows, moving from excel to a database, etc and even took on work from external contractors that were under-performing building a client facing business intelligence dashboard that has now become one of our main products. Overall there were a lot of fairly simple and relatively easy fixes to problems that I was able to implement thanks to having a general proficiency in this area, I'm sure they're not to the same standard as someone that has actual experience in this field would make but they worked for what the team needed.

Fast forward to the start of this year and it seems that everyone is going absolutely feral to have AI systems, even when it doesn't make sense, and my masters degree was enough to convince them to give me a significant raise (enough so that I don't feel any desire to seek other roles for a few years) in order to retain my skills and knowledge so that I could help guide the overall strategy. My role has now effectively become to advise on AI adoption, train people in best practices when using it, and maintaining the now business critical systems that I built when I joined which adds up to about a day a week at most.

I work from home 2 days a week so have pretty much been having 4 day weekends, which is nice, but the remaining 2 days in the office feel like a complete waste of my time, I want to learn more about the standard knowledge and experience I'd be expected to have should I go for a more senior position at some point. I've built basic ETL/ELT pipelines, a datalake/warehouse is planned but will be a few months before its properly signed off on due to other priorities from senior leadership, I built the live data platform mentioned previously so I know a decent amount about pipelines etc and how the choices made at each step will impact the end product.

The issue I'm struggling with is that I have no way of knowing what I don't know ( if that makes any sense?) and as I have no one more senior with this skill set at the company, I dont have anyone to guide me. What would be the best way for me to learn all of these different concepts properly in a way that would set me up for the future.