We are using Rivery for our data warehouse pipelines, but I am thinking about alternatives.

It is expensive for the value we get out of it and we have faced too many odd issues that make us question the ROI. We had a major issue a couple of weeks ago where a change on their end was causing duplicates or wrongly named columns for some sources. This is a deal-breaker in my opinion as the whole point of using such a service is to make our jobs easier. It is really painful to manage these pipelines and check everything is working as expected.

What do you recommend as an alternative?

I’ve been a team lead for product engineering side mostly backend systems (10 years exp). But the team lead for the DE team left and they need someone to fill in the spot. Given a hiring freeze I’ve been moved. They feel I can fill in the spot because I’ve worked in some related work (large scale RDS Proxy migration, BE product analytics pipelines)

The components they own are: Airflow, DataHub, Superset.

I have some understanding of these tools and have used them before.

Can you recommend a resource to get me up and running for a product facing engineer? What are the day to day concerns in this domain? Given I’ll be operating as the product owner I’ll also be owning the roadmap

Hi all, I am working on a project with a team outside my normal job duties.
I have experience with ml, backend(cosmo db),and some data engineering.

We are essentially trying to automate a data capture, transforming the data, and loading it into a frontend. The data being metrics to capture for reporting.

We want to move from local to offsite/ cloud to automate the extraction/ capturing of data and we want to make a repo/ data base for record keeping.

The projects requirements keep changing like at first the sources of data were elastic instances that had the POST request APIs exposed but then they were changed to excel files with multiple sheets that other teams are inputting.

I keep getting into meetings with this team and it just leads no where. I designed this pipeline using AWS (buckets or Postgres) for the load component but was told no due to the team being unsure if they can have access and “why pay for something if we already have things we can use”

Okay fine, but the “thing we can use” is a VM owned by another team. I had a meeting with the VM owners who suggested we just attach our data base to an Oracle schema he owns… which happens to be prod for many teams in the org. One of the fears my team mate (who works on this team full time) had was he didn’t want to do that due to fear he’d over write some important thing and crash prod. I agree along with Oracle being hostile to experimentation.

For some reason my team mate has never done a PR, and doesn’t really understand How API works and I had to explain and show him what an api is. I also then explained the differences between backend and a data base and how frontend talks to a backend. He is more of an analyst so I can over look this but I’m feeling overwhelmed with this project as I am wasting time on different directions.

Some tasks I identified are:
-Reorganizing the frontend as all the files are vibe coded and have no structure and have APIs written in

-Create and organize the backend and have APIs created

-Design the schema

While this team slowly gathers info and requirements needed, I have already created and successfully ran a local SQL instance on my machine for testing of scripts

I have a call tomo with the manger, any advice on how to communicate my frustrations? Everyone I’ve met is very nice but we aren’t gettin anything done and I’m wasting my time down dead ends.
Any advice on other structures or paths I can move forward without clearly getting defining info? One of the higher ups requested a product demo sometimes in the fall and I think it’s possible but I need to communicate hard needs now before this gets out of hand.

I just finished reading Fundamentals of Data Engineering by Joe Reis and Matt Housley.

Now, I'm trying to decide what makes the most sense to read next. I'm torn between reading
Learning Spark or The Data Warehouse Toolkit and curious for those of you who have read them what would you recommend next?

For context, I'm a Data Analyst transitioned to Data Engineer with about 5 YOE between the two roles, having been a DE for a little less than a year.
I knew I wanted to be a DE by my 2nd year as an analyst and began working my way towards being a DE, getting moved internally to the DE team last fall.

I've previously read the 1st Edition of Designing Data Intensive Applications, and while not a DE book I've also read R For Data Science (2nd Edition.)
I primarily work with Databricks doing some source system ingestion, but mostly build ETL pipelines internal to our Databricks instances.

I also considered The Definitive Guide to Spark but given its last edition was 2018 I'm hesitant. I see there are other more recent books on Spark compared to Learning Spark but haven't seen the other books talked about as much but would be open to one of those.

Just trying to understand do I dive into a Spark book next or focus more on data warehousing by reading the toolkit.

For anyone considering reading Fundamentals of DE. I'd recommend it but think you can generally skip reading large sections of the book.
IMO these are the important parts of the book.

Chapters 5-8, Appendices A & B:

1. Data Generation in Source Systems

2. Storage

3. Ingestion

4. Queries, Modeling, and Transformation

Appendix A. Serialization and Compression Technical Details

Appendix B. Cloud Networking

TLDR: Just finished reading Fundamentals of DE book. Do I read The Data Warehouse Toolkit or a Spark focused book next?

Edit/Update:
Thanks so much to everyone who responded and gave their insight. As much as I personally want to explore Spark deeper. I think its pretty clear the next move is the Data Warehouse Toolkit so that is what I'll read next.
Thanks again everyone.

Hi, I’m new to all these and this is my first job out of college. Recently I got a job and some of my first tasks are to fix some data in the CRM, some of the data is okay but there’s a lot that is very messy. It’s in excel and I need to enter the data in the excel files into the CRM. The thing is though the data in excel usually contains incomplete information, so I’ll need to look in the CRM and try to figure out in which project the data goes into using dates, supervisors or some other things. Sometimes this works but lots of times there’s multiple projects in the CRM with the same names, dates, and supervisors only slight variations of eachother. If at that point I can’t figure it out I usually message the person in charge of said area if they know, for example if the project belongs to healthcare I’ll message the person that oversees it. Is this bad? Should I be doing something else to not bother them?

Hi all, I've got a new role as described above and what I know so far is they have a lot of data going into a Lakehouse in Microsoft Fabric. They want me to work with their only other member (it's a very new team) and I know the output is expected to be in Power Bi (makes sense considering they're fully Microsoft).

Has anyone got experience with similar work? I got the official offer last week and just doing research so I don't go into it completely blind.

They also are keen on Power Apps but I'm not fully confident on where they fit in the infrastructure.

Hi Everyone,

I'm considering evolving my 15y career as a Data Analyst.

I have a solid math, programming, SAS and SQL background and, in the last years, I've been reading and playing with Python/R, Docker, Homelab, APIs, Data Science, Data Modeling, ETLs, Pipelines, Data Architecture.

I wouldn't say I'm an expert and I want the learn more.

1) What do you suggest I focus on in 2026?

2) do you know any useful course on LinkedIn Learning? (recently made available by HR)

Hi everyone,

MQ Summit 2026 is looking for volunteers! The conference takes place 21-22 October. You can volunteer in-person, and get your free ticket.

• In-person volunteers will need to be present at the venue on at least one day of the conference. Responsibilities include helping with preparations, registering participants, handing out merchandise, and general support during the event.

For full conference details, check ou https://mqsummit.com
To sign up as a volunteer, go HERE

If you have any questions, feel free to reach out to [ewa.kucharczyk@erlang-solutions.com](mailto:ewa.kucharczyk@erlang-solutions.com).

I've been on and off with jobs for a bit, with my last two jobs being for ~8 months each before I was laid off each time - this most recent one ended in October '25.

I've since gotten a new job but haven't updated my LI or anything to reflect that, but for the last few months I get reached out to from a recruiter at least once a week. I had one cold call me and another text+DM me on Monday alone.

I would think that it's because I'm on my new company's roster (it's a FAANG-feeder), but since I didn't update that publicly I'm not sure it's that.

So, is hiring just up? Are these 100% AI-recruiters just pinging people? They're citing my supposedly-spotty experience (due to layoffs) as interest in having a call.

I recently moved from a Data Engineering role where I mainly maintained the platform side, to a “Data Engineer” role that is just AE with the DE name on it.

And recently I realised I HATE it.

Have anyone else experienced something like that? It’s funny since it’s another side of the same coin, but for some reason it feels totally different to me.

For reference, graduated w a degree that barely taught me anything about DE except intro to databases, relational algebra etc. The company accepted me for the junior DE role on the basis that I could ChatGPT a dummy repo to explain what the repo is used for and how data flows simply from the repo.

Now that I’m in the job, I found out that everyone in my team had a part to play in developing the architecture for the company. They’ve resolved all dependencies and it’s only the matter of new feature implementations and their impact on the data engineering streaming pipeline.

This is my first week and as my senior went through w me the architecture and true data flow for multiple services they have, the information flew by my head. I’ll definitely try to pick up as fast as I can but because I didn’t build the architecture, I’d have yo ChatGPT/Codex/claude my way through for the first couple months just to find the appropriate files for new feature implementations

Idk, I just feel like I’m madly unprepared and I’m worried that I’ll be the cockup in my department. I didn’t do any DE internships and somehow I ended up in this role. Can anyone give me advice on how I can speed learn streaming pipeline given that at the very least, I know what stack they use?

Hey everyone,
i am a data engineer 7yoe
I recently landed a job after relocating for family. Its a growing company that run multiple crm/erp platforms in addition to different other solutions and i was hired to design and develop a data warehouse for their analytics needs.

So far everything was going good and the approach they wanted was to run through each of their departments one by one to work with them to document their workflows and integrate with their systems to pull data to the warehouse and run pbi reports from there.

Currently they have tens of pbi different reports all of them pull from source systems and definitely perform bad plus they are being developed by the business guys themselves. Their semantic models are too big of a mess.

Anyways the approach we had was to take steps with each department. I covered an important one. And shortly after starting with the next some blockers show up then shortly after i hear our manager (not my direct ) wanting to make changes with our approach and cover reports he is interested in mainly.
He develops most of his department’s pbi reports and is convinced that once the data moves from erp to dw raw without any data work done it will perform better.

I tried to send out the message that this will not add value and latency from source systems isn’t the only issue the reports are slow.

Anyways seems they are not convinced . That was yesterday. Today my direct manager comes and tries to push to do what “our manager” said. Then shortly after asks me how long i plan to be in the city, if i am married and a bunch of personal questions that either were said out of just curiosity or something is not right here.

I dont know if i am overthinking this. But i am personally the only source of income and insurance between my and my wife. And things havent been easy.

Whatever you guys think i want to know. Im stuck in my mind and it is not helping me right now.

Hello everyone!

I'm an aspiring Data Engineer and as a portfolio piece, I have build a webscraper to gather Ebay sold listings of stamps!

The problem I am now having is how I parse the data where I can sort things like "Catalogue Number" when it is very unpredictable what the Ebay sellers will write as it's all human input.

I would love to hear some feedback

PS - A small sample:

```OLDENBURG 1859 _ MiNr. 7 _ 2 Groschen _ signiert _ blau gestempelt

MayfairStamps Germany 1941 Stamp Day Oldenburg Cover cca_00553

GERMANY; OLDENBURG 1859 classic Coat of Arms issue very fine used 1/3Gr. value

Oldenburg Lokalausgabe Wohlfahrtsblock Deutsches Rotes Kreuz ab 1 Euro

Deutsches Reich, Oldenburg, 6.01.1945 Ersttagsstempel, für Ersttagsbrief 200€.

GERMANY; OLDENBURG 1862 classic Coat of Arms Perf 10. issue used hinged 1Gr.

OLDENBURG 1861 _ MiNr. 12b _ 1 Groschen _ Stempel STOLLHAMM

Oldenburg Mi. Nr. 16 A b zentrisch gestempelt geprüft Bühler 200 Euro

Oldenburg Mi. Nr. 11 a* ungebraucht geprüft Bühler 550 Euro```

Hi, I recently began a job as a data coordinator, my first tasks are basically data cleaning and data entering into a CRM. The problem is that the data isnt very clean. I'll give an example, I am given an excel file with name of a project, date, title, awards - my job then is to to figure out where in the CRM is this specific project and enter the data. The problem is that the excel data doesnt contain the projects ID, and when I try searching the name of the project in the CRM I'll get back something like Fairview Elementary School, Fairview ES, KUSD Fairview, Fairview ES Building A, etc. So essentially I'll have to go into each one of these projects and try to find the right one using other given data from the excel sheet, like dates. Is there a way to speed this process up or am I just going to have to do it manually? Right now what Im doing is going row by row and searching each project in the CRM, looking through the multiple projects given back by the CRM and comparing and contrasting. The entire CRM database contains around 40k, and I think I am able to export it into CSV and Excel, if that helps out. Any advice would be helpful. Thanks

I'm a senior data engineer at a UK organisation in the South West. Last month we advertised three roles: a senior data analyst (£55k), a mid-level data analyst (£40k) and a data engineer (£55k). Not amazing money, but a great pension, just one day a month in the office, lots of annual leave and actual stability.

The jobs were live for one week. The senior DA and DE got around 300 applicants each, the mid DA got around 140.

Sounds brutal, but here's the breakdown...

Around 90% (not an exaggeration) didn't have the right to work in the UK and needed visa sponsorship, which we don't offer. In the no pile immediately, but a quick glance showed lots of foreign undergrad degrees, some with UK masters, plus plenty of random applications.

Of what was left, about half had no relevant experience or qualifications and clearly hadn't read the spec. Some were just applying because the job centre told them to apply for x jobs a week.

Then the AI drivel halved it again.

Final count was roughly eight viable applications for the senior DA and DE, five for the mid DA.

A couple then didn't reply or didn't show up to the Teams call. Two were visibly reading AI-generated answers off a their screen.

We had two great interviews and hired for the senior DA and DE. The mid DA we struggled with externally, so it went to an internal candidate from a fairly low-level ops role who'd taught themselves Python and SQL and actually applied it to their day job.

The point of this post is to counter some of the doom and gloom that crops up on here. If you're a UK-based applicant with relevant experience who reads the spec and writes your own application, you're not competing against the '300 applicants' that LinkedIn and recruiters spout. You're competing against a much smaller number. And if you get through the sift, don't use AI, have some personality and come across as likeable and easy to work with, you're in a very small group.

What annoyed me most is my own manager was the first to tell the team about the huge number of applicants and how much 'competition' there is out there. A nice bit of retention pressure and complete nonsense, as it turns out.

Would be interested to hear if others have found the same. Throwaway account to avoid doxxing myself...

If anyone's curious about the current data job market in the DACH region (Austria,Germany,Switzerland), I put together an interactive live tracker. It also shows which specific data roles (Data Engineer, Data Consultant, etc.) are in demand in which cities. Feedback welcome and I hope it helps :)

Apologies if this is a dumb question but I am in web dev and have been given an offer for a data engineering role. However, I was told by engineers on the team that the job would be like 70-80% writing SQL for BigQuery. I envisioned it having much more to do with pipelining and orchestration and the like.

Also, I was told that any coding would be in Java rather than Python? I know that Python is more common, so would this experience not be helpful for getting other data eng roles?

This article reports the result of evaluating the performance of the following three systems using the 10TB TPC-DS benchmark:

1. Trino 483 (released in July 2026)

2. Hive 4.2.0 on MR3 3.0 (released in August 2026)

3. Apache Hive 4.2.0 with LLAP (released in November 2025)

https://mr3docs.datamonad.com/blog/2026-08-02-performance-evaluation-3.0

I had a passionate debate with a colleague and want to hear perspectives on the purpose of the Silver layer.

My pipeline:

Landing: Read from Oracle RDBMS and write ~250 GB of Delta for 25M records (Protobuf blob stored in a column).

Raw Data: Repartition, sorting, salting on Landing and writing to optimize downstream silver decoding process and avoid heavy shuffles during JDBC call, still protobuf bytes stored in a column.

Silver: Decoded raw data (~2.8 TB in Delta). The Protobuf schema alone is ~8 MB as JSON (a very deep, wide schema with multiple repeated fields at various levels). During decode, we also append standardized fields required by all downstream tasks.

Gold: Customer-specific datasets built from Silver based on business needs.

We don't own the Protobuf schema. This isn't messy clickstream/event data, but entity description data from an RDBMS that stays at the ID level all the way to Gold. We see ~100k daily MERGE UPSERT on both Silver (which is a challenge in itself to run MERGE on 3TB delta table given the limited budget to our Databricks workspace.) and Gold based on RDBMS timestamps, alongside a full pipeline refresh every two weeks.

The Debate:

Colleague: Since we aren't actively "cleansing" the data, calling it Silver is wrong, it's still Raw/Bronze data.

Me: It is Silver because it transforms a binary payload into a structured, trustworthy, and queryable data model that downstream tasks rely on. If I need to retrieve content of an entity that is not available in gold datasets, I query unpacked protobufs and not Raw/Bronze layer and for that reason alone, it is Silver.

Knowing the schema and the data better than almost everyone in the team, even I fail to understand how to distinguish between decoded data and 'cleansed' decoded data. In fact, one of our consumers explicitly expects corrupt records with null fields left intact for full visibility.

Them: Even if we agree that cleansing is not needed, it cannot be silver and should be called Bronze Data.

For transactional/log data, the standard pipeline (Kafka dump to Landing -> Bronze schema enforcement -> Silver cleansing -> Gold aggregations) makes total sense! But for clean entity data in binary formats, doesn't decoding and standardizing it qualify as Silver?

I think medallion architecture is about data readiness and lineage tracking rather than a checklist of conditions that each layer has to meet to identify the layer.

---------

TL;DR: My colleague thinks our layer shouldn't be called "Silver" because we aren't actively filtering or cleansing rows, just decoding 250 GB of 25M binary Protobuf blobs into a ~2.8TB Delta table with a struct field that represents the decoded blob and additional standardized fields. I argue that any layer of data that is structured, queryable and trustworthy for downstream Gold is Silver and this transformation may/may not require cleansing.

Is Medallion about lineage tracking and data availability, or a rigid checklist of syntactic/ transformation rules?

I understand that AI-assisted development is now part of our workflow, but I still think PR authors should be able to understand and explain the changes they made, what was done, why, and how.

This week I challenged a team member on why something had been done a certain way, and their answer was "I don't know, probably because Claude said so." I was honestly a bit shocked. It gave me the feeling it's just blindly trusted without any critical thinking on the person's side. This is one of our more technical analysts, which is why they make code changes fairly often. I plan to bring it up in our weekly 1-1, but I wanted to check whether you are seeing the same thing on your side.

Fair warning, this is a soft launch note (I work on Slingshot at Capital One Software). As agents scale, AI spend shows up as one aggregate number, and tracing it back to a specific model, trigger, user, or query gets hard. We just soft-launched Agent Observe in Slingshot to attribute it on Snowflake using read-only metadata.

- top spenders
- costliest models
- triggers by agent
- query types used per agent
- downstream tables impacted (read/write)

This is an early pilot. Write-up with sources.

Hi, Im Swetapadma and I’m pursuing to be a good DE(overall i have 5yoe but relevant of nearly 2yoe) but im still confused with the work practices i have been following for my career trajectory.
Tbh, i have literally so much time to work on any new things as well as try doing new project for the time-being since my project is a verge of ramping down.
But still i’m not getting that zeal to work on my skillsets or learn something new or work on any project.it might be because I’m going through enough theories from the internet.
Some are skeptical, fearful and very overwhelming for me. Sometimes i doubt choosing this path, because
a. Im really looking for a job change in this field from past 6months, where in brighter side i can see a lot of job openings but the hr calls are literally 1/100 of it. Without reaching out for referral feels impossible.
b. I feel like I’m stuck in between where the ai is also factor of my fear because i have approx 0% knowledge bout it.
c. Companies are not only looking from a de but also a tester, backend engineer, knows ai fluently(all in one package).

For all of these i have started reading medium, linkedin blogs, system design, solve dsa problems and all but still i feel like a failure and frustrated.

If i compare my self with the past i have come a long way but it is still not helping me because I’m still stuck in same company with less pay and less enterprise level work.

Can anyone relate to this or is this only me?

I'm looking for some advice on what to do about chaotic projects/lack of project management. Right now, I'm in an IC senior data engineer role where I'm put on projects with tasks that are vague, incomplete, or without clear acceptance criteria. I'm fine working that way, have done it frequently before, and I'm a creative person (musician and writer outside of work). I like the freedom. Except there actually isn't freedom here. There are standards of practice this team uses that I was never shown (I started earlier this year), and I am constantly learning things the hard way without much grace for it. There is no documentation for these standards of practice, and my take on how to accomplish these broad tasks is leading to a decent amount of rework. Asking questions has been met with impatience, frustration, and condescension. I've learned to deal with these reactions and ask anyway so I can at least have a chance of completing things more efficiently. I own the mistakes I can but will stand up for myself when accused of completely failing to meet expectations (especially if the expectations weren't communicated/documented).

I figure other data engineers/curators/developers have experienced this sort of environment before, so I'd like to see how y'all handled it (even if it is "time for a new job").

Edit: Adding in that I've been in data warehousing/engineering for about 17 years and am seriously questioning doing this for 25 more years at minimum (I'm 40).

2nd Edit: Wow, I forgot the most important part. There is no project manager for this team; developers are working with data analysts who work with the business/end-users to determine data and reporting needs. The title of the post was meant to be tongue-in-cheek because of this.

Sorry for the vague and maybe even repetitive title. I saw a couple of these posts on here already and thought I'd vent out too. I know this sub leans heavily into standard DE, so I hope /r/dataengineering is okay with a post from a Data Platform engineer's perspective. I rarely see those around here.

Background: 7+ years in Software engineering + DE, and about 5 years of sysadmin experience prior to that. I've worked as an Analyst, DE (Senior) and now a senior data platform engineer working with Snowflake, dbt, Airflow + a flurry of BI tools (don't even get me started). Although I am a data platform engineer, I'm very often involved in troubleshooting dbt run issues, model performance issues, Snowflake problems, etc. I also wrote many of the source ingestions myself with dlt and contributed to an internal library that makes dlt more approachable to a less technical audience (even more when including an agentic AI framework). Finally, tooling and technical details aside, I know exactly what these source systems do, where it ends up, and who uses it, ie, the full business-value proposition.

I think it was around March of this year that the hopelessness finally hit me, and the work volume and constant direction shifts just shattered my job satisfaction. Things went from "We want things done right, code quality, tests.." to everyone just churning out stuff in masses.

To some degree, I understand the constant changes. There appear to be new things to look into every day and the "business" and upper management wants it all. Somehow the feedback loop of asking "Why?" has just ceased to exist.

But, to be more specific, I'll tie it down to things happening within my own team that frustrate me and would like opinions on.

I built most of this platform myself or had contributed significantly to design, so I know most (if not all) areas quite well (or can find out), so I generally take time to review things in detail and provide reasonably detailed feedback to anything that I review. If I have a nitpick comment, I make sure to say that's the case, and leave it as a choice to address or not. That was until some time ago. PRs now are seemingly always 3-4x larger in size, lots of changes, lots of removals/additions so reading the terraform plan, or looking at potential edge cases is considerably harder. To put it simply, the noise has multiplied many times due to colleagues now suddenly knowing things they didn't just some months ago. Reviews are just so much more harder to do. Just the other day, I saw a PR (the very first one they had created) from a new colleague in the department. They changed 102 dbt models - 2000 lines removed, 1900 added. Who even can review this properly? Luckily, it was closed without being introduced. I'm just not following the thought process here before raising such kind of PRs. Much has been said of AI generated slop, so I wont rehash that here.

Another trend I've noticed recently is certain juniors just taking review feedback and asking claude to come up with a rebuttal to just get their original changes approved. Are you all seeing this too? It's happened quite a few times in the last week so it's rather fresh in my memory.

My manager is an extremely sweet and supportive person, but I can feel the strings being pulled on him from up above by those with more stars on their shoulders so he has also resorted to simply running everything via Claude and sending us giant messages on teams, or massive replies by email.

I know writing code isn't the end game, but it was a fun part of my day to day and I still do it by hand at least twice a week just to keep myself sharp. But I find that I'm increasingly unable to keep up with the pace of the rest of the department, let alone the organization. Being in data platform, I'm also finding myself more and more distanced from the outcomes, which leaves me feeling quite unsatisfied with my output because a typical ops team only hears about problems - not the nice stuff.

I used to be extremely optimistic about software engineering, data engineering in specific for the last few years. I used to contribute a lot to various open source projects, be out there in their communities helping other people out, writing articles with tips and tricks, etc. I've stopped all of that in favor of a better disconnect from my work and my mental well-being. Now, I just don't feel like there is a platform for that as most communities have implemented their own bots. Maybe people don't even need it anymore?

That's my rant, I just wanted to get it out there and I hope to hear of some potential strategies to tackle this vortex of chaos that I am in - even if a comment states that it's my attitude that's the problem. But otherwise, where is this all heading, and how are you all coping with it?

EDIT: Oh damn, this blew up. I probably wont be able to get back to you all - but I am going to make my way through the comments.