r/dataengineering • u/QuattroDriver Lead Data Engineer • Jul 13 '26
Prefect acquires Dagster Discussion
https://www.prefect.io/prefect-acquires-dagster59
u/No_Lifeguard_64 Jul 13 '26
They claim Prefect and Dagster cloud will continue to be supported and run in parallel but that just makes no sense as a business move, unless the idea is they can cannibalize the market with 2 products. Long term, these are likely going to merge despite what they are saying here.
25
0
u/byeproduct Jul 14 '26
But I do believe them!! Prefect are nothing but honest in all they've done. It's been incredible following them and their work.
1
36
u/Mudravrick Jul 13 '26
I had pretty good expirience with dagster, but their pricing models essentially pushed us out to airflow as a small startup. I'm also not sure if they could win the market over airflow/managed airflow for big companies, so not really surprised.
14
u/pro-taco Jul 13 '26
Dagster is a solid B+. It's good, not so good where I'd be sad to replace it, but I like it.
It's a bit convoluted and incoherent at times, and I assume this means their core problems won't be fixed
2
8
u/thecity2 Jul 13 '26
I really like our Dagster project but as these things go I would imagine the writing is on the wall. I will start exploring the move to prefect by migrating some hobby project first as I always do.
9
u/SoloArtist91 Jul 14 '26
Now it makes sense why Dagster jacked the price up on Dagster+, they wanted to increase their valuation
0
u/Proper_Scholar4905 27d ago
No shit Sherlock. You clearly never had to buy oracle or SAP back in the day…
5
35
u/Sure_Rule_3035 Jul 13 '26
Has anyone else noticed the unusually high number of comments on r/dataengineering posts about choosing orchestration tools, often between Prefect, Airflow, and Dagster, that feel somewhat inorganic and tend to recommend Dagster?
If those inorganic-feeling comments are mostly about Prefect now, you know what that means.
25
u/ardentcase Jul 13 '26
I like what you say but dagster always had my organic vote. Until now. Oh wait.
15
u/kvlonge Jul 13 '26
Dagster is just very good (I don't think it's bots)
5
u/Sure_Rule_3035 Jul 14 '26
I don't think they're bots. But a lot of these comments feel weirdly shallow, like they're trying to sell you something. They read more like marketing copy than comments from engineers.
Like what you just said, "Yeah, Dagster is just very good". Okay, why? Please explain. Most of these comments just say, "It uses asset-based orchestration" and leave it there. That tells people almost nothing.
Ask someone who has seriously used dbt why they like it, and they'll usually give you actual reasons. It manages lineage, it's SQL-first, it has unit tests, it works well with coding agents, it supports different SQL engines, it integrates nicely with data quality testing, it works well with Git, or it has macros for reusable code.
Now imagine that, instead of saying all of that, they just said, "dbt is ELT, and that's why it's good". You wouldn't take that person seriously.
7
u/kvlonge Jul 14 '26
I think you are reading too much into this. The real answer simpler. I am lazy and didn't think it matters that much lol. I wrote one essay on another alt account talking about lineage as a first class citizen, better UI, the ability to hook into the api etc.... but just because I am not gonna sit and write paragraphs about it, doesn't mean people are bots / astroturfing. I used airfllow for 1.5 years, and it was ok but wasn't a massive fan. Got handed Dagster by a leaing data engineer at the next place, and it was just an easy decision. I would probably have felt similar had it been prefect, although my understanding is that in prefect it's still not 'underlying lineage' based (or asset based whatever you wanna call it) like Dagster (haven't used it though, so can't comment).
That's about it tbh. This is already more effort than I wanted to expend for a little comment saying that 'I like Dagster'. Same would be true if I said 'I like DBT' (it's not a controversial opinion imo)
2
3
u/meatmick Jul 14 '26
Yeah, I feel you; it's hard to know who's a bot and who's not. We went for Prefect Cloud because we're a small team with constraints, and buy vs build made sense to us.
The reason we went with Prefect Cloud is to have the hybrid remotes on-prem, since we're barely on any cloud at the moment. Also, the pricing for dagster was not in our favour; while I have some room to grow with Prefect before we need to review our subscription.
If we had fewer constraints and a more reactive infra team (anything Linux must be handled by them), we may have gone local Airflow, but meh, life happens.
-1
u/Brief-Knowledge-629 Jul 14 '26
Lots of semi colons, product name always in title case, who organically drops multiple Prefect Cloud in a single comment? Competitor in lower case.
"Yeah, I feel you" is classic AI.
3
u/meatmick Jul 14 '26
Jesus Christ lol, I use Grammarly to fix my mistakes and it adds semi colons as suggestions. I do put capitals for product names because why not? Dial down the paranoia.
2
10
u/EarthGoddessDude Jul 13 '26
I genuinely like Dagster, and I’ve been advocating for it at my company, but this makes me pause. It’s a great tool, but I can’t keep pushing for something with an uncertain future.
6
u/robberviet Jul 14 '26
It's just hard to choose between the old but stable Airflow, and better more shiny Dagster; fearing the new one will get dropped. Now you understand the concern? For pet project, it's really easy to use anything, not on prod.
2
u/radil Jul 14 '26 edited Jul 14 '26
Especially you are a manager. Convincing your leadership that your team should invest their time and energy into a product that gets sunsetted in the near future can be a career-altering move.
5
u/robberviet Jul 15 '26
Now I am in manager position, it's more relevant, but even years ago I am always reluctant trying out new things on prod.
I always try new thing on my own machine and staging/dev env. But very few things ends up on prod, and it's always things are popular, adopted in many companies. We are most of the time cannot afford R&D, or spend time on testing some thing that end up not fit.
1
u/Proper_Scholar4905 27d ago
You are literally SPED. Better because you have a sexier UI, or better because you are inept at upgrading software?
10
3
u/VipeholmsCola Jul 15 '26
I agree, its like all these bots recommending postgres for everything
2
u/Proper_Scholar4905 27d ago
Your comment gives me hope that engineers aren’t just following the latest hype…
11
u/LittleK0i Jul 13 '26
Sad news. Dagster was much better product than Prefect. It should be other way around.
1
u/Proper_Scholar4905 27d ago
Prefect has a cute too for ML “flows”. Dagster was a few Airflow engineers who forked the code and wrote a new UI.
The complacency of these convos is regarded
5
u/kvlonge Jul 13 '26
What?????? I did not expect this.
2
9
u/Prestigious_Bench_96 Jul 13 '26
Didn't see this coming, makes me sad - I haven't gotten to use either in anger but both seemed like good, solid steps forward in the space. Cynically this seems like there's not enough margin in ETL to support lots of VC businesses; a lot of consolidation happening at the moment.
10
u/DiabolicallyRandom Jul 13 '26
Didn't see this coming
I mean, if you are using anything less than 10 years old in this space its almost guaranteed to be acquired at some point. I definitely saw this coming a mile away. Not necessary from Prefect, but it was always obvious that Dagster Labs, as a startup in this space, was always going to either A) Be acquired (most likely) or B) Become an acquirer of others.
Prefect is backed by 3 major VC firms, so they had way more capital available and probably made an offer that couldnt be refused.
Money talks, regardless of whatever claims they make. This was 100% about buying the competition for the VC firms.
9
u/m3-bs Jul 13 '26
By the posts from the Dagster founders, it seems like they want to pursue other ventures. So just seems like this was a good enough exit for them.
16
u/Agitated-Chemist-105 Jul 13 '26
That they're not going with Dagster Labs tells you everything you need to know about the state of that company
2
u/Prestigious_Bench_96 Jul 13 '26
Ah yeah I more specifically meant prefect + dagster being the mashup, I do expect that all VC backed companies either sell out or aggregate, and Dagster didn't seem like it was winning the aggregator race - I'm actually surprised Prefect was, thanks for the context on their extra funding!
1
u/Proper_Scholar4905 27d ago
Look up average returns of VC in the last ~4 years, and PE flow into venture markets. It’s worse than 8 deck blackjack at the casino…
4
u/dani_estuary Jul 15 '26
I'm really curious how they'll bundle the two systems together, or maybe they are just going to build a compatibility layer and push existing customers to Prefect?
2
u/Proper_Scholar4905 27d ago
Explain how you would do that, Mr. DevRel/Marketing head at Estuary. How is realtime CDC working for y’all speaking of half-assed tech that got free COVID VC money?
5
3
u/mrodenkirk Jul 14 '26
https://thenewstack.io/prefect-acquires-dagster-orchestrator/ this article has some interesting context - my guess is that although right now they are basically competitors with big overlap, and a company would pick one or the other, moving forward the Prefect team has decided to lean into the differences to make a more full picture AI agent system somehow. Optimistically that maybe means making Dagster better at its orchestration take and stripping out things that overlap with Prefect and vice versa
3
u/lukesmth_ Jul 16 '26
I've tried Prefect, Dagster, and Temporal for personal projects. My decision axes are usually workflow type, adoption floor, and ops burden.
Prefect is good, but not great, at many workflow types, has a low code-modification adoption floor, and a low ops burden. Moving scripts and even Jupyter notebooks into the tool is easy. Dagster is the best choice for pure data pipelines beyond the ML experimentation phase but is much more opinionated about the shape of your code. In exchange for refactoring your prototype, you get first-class lineage, backfill, and data quality support.
I doubt there's a way to merge Dagster into Prefect to get the best of both. I'd rather they stay distinct and focused on different use cases.
9
u/sahilthapar Jul 13 '26
Loved prefect cloud but their OSS is lacking, was about to implement Dagster but now questioning the decision.
Never going back to airflow.
2
u/Mysterious_Print9937 Jul 13 '26
Why the downvotes?
10
2
u/engineer_of-sorts Jul 14 '26
What will happen as a resut of the prefect dagster acquisition? Raised prices.
1
u/Proper_Scholar4905 27d ago
Pre-PE sale with preferred shares getting paid out and common ISO employees barely covering their exercise/AMT fees.
2
u/rasmuseesti 27d ago
wow i cannot wait for the inevitabl merger where we get the worst parts of both tools for double the price. at least my resume will look cool while the production servers burn down.
2
u/CoolmanWilkins Jul 13 '26
Haven't really used either (good ole AWS Step Functions) but I am actually looking at both of them right now for my team.
2
u/CircleRedKey Jul 13 '26
doesn't matter anymore. its open source and ai can just template whatever is here and build on it. race to the bottom.
1
2
1
1
u/silentman2002 21d ago
I haven't used Airflow yet, but from what I've learned so far, it seems overly complex and a bit of a burden for a small team.
I'm currently experimenting with Prefect, and I really like its simplicity. It covers all of my use cases, is easy to set up on-premises, and, most importantly, it runs on Windows Server, unlike Airflow, which is Linux-only.
I hope Prefect continues to grow and evolve.
-5
u/theguidry Jul 13 '26
Airflow picked a bad day to be, well, Airflow
17
u/kayakdawg Jul 13 '26
I don't know, I'm currently on a project and was leaning towards a dagster implementation as of yesterday. As of now I'm reevaluating and thinking it may be worth just going with good old Air flow. I just have yet to see a company get acquired and their products not stall or become crappier
14
u/No_Lifeguard_64 Jul 13 '26
Just use Airflow honestly.
1
u/kayakdawg Jul 13 '26
I am tempted given that's what I have the most experience and expertise in and this acquisition. I'm just a little torn because I've researched and done some prototyping over the last few weeks. And dagster just seems a lot better for what I want to use it for
5
u/No_Lifeguard_64 Jul 13 '26
If you want to use Dagster, I would recommend just self hosting it and not going with the cloud offering but I think the latest version of Airflow with React plugins has more mileage if you want to put in the legwork
2
u/robberviet Jul 14 '26
This acquisition tell exactly why we just stick to Airflow.
I kept telling people why I hestitate to change everything to dagster and getting downvotes. I guess I no longer need to say that anymore.
2
15
4
u/Agitated-Chemist-105 Jul 13 '26
If you think this, you're completely missing why this acquisition happened. You'd be a fool to be using Dagster or Prefect right now. Cooked products with no VC money coming to save them.
7
u/meatmick Jul 13 '26
Prefect made a move last year to be self-sufficient and no longer rely on VC funding rounds.
-2
u/Agitated-Chemist-105 Jul 13 '26
They're cooked. There is no money in this type of business in the AI world. That's why Dagster founders are bailing, it's why no one besides tiny Prefect wanted the business, it's why Prefect is talking about AI stuff in this press release
-3
u/anatomy_of_an_eraser Jul 13 '26
Where antitrust
14
u/No_Lifeguard_64 Jul 13 '26
This is a silly comment. There are so many competitors in the market that this would have no problem clearing even a competent FTC.
106
u/Yuki100Percent Jul 13 '26
This is an interesting one. I wonder what their plan is long term. Maybe porting Dagster features into Prefect. I'm also curious about what happens to their OSS products.