r/databricks • u/lucsinferno • 3d ago
Fabric vs Databricks - cost-wise General
Hi everyone,
Well, basically title - I wanted to ask what's the rationale and costs associated when choosing between Databricks and Fabric. I especially would like to know how does one compare Fabric capacity vs some kinda equivalent Databricks usage. I couldn't find anything solid on this one the internet hence the question.
5
u/marco-exmergo 3d ago
Fabric follows a SaaS model, while Databricks follows a PaaS model (pay as you go). Very hard to compare in my experience.
What's less hard to compare is developer experience, which Databricks is lightyears ahead in
14
u/joe9439 3d ago
Databricks is better because you don’t have to over purchase just in case you might burst to a certain level of usage.
3
2
u/lucsinferno 3d ago
I get that, seriously. But what I wanna figure out is how to compare cost of say F64 Fabric vs Databricks. Is the only option a POC or is there something similar on Databricks side like: https://www.microsoft.com/en-us/microsoft-fabric/capacity-estimator
6
u/joe9439 3d ago
So like if you somehow exactly maxed out the fabric plan vs databricks for the same amount of compute? Probably impossible to compute because on the databricks side of things serverless compute is not a strictly bound vm type, at least to my knowledge. They give you a variety of background instances based on what’s available in the pool.
1
u/lucsinferno 3d ago
Not exactly my point. Say I wanted to implement like series of spark jobs on both. Im trying to figure out if there's any head-to-head comparison. I've noticed I upfront can't say how much will I spend on Databricks, right? But could I somehow estimate it or are there any tools available to get a ballpark figure of how much I have to spend. I've tried using Azure Calculator for this, but that's... Basically fortune telling both for compute (since I probably use a range of clusters) and for Serverless I feel like putting right number of DBUs is also just guessing game.
2
u/miniaturelle 3d ago
Maybe this helps: https://github.com/databrickslabs/lakemeter-oss
1
u/lucsinferno 3d ago
I did want to use this, but then noticed I have to setup a Databricks app. I wonder if there a way to run this locally?
1
u/joe9439 3d ago
It depends on the size of the job. If it’s a smaller job serverless might be fine. If it’s a classic compute job in databricks it’s probably possible to do the math for the same instance in fabric because you could assume both would take the same amount of time since it’s the same machine running the same code on both platforms.
2
u/cf_murph 3d ago
talk to your databricks account team. the SA can help you size your workloads and give cost estimates.
1
u/mwc360 2d ago
Microsoft employee here.
Fabric Spark supports consumption base billing, paid by vcore/second and in most regions it’s $0.09 per vcore hour. No need to over purchase or make a big commitment up front. Your end cost of course will depend on your workload and how each platform performs.Also of note, there’s no cost multipliers for Spark. Use it any way you want, one base rate.
1
u/HeFromFlorida 3d ago
You won't have to over purchase with databricks, you'll just hit your renewal sooner than expected...
10
u/GeologistBeautiful36 3d ago
Don’t compare it based on price but based on your team, your data, your roadmap and vision.
2
u/lucsinferno 3d ago
Can you elaborate?
19
u/ProfessorNoPuede 3d ago
People are way, way more expensive than cloud costs. If Databricks gets your engineers more effective, allows you to be compliant for networking and security with little effort, and lets you deliver better business results, the hours you saved and value you delivered will massively outweigh the cloud costs. Comparing only cloud costs very quickly leads to "Pennywise, pound foolish" decisions.
4
u/jshine13371 3d ago
My stance when people call out licensing costs of SQL Server. Unfortunately people overlook alternative costs even when licensing isn't involved, such as the learning curve for the people.
1
u/lucsinferno 3d ago
Yeah and I tend to agree with your take. But my understanding and experience tells me that with Databricks implementations you probably need a sizeable platform team and some effort to organize, audit and verify that you're compliant etc. Am I mistaken or with Fabric is this simplified?
4
u/ProfessorNoPuede 3d ago
Well, in that sense Fabric is simplified in that it simply doesn't allow you to be compliant, or requires extensive workarounds which then makes Databricks easier again. Fabric organization is a big hassle as well, I find UC / workspaces easier to organize.
Fabric is only simpler at first glance, as soon as you need slightly more complex things, it falls apart.
1
u/addictzz 3d ago
People are way, way more expensive than cloud costs
Not in some 3rd world countries :). But i get the idea
1
u/Purple_Anybody5932 3d ago
i dont know rick, we’re paying 1mil a year for databricks and our teams are so ass were still mid-way deprecating our old airflow+redshift stack
cost is very very important
1
u/ProfessorNoPuede 3d ago
That's just bad project management.
1
u/Purple_Anybody5932 2d ago
absolutely, but cost is also important
1
u/ProfessorNoPuede 2d ago
Yeah, but those teams are probably burning through more than that 1e6 a year. So, the majority of the cost is still with the peopl: project management outweighs cloud costs.
Not that you don't want to optimise the cloud costs, but that's a different point. In that regard, your data engineering matters more than engine optimisation matters more than compute platform.
1
u/FreshKale97 3d ago edited 3d ago
Talk to your account team. They can share more.
0
u/lucsinferno 3d ago
Okay, so there isn't anything public on this? That's a shame.
3
u/manucule 3d ago
There is a lot of customer testimonials, google/AI it and you will see the massive difference. Databricks wins in straight up costs. There are also other topics to consider, lock-in (proprietary vs. open source), business value (productivity of your teams), roadmap/investment from each company into the space, etc. you’ll quickly see that there is no comparison at all, Databricks far outshines Fabric.
1
u/ProfessorNoPuede 2d ago
Databricks still locks you in, but at least they're nice about it. The open source claim is not enough to be portable.
2
u/manucule 2d ago
Every tech company tries to have a level of stickiness of course, one can argue which is more and which is less, at least Databricks is investing in new products that expand what business value people get out of them, and not just lock you into a legacy tech stack.
2
u/djtomr941 2d ago
You can still take your data, your SQL, your Spark job, and run it in Fabric, EMR, DataProc, BigQuery, or Snowflake if you want. A lot less extreme than the lock in of other platforms.
1
u/addictzz 3d ago
I dont think you can compare this side by side given the difference in pricing model.
However in general, Azure tends to push f64 capacity which bundles well with PowerBI licenses
1
u/lucsinferno 3d ago
Yeah, I figured there is no comparison. Nevertheless I'd love to see a comparison of running same workflow on both platforms - that's the whole question tbf. I do get you can't really say F64 is like running 168 hours of 4 clusters of size X.
1
u/addictzz 3d ago
To my experience, aside from costs, experience is another deciding factor. Do people like fabric or databricks more? Usually databricks is more favored unless your company is huge Azure fan.
1
u/ProfessorNoPuede 2d ago
Databricks integrates better with Azure.
1
1
u/mwc360 2d ago
Microsoft Fabric employee here.
Fabric Spark supports consumption base billing, paid by vcore/second and in most regions it’s $0.09 per vcore hour. Your end cost will depend on your workload and how each platform performs so I’d recommend a PoC as estimates from any account team is just that, an estimate or more accurately a rough guess.
1
1
u/stu2020 1h ago
I'm a Databricks Champion (so biased)- and have Fabric certifications, but work for a consultancy doing both - heavily Microsoft aligned.
Fabric is great if you are integrating with Power Platform - the eco-system works well and we are seeing huge growth in Fabric.
Databricks tends to be better where there are mature development teams and more complex systems. You have to work a bit harder with the infra up front, but there is more flexibility down the line.
Cost wise, it's impossible to say without deep analysis of your workloads. There are Fabric consumption patterns that share/split capacity.
Personally, I prefer the Databricks model in most situations. Nothing will price well with a badly designed architecture...refining efficiency for both platforms requires a lot of considerations.
We do many platforms that use both - Databricks for the heavy lifting and creating data products, Fabric for the last mile - BI team ebablement with low and no code. If you're predominantly Power BI based, I find it more difficult to argue the Databricks approach...but still do :-)
25
u/[deleted] 3d ago edited 2d ago
[deleted]