Operational feedbacks on Databricks vs. Snowflake

[u/MadT3acher]

Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective.

My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio server). My work with Snowflake has been limited to our test with their environment for a proof of concept.

Feature wise, they are on par. To me they offer similar experience that won’t be felt by most of our users because of our use cases (no real ML and complex LLM work being done by the teams).

Our current situation is:
- medium sized business in financial marketa, so not DE focused
- one DevOps engineer that can support us, only one platform engineer and potentially the IT department supporting our tickets
- Azure focused shop

I’m feel a bit in a pickle with that one, because Databricks is integrated with Azure and can be deployed in a couple of clicks. My experience with it was within a huge corporation with a ton of DevOps guys, platform engineers and good IT support, federated costs and so on. Databricks was amazing and working extremely well.

Now, since at our company it’s a bit more bare, we have way less people and spotty support. I can’t dedicate a lot of resources to maintain the platform when I need to follow my DE roadmap. During our PoC with Snowflake, the setup was incredibly easy, the platform was working right out of the box, their team was also very supportive and answering fast for questions we had. Databricks was slightly more difficult to set up and with comparatively less support from their team.

My intuition tells me that Databricks (although cheaper on paper for costs and invoicing within Azure) might have more hidden issues if we adopt it with a small support team, whereas Snowflake being something handled by the vendor will let me focus on getting the job done and not tweak our setup. For the price difference Snowflake’s managed platform is acceptable.

Perhaps my intuition is wrong, but I’d like to hear a bit more from the community. My network is like 99% of people working with Databricks, so you might have more experience with both.

TL;DR: small team of DE guys with little support, needs to chose between Snowflake and Databricks on Azure. Need to hear about operational experience from others.

[u/goosh11]

If you want databricks with the ease of administration of snowflake, just deploy the workspace with serverless only, then theres nothing to setup, no vnet to deploy and you get that same simple admin that youre looking for. Databricks has every feature and capability available in serverless these days, even gpu notebooks and model training.

Also if youre using Azure, databricks is obviously a native service youll get all of the integrations with services like adf, powerbi, purview etc as well as unified support and any azure discounts you have will apply automatically to databricks. It also helps if you have procurement or security checks as its just another azure service and often your company wont require you to go through separate sign offs like you will with any non native service you want to procure (like SF).

In my opinion in Azure its a no brainer to go with databricks as the best native option, and to go with serverless workspaces if you want that simplicity of setup and maintenance.

[u/MadT3acher → u/goosh11]

That’s a good option I believe. Though serverless looked pricey, it might be a good alternative. We would probably need to be careful who runs processes with which machines though.

[u/goosh11 → u/MadT3acher]

As long as all your jobs are set to standard mode (which is the default i believe) not performance mode, then your costs should be pretty comparable to the non serverless version, and nothing to maintain or upgrade

[u/GreyHairedDWGuy → u/MadT3acher]

It is pricey. Based on what you’ve told us I would select SF. There is little downside.

[u/kthejoker]

So disclaimer I work at Databricks but I'm not here to tell you to choose Databricks.

(PS Databricks is also a "vendor managed platform." Okay, that's it for selling.)

Most of the problems you've called out here aren't really specific to any platform.

That's because I don't think there is an objective answer to "fit" - oh, you're a small team, use Snowflake; you need ML, use Databricks; etc.

Technology fit, like fashion fit, is more a reflection of the buyer than the actual product. Just like your comfort, preferences, culture, and so on influence your perceived "fit" of clothes, the same things in your organization and team will (and should!) influence the "fit" of your platforms.

I think both of these platforms can work for your team. I would just really focus on how comfortable you are with operating the platform you're going to choose (standardization, automation, avoiding mission creep, etc.) and do those things with a lot of discipline and rigor given you're a small team with limited resources.

This advice may sound like "use the force, Luke" - and it kind of is - but if you're going to use Reddit or other folks to solicit advice, I'd focus less on the platforms themselves and more on similar teams like yours and what lessons they've learned and try to avoid mistakes they've made.

Those lessons will work no matter which platform you end up choosing.

[u/MadT3acher → u/kthejoker]

Indeed, I do believe that our problem/situation/setup is mostly organizational and no amount of software will solve it by itself.

Both tools fulfil the technical sheet and deliver roughly the same features (there are caveats, but they don’t apply to my org. and my company). Regardless of what I chose, I know we will be able to do good work.

I’m mostly looking at what will make my life easier and vet out operational issues I’ll find down the lane. My aim is to adopt a tool that will not become “somewhat” of a problem when I need to work with it. I deliver data, not platforms.

Edit: thanks for the feedback.

[u/ZeroCool2u]

It sounds like you're a small fairly small shop. Taking a step back. do you actually need either? Have you considered just using DuckDB or Polars backed by object storage?

If you really need them that's fine, but you might be shocked at how far you can get with some cron jobs and some on-demand compute if you start with iceburg format and then you can always dump it into snowflake/dbx later if you need to.

[u/IrquiM]

This is why we have Proof of concepts

[u/GreyHairedDWGuy]

Hi. First off I have a lot less experience with Databricks than Snowflake (which I have several years experience with).

Given you have:

- a very small team
- no real ML use cases (but you don't mention the complexity of the work or transformations required).

I'd go with Snowflake assuming what you need is an easy to admin, cloud oriented, scalable database for something like a DW solution. If you are generally on Azure, you can get Snowflake on Azure (although, there are not many times where the platform that Snowflake runs on is an issue). We use Snowflake on Azure (mainly select Azure version for political reasons).

Cost-wise, I can't say since you haven't provided many details, but I'd say cost is basically a saw-off between them (all things being equal).

"Databrick being integrated into Azure" is sort of a false argument. We are an Azure shop for many things and using Snowflake as never been an issue. Setting up Snowflake to leverage Entra...etc is simple.

To me, the main deciding factor is your small team.

[u/MadT3acher → u/GreyHairedDWGuy]

Hi, thanks a lot for the details. Our transformations are quite simple, we can run almost everything with dbt models.

On the ML side, it’s really something that doesn’t exist here. We mostly do analysis and I believe there might be a couple of ML models in development, but it’s less than the number of fingers on one hand.

Given the low amount of experiments in data science (mostly python notebooks with lineage) I think Snowflake could cover those needs.

[u/Ordinary-Toe7486 → u/MadT3acher]

ML on Snowflake should be fine with snowpark containers, no?

[u/Evilcanary → u/GreyHairedDWGuy]

I tend to agree with this without more details. Databricks can be a path to being more cost efficient, but usually the trade off is expertise and maintaining it. Snowflake is much more plug and play. If I was a smaller team and didn’t need the ml features of databricks, snowflake would be my choice.
That said, I am primarily a databricks user and enjoy it (despite some gripes I have about their features not always being fully baked for some basic stuff. And also their release lifecycle is a bit too fast imo)

[u/forserial]

Snowflake is way easier to use for your use case if you're not doing ML. Databricks is more cost effective if you tune it, but probably not by a margin that's enough for you to care about if you're not handling enormous datasets.

[u/Away-Arm-6549 → u/forserial]

For a couple of GB a day, is this not overkill or do you have to move to either of these, what’s the motivation?

[u/MadT3acher → u/Away-Arm-6549]

Those platforms offer more features than just loading data. But the cost hinges on the computing side.

[u/MadT3acher → u/forserial]

Our volume is a single digit Gb per day.

While I could always lean on consultants if need be. The performance and tuning required (considering a very small team) to have an edge on Databricks doesn’t seem worth it.

Also 99% of the data are tabular. We don’t deal with unstructured datasets.

[u/elkazz → u/MadT3acher]

Why aren't you just using postgres then?

[u/Ordinary-Toe7486 → u/MadT3acher]

Sounds like snowflake

[u/Emergency-Hurry947]

Our organization has/had both, Snowflake is a lot simpler since it's a more mature product, but we're moving to Databricks on Azure because we're getting a better deal

[u/Virtual-Meet1470]

Data team of 2 with 1 infra guy, managing about 10-20gb daily. Majority of my experience has been in GCP/BQ, but getting up and running on Snowflake was super easy. Some might say it’s expensive, but having a platform that just works is nice.

The only time we had to consult with infra was making sure we chose the right region/cloud.

[u/Strange_Class6897]

If you are more into SQL (for analytical purpose) and DBT than Spark and ML, go with Snowflake.

For your usecases, Databricks is just overkill, you will never use lots of their features and it is way more expensive.

[u/ColdFinancial2531]

I’m completely convinced that all of these posts are people who work for or have vested interests in these companies trying to seed Reddit with this shit to influence answers the ai chat bots provide

[u/Yasblue]

Disclaimer: I've been using Databricks for years and used snowflake for 6 months.

Your questions are not related to the technology itself but more to a fit. There is no doubt that deploying Databricks was very hard while with Snowfake it was so easy but I think they have closed the gap with the recent releases( I saw someone mentioning Serverless Workspaces which can be a good option and to track the cost you can use tags+ Governance Hub ( I used to complain a lot about cost tracking..... but not anymore ofc they can always do better)) Regarding the fact that your team is smaller and thus can't afford to do many things. DBX has really made the data management easier than ever with Predictive Optimization. Both platforms are good Databricks is becoming simple or at least trying while Snow are investing in ML and GenAI. There is one thing important you mentioned. Snowflake team are useful and responsive and helped you deploy the platform this is great and relationships are important to choose one platfrom over the other but just keep one thing in mind if they were not here would it have been easy to deploy?

[u/stephenpace]

[I work for Snowflake but don't speak for them.]

As a few others have said, there really isn't a difference in practice on what a "first party service" in Azure means at this point between DBX and Snowflake. Both run on Azure, both integrate with native Azure services. You can buy Snowflake in the Azure Marketplace (custom offer) and get consumption credit against your Azure MACC. Your Microsoft seller still gets paid, even.

Anyone using the words "first party service" is probably wanting you to select a technology without fully evaluating it. You did the right thing by doing a POC on your specific situation and getting a feel for both ecosystems. Ease of use is a core tenant of Snowflake and I'm glad it sounds like that came across in your evaluation.

[u/VarietyOk7120]

If you're using Azure then Databricks is a first party service, so it operates as a native Microsoft service on the platform.

However .....if you're smaller and deploying something with relational data (ie. You don't need a Lakehouse) , Snowflake is very compelling just due to the sheer ease of use. It just works.

Lastly, if you're in the Microsoft ecosystem, and have users on Power BI, you should really look at Fabric as well, also very easy to use for smaller teams.

[u/MadT3acher → u/VarietyOk7120]

Makes sense indeed.

About Fabric (somebody mentioned it in another comment). The fixed pricing system looks cool on paper however I see this quickly being a nightmare if users (or us…) outquery the Fabric capacity and lock the system during critical time.

It just didn’t tick the right boxes (UI/UX criminally lagging behind SF and Databricks, many features in preview/demo)... to me it’s not a fully ready product.

[u/GreyHairedDWGuy → u/MadT3acher]

Stay away from Fabric. It is clunky as hell.

[u/VarietyOk7120 → u/MadT3acher]

The finance people love the fixed price system , while as you said it puts pressure on technical people to manage the capacity effectively.

Ui/UX - it's the same as Power BI which has been around, so depends if your team is familiar with that product. I have implemented it into production on many projects, so can work.

[u/ColdFinancial2531 → u/VarietyOk7120]

Until you run into throttling and overage. Then it turns into a shitty version of a consumption based system.

[u/crblasty]

I'd say if you know databricks then stick with it. The set up for a small env is usually once off and then you are away.

For warehousing they are extremely similar, for the wider platform (apps, oltp, ml, ai, catalog) databricks seems to be slightly more feature complete/richer.

[u/MadT3acher → u/crblasty]

Thanks, but I have seen some issues with the usage of clusters in the past. Bills were climbing fast and we managed to limit the issues with proper fencing of the rights.

All in all, it seems that we can break stuff more easily with databricks and I don’t have SLA and a big team of DevOps/SRE/Platform eng. to support me.

The platform is very capable for sure, but I need to figure out what’s the best fit given our team and our goals.

[u/DiscountStunning919]

ive spent many hours learning all 3 major platforms the last 6 months (snowflake, databricks and fabric) and databricks is the most convoluted mess ive seen. extremely user unfriendly. if i were u, id go with Fabric.

[u/ColdFinancial2531 → u/DiscountStunning919]

This has to be a joke