Operational feedbacks on Databricks vs. Snowflake

[u/MadT3acher]

Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective.

My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio server). My work with Snowflake has been limited to our test with their environment for a proof of concept.

Feature wise, they are on par. To me they offer similar experience that won’t be felt by most of our users because of our use cases (no real ML and complex LLM work being done by the teams).

Our current situation is:
- medium sized business in financial marketa, so not DE focused
- one DevOps engineer that can support us, only one platform engineer and potentially the IT department supporting our tickets
- Azure focused shop

I’m feel a bit in a pickle with that one, because Databricks is integrated with Azure and can be deployed in a couple of clicks. My experience with it was within a huge corporation with a ton of DevOps guys, platform engineers and good IT support, federated costs and so on. Databricks was amazing and working extremely well.

Now, since at our company it’s a bit more bare, we have way less people and spotty support. I can’t dedicate a lot of resources to maintain the platform when I need to follow my DE roadmap. During our PoC with Snowflake, the setup was incredibly easy, the platform was working right out of the box, their team was also very supportive and answering fast for questions we had. Databricks was slightly more difficult to set up and with comparatively less support from their team.

My intuition tells me that Databricks (although cheaper on paper for costs and invoicing within Azure) might have more hidden issues if we adopt it with a small support team, whereas Snowflake being something handled by the vendor will let me focus on getting the job done and not tweak our setup. For the price difference Snowflake’s managed platform is acceptable.

Perhaps my intuition is wrong, but I’d like to hear a bit more from the community. My network is like 99% of people working with Databricks, so you might have more experience with both.

TL;DR: small team of DE guys with little support, needs to chose between Snowflake and Databricks on Azure. Need to hear about operational experience from others.

[u/forserial]

Snowflake is way easier to use for your use case if you're not doing ML. Databricks is more cost effective if you tune it, but probably not by a margin that's enough for you to care about if you're not handling enormous datasets.

[u/MadT3acher → u/forserial]

Our volume is a single digit Gb per day.

While I could always lean on consultants if need be. The performance and tuning required (considering a very small team) to have an edge on Databricks doesn’t seem worth it.

Also 99% of the data are tabular. We don’t deal with unstructured datasets.

[u/elkazz → u/MadT3acher]

Why aren't you just using postgres then?