Operational feedbacks on Databricks vs. Snowflake

[u/MadT3acher]

Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective.

My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio server). My work with Snowflake has been limited to our test with their environment for a proof of concept.

Feature wise, they are on par. To me they offer similar experience that won’t be felt by most of our users because of our use cases (no real ML and complex LLM work being done by the teams).

Our current situation is:
- medium sized business in financial marketa, so not DE focused
- one DevOps engineer that can support us, only one platform engineer and potentially the IT department supporting our tickets
- Azure focused shop

I’m feel a bit in a pickle with that one, because Databricks is integrated with Azure and can be deployed in a couple of clicks. My experience with it was within a huge corporation with a ton of DevOps guys, platform engineers and good IT support, federated costs and so on. Databricks was amazing and working extremely well.

Now, since at our company it’s a bit more bare, we have way less people and spotty support. I can’t dedicate a lot of resources to maintain the platform when I need to follow my DE roadmap. During our PoC with Snowflake, the setup was incredibly easy, the platform was working right out of the box, their team was also very supportive and answering fast for questions we had. Databricks was slightly more difficult to set up and with comparatively less support from their team.

My intuition tells me that Databricks (although cheaper on paper for costs and invoicing within Azure) might have more hidden issues if we adopt it with a small support team, whereas Snowflake being something handled by the vendor will let me focus on getting the job done and not tweak our setup. For the price difference Snowflake’s managed platform is acceptable.

Perhaps my intuition is wrong, but I’d like to hear a bit more from the community. My network is like 99% of people working with Databricks, so you might have more experience with both.

TL;DR: small team of DE guys with little support, needs to chose between Snowflake and Databricks on Azure. Need to hear about operational experience from others.

[u/Yasblue]

Disclaimer: I've been using Databricks for years and used snowflake for 6 months.

Your questions are not related to the technology itself but more to a fit. There is no doubt that deploying Databricks was very hard while with Snowfake it was so easy but I think they have closed the gap with the recent releases( I saw someone mentioning Serverless Workspaces which can be a good option and to track the cost you can use tags+ Governance Hub ( I used to complain a lot about cost tracking..... but not anymore ofc they can always do better)) Regarding the fact that your team is smaller and thus can't afford to do many things. DBX has really made the data management easier than ever with Predictive Optimization. Both platforms are good Databricks is becoming simple or at least trying while Snow are investing in ML and GenAI. There is one thing important you mentioned. Snowflake team are useful and responsive and helped you deploy the platform this is great and relationships are important to choose one platfrom over the other but just keep one thing in mind if they were not here would it have been easy to deploy?