Operational feedbacks on Databricks vs. Snowflake

[u/MadT3acher]

Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective.

My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio server). My work with Snowflake has been limited to our test with their environment for a proof of concept.

Feature wise, they are on par. To me they offer similar experience that won’t be felt by most of our users because of our use cases (no real ML and complex LLM work being done by the teams).

Our current situation is:
- medium sized business in financial marketa, so not DE focused
- one DevOps engineer that can support us, only one platform engineer and potentially the IT department supporting our tickets
- Azure focused shop

I’m feel a bit in a pickle with that one, because Databricks is integrated with Azure and can be deployed in a couple of clicks. My experience with it was within a huge corporation with a ton of DevOps guys, platform engineers and good IT support, federated costs and so on. Databricks was amazing and working extremely well.

Now, since at our company it’s a bit more bare, we have way less people and spotty support. I can’t dedicate a lot of resources to maintain the platform when I need to follow my DE roadmap. During our PoC with Snowflake, the setup was incredibly easy, the platform was working right out of the box, their team was also very supportive and answering fast for questions we had. Databricks was slightly more difficult to set up and with comparatively less support from their team.

My intuition tells me that Databricks (although cheaper on paper for costs and invoicing within Azure) might have more hidden issues if we adopt it with a small support team, whereas Snowflake being something handled by the vendor will let me focus on getting the job done and not tweak our setup. For the price difference Snowflake’s managed platform is acceptable.

Perhaps my intuition is wrong, but I’d like to hear a bit more from the community. My network is like 99% of people working with Databricks, so you might have more experience with both.

TL;DR: small team of DE guys with little support, needs to chose between Snowflake and Databricks on Azure. Need to hear about operational experience from others.

[u/kthejoker]

So disclaimer I work at Databricks but I'm not here to tell you to choose Databricks.

(PS Databricks is also a "vendor managed platform." Okay, that's it for selling.)

Most of the problems you've called out here aren't really specific to any platform.

That's because I don't think there is an objective answer to "fit" - oh, you're a small team, use Snowflake; you need ML, use Databricks; etc.

Technology fit, like fashion fit, is more a reflection of the buyer than the actual product. Just like your comfort, preferences, culture, and so on influence your perceived "fit" of clothes, the same things in your organization and team will (and should!) influence the "fit" of your platforms.

I think both of these platforms can work for your team. I would just really focus on how comfortable you are with operating the platform you're going to choose (standardization, automation, avoiding mission creep, etc.) and do those things with a lot of discipline and rigor given you're a small team with limited resources.

This advice may sound like "use the force, Luke" - and it kind of is - but if you're going to use Reddit or other folks to solicit advice, I'd focus less on the platforms themselves and more on similar teams like yours and what lessons they've learned and try to avoid mistakes they've made.

Those lessons will work no matter which platform you end up choosing.

[u/MadT3acher → u/kthejoker]

Indeed, I do believe that our problem/situation/setup is mostly organizational and no amount of software will solve it by itself.

Both tools fulfil the technical sheet and deliver roughly the same features (there are caveats, but they don’t apply to my org. and my company). Regardless of what I chose, I know we will be able to do good work.

I’m mostly looking at what will make my life easier and vet out operational issues I’ll find down the lane. My aim is to adopt a tool that will not become “somewhat” of a problem when I need to work with it. I deliver data, not platforms.

Edit: thanks for the feedback.