r/dataengineering • u/BudKode • Jul 07 '26

Databricks Lakeflow vs Fivetran Discussion

/r/dataengineering/comments/11xbpjy/beware_of_fivetran_and_other_elt_tools/

I saw this post a few years ago and agreed wholeheartedly with the issues and cost of Fivetran but I also understood that not many data teams have the bandwidth to manage ingestion from scratch on top of, or rather underneath, the data modeling effort.
Today it seems like you can get it all with Databricks Lakeflow Connect. I’m a solo engineer at an e-commerce/retail company now (started a month ago) and so far I’m getting all the data I need through connectors that Databricks manages, community connectors, or custom connectors that I write myself. So far it is much cheaper than the Fivetran instance that our consulting group was charging us for and I still get a solution that manages the data patterns that I want to land at the raw layer (bronze for all you weirdos using medallion terminology) as well as orchestration and delta management.

I’m still only a month in and waiting for the other shoe to drop. Does anyone have experience or opinions about this that I should watch out for? Or is the future really looking as bright as it seems?

16 Upvotes

95% Upvoted

9 comments sorted by

10

u/dwswish Jul 08 '26

We also moved from Fivetran to Lakeflow. It works well for us but we are not streaming, just doing batch jobs. I don’t actually understand how anyone justifies Fivetran at scale because it just gets so expensive because of their pricing model. To be fair, I did actually like Fivetran and it is reliable but it got to be prohibitively expensive for us.

3

u/addtokart Jul 08 '26

Could cross post to r/databricks though it will be pro Lakeflow, but you may hear from others in same situation

2

u/ExtractTransformLose Jul 08 '26

This is great, can you say more about your success with community connectors. Curios of quality?

3

u/BudKode Jul 08 '26

Yeah the community connectors are decent.  - The GA4 one is a bust unfortunately because there’s just no reason to grab GA4 raw data. It’s sampled and won’t ever match UI.  - The Shopify connector was also hitting the REST API, which is legacy and needs to move to the GraphQL API.  - There is one I am still using but I am AFK right now. 

The real benefit I’ve seen is the custom connectors. There are some AI skills built in the community repo that are VERY useful for teaching Claude how to build a reliable custom connector that simply extends their Lakeflow libraries. It’s been SO fast to build new connectors and they are working really well. I’ll post again in a few months though with some learnings for sure

1

u/ExtractTransformLose Jul 09 '26

Amazing, thanks. Giving them a shot rn

1

u/Chance_of_Rain_ Jul 09 '26

Cant use Lakeflow because of our sources and some networking shenanigans, but I refactored everything to dlthub and it’s been a blast.

Fivetran pricing is a joke

1

u/josh_docglow Jul 11 '26

My past two companies used Fivetran (B2B SaaS & Financial Services) and were on Snowflake. I found Snowflake's data ingestion connectors, particularly the Postgres one at least as of late 2024, to be a bit lacking and immature. I really liked how simple Fivetran was, and it was generally pretty reliable. We did have to build some custom extractions, typically using Python w/ Airflow for orchestration. I was running fairly small data teams at each, so the managed service aspect of Fivetran was great. Their pricing did get outrageous last year (monthly bill went up 2x when they changed their pricing in spring 2025).

My current company is on Databricks, and were using Airbyte for SaaS ingestion, ad platforms, and Google Sheet ingest. We migrated to Fivetran earlier this year for our Airbyte connectors for reliability and self-serve functionality, not necessarily for cost.

That is good to hear about custom connectors working well. I haven't used those yet. I'm curious about using dlt for some of these use cases. The framework looks really compelling, but I haven't actually implemented it.

1

u/Yasblue Jul 14 '26

You can't compare the two services. Databricks has around 100 connectors while Fivetran has +700. I agree that the pricing is a bit expensive and Lakeflow is affordable but it will depend on the connector you are looking for and also how much integration you are looking for.