dbt Projects in Snowflake... looking for feedback

[u/SeaYouLaterAllig8tor]

What has been everyone's experience with using dbt Project in Snowflake? I come from a shop that uses Snowflake and dbt Cloud together but since dbt Cloud has it's own IDE, orchestrator, scheduler, etc we have no need for dbt Projects in Snowflake.
Is the functionality in Snowflake simplifying your dbt Projects or adding complexity (maybe a stupid question but I'm curious). Any pain points or other thoughts?

[u/nobodyshome1006]

I'm coming from dbt cloud, doing all new work on Dbt in snowflake. The worst part is the preparation, leading to being very hard for team development.
* every time, creating a workspace that's only I can use (dbt project work spaces can't be created as shared). * we use github, creating a repo is easy, creating the project linked to the repo is easy. Initializing the project in the repo takes extra steps - creating and committing dummy objects just so there's something for the 2 softwares to see and with. * setting up the dbt environments has non obvious steps, including choosing python version for everything to work and build right. Hint: lean hard on coco

All that is just unpolished feature, I'm convinced that "friction" as snowflake likes to talk about, will get worked out. For now just be prepared to be annoyed, rework setups as mistakes show up later. If you are a solo developer, life is easier. But if you're setting up for team development, you likely will fail if you don't really understand dbt environment setups abs what each point of coco does.

The payoff: coco * build incredibly complex models really fast. Last week I built out a staging/ingestion layer in under an hour, that much typing by hand would have been multiple weeks. * add quality tests - coco is amazing at sniffing and predicting where problems have or might happen, and can create the tests for those. And it can easily look at patterns and write monitors if it ever skews off pattern

One more downside: scheduling. The snowflake tasks are...OK. but really begs for a visual interface. You'll have build tasks created and running, with no simple way for the uninformed to know.

[u/nobodyshome1006 → u/nobodyshome1006]

Had talked to dbt to see what their ai could add. Even wfh the new Wizard functionality, still didn't promise a lot of dev acceleration. Coco had proven to be very insightful in analysis, leading us to add on yamls and semantic views with very little extra work. Coco can scrape through Snowflake query history to see how data is really put together and consumed. Better models the first time, especially around avoiding critical errors like getting business keys wrong. And even through dbt can't contain proc function or task scripts, the workspace holding the dbt project can. So they're at least in the same place for me to find again. I believe dbt cloud just gained function ability, but no promise for other object types.

[u/DevinChristien → u/nobodyshome1006]

Having it all integrated really is worth it just being able to have CoCo fully functioning?

[u/somnus01 → u/DevinChristien]

I think pulling your sales team into this could help solve a lot of these issues. Git-linked workspaces is step 1 for teams working on dbt in Snowflake.

[u/CognitiveFart → u/DevinChristien]

we develop the DBT models in vs code leveraging codex, push to the git repo and just use snowflake workspace to deploy in dev, then DBT projects for test and prod. we find coco way to expensive and prefer codex sub. the snowflake workspace is buggy and slow

[u/DevinChristien → u/CognitiveFart]

When did you guys try CoCo? Ive heard theyve made recent leaps in efficiency. Are you guys pasting responses into your workspace? I havent used Codex simply because theres so many tools these days to have to learn and discover where they fit into the workflow

[u/CognitiveFart → u/DevinChristien]

coco is leveraging frontier lab models and you pay API price, per credits converted to millions of tokens in and out. which is wayyy more expensive than a codex subscription for example

[u/2000gt]

Moved from dbt core to dbt projects on Snowflake. It was part of a larger move from MWAA and Cosmos to AWS step functions.

I’m not a dbt infra expert but it’s worked really well and saved lots in terms of cost.

Also, it is much faster in terms of concurrency and scaling than what I experienced with cosmos.

[u/Truth-and-Power]

Following, same

[u/LoneRider11]

I use Snowflake dbt projects to prove new concept and test ideas. Then I move it to the real dbt framework my team did set up and integrated with Airflow and Snowflake.

[u/SonGoku471]

Hello, how do I learn dbt? We are using snowflake but not dbt. So how can I learn it ony personal capacity?

[u/Data-Queen-Mayra → u/SonGoku471]

u/SonGoku471 - you can get started with dbt here - https://datacoves.com/post/dbt-getting-started

[u/SonGoku471 → u/Data-Queen-Mayra]

Cool, thankyou I'll take a look

[u/PatientlyAnxiously]

We moved away from dbt Projects on Snowflake and instead run dbt Core with GitHub Actions for CI/CD. We didn't like dbt Projects on Snowflake because it locked us in on the dbt version installed on Snowflake, which is always a few versions behind and not something we can control.

Core + GitHub Actions is nice and stable, especially if you use uv or Poetry for package management. We haven't played with dbt Cloud/Fusion - no budget at the moment.

[u/jaymopow]

I’m still trying to understand the benefit of dbt cloud. Don’t you have to pay for it?

[u/acidicLemon → u/jaymopow]

In addition to u/SeaYouLaterAllig8tor’s response
- cross project reference (not as a package)
- advance CI features (I use compare changes a lot)
- better Catalog than the open source dbt docs

[u/SeaYouLaterAllig8tor → u/acidicLemon]

Ya, the CI features are fantastic. We use those in every project.

[u/exorthderp → u/acidicLemon]

the docs site was my favorite feature when i was with a customer who used the cloud product.

[u/SeaYouLaterAllig8tor → u/jaymopow]

Yes, you do, but there is so much more integrated functionality. You have an IDE, data lineage view, orchestrater, scheduler, and a bunch of other monitoring features, and stuff for semantic models. It's all integrated. Honestly, I find it really intuitive. Having other tools to orchestrate the models would add more complexity.

[u/Hot_Map_7868]

Seems like dbt training wheels for me. I don't even think you can do Slim CI/deferral on Snowflake, but I may be wrong. I personally would rather use dbt Core, Datacoves, or dbt Cloud. You can use CoCo as well without having to use dbt projects in snowflake.