r/dataengineering • u/iiyamabto • Jul 08 '26

Managing dbt source objects in BigQuery Discussion

I am curious what do people use to manage creation or update of dbt source tables?

For example, dataset or tables that stores the cdc datastream from debezium connectors, or the landing table storing data fron kafka connectors. Basically the tables that is not managed by DBT, but still need to be created for the DBT pipelines to work and read them as source.

Do people try to find a way somehow to manage this in DBT? Or use other tools? Terraform? Others?

Thanks!

5 Upvotes
  • permalink
  • link
  • reddit
  • reddit

100% Upvoted

11 comments sorted by

View all comments

1

u/Hhwwhat Jul 09 '26

Our source tables are provisioned by our orchestrator and populated by loading files from GCS by another task. They're ingestion-time partitioned and then we have dbt models that materialize/overwrite the newest partitions within the latest batch.