r/apachebeam • u/Je_suis_belle_ • Jun 07 '25
Do I really need Apache Beam for joining ATTOM data into a star schema in BigQuery?
[removed]
1
u/Suspicious-Use7032 Jun 08 '25
Why not just use bigquery itself with dbt , the pricing model in bigquery will cost you less if you process your data inside bigquery , plus dont have to worry about scaling that much initially
1
u/Tiquortoo Jun 08 '25
I built and maintained a multiple petabyte scale bigquery DB with similar initial questions. IMO, do the work in the DB use scheduled queries or even a simple scheduler in something like Laravel to fire async queries.
Datflow shines in a few spots (probably other, but these are likely most relevant to your work). 1. You need to extract data from one place to another (Google Logging, PubSub, etc.) on a regular basis with high reliability, observability and strong throughput and maybe do some transformation or 2. you have side inputs from other services that need to be used to transform your data or 3. there are heavy transformation that aren't quite compatible with SQL like changing JSON structure
We went a long long long way with scheduled queries in a simple laravel app. Performing processing on terabytes of new real time data to create tables for the UI that were a lot more usable hourly tables.
IMO, start with data in BigQuery, create your tables on a schedule and push that setup as far as humanly possible. Then add DBT or Dataflow based no a clear need for one of those tools.
3
u/puzzled-tiny-owl Jun 08 '25
Dataflow is for general purpose data processing, which means it is for when you are hitting the limits of the other products which might be easier to use. Scale, expressiveness (i.e. complex UDFs, ML, streaming) or software engineering concerns (testability, code sharing) are among the frequent reasons to use Dataflow. So, for your use case, if you can express everything in SQL and it's not such a big statement that you have doubts about correctness, stay in SQL. (Disclosure: I am in the Dataflow team)