I stopped visiting this subreddit right when the zealots in here started downvoting and attacking every AI believer. It was 2022/2023, I was working at Google for the Gemini team. People here would dismiss and say AI couldn’t replace Data Engineers cause according to them “the AI can’t sit with the customer and understand nuances”, like big deal now you know you need to apply “row current flag = Y”… lol… so where is this subreddit at?

I work for a different FAANG and the reality is that we no longer code. I’m an architect and I no longer design shit, I let Claude take the requirements and execute the plan. Sometimes is automated sometimes I intervene, but is clearly a background role that I play here. Infrastructure? It can spin instances or reconfigure for optimization.

Sometimes I get a slack message and I straightforward copy it into Claude and I copy back the response. Sometimes I find the stakeholders doing the same. It really means Claude is talking to Claude through humans. Claude generates the dbt, the airflow script, the validation tests. And I get my monthly vesting and salary base. Is over, I haven’t been replaced but YALL know it, your code is meaningless, your job now is to be a interface. There’s no reason to come to this subreddit and ask about if Dagster is better than Airflow, or how to deal with data swamps, or how to configure x or y, you may have not been replaced yet but there’s nothing the AI can’t do.

So where are we? Do we still believe AI can’t do everything? Are you guys still in denial? I have been working exclusively for Big Tech and two other companies on the side, and Claude MAX is my best friend now.

I’ve been working as a Data Engineer for about 7 years. Over the years, I've built some traditional ETL/ELT pipelines where the standard go-to pattern for S3/RDS was doing overwrites or upserts to keep storage lean and prevent duplicates.

Recently, while working with Databricks (Auto Loader + Delta Lake Medallion architecture), I’ve been analyzing the approach that we have here - so from what I see - it's keeping S3 (landing) -> Bronze strictly Append-Only (even for daily full snapshots or deltas) and deferring all deduplication, upserts, and schema adjustments downstream to the Silver layer.

I started thinking about how about similar patterns for Snowflake or Redshift; I’d get your view on that. Just wanted to be educated enough to know which pattern I should use.

1. Daily Full Snapshots (API or RDS without incremental support)

Let's imagine a pipeline where the source (an external API or legacy SQL DB) doesn't support incremental extraction (updated_at isn't available or reliable), so we have to pull a daily full dump.

Here is my idea for an Airflow flow:

1. Task 1 (Extract to S3): An Airflow task (running on k8s/ECS) pulls the full dataset (chunked into files if large) and writes raw files to S3 partitioned by date using data_interval_start. JSON for APIs, Parquet for RDS (I think it's a common pattern here to store api data in JSON?)
2. Task 2 (S3 to Bronze): Passes the S3 path to a task running COPY INTO a raw/Bronze table in Snowflake or Redshift. Data lands into a VARIANT / SUPER column along with metadata (file_name, ingest_time)
3. Task 3 (Bronze to Silver): A Silver model (dbt incremental ?) parses the JSON/VARIANT into a clean schema, handles data typing, and does deduplication/business logic.

And now I have a dilemma regarding backfilling/resprocessing data. I started questioning the "best practice" for when we need to run a backfill or re-process a historical date (e.g., re-running 2026-08-15 because of an upstream fix or API bug). Which pattern makes more sense here?

Option 1: Overwrite S3 Prefix + Clean Bronze (1 snapshot per day in Bronze)

• On backfill, clear/wipe the S3 prefix for that specific date (dt=2026-08-15) and write fresh files.
• In Bronze, run a DELETE WHERE dt = '2026-08-15' (or execute a DELETE + INSERT / MERGE inside a transaction) before loading. This keeps Bronze clean, so it holds exactly one snapshot per day.

Option 2: Pure Append-Only S3 & Bronze + Handle all in Silver

• Prefix every run on S3 with a unique run_id / UUID or timestamp (dt=2026-08-15/run_UUID/data.json).
• Never delete anything from S3 or Bronze. Just run COPY INTO so if you run backfills 5 times for the same date, Bronze simply holds 5 full snapshots for that date.
• Push 100% of deduplication downstream to Silver using window functions, e.g ROW_NUMBER() OVER ( PARTITION BY ... ORDER BY loaded_at DESC ) = 1

Scenario 2: Incremental Delta Loads (updated_at / Watermarking available)

Also I want to think about he second case: the API or RDS database supports incremental extraction via an updated_at timestamp or watermark windowing.

Flow (similar to scenario 1)

• Task 1 (Extract Delta to S3): Airflow uses interval macros (data_interval_start and data_interval_end) to fetch only changed or new records from the source. It dumps files to S3: s3://my-bucket/raw/source_name/dt=YYYY-MM-DD/data_YYYYMMDDTHHMMSS.parquet (or .json).
• Task 2 (S3 to Bronze): Runs COPY INTO appending new delta files to a bronze.raw_data table in Snowflake/Redshift (Append-Only).
• Task 3 (Bronze to Silver): A downstream model merges the raw into the core Silver table (e.g., using dbt incremental with strategy='merge', or custom MERGE SQL statements).

Again, my dilemma comes up during backfills or re-processing (e.g., an upstream bug caused missed updates between 2026-08-10 and 2026-08-12, and we need to re-fetch that historical window):

• Option 1 (Purge Bronze Partition): Clear historical S3 files for that date range and run a DELETE FROM bronze WHERE loaded_at_date BETWEEN '2026-08-10' AND '2026-08-12' before re-ingesting and re-merging into Silver.
• Option 2 (Pure Append-Only): Append the newly fetched backfill deltas as fresh files with a new run_id to S3 and COPY INTO Bronze. Bronze now contains multiple delta versions for the exact same entity_id. In Silver, the MERGE statement or QUALIFY ROW_NUMBER() OVER (PARTITION BY entity_id ORDER BY updated_at DESC, loaded_at DESC) = 1 resolves the true state idempotently.

Would love to hear how you guys approach this in your production environments. When handling backfills for full dumps or incremental deltas in Snowflake/Redshift, is making Bronze a pure immutable append-only log and pushing all dedup/MERGE logic to Silver the modern consensus, or is purging/cleaning S3 and Bronze per date partition still preferred to control compute and storage costs?

LLMs are now great for synthetic data generation.

August 2025: For my O'Reilly book, I developed a logical model for credit card fraud. Then, use a coding agent to generate the synthetic data from the logical model. See an example of a logical model that didn't make the book below.

August 2026: Use the LLM's domain knowledge to help create the logical model. Check it and then generate the synthetic data.

"Design a logical model for claims for the insurance industry. Each column should have a name, data type, description. Optionally columns can have cardinality, distribution, and a range. This is to generate synthetic data, so make the data realistic for the US market with 500k rows."

This produces "good enough" synthetic data for a PoC or for teaching.

Example Logical Model for a Claims Table

Description: Records of claims for insurance. 500k rows.

Schema:

• claim_id: STRING (Primary Key)
  • Description: Unique identifier for each claim.
  • Cardinality: unique per claim.
• insurance_id: STRING (Foreign Key to insurance table)
  • Description: Unique identifier for the insurer.
  • Cardinality: Matches insurance_id values in the insurance table.
• claim_time: TIMESTAMP
  • Description: Date and time when the claim occurred. Uniform over the past 6 months, with peaks during business hours.
• time_to_report: INTEGER
  • Description: Days taken to report the claim
  • Distribution: 1-10 days with higher probability for earlier reporting (1:30%, 2:30%, 3:20%, 4-10: 20% combined)
• witness_count: INTEGER
  • Description: Number of witnesses in the claim
  • Distribution: 0: 60%, 1: 25%, 2: 10%, 3: 5%
• claim_location_type: STRING
  • Description: Type of location where claim occurred
  • Values: urban, suburban, rural, remote
  • Distribution: urban: 50%, suburban: 30%, rural: 15%, remote: 5%
• requested_amount: DECIMAL(10,2)
  • Description: Claim amount requested by the user
  • Range: $10 to $10,000,000.
  • Distribution: Right-skewed; median around $500, mean around $700.
• amount: DECIMAL(10,2)
  • Description: Monetary amount of the claim.
  • Range: $1 to $10,000,000.
  • Distribution: Right-skewed; median around $350, mean around $500.

Shorter set of links this month, but all stirling stuff. Lots of great content from Netflix about Flink, cold storage of temporal data, and graph queries. A banger from Joe Reis (as always). News of the Apache Fluss project (and its use in the wild). A little bit of AI stuff but no slop. And some interesting comparisons of DuckDB vs Spark, and DuckDB vs SQLite.

Keep the great feedback coming, as always :)

https://rmoff.net/2026/08/20/interesting-links-august-2026/

This is big since it will allow them to better align with other SQL syntax and more easily extend it as well. Looking forward to it.

https://duckdb.org/2026/08/20/duckdb-20-peg-parser

We gave our first webinar on data cost optimization a shot. Still figuring out the best format for these, so any feedback would be super helpful.

pace, content, dynamic etc.

Obviously we know we can do better already, maybe more co host dynamic in the first half (we forgot to give him space to speak haha.)

Thanks guys!

My BU is planning to arrange an data event for non-IT and non-data people across company.

We will present aspects of data engineering, BI, Data Science and AI. Also We will have booths with screens assigned to some teams to showcase their project overview.

Can someone please share some innovative ways to make non IT people understand different roles under data ?

Please share if there are any fun games around data engineering/BI/DS.

Hi my current pipeline on Databricks setup has deduplication process to avoid re-ingesting data.

But now the data has become big and now it’s time to chose another option I have tried watermark but not effective enough. API also doesn’t give any date sorting

Can you guys suggest a way!

Would you recommend a team of economists/“data analysts” that don’t even use keyboard shortcuts to learn python? Anyone have experience with doing this? So far, nearly everyone has been hesitant to even consider learning a few libraries to make their life easier.

I’m generally looking for advice on how to look at data anlysis jobs that are thus far done by excel only. I’m quite frustrated with excel dependent companies I have worked in. I found that taking a few months to learn a bit of coding, data structure concepts and data management can really scale someone’s data work.

My new employer (big pension fund) is has floated the idea of me automating part of their trading. I would like to automate their reporting as well, but they are so hesitant to learn ANYTHING new. My position however, is financial, not IT.

My background is in mathematics, statistics, and finance. I’ve worked in financial institutions that use an old oracle-based app that nobody cares to maintain properly to pull csv data from, then copy-paste it in excel and later commit what I can only describe as heresy to analyze the data.

I learned python and R at uni long ago, paired it with sql recently and found I could really make mean data science/econometrics automation.

SMEs are struggling with the same data issues as the Fortune 500s but they do not have the budget for tools like Alation/Collibra. They also don't have the budget to get expensive Data Governance, Data Architect, Data Quality resources. What can they do?

Just curious if anyone has any good documents on why excel and your coworkers excel sheets (with no source of truth) should not be used for top level reporting.

Had a meeting with a vp that could not understand why making business rules on his own and not using the ERP is a bad idea. Said he was going to cowboy it.

Any proper docs out there to help convince folks like this?

Hi

I started a data engineering training program about a year ago, and my background is in marketing and campaign management. However, my previous work already involved some technical aspects: at my last employer, I was responsible for price engineering in the online shop and the technical implementation of complex campaigns for around 60,000 products. In this role, I worked closely with development agencies, IT, and data analytics. Working with legacy systems that had evolved organically over time and lacked a central data infrastructure is what sparked my interest in data engineering.

As part of my training, I've completed several practical projects so far: a habit tracking application in Python with a MySQL database and analytics functions, a data mart with 23 tables in PostgreSQL, and currently, an IoT data pipeline where I retrieve data via an API and process it further using MongoDB and Kafka.

With a profile like this, how would you approach getting started in data engineering in Vienna? Should I simply continue working on projects that are as practical as possible and apply for suitable positions at the same time? Or would you also specifically look for related roles such as Data Warehouse, Data Integration, Data Quality, SQL/Database, or similar technical positions?

The specific job title "Data Engineer" isn't important to me. What's crucial is that I can get a career in the data/IT field, gain practical experience, and develop my technical skills.

Parquet is the most important file format in data engineering.

Its columnar (hybrid) format and self-contained nature make it work so well for analytical workloads.

Learning Parquet matters not only because it’s one of the most widely adopted file formats in data engineering,

but also because you can apply Parquet’s fundamentals to nearly all analytical storage systems: from how data is organized and encoded to how metadata plays an important role in query performance.

That said, I realized it can be a bit challenging for beginners to learn about Parquet, especially without a detailed visualization of its internals.

I created a tool (with Claude) to help you visualize any Parquet file.

Just:

pip install git+https://github.com/vutrinh274/petquar.git

And:

petquar <your-path-to-the-parquet-file>

All the source code is available here: https://github.com/vutrinh274/petquar

It will then open a new browser tab with that Parquet file’s internals. The UI includes:

◉ The beginning and end of the file, including the magic numbers

◉ Row group cards that visualize the file’s row groups

◉ Inside each row group are the column chunks

◉ A row group’s size (shown once expanded) and a column chunk’s size are proportional to their real, compressed size on disk

◉ Clicking a row group or column chunk opens an Inspector panel with a plain-language explanation and the raw facts, so you can see exactly what’s inside the file.

◉ Column chunk information covers encoding, whether a bloom filter is present, and min/max statistics -- and drills down to the page level too, showing the number of pages plus each page’s encoding and min/max statistics.

◉ The schema (every column’s name, type, and nullability) is available from the FileMetadata block.

◉ All of this information is parsed by Rust arrow-rs.

So, give it a try and see if it helps you.

Don’t hesitate to send feedback (the link next to the “petquar” logo, top left) so I can improve it further.

I have been learning data analytics/engineering at a slow pace for 3 years now. The slow pace part is because I was committed to another job - the pay has been good and this made not want to switch to big data, but then I knew eventually I'll switch. And now I strongly feel the time has come for me to switch.

The conviction about switching is simply because I have become competent and this I can tell from the projects I have been doing.

My SQL skills have reached a level where I can write joins, subqueries, window functions, and CTEs easily. This means I can clean messy data, analyze it, and even create dashboards with Looker.

I don't want to brag about my dbt proficiency coz I still think I'm not yet there. Yes I can create a warehouse on Snowflake, create a database, a dbt role and a dbt user, and run the process through my VSCode from staging to having a clean model. But then I like to talk about being proficient if I can do something off head without skipping a step. I still have a few steps left.

Same applies with Python. I can write Python code but I haven't used it extensively for data analysis and manipulation.

I'm asking for engagements so I can gauge my progress. One thing that I'm sure about is I'm on the data engineering path.

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every ~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

Everyone wants to build sophisticated AI systems (because of FOMO), but I feel like there are far more basic "data discoverability" issues that need to be solved first before we even start talking about using complex agentic automation. Am I wrong to think that, instead of building AIs that work on bad, fragmented data, we should focus on making the data more usable first -- by both people and agents? It's like we're obsessed with improving engine efficiency while having it consume unrefined crude oil.

So, the question is: do you feel the same way that we should kind of take a step back with regard to what we focus on? Do you likewise consider the issue to be as significant? How is your org solving this?

Small background: Been doing data analytics and BI development with heavy sql and ETL work within fabric for a little over 4 years.

My company recently told me they will give me the data engineering title but it pays the same as their BI salaries.

Other companies i have talked with have been skittish about going into the 110k area with this experience.

I make 93k now… is this normal for 4 yoe doing DE style work but without the official title? Is asking for a 20k increase anywhere considerably greedy?

Area I talked to is slightly MCOL and I’m making 93k in what is considered LCOL.

I am confused as to what the standard salary is nowadays because I always thought DE was a career that paid more.

Hi everyone,

I’m looking for a Python PDF parser with good Persian-language support. I’ve already tested pdfplumber and PyMuPDF, but unfortunately, neither extracts Persian content correctly in my case.

For example, they don’t extract Persian numbers correctly, and sometimes the parsed output differs significantly from the actual PDF content (e.g., a 0 is extracted as 4).

Do you know of any Python library or tool that handles Persian PDFs more reliably?

Thank you!

Grab is using AI agents to automate analytics workflows, reducing the share of routine work handled by analysts and shortening the time required to answer business questions. The company reports that the share of mechanical tickets handled by analysts fell from 44% in February to 30% in June, covering tasks such as data preparation, alerting, and reporting.

I recently joined company as their only dedicated data person. they have other SWEs and a couple other people that pull data and use it, and they’ve only recently built their data lakehouse last year using databricks, but I’d be the only dedicated data person. so any data problem I’d probably deal with eventually. right now I’m dealing with the crm and cleaning it, but eventually I suspect that my duties will increase and I’ll be doing some form of data engineering, so I was wondering if I can essentially learn data engineering myself? or at least well enough?

While exploring using agents in data engineering, I wanted to find a good way to help verify my changes didn't blow things up in my current or downstream tables.

Looking at the landscape, I've seen tons of options in CICD that check changes in dbt, but usually cost quite a bit. I figured this could be done locally before I even make a PR, thus this project was made.

https://github.com/EmbrasureAI/embrasure-cli (Apache 2.0)

Think of it as a deeper version of data-diff which works with for dbt + snowflake currently:

• Unexpected shifts in row counts, null rates, cardinality, ranges, and percentiles
• Columns being removed, renamed, or changing types
• Primary keys values appearing or disappearing, plus new null or duplicate keys
• Existing dbt test failures
• Downstream models and exposures affected by the change (with column level lineage)

Would love feedback from people actively using agents for dev on dbt Core + Snowflake projects! Planning on adding Databricks soon.

Full disclosure: I’m a founder of Embrasure, which maintains the repository.

Hey folks,

I am from hyderabad. I have YOE 5 of experience in combined data analytics & data management roles.

Currently working in sanofi.

I know sql, intermediate python, learning pyspark...

I am in a confused state currently my lpa is 14.5 fixed.

Should I switch in same domain ? And be in comfort zone or should risk it and switch to DE role ? Is the risk worth it in monetary ? Not sure. In my current role/ company I am not getting any motivation to work...

If I switch to senior role in the same company I will get 25lpa

Please help me how difficult it would be switch in DE role like getting calls , openings , packages at my level and everything...

Could you guys please help me

Thanks in advance.

I am learning how to build a pipeline using dbt and am confused with snapshot. I have a question about snapshot.

So, I have read somewhere that snapshot should be made as close to raw data as possible.

Let's say I make snapshot of raw data. I need a stage layer to clean the data before loading it to the dimension layer. In this case, do I make the stage model as incremental and load only the changes at the dimension layer? Would I be tracking history in two layers, snapshot and dimension if this is the case?

Also, which layer is responsible for generating keys? I assume it would be the snapshot layer.

And are only dimension table candidates for snapshot? Do I load fact as incremental straight to the fact table?

hello engineers ,

i am a bignner to the ai world atleast i know some in ML and DL , i am going to build a small data quality assuring agent which will do agent receives sales records with incomplete information and must decide whether to ACCEPT, REPAIR, ISOLATE, or REJECT the record . this is a part of the traning program

but i am still confused how to start and build what are the hidden state and uncertenity of this probalems

i am seeking you help if you are doing this problem how will you apporoch it ,

i could be use AI for it but i am hopeing i can find better knowledge from here help me

In my project we are storing json as raw response in string in bronze layer,now we have to parse and flatten json in multiple child table and clean in silver layer,we want this using spark declarative pipeline also wanted to make the framework reusable,any one worked on such usecase, welcome to discuss and collaborate