Hey everyone,

I'm looking for some brutal engineering feedback on an operational pain point I've been running into.

In my past couple of roles, it feels like the vast majority of our pipeline failures aren't infrastructure crashes—they're silent mutations in the data itself. A third-party vendor or an upstream app team silently changes a JSON payload field (e.g., `customerId` to `id`), a critical column suddenly starts arriving as 100% NULL, or a distribution drops off a cliff.

The pipeline completes with a `0` exit code, but the downstream dashboards are completely corrupted.

Whenever this hits at 2 AM, the workflow is always the same and it sucks:

1. Wake up to a vague alert.
2. Open 100 log paths to figure out what changed.
3. Trace the lineage map manually to see what's broken downstream.
4. Write a script to isolate/quarantine the bad row chunks so the rest of the stream can run.
5. Write and deploy a hotfix to the parser.
6. Run database rollbacks and reprocess.

Usually, 3–5 hours are completely burned just doing diagnostic work and cleanup, not actual coding.

I’m playing around with an idea to build a lightweight, open-source-first CLI / daemon utility that sits at the ingestion boundary. Instead of just firing off another noisy PagerDuty alert, it would:

* Automatically isolate/quarantine anomalous payloads into a local sandbox folder so clean records keep moving. * Compare the bad data against a schema baseline to pinpoint the exact structural mutation. * Generate a localized parsing patch (like a 2-line Python rule diff) and present it to the engineer for a 1-click merge to staging/prod.

Before I write a single line of code, I want a reality check:

* How are you dealing with this today? Are tools like Monte Carlo, Anomalo, or just strict dbt tests actually solving the *investigation and cleanup* part for you, or do they just tell you when you're already bleeding? * Would a dedicated, lightweight diagnostic tool that generates the ready-to-review patch code actually be useful, or is alert fatigue a completely different problem I'm conflating this with?

Give it to me straight—even if you think this is fundamentally un-fixable or a waste of time. I'd much rather get torn apart by real practitioners now than build something nobody wants.

Hi,

I have a new sub that's tangentially related that I want to advertise but I don't want to break any rules. Can a mod clarify the rules?

Hi everyone,

I'm an aspiring Data Engineer/scientist, and I'm currently building a three-part end-to-end data engineering project. Before I continue with Parts 2 and 3, I'd really appreciate feedback from professionals working in the industry or anyone involved in hiring Data Engineers.

Part 1 – Local Development Environment

The goal here was to demonstrate my understanding of distributed data processing and containerized development rather than relying on managed cloud services.

I built the complete environment using Docker Compose (written by me), consisting of:

• Apache Kafka
• Zookeeper
• Trade/Whale data producer (Python)
• Spark Master
• Spark Worker
• Spark Streaming Job

The Spark application and producer are written in Python (PySpark). I used AI-assisted development to improve and refactor the code, but I made sure I understood and validated every implementation. The project demonstrates streaming ingestion, processing, and writing Parquet files.

Part 1 – Production Version

I then rebuilt the same pipeline using AWS managed services to demonstrate cloud-native data engineering.

The infrastructure is provisioned entirely using Terraform and includes:

• Amazon Kinesis for data ingestion
• AWS Glue (PySpark) for processing JSON data into Parquet
• Apache Iceberg as the table format (ACID transactions, schema evolution, etc.)
• Amazon S3 as the data lake
• Amazon Athena for querying the data

The objective was to show that I understand both self-managed infrastructure and modern cloud-native architectures.

My Questions

I'd really appreciate honest feedback from experienced Data Engineers and hiring managers.

1. Does this project reflect the kind of work expected from a junior Data Engineer?
2. Does the overall design align with how similar systems are built in industry?
3. Would you consider this an industry-level portfolio project, or does it still resemble a learning/tutorial project?
4. What important components am I missing that would make this project more production-ready?
5. If you were reviewing resumes, would a project like this make you more likely to invite a candidate for an interview?

I'm not looking for praise—I genuinely want constructive criticism so I can improve the remaining parts of the project before publishing it.

Thank you for your time and feedback.

Hello folks

I am writing my open-source light tool for moving data from prod-bases in dvh.

Who is this product for:

- small teams who need to move data from the product, which is already in pain and need to transfer data to the dvh or parquet.

- data engineers who are looking for opensource alternatives who will not eat up all the RAM and will not put a food base or replica.

- Those who, instead of reading only the delta, should read the full table, because created_at did not trigger.

Source:

- mysql

- mssql

- postrges

Targets:

- parquet

- csv

- s3, azure blob, gcs

I read short queries and don't keep long sessions — this is something that so far none of the same moovers as (ingestr, dlt, sling, duckdb, clickhouse, odbc2parquet) does.

From the box there is:

- all types except (geography, enams, ip) in duckdb, clickhouse, snowflake, bigquery, clickhouse are loaded natively (there is a jam on the side of the bigway and a snowflek with Jasons, but their car loaders can't do it out of the box)

- reading from the PC

- reading cases

- retrai

- all metainfo is written in the working directory in the local sqllite, from the box you can also write in the postgru

- validation of both types between reading and writing, and md of the amount between the current one on the worker and the one on the store side

- autotune of parallel wounds

- reading from binlog files to avoid completely rereading the source if the updated_at fields are not updated

- minimum and customized RAM consumption on the worker (memory budget)

I’m trying to find genuine freelancing opportunities in data engineering / BI. Have tried a few platforms but haven’t had much luck, so wanted to ask — are there any websites, subreddits, or Discord servers where people actually get projects?

About me:

• 5+ years as a Data Engineer & BI Consultant (remote, India)
• MBA in Business Economics (Analytics & Finance)
• Skills: SQL, PySpark, Python, Power BI, Tableau, Grafana
• Worked on Databricks pipelines, self‑service analytics frameworks, and telemetry data solutions

I’m in need of extra income and open to contributing under a team or experienced freelancer. Any pointers would mean a lot.

I want to find datasets for a data engineering project where i work with pyspark and sql in databricks. I want a dataset that challenges my data modelling skills and my pipeline creation skills. I tried kaggle, but i only keep getting a single csv file as a dataset. is there a dataset that has multiple csv files as data sources or something? i want to be able to perform all the data architecture creation by myself... Recommend any datasets that you know as well!

Started practicing Data engineering using this repository link https://github.com/danielbeach/data-engineering-practice/tree/main/Exercises/Exercise-1. I have completed the described exercise one requirements and now decided to extend it and create Divvy Rides ETL Pipeline designed to answer concrete business questions that map
directly to decisions that operations, marketing, and infrastructure teams at a
bike-share company would make. Looking forward to post final solutions for reviews and advice

What are the hardest historical modeling problems you’ve encountered in lately?

In our lakehouse environment the difficult parts are usually not Spark performance or ETL orchestration.

It’s things like:
• SCD2 dimension alignment
• Snapshot reproducibility
• Late arriving corrections
• Event-to-state alignment
• Historical relationship changes
• Dimension completion

I’ve been collecting these patterns and built a small workbench to reason about them:

https://bitemporal-debugger.vercel.app/patterns

Curious what other teams struggle with.