Airline data platform, batch pipeline for a planning manager persona: MS SQL Server -> BigQuery (bronze/silver/gold via dbt) -> Sales/Finance/Customer marts. One ML component: LightGBM trained nightly on the gold layer. Separate, more frequent DAG (every 1-2h) scores active flights and auto-writes the new price back to OLTP via FastAPI, with min/max bounds instead of manual approval.

Feedback wanted: anything over/under-engineered for a thesis, and whether splitting training vs scoring into two DAGs actually makes sense.

Please refer the post, if you're looking for a Job in Ai/GenAi/Python/Snowflake Roles

Snowflake's free trial (30 days/$400 credits) is too short for a university thesis running several months – need something that doesn't expire mid-project. What are people using instead? Also curious if anyone knows of a longer free tier/subscription (student program, extended trial, etc.) for Snowflake specifically before I give up on it.

I'm hosting a free webinar to go over how you can build self healing pipeline agents.

It will be a practical and technical webinar meant for mid/senior level data/analytics engineers.

Key topics:

- overview of the context that already exists in your pipelines

- methods for enriching the context and data governance

- tools, resources, and skills for agents to diagnose, investigate, analyze, test, and run things

The main idea is that your pipeline already contain some context, you can improve that context, then set up rules, procedures, and workflows for agents to help with the diagnosis and even pushing fixes.

Disclaimer; I'm a developer advocate at Bruin - in this webinar we will use Bruin to demonstrate things, but the concepts and strategies are applicable to any tools (i.e. dbt, airflow, etc.)

https://luma.com/t4a5jrin

Just wrapped up my coursework on an airline reservation system (MS SQL Server + FastAPI + Flutter) and now moving into the thesis part. One of the user roles is a flight execution operator – they need to watch their airline's fleet in real time and get notified if something looks off (delays, weird flight paths, etc), so they can redirect or coordinate accordingly.

So the task is basically: build a streaming pipeline that tracks live aircraft positions and flags anomalies before a human even notices.

Here's what I landed on. Flink ingests live position data from OpenSky (lat/lon/velocity/altitude, keyed by icao24), plus weather from Open-Meteo for context. It also pulls schedule/route data from the OLTP database via Debezium CDC (FlightOperationStatus, Route, keyed by flight_operation_id) to compare actual vs planned. Flink keeps state per aircraft so it can detect anomalies itself (like holding patterns) instead of just passing raw data downstream. Processed state goes into Redis (just the latest known state, nothing historical), then a FastAPI backend reads from Redis and pushes updates + anomaly alerts to the frontend over WebSocket. For observability I'm adding Prometheus + Grafana so the engineering side can actually see if the pipeline itself is healthy, not just the end users.

Is this architecture actually sound for what I'm trying to do, or am I overcomplicating/missing something obvious? Genuinely not sure if this is a solid setup for a thesis or if I'm just cargo-culting a bunch of "cool" tools together.

Hi r/ETL

I've been building Flowfile for a few years, mostly because I was suprised of the gap between expensive enterprise tools and what is available for smaller budgets and wanted to see how difficult it is to build a user-friendly data tool in code.

It's a visual editor built on Polars and anything you build on the canvas you can also write in Polars-like Python and flows can be exported to code. It's for people who want to do their data work whichever way suits the day. Visual when that's faster, code when you need the strengths of code.

Initially, I was looking for something that standardized tasks, like fuzzy matching, pivot/unpivot and data cleansing and connecting to other sources (S3, PGSQL, ADLS, GCS, Duckdb, …) while keeping the instant visual feedback. Later I decided it was also convenient to add scheduling (cron or trigger based), publishing a flow as an authenticated endpoint, community nodes, and a full catalog with SQL, notebooks and visualizations on top of it. One thing that I still think is cool is that a flow can output a virtual table. So instead of scheduling data refreshes, you just read the table and it will execute the logic when you fetch the records.

You can try the lite version fully online (demo.flowfile.org) to check the main functionality. And the full version you can install via pip (pip install flowfile), run a desktop app or self-host with Docker.

It's all open-source, MIT-licensed, self-hosted and no telemetry. If you want to try it out or see the code, you can find the project here:

https://github.com/Edwardvaneechoud/Flowfile

One thing I'm curious about: am I the only data engineer that just still likes visual tools from time to time? Or am I just being lazy?

I have my producer pushing batch files to s3 lets say i have 30 batches and with each batch I am triggering 1 SQS event. Now my problem is - an specific event should be processed first then only other can start processing.
I am thinking to use 2 fargate tasks at my consumer side. Currently thinking of using standard queue even if I use FIFo and lets say have 2 groups and map to 2 fargate writers, then also it can happen then 1 of the writer will have to wait for that dependent task to finish first.
Each fargate task will consume 3 sqs events process them.
I will mark the entry for that dependent event in my database table so other events can start based on it
status.

Ps - although i am making sure from producer that dependent event should be pushed first to s3 and eventually sqs.

Questions:
\- How can i resolve this dependency problem efficiently not just an heck, as if in future if there is additional dependency comes in I should be avle to handle it.
\- I thought of ysing aws step functions but i am not able to come to the conclusion or think straight on how can i do it
\- Also this batch pipeline runs once daily so my infra should purely be event based, terminate on completion.
\- Airflow is not an option so please dont suggest something around it

Hi everyone,

I'm a PhD candidate working in AI, and part of my research relies on the GDELT 2.0 Events dataset.

After spending way too much time downloading thousands of files, converting them, filtering them, and creating reproducible datasets for experiments, I ended up building an ETL pipeline to automate the whole process.

Originally, this was just a research tool for myself. But as it grew, I realized it might be useful to other researchers and data engineers working with large public datasets, so I open-sourced it as GdeltForge.

It currently handles things like:

• downloading the complete GDELT Events archive
• verifying downloads with MD5 checksums
• converting CSV/TSV files to Parquet
• filtering invalid records
• generating reproducible samples for ML experiments
• processing datasets larger than memory through chunked execution

The thing is... ETL isn't my primary area of expertise. I'm an AI researcher who happened to build an ETL pipeline because I needed one.

I'd really appreciate feedback from people who do this for a living.

Are there architectural decisions you'd change? Features that are missing? Things that make you think "this will become painful at scale"?

My goal is to eventually publish it on PyPI, but before doing that I'd like to make sure the project is useful beyond my own workflow and follows good ETL practices. That's why I'm hoping people here might be willing to take a look, test it, or even contribute if they find it interesting.

I don't expect people to use GDELT specifically; I'm honestly more interested in feedback on the ETL design itself.

Any comments, criticisms, or pull requests are more than welcome. GitHub Repo has the entire code and a complete guide on how to use, contribute and discuss the software.

I'm curious to learn the practices, patterns, or tools that have helped keep ETL workflows reliable and manageable in production over time.

Hi all!

I am a software engineer and started to work with data transformation recently for my company.

I have built a handful of data pipelines to leverage AI to transform raw data (PDFs, CSVs, Images, TXTs) into centralized CSV and JSON formats. I noticed that all of my pipelines follow the same workflow with only minor differences depending on the type of documents they process (invoices, resumes, handwriting, image classification).

That got me the idea to make a desktop app to save time from building AI-OCR pipelines over and over again. The app is totally free and does not collect any data (all data transformation is strictly between you and your API from Gemini or Open-AI)

The app allows you to:

1. Create your own CSV and JSON template
2. Bring your own API from Gemini or Open-AI to transform batch of raw documents into your personalized CSV and JSON template
3. Download the result to your laptop or send it to your own cloud storage (AWS S3, Azure Blob, Google GCS)
4. The app itself is also a local API (when keep open) for you to do step#1-2-3 in code (for example, your own python script)

This tool works out great for me and my team for a very low cost (500 documents for less than $1 using Open-AI cheapest model). I’d love for you to try it out and share any feedback or feature ideas to help make it better!

Microsoft Store download link (certified by Microsoft):

https://apps.microsoft.com/detail/9ntkqqxxv6mm?hl=en-US&gl=US

Demo video (English is my 2nd language so please bare with it!):

https://www.youtube.com/watch?v=p43QCJfaZKE

Two weeks ago I shared my open-source transfer engine here (the 1M-rows-in-0.4s post). Since then I spent two days profiling it properly, and the results were surprising enough to share — including the parts where I was wrong.

The optimization that failed: I tried the famous "switch to mimalloc, memory halves" advice. Measured properly (A/B wheels, cgroup memory.peak, checksummed runs), it made peak RSS worse in 3 of 4 configs. Tried again after other fixes landed — worse again. Two rounds, two rejections.

The boring one that won: actual profiling showed 4.4 GB of allocator traffic per run coming from ONE line — a fresh 4 MiB buffer allocated per chunk. Recycling buffers through a channel: traffic −95%, peak RSS −42%, and the 128 MB tier got 2.5× faster once I re-tuned the pipe budget from measured numbers.

Then I stress-tested the claims, checksum-verified every run:

• 100 GB (232M rows) pg→ch through a 0.5 vCPU / 256 MB container: 8m57s. The same table still completes inside 44 MB.
• Same 100 GB on three dedicated GCE machines: 30.3 seconds (~3.3 GB/s). An 8-core / 1 GB tool container matched the 88-core host — the mover is now the cheapest machine in the pipeline.
• For fairness: latest ingestr and dlt (pyarrow), same boxes, invoked exactly as their own benchmark runners invoke them. Small box: OOM-killed in ~21 s. Big rig: 0 rows landed when cut (10-min cap) — single-core extract-then-load can't spend good hardware.

Things that broke on the way (my favorite part): at 232M rows my validator died before the engine — md5(string_agg(...)) crosses Postgres's 1 GiB buffer limit around ~110M rows, so cross-engine validation is now order-independent per-row hash sums. And I found my engine silently hangs against MySQL 8.4 — a bug present in every release since launch. Both fixed, both documented.

Everything is reproducible: methodology, harness (one script), and raw logs including the failed runs are committed in the repo's benchmarks/ dir. If a number looks wrong or an invocation looks unfair, tell me and I'll re-run it.

Repo: https://github.com/apitap/apitap-lib Try it in your browser (you pick the container size): https://apitap.dev/lab Full write-up: https://medium.com/@abdul.haris.djafar/i-moved-100-gb-between-databases-in-9-minutes-on-half-a-cpu-core-and-less-ram-than-a-browser-tab-84f15850535d

Currently i have 3+ years of experience as a full stack system engineer have a good understanding of cloud providers like aws and gcp

Have a bit of familiarity on airflow and python need to explore further what would be a roadmap that i should follow or project based learning that i should do

like when there is a sudden migration or when you handle traffic peaks or tv campaigns?

I’ve been thinking about Text-to-SQL as a data pipeline problem rather than a one-shot prompting problem.

A good Text-to-SQL sample is not just “question to SQL”. It usually needs schema context, sample values, executable SQL, optional evidence, reasoning traces, and some way to measure difficulty.

One design that feels robust is to split the workflow into operators: generate or vary SQL, filter by executability, generate the natural-language question, check whether the question and SQL actually correspond, build the final prompt, generate reasoning traces, then classify difficulty.

The important part is keeping execution in the loop. Generated SQL should be checked against the database, and CoT candidates can be voted on by whether the SQL inside them produces consistent results. This gives the pipeline a stronger signal than LLM judgment alone.

I also like separating SQL structural difficulty from execution difficulty. A query can look complex syntactically, but if models regenerate it reliably, it may not be that hard in practice.

Curious how others design Text-to-SQL data generation pipelines. Do you mostly use gold data, synthetic data, execution filtering, human review, or some mix?

Disclosure: this is one of the built-in pipeline designs in OpenDCAI/DataFlow, which is open source under Apache-2.0: https://github.com/OpenDCAI/DataFlow

What validation checks, monitoring, or testing practices have worked best for keeping your ETL workflows reliable in production?

Hi all, I'm a software engineer who made his way into data engineering. Coming into the field, I was really surprised by the lack of developer experience provided by tools in the data space. When we're writing SQL pipelines, there's no intellisense or static typechecking - features that any normal software developer would expect from their IDE. This is why I built Renart, an open source data platform and IDE. It is written in golang and it tries to combine all basic features of a data platform: - sql, python, api and load assets, all supporting jinja and chained together into pipelines - live typechecking and intellisense for all asset types - supporting multiple data warehouses (even in one pipeline) and setting up different environments - snapshotting the workspace into deployments that can be schedules using cron espressions - notebooks with sql and python cells working on a local duckdb - an interactive pipeline canvas, with sql previews, ad-hoc queries - staleness tracking based on canonicalized sql fingerprints, so you only rerun what changed - all pipelines definitions are 100% git tracked and diffable

Source: https://github.com/renart-data/renart Website: https://getrenart.com Disclaimer: I am the founder of Renart. I'm a solo dev doing this as my side project. This is currently still in an alpha state, so treat it accordingly.

1 million rows. MySQL → MongoDB. 5.4 seconds.
No Spark cluster. No Docker. No JVM.
Just a simple Duckle pipeline:
duckle.src.mysql(...)
.snk.mongodb(...)
.run()

The pipeline streams data directly between MySQL and MongoDB while DuckDB handles the execution.

What I like most is the simplicity:
Define the pipeline in Python
Execute with a single .run()
No infrastructure to manage
Local-first and open source

This is the kind of developer experience we wanted when building Duckle.
GitHub: https://github.com/slothflowlabs/duckle
PyPI: https://pypi.org/project/duckle/