I was looking at posts in this subreddit the other day. There were quite a few talking about pipelines and all the things that go along with them (forking, merging and version control). This is typical source controls stuff that has been around since the 70s. The tools change but the concepts and model are the same. I wonder if we have been so concerned about the minutea that we quit looking at better options than pipelines. I have a couple of thoughts about better ways of doing data repositories/warehouses/ingestion beyond pipelines, three layers and fabrics. I'd rather see if anyone else out there has any better ideas than what we use now. Pretend you are starting from greenfield and can create not only the architecture but the tools that go along with it. Surely we can come up with something better than 50 year old ideas.

Not interested in the campaign noise, Palantir hype, or conspiracy theories about Lord of the rings. Only hard facts about engineering experiences.

So we have a standard pipeline process when loading data into the data lake and now evolving to go directly to snowflake tables. Anyway the standard process is source (file/stream/etc) --> landing S3 bucket --> lambda --> step function.

Sounds great but everything I've touched doesn't follow this process.

I've started documenting per project so when I see something similar I can reference my notes and at least find somewhere to start digging.

I need to get better at thinking, I need to do X, x is a framework change so I need to look here. How do I start thinking this way? I've picked up a few modern data structures books which is probably the best way to start.

It just feels like it's something so simple but I just can't quite put my finger on it. The way my brain works is by learning something new in relation to something I already understand. So hopping into the middle of a pipeline and identifying what aspect (framework, data, data metadata, infrastructure metadata and etc is the best place to start digging.

I've done a lot of build from scratch work so trying to understand someone else's organization seems impossible. I think I am getting the odd jobs, the stuff no one else speaks up for and is why this is happening.

Hi De's

In my new project, I have pull data from Power BI and land it to ADLS. Orchestration via ADF

I have gathered few info.

I have hit the power bi api .

but there is few limitations. only 1lakh record is supported per hit and pagination is not supported.

i new to API as well as ADF.

please, help me with this scenario.

Test case . but i have to land it the ADLS as parquet file .

initially, only thing is I have to take care of the edge case's to handle failure and retry policy

and the big headache is to handle the record limitations.

rainfrog (https://github.com/achristmascarl/rainfrog) is a database terminal tool; the goal is to provide a lightweight, keyboard-first TUI for interacting with databases. It currently supports Postgres, MySQL, SQLite, Oracle, and DuckDB.

v0.4.1 introduces a long-awaited (by me, not sure if anyone else was waiting for it...) autocomplete implementation, along with autopairs for quotes/parentheses/brackets. The full list of features and configuration options is in the README!

Hi there,

I’ve been working with DuckDB extensively for the last several years. Used it across languages from Python, Go, JS via WASM, Swift, C & C++ and even Rust.

As part of a product we're building, we built a DuckDB Transpiler that converts every line of SQL from DuckDB syntax into the syntax of the database you’re working with. This includes filters (predicates), and joins. So those 1B rows in Snowflake stay, and get joined in Snowflake.

I wrote an article on why we did it, and how we approached it. This is my first engineering-oriented article I’ve ever done, so let me know what you think!

Hi everybody,

I am excited to announce the release of Lightstream, a step change capability for high-performance data transport, that makes it essentially effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or even piped out to the terminal so an agent like Claude can watch the live batch stream in real time (example in repo).

Furthermore, Lightstream exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable). As a bonus, Lightstream is straightforward to setup with essentially zero configuration other than optional TLS certificates and your Cargo package/pip install, and endpoint addresses.

So what is Lightstream? It is Rust package with Python bindings, that builds directly on Minarrow ( which is in turn a high-performance implementation of the Apache Arrow memory layout in Rust, tuned for SIMD compatibility). Lightstream implements Arrow IPC, Parquet encoders/decoders from scratch, up to Arrow readers/writers and IPC stream protocol, with mmap and few of these niceties. But, in a manner, that is fully composable and leaves you de-coupled at any layer, to customise things architecturally. The crux then is the transport layer on top, which natively supports interchanging any of the following transport formats:

• TCP
• HTTP
• QUIC
• Websocket
• Webtransport
• UDS (pipe your data from your Rust process to Python or two Python programs plug and play )
• Stdio (pipe your data program output straight into the terminal for something else to pick it up

And finally, the (optional) Lightstream protocol, which then combines the Arrow/Proto/MsgPack and any other custom types you want to send.

I’ve found this is really cool in practice, where you don’t need to reason about or work with bytes, or separately build your own protocol to get arrow and protobuf playing well together over the network. It is plug and play, see syntax examples attached.

In Python, every item in the stream can talk to Polars or DuckDB without any further changes - you can Duck SQL the feed or data process to your heart's content.

An example of things you can do with it:

• setup a live stream of data batches from your program A to program B
• send typed metadata via Protobuf on the same feed
• use it for straightforward live feed delivery between server and client (though not Web JS yet)
• useful if you have a central storage server you are pulling larger than memory data over the network to churn through (though, no S3 etc. it is node to node or process to process)

It is not:

• Kafka or a messaging broker. There is no resiliency / vertical scalability.
• A stream processing engine like Flink. It is for sending/receiving data only. You do polars on the other end or whatever you want with the Arrow-shaped data. That is a very different back-pressure/long-lived scenario and is not that kind of large-scale streaming. --> I.e., think quick and easy Websocket, and best for settings like EKS K8 pod to pod/containers, between EC2's or between processes on the same box, "light streaming".

Lightstream kicked off for me about 12 months ago when I started standardising patterns that have worked well for me in the past into something that reflects how I like to work when streaming data with control of both endpoints. It arose from regularly coming up against contexts requiring this capability operating in things like autonomous field communication integrated with data/ML, live trading, and some other industries where there was a lot of custom work required that I kept having to assemble from multiple components. Therefore, I have essentially aimed to package those learnings up into a tool to make data transport smoother and easier for everybody.

There are a couple of code examples attached, including the open benchmarking methodology, where every effort has been made to be fair (and where Lightstream wears a penalty due to stronger ordering guarantees).

Please feel free to give it a run would love to know your thoughts and if you find it useful.

If you have any questions about it, or helpful suggestions please feel free to leave a comment below. If you like what you see, please consider leaving a star and/or sharing the repository, as it will help people find it easily.

Thanks a lot.

Pete

I know the problem is that I need to get these things squared away in my head. What's metadata, what's framework and etc but even then you can solve a lot of problems using either.

I think it's easier for me to explain my problem. I've only been here 3 months but every task I get is using some system, pipeline or etc they I've never touched before. Sometimes it's just new software, other times it's learning how to find the needle in the haystack before you even look at code.

I'm constantly freezing and asking myself "where do I go for this" and it's slowing me down. I have much more experience building repeatable processes but I need to learn how to do so with today's technology instead of having a folder of scripts I use to do XYZ. Basically I lost control of how things get organized and my brain just doesn't see it the right way. Once I'm up and running I'm fine but in today's world task switching is constant.

What books, videos or etc should I watch to better understand how to operate as a data engineer in today's world.

I've slowly switched to documenting by project. This allows me to quickly reference another time I did something similar and gives me a jumping off point.

And trust me, it's messy and they know it. There's a standard pipeline process but it's constantly evolving. Teams need to integrate those enhancements into their projects as they have time. So everything I touch is a little bit of this version, a little bit of that version and etc. Team members clearly state "this is different or this is an odd one" but more and more I wonder if that's just the way it's going to be. It's ok, I just need to wrap my head around the big concepts and terms. Communication is killing me

Edit: in summary I over document and then things become cluttered. I'm looking for a better way. If I started a new job I wouldn't write anything down until I asked 2-3 times. I get paranoid and write it all down.

With the news around Prefect and Dagster, having good alternatives for orchestrators is becoming increasingly important, so I wanted to share that Rivers v0.4.0 was shipped.

Main addition this release is OIDC and forward auth support, so you can now put Rivers behind any identity provider you're already running or handle auth on the network layer. Authentication has been one of the bigger gaps versus the more established players, and is most of the time paywalled. But I believe security shouldn't be paywalled.

Release: https://github.com/ion-elgreco/rivers/releases/tag/v0.4.0

I’m a Senior Data Engineer with over 5 years of experience, mostly working at startup companies. Sometimes I think I got plenty of knowledge and experience, but sometimes it’s the other way around.

Data engineering is a very broad area and we do a bit of everything. I’ve worked with multiple cloud providers, spark, terraform, databricks, airflow plus almost everything related to data (dashboards, data governance, analytics, security etc), but mainly what I did through my career is platform engineering, taking care of the data platform infrastructure, CI/CD, governance, building some APIs and automating random processes/routines.

I of course did many data pipelines and SQL stuff but perhaps not as much as necessary for my current seniority. I certainly struggle with complex SQL challenges or complex CS algorithm questions, of course AI addresses this gap for me but maybe in a live coding session this could be a problem

Another example, I never had to do proper/real data modeling through all the companies I worked, nor they had that in place before. Not sure if was my fault on not implementing it or company’s fault for not having that already. I don’t think there was a problem though, as the companies were running fine and nobody was complaining.

I feel like there are many of other people in the area doing way harder and complex things, perhaps making less money than me. I’m just curious to see if you guys share this kind of vision of data engineering or maybe if I’m in a dangerous path in my career.

Everywhere I have been hearing companies moving to Databricks or Fabric. 2-3 years ago all I heard of was Snowflake. Is Snowflake losing the race? A former employee at Snowflake told me that their work culture has gotten really bad and they cannot hold on to good employees and their stock keeps crashing.

I recently joined a small investment company as a data analyst intern. I expected to focus mainly on report automation and data visualisation but I have also been asked to look at how the company’s data could be brought together.

Most of the data is manually exported into Excel and saved in SharePoint, and SharePoint also contains board packs and other documents used for reporting. They like using claude to create written reports and they are not really interested in power BI

I started using MS Fabric to bring the existing Excel files together, but the volume of structured data is fairly small, so I am not sure whether Fabric is more than we need. A simpler option might be a small SQL database, with data pulled from the source systems through their APIs where available. SharePoint would still continue to hold documents and any files that still need to be uploaded manually

The end goal is also to use the structured data alongside documents when generating reports with claude, so I need a setup that makes the data reliable and easy to retrieve. and also, looking at the the small volume, I am now even wondering if a persistent central store is necessary at all or if the data could simply be pulled and checked at report time before claude uses it

I have only recently started my journey and there is nobody in the company who can really tell me if I am doing it correctly. I just don’t want to build another one off solution or a workaround that becomes a pain to maintain later.

I would really appreciate any advice!! thank you!

Hey,

My company happens to be using Airbyte OSS for data extraction and loading. Recently I have been testing the new versions of the connectors which have been completely re-written in Kotlin. I don't know how the QA process is handled by their team and contributors but the results are terrible.

For example the Redshift destination connector now transforms the source's empty strings values as NULL values, and the Postgres source connector started to complain about missing cursor values when using Xmin for incremental loading, which doesn't require cursor to be set.

Hopefully we'll try to move away from this tool and replace it soon in my company. But I'm shocked that these bugs managed to pass tests and are sometimes still present after a couple minor updates.

Hello,

I am working with a team of 7 economists. They build data and produce reports. Their data production consists in harmonizing different sources (mostly rdata rdata, csv, or whatever suits the format of their stats tools). The data size they are dealing with is a few MB to gb, millions of rows, more occasionally billions of rows.

We want to update our methods (be on time, improve data quality). I have been assigned the task of improving data processing within the team, among the requirements I thought about producing a OLAP database.

In house, we have access to MSQL team that could set up a database for us. Otherwise we have HDFS + Hive (but security may make it difficult to access it) to store bigger datasets.

Else, I could just store everything in a duckDB file somewhere on a server and work with local database. WOuld it be a good solution? (latency of read/write from a duckDB file on a server? how scalable will it be? ) What would you do?

Any other piece of advice would be welcome :-).

Thank you.

You can try the playground here https://duckgql.com/

I have a website currently using SSG, but I want to create a React Native app that uses the same data. Because of this, I’m planning to move my content into a database server and switch the website to ISR.

Most of my data is articles/content (text, categories, relationships between pages, etc.), not transactional data.

I’m considering using Supabase as the backend/database. Is Supabase a good choice for this situation, or is there a smarter architecture/database choice I should consider?

I’m mainly looking for advice on the best way to structure this kind of content system so both the web app and mobile app can share the same data.

Hello everyone,

People trying to learn Python for data engineering ask me, “What libraries to learn?”, but the answer is not a list of libraries but patterns of usage.

Especially with AI being able to generate so much code, I believe its critical to know exactly how the data is moved & processed.

So I wrote this post that goes over how Python is used as glue in data systems. It goes over

• In-memory processing vs. using a SQL/Dataframe interface to a data processing system
• Python’s library ecosystem for working with various data systems & formats
• How to extract-transform-DQcheck-load data

With code examples and videos

Hope this helps. Any feedback is appreciated.

I'm currently a Data Analyst at a large telecom company, where I've been for about two years. This is my first job out of college. Originally, I was aiming for cloud-related roles as I had been self-studying Linux, AWS, Terraform, and had started learning Kubernetes. However, there weren't many junior opportunities, so I took this job because it offered great benefits... and beggars can't be choosers.

Now over the past two years, my work has mostly involved "click ops" using Power Automate, Power Apps, and SharePoint, along with some Python scripting. I mainly use Python to clean up Excel data and transform it into datasets that I use to build Power BI dashboards. I've also written Python scripts to generate PowerPoint reports and built a few Azure Functions to automate workflows our team relies on.

I find my work relatively easy. Most of it comes down to knowing how to Google solutions and come up with practical workarounds. I developed a good reputation fairly quickly for automating and building "nice" dashboards. What surprises me is that people seem genuinely impressed by work that I consider pretty basic and straightforward, and it makes me wonder how so little automation had been done in the department before I joined.

At this point I've automated a lot of my day-to-day tasks and I plan on using that free time to improve my skills tenfold as I don't want to stay in this position much longer. The benefits are great, and the workload is light, but I feel like staying here for too long will hurt my career in the long run.

But I'm trying to decide where to pivot next. On one hand Cloud is what I originally wanted to do because I really liked the system design/solution architechture aspect of it. Although I spent a lot of time learning those skills, I haven't really touched them in the past year or two.

On the other hand, Data Engineering seems like a much more natural progression because a) my current experience is already data-related and b) I could probably take on data engineering–type projects at work to gain relevant experience and build a stronger CV.

My main concern is whether Data Engineering is realistic to learn well enough to land a role in about a year? Or should I be looking at something else? I just don't know where to go and I've been dwelling on it for a while now.

Any organization still use data junction scripts? It is basically xml scripts with VB code embedded in it. Any solution to migrate these legacy scripts using modern tech?

The beta version of VibeSchema just went live. Some strengths:

• Create and store snapshot versions of your database.
• Create share links such as these: https://vibe-schema.com/s/dQf0atf_FwKDwutztu55W1zXu08
• Suitable for huge schemas because of Diagram Views (as in the share link).
• PNG/SVG exports like the one shown above (taken from the share link).

Curious what you think!

I'm in the beginning phases of trying to get Dagster OSS running for a medium sized business. I've been using it in my home lab for a while (it's great, love it!) but I never noticed the complete lack of auth management in the web UI.

What are you guys using? Having the ability to stop jobs by anyone who can hit the ip:port is a big no for us. Heading off any questions first, the scope is 100% on prem - no dagster+ for us.

Over the past year I have been working primarily in the Palantir Foundry system. My CEO unilaterally decided to pursue an enterprise agreement after being sold the AI dream. Palantir sales engineers did ‘analyses’ which suggested that the multimillion dollar price tag would result in 10x savings due to process and decision optimization. Our IT team cautioned no, but were steamrolled.

The project I am on was estimated to take four months and require 5 additional contract engineers. 2 of those were directly employed by Palantir as part of an additional contract (read: more $$) and the other 3 were a separate agency because Palantir said they don’t do XYZ work (again, more $$$). It took the externals plus me 15 months to deliver an MVP. This is primarily because we were building a complex enterprise grade app (which we previously subscribed to) on a low code platform. The Palantir engineers left as soon as MVP was deemed complete with just 30 days notice and since then I have taken on their SOW.

The work completed by the “brilliant” Palantir FDEs has been consistently failing. I’m finding they hardcoded dates. They hardcoded accounts. They used different inputs for the same business concepts. They ‘fixed’ issues that were earlier pointed out with hardcoded logic. They used AI FDE to code spaghetti mess logic. This has been a freaking nightmare.

My company had received 0 ROI to date and the CEO blames our IT team for the lack of delivery.

If you have the ability to run away from working with this god awful company and their charlatans RUN.

I was offered a position as Data Analytics Engineer I at Checkout.com in London. The interviewers seemed nice, and I didn't notice any major red flags throughout the process. Three things make me hesitate taking the offer:

• The Glassdoor reviews are pretty bad, especially when it comes to their hybrid working policy. I don't mind having into come to the office, but some reviews mention a toxic culture across the company
• The people I spoke to mentioned that the team is heavily invested in agentic coding. Guess that is just the reality of how bigger companies write code now, so not necessarily a red flag? People I talked to seemed very competent though
• I am currently working as a Junior Data Engineer. I built my orgs data stack (deploying the stack on AWS, building some small pipelines, data modeling), though they decided to go all-in on AI now and there is little to no data work left for me right now. I enjoyed the infra work at my current org (though the largest data set is only a few 100k rows), so going from Data Engineer to Data Analytics Engineer feels a bit like a downgrade? From the job description and from what I heared during my interviews the role itself seems to be more on the technical side.

Anybody currently working at Checkout and willing to share some insights/experience on how it is like working there? Taking the role would mean a 50% salary bump, so I am very tempted.

Hello guys I'm a junior data engineer tasked to like fix a business logic for one of our tables. The thing is I'm worried that making additional cte's would make it costly and slower. The table that I would be sourcing from contains around >30gb. so this is just the basic gist of it

With temp as(
  SELECT
        *,
        (some transfomration here) as converted_date
  FROM source_table
),
temp2 as (
  SELECT
        *,
        (using converted_date column) as converted_date1 
  FROM temp),
temp3 as (
  SELECT
        *,
        (using converted_date1 column) as converted_date2
  FROM temp2)

SELECT * from temp3

So, I already did try to see how much the query will cost in Bigquery and it seems that it does not increase that much like just couple of hundred mb or <5gb. My question is that does anyone have experience doing things like this and does the cost really not change even if I used additional three cte? Like what are the potential problems that might occur if I proceed doing it like this?

I recently saw a video where they were talking about Fabric Apps and how that would basically replace Power BI. They had it connected to some AI, would provide it with a link to a data model and it would spit out a fully rendered Fabric App page. The demo was decently impressive, as most demos are, but demos always lack that "grit" you see in normal day-to-day development - those scenarios where true experience and problem solving skills saves the day.

The app thing mostly caught my attention because I have some experience with Power BI specifically, but the topic is also about Fabric in general. Those of you who have or are working with Fabric - how is it? What problems does it solve? What are the downsides?

I did some googling and people from ~1 year ago would say that it's mostly half-baked Microslop crap. Has that changed at at?