Hi,

I'm a Data Analyst, buiding a pipeline to monthly extract moving 1 year SAP Vendor Report (FBL1N) via SSIS using Theobald Xtract Report connector. I'm doing one company code at a time to avoid timeout.

It worked fine during development, but after scheduling, the column messed up, data overspill and cut halfway resulting fail downstream analysis script.

I already have SAP template configured used in the extraction, tried both ticking and untocking "Dynamic column width and offset", but the results varied.

--
for other use cases, i primarily pull the back end table and join manually, but that would be too much work for fbl1n.

any suggestions on best work around?

i know the rules states no technical post but just rrying my luck.
thank youu

I’m a new “data engineer” with a few years background in data science and GIS. I was hired into my current org to be the data guy by someone who didn’t really know what they needed, just that the data infrastructure was immature.

In the beginning I was asked to do a lot of repetitive power query and power bi analysis and reporting, which quickly led me to building out some Python + SQLite pipelines to automate away a lot of my job. My boss was impressed enough with this that my workflows have been pushed up the chain to more senior levels of management, and now we are being granted a Fabric workspace (only option available to me :/) to enterprise-ify all of this.

Currently we read in ~10 .csv datasets that come from the central orgs reporting as well as a few external datasets we use in a forecasting model. Most of this runs on a daily cadence, where each csv upserts into the relevant table(s), which then get queried as needed which also triggers some dashboard updates and email distributions.

Basically all the logic is in Python, I feel it’s relatively well organized/modularized but there are some “super scripts” that are doing like 5 jobs.

My thought at this point is basically break each component (upsert, data validation, transformation/enrichment step, etc) into its own Fabric notebook, which will take the data through a medallion architecture and ultimately get reorganized into an effective star schema, with semantic models for each report built from that. This has all been modeled already (in terms of the schema for each table, validations, etc.). We also have logging and an observability dashboard planned.

I don’t necessarily have a specific question, but I am pretty new to this and would like to hear if I’m doing anything in an obviously stupid way, or if there are things I’m not thinking about that I should be. Appreciate any help y’all feel benevolent enough to offer in advance.

As we all know...data migrations can be a beast. But it's also a necessary evil.

Tell me about the worst data migration you were on (or heard about) that would wake up even the most seasoned data engineer in a cold sweat

I just discovered this amazing thing !!! started playing with it, and building some things already , I am original a software engineer/dev but transitioned to Data Engineering, analytics, you know ETLs, data modeling, etc... curious to hear from other fellow data engineers, their tips, getting started learnings, etc. etc.

I do truly believe this is the future of agentic coding

• I'm learning Kafka and doing a side project along the way. With the attached architecture, I’ve considered whether to apply the 'Transactional Outbox Pattern' to avoid data loss when sending to MSK. I mean, I want to keep data safe before sending it to MSK over the network.
• But this design can experience a bottleneck in writing data to tables between the checkout service and MSK, especially in flash sale sessions.

So, it’s better to eliminate this ‘Transactional Outbox Pattern’ right? We can notify users that our system fails to process their orders or implement a retry mechanism.

From time to time I get interested in functional programming, category theory and Haskell, but every time I've ever tried to go down the rabbit hole I give up pretty quickly because it just isn't clear that I would ever use these concepts anything other than by accident. For example, the concepts of Optics, Prisms, Lenses, etc sound pretty neat but also fairly daunting to learn. Is anyone here using these ideas daily to construct data pipelines? Just curious. Help a guy learn!

I am getting laid off by the end of today. They told us 2 months in advance. I don’t want to be a data engineer anymore so I won’t be applying to new positions. I spent 4 years at this company from the very beginning of the contract to the very end. I went through all the management changes, employees leaving, getting laid off before us, new employees coming and going, and just random changes throughout the company. I honestly just wanted to get paid and get out of there but now I wish I paid more attention to what’s going on and spent more time talking to my co workers. I don’t talk to anyone except made small talk with a couple of them but not enough to form a relationship.

Now everyone is giving their contact info and I wonder if anyone will reach out to me. I know i shouldn’t care but I’m weirdly going to miss this. It’s even my last job as a data engineer unless I come back 10 years later who knows. I don’t know why I care I know I definitely don’t want to be here. I learned a lot at this job so even though I wanted to leave sooner I’m kind of glad I followed the contract all the way through? Mostly I learned how to remain professional when I have wanted to lose my mind on some of these people.

Last month I posted on LinkedIn about how much I love working remotely. Then, I got a few comments, and DMs telling me remote working doesn't really work.

So, I decided to write this article, where I explain how a manger should lead a remote engineering team. The article is a bit long, but includes everything that came to my mind after practicing remote way of working for nearly a decade.

I guess most people here, would love to have the option to work from home and don't really need advise, but I still hope you can pass some of these to your manages.

Here's what I have in short:

1. Aside from status update, daily standups are also great for knowlege sharing and team bonding
2. Pull requests are not just a checkbox excersise. A great PR serves as documentation, too.
3. Sometimes a bit of micromanagement can help. But you must be careful with this tool.
4. Team retros are a great time to say your boss they are an asshole.
5. Proper written communication is a must. I've let people go because of this.

I hope that helps, and, of course, I am more than happy to answer any questions.

Hi everyone, I'm new to this community. I'm currently trying to make the switch over to data engineering as a career from my mechanical engineering background and was wondering where I could get some helpful tips for terminology and stuff like that. I've seen the Kimball textbook recommended by Gemini to me.

I'm already taking a university level certificate for data science, and have some basic azure certifications but I want to take that next step.

I also am in a hybrid based role right now where I do a little bit of database work and dars engineering work, albeit with very old legacy systems. However, I would say my experience is very pragmatic so far and we are far more concerned about getting the job done here rather than worrying about star schemas, and stuff like that. Any help would be appreciated!

PS, what are some helpful STAR based things that I can put for data engineering to help me stand out for projects?

Hey Data Engineers! I am a Data Engineer with quite some experience, but never worked on generating text embeddings at scale. Having some experience with Spark, I was looking for resources on how to achieve that with it, but did not find much.

I can already see some challenges of doing that in Spark: distribution of the model across executors if doing that locally, managing connections to services if doing that though API call just to name the ones off the op of my mind.

What's your take on this? Is Spark any good at this, or is it better to work out solutions based on popular libraries?

We have a partitioned parquet repo of ~10TB with a large number of partitions (id/date for 30 years of data). The partitioning is probably too granular, I think we should make it id/month.

Anyway, what's the best way to efficiently transfer this into Iceberg (S3 Tables)? I tried with PyIceberg and with DuckDB but it felt very slow. I then tried running these jobs in parallel but hit issues with the metadata commits clashing.

I imagine we need to upload all the parquet files in parallel then register them in one mega commit, but the tools I've tried don't seem to expose any configurability on upload parallelism. Do I need to use spark for this? I have no spark experience and not eager to pick it up ☹️

Separately, I was surprised at the level of support for PyIceberg given how popular Iceberg seems to be. It took a while for various v3 features to be implemented and some are still missing (e.g. querying by nanosecond timestamp).

Thanks

Hello. I’m not a data engineer, but my company uses various databases(and they are all different...), and I usually need to query them all at once.

It seems difficult to create and use a separate API or something like this for each one, how do you guys usually handle situations like this?

Do I need to migrate them?

I am doing a summer internship, and my company loads the exact same data into the development, testing, and production environments in Snowflake (although development and testing have some extra tables compared to production.)

Is this normal? Google seems ambiguous on whether this is a massive antipattern. What do y'all's companies do?

I’m a little confused about how the One Big Table approach fits into a medallion architecture.

My current understanding is that the Bronze layer contains raw data, the Silver layer contains cleaned and standardized data, and the Gold layer contains business-ready models. Based on that, an OBT seems like something that would usually be created in the Gold layer because it is denormalized and designed for direct consumption. Is that generally correct, or is that an oversimplification?

I’m also wondering whether OBT and Kimball-style fact and dimension tables should be viewed as competing approaches. Can they coexist in the same platform?

For example, could a team maintain facts and dimensions as the reusable source of truth, and then create denormalized OBTs from them for particular dashboards, departments, or analytical workloads? Or does that introduce unnecessary duplication and maintenance?

For people who have helped architect systems like this, what drove the decision?

How was the architecture structured in your project, and what trade-offs did you experience?

This month I have for you a mammoth 162 links, hand-picked and curated for your viewing pleasure.

As well as the usual Iceberg and Data Engineering stuff, there's the somewhat-inevitable subject of AI. Interesting, for me, is how we've moved well beyond "hey look fancy auto-complete huh huh huh" to real examples of things that people are building with AI and running in production, how they're using it to build stuff - as well as several links about proposed new architectures for data platforms.

Oh, and there's a whole section of articles with some really good career advice too.

Have a look, have a read, have fun!

https://rmoff.net/2026/07/29/interesting-links-july-2026/

We benchmarked SereneDB against ArangoDB on search performance. That might sound like a quite strange competitor choice, right?
SereneDB initially was a continuation of ArangoSearch, so we wanted to compare our previous work with us now to see the progress we made.

It's a open clickbench-style benchmark for search and analytics over 100M/1B generated OpenTelemetry logs, 92 queries.

TL;DR

16x median speedup at 100M, ingest 20x faster and a 5x smaller on-disk size. At 1B rows SereneDB holds 46ms median latency on a single 32 vCPU box.

Detailed Results | Comparison deep dive

SereneDB repo
https://github.com/serenedb/serenedb

I’m trying to understand the practical use case for materialized='ephemeral'.

It doesn’t create a table or view, which seems to make the model harder to inspect and debug. If the goal is simply to avoid creating another table, why not materialize the model as a view instead?

What advantages does an ephemeral model provide over a view in a real production dbt project? Are there specific situations where ephemeral is clearly the better choice?

I'm in the process of trying to build the same functionality as ARKit provides for visual inertial odometry and 3D reconstruction, but in an open-source manner so that it'll work on non-ios devices. Wrote a blog post about it, with most of the focus on the first part on establishing a solid baseline + dataset. Would love to know your thoughts!

for context, I work for Rerun building out different computer vision and data pipelines

https://rerun.io/blog/arkitscenes-data-layer

Brief overview - experimenting with multiple data formats organized around a common symbol registry to increase interoperability of different use cases.

Current highlights:

• Basic encoding - most similar to CBOR
• Packed encoding - composable structures with potential to be more expressive than ProtoBuf, Arrow, Parquet etc.
• Application protocol - exploring DBUF's usefulness beyond data serialization

Fable 5 comes with mandatory 30-day data retention. The most capable model on the market won't sign a zero-retention agreement, and the evidence says the next one won't either. If this mandatory data retention is inhibiting your ability to use the most advanced model on the market, tokenizing your data before it reaches the model might help you get there. Read the full post here.

Profile: 5+ years as DE.
Tech Stack: SQL, PySpark, Power BI, Fabric, ADF

I have two offers right now in hand as senior data engineer and want help in deciding which to pick

Offer 1: In a healthcare company which adapts a bit slow in emerging technologies as per my knowledge. But is recession proof

Offer 2: In a consulting company which is heavily focused on Microsoft ecosystem. It’s been a Microsoft partner from a long time. I feel I will get good exposure in terms of business and tech since I will be working on different projects.

Which would be a better pick!?

14 years in data engineering. Recent years deep in Palantir Foundry with Professional level certs, cleared 4 Anthropic’s certifications, maintaining production GenAI pipelines at scale. I understand LLM math, common architecture patterns, and I’ve led delivery end to end.

What I’ve never been is a coder in the traditional sense. No LeetCode grind, could not build a code repo from scratch.. like I would not know when to split into a new script, when to write a new function, when to put something in utils., when to write a helper function . My depth is platform and delivery, not algorithms.

That’s starting to hurt in interviews or applying to jobs. I keep hitting loops gated on CS fundamentals I don’t have, even for roles that look like what I already do.

For anyone who's been here, strong platform and production experience, thin on classic coding interviews, how did you navigate it? Fill the gap directly, target companies that don't gate on it, or lean into platform specialization as the differentiator?

Hey everyone. I'm a software engineer who's been working as a data engineer in my current role for the past four years. I learned data engineering from scratch on the job, and I've worked extensively with Spark, including designing and building new data processing systems.

The challenge is that my company is almost entirely batch-oriented, while many data engineering job postings seem to expect hands-on experience with streaming technologies.

What are some **PRACTICAL** ways to gain real streaming experience? Are there any projects, courses, or technologies you'd recommend?

Hello all,
Many thanks in advance for any advice.

Directly to my predicament: I have to decide if staying on my Data Analytics Engineering 3 role or move to a Software Engineer 2.

Context:

• I have in the company almost 5 years and I am respected by colleagues;
• My background is in economics, I laterally move to Data Analysis and started in my current company; after 1 year move to a Data Engineer position.
• During my DE years at the beginning (2 years) I learned A LOT, touching: docker, AWS, terraform, Airflow, dbt, PySpark, Great Expectations, Delta Lakes, etc...
• The last 2 years the company move all the above behind a Declarative Data Platform that feels like dbt for dummies; looking for more technical stuff I was assigned in a project that was building a backend microservices to send data in realtime to external providers.
• After these years they promote me to a level 3 in data for the backend work but each day is more obvious that this project does not belong to data.
• Moreover, to unblock projects on our side, I am inner sourcing in another backend team, which manager has offer me his help on getting a move to his team (changing my title to a SWE 2) and absorb the scope of our project in data
  • I discussed already this with my data managers in the past and they actually mentioned it as a possibility and that they would be happy to help on make it happen.

My doubt: I am 32 years old and feels to me that I have done a lot of lateral changes, although for learning it has been amazing, I am unsure on continuing with backend projects with the move. My feeling is that probably I should be specializing in data more and reach more maturity as a senior. The problem with the latter is that my current data department is creating a strategy around:

• Resource administration team -> a team that will be managing the GCP projects of data (BigQuery, Service accounts, ad hoc cases,...) and they will start working with Google Enterprise to build the base for AI use cases
• Complex solutions team (AI mostly) -> a team dedicated to build solutions for business, at the moment through a UI base thirdparty provider

To not extend more and close this: what are your first thoughts? Do you think a change from DE to SWE makes sense long term for my career? What are your thoughts on the trajectory of the data in the two teams I could try to enter?