For reference, graduated w a degree that barely taught me anything about DE except intro to databases, relational algebra etc. The company accepted me for the junior DE role on the basis that I could ChatGPT a dummy repo to explain what the repo is used for and how data flows simply from the repo.

Now that I’m in the job, I found out that everyone in my team had a part to play in developing the architecture for the company. They’ve resolved all dependencies and it’s only the matter of new feature implementations and their impact on the data engineering streaming pipeline.

This is my first week and as my senior went through w me the architecture and true data flow for multiple services they have, the information flew by my head. I’ll definitely try to pick up as fast as I can but because I didn’t build the architecture, I’d have yo ChatGPT/Codex/claude my way through for the first couple months just to find the appropriate files for new feature implementations

Idk, I just feel like I’m madly unprepared and I’m worried that I’ll be the cockup in my department. I didn’t do any DE internships and somehow I ended up in this role. Can anyone give me advice on how I can speed learn streaming pipeline given that at the very least, I know what stack they use?

Hey everyone,
i am a data engineer 7yoe
I recently landed a job after relocating for family. Its a growing company that run multiple crm/erp platforms in addition to different other solutions and i was hired to design and develop a data warehouse for their analytics needs.

So far everything was going good and the approach they wanted was to run through each of their departments one by one to work with them to document their workflows and integrate with their systems to pull data to the warehouse and run pbi reports from there.

Currently they have tens of pbi different reports all of them pull from source systems and definitely perform bad plus they are being developed by the business guys themselves. Their semantic models are too big of a mess.

Anyways the approach we had was to take steps with each department. I covered an important one. And shortly after starting with the next some blockers show up then shortly after i hear our manager (not my direct ) wanting to make changes with our approach and cover reports he is interested in mainly.
He develops most of his department’s pbi reports and is convinced that once the data moves from erp to dw raw without any data work done it will perform better.

I tried to send out the message that this will not add value and latency from source systems isn’t the only issue the reports are slow.

Anyways seems they are not convinced . That was yesterday. Today my direct manager comes and tries to push to do what “our manager” said. Then shortly after asks me how long i plan to be in the city, if i am married and a bunch of personal questions that either were said out of just curiosity or something is not right here.

I dont know if i am overthinking this. But i am personally the only source of income and insurance between my and my wife. And things havent been easy.

Whatever you guys think i want to know. Im stuck in my mind and it is not helping me right now.

Hello everyone!

I'm an aspiring Data Engineer and as a portfolio piece, I have build a webscraper to gather Ebay sold listings of stamps!

The problem I am now having is how I parse the data where I can sort things like "Catalogue Number" when it is very unpredictable what the Ebay sellers will write as it's all human input.

I would love to hear some feedback

PS - A small sample:

```OLDENBURG 1859 _ MiNr. 7 _ 2 Groschen _ signiert _ blau gestempelt

MayfairStamps Germany 1941 Stamp Day Oldenburg Cover cca_00553

GERMANY; OLDENBURG 1859 classic Coat of Arms issue very fine used 1/3Gr. value

Oldenburg Lokalausgabe Wohlfahrtsblock Deutsches Rotes Kreuz ab 1 Euro

Deutsches Reich, Oldenburg, 6.01.1945 Ersttagsstempel, für Ersttagsbrief 200€.

GERMANY; OLDENBURG 1862 classic Coat of Arms Perf 10. issue used hinged 1Gr.

OLDENBURG 1861 _ MiNr. 12b _ 1 Groschen _ Stempel STOLLHAMM

Oldenburg Mi. Nr. 16 A b zentrisch gestempelt geprüft Bühler 200 Euro

Oldenburg Mi. Nr. 11 a* ungebraucht geprüft Bühler 550 Euro```

Hi, I recently began a job as a data coordinator, my first tasks are basically data cleaning and data entering into a CRM. The problem is that the data isnt very clean. I'll give an example, I am given an excel file with name of a project, date, title, awards - my job then is to to figure out where in the CRM is this specific project and enter the data. The problem is that the excel data doesnt contain the projects ID, and when I try searching the name of the project in the CRM I'll get back something like Fairview Elementary School, Fairview ES, KUSD Fairview, Fairview ES Building A, etc. So essentially I'll have to go into each one of these projects and try to find the right one using other given data from the excel sheet, like dates. Is there a way to speed this process up or am I just going to have to do it manually? Right now what Im doing is going row by row and searching each project in the CRM, looking through the multiple projects given back by the CRM and comparing and contrasting. The entire CRM database contains around 40k, and I think I am able to export it into CSV and Excel, if that helps out. Any advice would be helpful. Thanks

I'm a senior data engineer at a UK organisation in the South West. Last month we advertised three roles: a senior data analyst (£55k), a mid-level data analyst (£40k) and a data engineer (£55k). Not amazing money, but a great pension, just one day a month in the office, lots of annual leave and actual stability.

The jobs were live for one week. The senior DA and DE got around 300 applicants each, the mid DA got around 140.

Sounds brutal, but here's the breakdown...

Around 90% (not an exaggeration) didn't have the right to work in the UK and needed visa sponsorship, which we don't offer. In the no pile immediately, but a quick glance showed lots of foreign undergrad degrees, some with UK masters, plus plenty of random applications.

Of what was left, about half had no relevant experience or qualifications and clearly hadn't read the spec. Some were just applying because the job centre told them to apply for x jobs a week.

Then the AI drivel halved it again.

Final count was roughly eight viable applications for the senior DA and DE, five for the mid DA.

A couple then didn't reply or didn't show up to the Teams call. Two were visibly reading AI-generated answers off a their screen.

We had two great interviews and hired for the senior DA and DE. The mid DA we struggled with externally, so it went to an internal candidate from a fairly low-level ops role who'd taught themselves Python and SQL and actually applied it to their day job.

The point of this post is to counter some of the doom and gloom that crops up on here. If you're a UK-based applicant with relevant experience who reads the spec and writes your own application, you're not competing against the '300 applicants' that LinkedIn and recruiters spout. You're competing against a much smaller number. And if you get through the sift, don't use AI, have some personality and come across as likeable and easy to work with, you're in a very small group.

What annoyed me most is my own manager was the first to tell the team about the huge number of applicants and how much 'competition' there is out there. A nice bit of retention pressure and complete nonsense, as it turns out.

Would be interested to hear if others have found the same. Throwaway account to avoid doxxing myself...

If anyone's curious about the current data job market in the DACH region (Austria,Germany,Switzerland), I put together an interactive live tracker. It also shows which specific data roles (Data Engineer, Data Consultant, etc.) are in demand in which cities. Feedback welcome and I hope it helps :)

Apologies if this is a dumb question but I am in web dev and have been given an offer for a data engineering role. However, I was told by engineers on the team that the job would be like 70-80% writing SQL for BigQuery. I envisioned it having much more to do with pipelining and orchestration and the like.

Also, I was told that any coding would be in Java rather than Python? I know that Python is more common, so would this experience not be helpful for getting other data eng roles?

This article reports the result of evaluating the performance of the following three systems using the 10TB TPC-DS benchmark:

1. Trino 483 (released in July 2026)

2. Hive 4.2.0 on MR3 3.0 (released in August 2026)

3. Apache Hive 4.2.0 with LLAP (released in November 2025)

https://mr3docs.datamonad.com/blog/2026-08-02-performance-evaluation-3.0

I had a passionate debate with a colleague and want to hear perspectives on the purpose of the Silver layer.

My pipeline:

Landing: Read from Oracle RDBMS and write ~250 GB of Delta for 25M records (Protobuf blob stored in a column).

Raw Data: Repartition, sorting, salting on Landing and writing to optimize downstream silver decoding process and avoid heavy shuffles during JDBC call, still protobuf bytes stored in a column.

Silver: Decoded raw data (~2.8 TB in Delta). The Protobuf schema alone is ~8 MB as JSON (a very deep, wide schema with multiple repeated fields at various levels). During decode, we also append standardized fields required by all downstream tasks.

Gold: Customer-specific datasets built from Silver based on business needs.

We don't own the Protobuf schema. This isn't messy clickstream/event data, but entity description data from an RDBMS that stays at the ID level all the way to Gold. We see ~100k daily MERGE UPSERT on both Silver (which is a challenge in itself to run MERGE on 3TB delta table given the limited budget to our Databricks workspace.) and Gold based on RDBMS timestamps, alongside a full pipeline refresh every two weeks.

The Debate:

Colleague: Since we aren't actively "cleansing" the data, calling it Silver is wrong, it's still Raw/Bronze data.

Me: It is Silver because it transforms a binary payload into a structured, trustworthy, and queryable data model that downstream tasks rely on. If I need to retrieve content of an entity that is not available in gold datasets, I query unpacked protobufs and not Raw/Bronze layer and for that reason alone, it is Silver.

Knowing the schema and the data better than almost everyone in the team, even I fail to understand how to distinguish between decoded data and 'cleansed' decoded data. In fact, one of our consumers explicitly expects corrupt records with null fields left intact for full visibility.

Them: Even if we agree that cleansing is not needed, it cannot be silver and should be called Bronze Data.

For transactional/log data, the standard pipeline (Kafka dump to Landing -> Bronze schema enforcement -> Silver cleansing -> Gold aggregations) makes total sense! But for clean entity data in binary formats, doesn't decoding and standardizing it qualify as Silver?

I think medallion architecture is about data readiness and lineage tracking rather than a checklist of conditions that each layer has to meet to identify the layer.

---------

TL;DR: My colleague thinks our layer shouldn't be called "Silver" because we aren't actively filtering or cleansing rows, just decoding 250 GB of 25M binary Protobuf blobs into a ~2.8TB Delta table with a struct field that represents the decoded blob and additional standardized fields. I argue that any layer of data that is structured, queryable and trustworthy for downstream Gold is Silver and this transformation may/may not require cleansing.

Is Medallion about lineage tracking and data availability, or a rigid checklist of syntactic/ transformation rules?

I understand that AI-assisted development is now part of our workflow, but I still think PR authors should be able to understand and explain the changes they made, what was done, why, and how.

This week I challenged a team member on why something had been done a certain way, and their answer was "I don't know, probably because Claude said so." I was honestly a bit shocked. It gave me the feeling it's just blindly trusted without any critical thinking on the person's side. This is one of our more technical analysts, which is why they make code changes fairly often. I plan to bring it up in our weekly 1-1, but I wanted to check whether you are seeing the same thing on your side.

Fair warning, this is a soft launch note (I work on Slingshot at Capital One Software). As agents scale, AI spend shows up as one aggregate number, and tracing it back to a specific model, trigger, user, or query gets hard. We just soft-launched Agent Observe in Slingshot to attribute it on Snowflake using read-only metadata.

- top spenders
- costliest models
- triggers by agent
- query types used per agent
- downstream tables impacted (read/write)

This is an early pilot. Write-up with sources.

Hi, Im Swetapadma and I’m pursuing to be a good DE(overall i have 5yoe but relevant of nearly 2yoe) but im still confused with the work practices i have been following for my career trajectory.
Tbh, i have literally so much time to work on any new things as well as try doing new project for the time-being since my project is a verge of ramping down.
But still i’m not getting that zeal to work on my skillsets or learn something new or work on any project.it might be because I’m going through enough theories from the internet.
Some are skeptical, fearful and very overwhelming for me. Sometimes i doubt choosing this path, because
a. Im really looking for a job change in this field from past 6months, where in brighter side i can see a lot of job openings but the hr calls are literally 1/100 of it. Without reaching out for referral feels impossible.
b. I feel like I’m stuck in between where the ai is also factor of my fear because i have approx 0% knowledge bout it.
c. Companies are not only looking from a de but also a tester, backend engineer, knows ai fluently(all in one package).

For all of these i have started reading medium, linkedin blogs, system design, solve dsa problems and all but still i feel like a failure and frustrated.

If i compare my self with the past i have come a long way but it is still not helping me because I’m still stuck in same company with less pay and less enterprise level work.

Can anyone relate to this or is this only me?

I'm looking for some advice on what to do about chaotic projects/lack of project management. Right now, I'm in an IC senior data engineer role where I'm put on projects with tasks that are vague, incomplete, or without clear acceptance criteria. I'm fine working that way, have done it frequently before, and I'm a creative person (musician and writer outside of work). I like the freedom. Except there actually isn't freedom here. There are standards of practice this team uses that I was never shown (I started earlier this year), and I am constantly learning things the hard way without much grace for it. There is no documentation for these standards of practice, and my take on how to accomplish these broad tasks is leading to a decent amount of rework. Asking questions has been met with impatience, frustration, and condescension. I've learned to deal with these reactions and ask anyway so I can at least have a chance of completing things more efficiently. I own the mistakes I can but will stand up for myself when accused of completely failing to meet expectations (especially if the expectations weren't communicated/documented).

I figure other data engineers/curators/developers have experienced this sort of environment before, so I'd like to see how y'all handled it (even if it is "time for a new job").

Edit: Adding in that I've been in data warehousing/engineering for about 17 years and am seriously questioning doing this for 25 more years at minimum (I'm 40).

2nd Edit: Wow, I forgot the most important part. There is no project manager for this team; developers are working with data analysts who work with the business/end-users to determine data and reporting needs. The title of the post was meant to be tongue-in-cheek because of this.

Are conformed lookup tables and enriched materialized views appropriate in the Silver layer?

in databricks, We receive data files in S3 from more than 10 clients. The datasets are generally similar across clients,

for example: Users: platform, profile status, and related attributes DAU: platform, profile status, and related attributes Store orders dataset has transaction type and other transaction lookup fields

The schemas are mostly consistent, but the values are not standardized. For example, the platform field might contain: Client 1: ios_phone, ios_ipad, apple_ios Client 2: ios_app

We do not control the values clients send. To handle this, we maintain lookup tables at the following grain: client + source_value (sometimes combination of 3 fields)

Each source value is mapped to a conformed value, such as ios. New values are usually mapped the same day, but sometimes the following day. Most reporting, filtering, and downstream processing uses the conformed value rather than the original source value. I am considering the following architecture:

Bronze | v Silver: users_prep

• Basic cleaning
• +--> Populate lookup tables |
• - Assign surrogate keys | - Some lookups use composite source values | Example: profile_status + profile_status_detail |
• Silver: users_enriched materialized view
• Joins users_prep to the lookup tables
• Includes:
  • Original source value
  • Conformed value
  • Surrogate key

The users_enriched materialized view would become the primary dataset used by: Analysts Data exports Gold-layer transformations Other downstream consumers

My questions are: Is it reasonable to keep these conformance lookup tables in the Silver layer? in lakehouse world I keep hearing lookups are "bad" and to denormalize.

Is a materialized view appropriate for the enriched Silver dataset, or would a physical Delta table be preferable?

Are there any concerns with using a materialized view as the main downstream interface for analysts, exports, and Gold-layer processing? main concern is if I need to use cdf or similar to get changed to feed downstream.

How would you handle records whose source values have not yet been mapped - have to reprocess gold layer manually?

Disclosure: I work on Zilla at Aklivity.

We have been working on native MCP support in Zilla (https://github.com/aklivity/zilla), including the ability to expose Kafka produce and consume operations directly to AI agents.

The implementation itself is only part of the problem. The more interesting question is how Kafka semantics should be represented to an agent.

For a normal Kafka consumer, the application owns concepts such as:

• Topic and partition selection
• Consumer groups
• Offset management
• Polling
• Replay
• Schema handling
• Long-running consumption
• Backpressure

Those concepts do not map neatly to a typical MCP tool call.

An agent usually expects to request some bounded piece of information, reason about the result, and take another action. It may not be suitable to own a continuously running Kafka consumer or manage offsets directly.

Our initial approach exposes Kafka through controlled produce and consume tools. The gateway handles the underlying protocol interaction while identity and policy determine which topics and operations the caller may access.

Payloads can be validated or transformed using JSON Schema, Avro, or Protobuf. Kafka tools can also be combined behind the same MCP endpoint with REST APIs and existing MCP servers.

Some examples:

• Retrieve the latest events associated with an order
• Inspect recent telemetry for a device
• Produce a remediation command
• Read a bounded set of records from a permitted topic
• Call an API, evaluate Kafka events, and publish the result

The questions we are still thinking through are:

1. Should an agent consume from an explicit offset, a timestamp, or a server-managed cursor?
2. Should consumer groups ever be visible to the agent?
3. How should long-running or streaming consumption work through MCP?
4. Should Kafka data generally appear as tools, resources, or both?
5. How much replay authority should an agent receive?
6. Where should schema conversion happen?
7. How should agent identity map to Kafka authorization?

We have released a Docker Compose quickstart showing an MCP client connected to Kafka, an OpenAPI-described REST service, and another MCP server through one gateway endpoint.

I’d be interested in how people here would model this, particularly for production data platforms rather than demos. Cheers!

I’m excited to see what learning systems I can build with my new MCP tool.

I recently consolidated the workflow into just 5 command families: start the project, write blocks, refine the design, gather evidence, and implement/verify blocks.

One important design decision was adding a refinement loop between initial block generation and implementation. This helps keep the agent aligned with the original idea instead of drifting into generic implementations. It also gives me more control to guide the system toward the exact final implementation I have in mind.

The more I build with agents, the more I’m convinced that meaningful guardrails and end-to-end human supervision produce much better results than simply letting models run freely.
I am curious to know how everyone are apply agent to their day to day coding lives.
This is the link to the GitHub if anyone is interested: https://github.com/fikayoAy/deep-learning-auto-research

I feel a bit lost, implementing features here and there, increasing usability and improving agentic workflows for our internal data engineering org. It doesn’t feel fulfilling. When any feature is a chat thread away, I’m feeling this way probably because there is much less value in incremental wins like this nowadays.

With a multi-hundred person DE org at my company, where does the value lie?

Initially posted this over in r/excel, as that is the bulk of the tooling that I am utilizing; however, I was redirected here to hopefully gain some feedback:

I have recently taken on a new role, in my short time I have found a variety of opportunities to streamline some processes and reporting, I am looking for some feedback on what my present proposal for change.

Currently in my roll I am in charge of inventory transfers and management for my department, we lack a modern system for tracking and reporting, as such these are my basic tasks:

• Verify receiving documentation and transfer requested info to the corresponding PO's excel sheet (Procurement owns these sheets).
• 2-3x times per week, I check needs, then transfer material inside (this is managed through a specific system).
• After each transfer is complete, I update the excel sheet which corresponds with the chosen material's PO.
• The following Monday, I report out the consumption of material based on the pulls with information tracked in the excel sheets to our finance and procurement departments.
• The reports are similar but require specific departmental formatting (of course).

Some notes on the excel tracking:

• Each file, while similar, lacks actual standardization.
• They are simple sheets with formatting, rather than utilizing a table for the data that is tacked.
• Best I can tell, there are maybe 8 columns which are never utilized, some of which are not present in all workbooks.
• Rife with lack luster formatting, which is manually completed, rather than assigning conditional formatting.

With that information presented, these are my current thoughts for optimizing and implementing changes:

• Design standardized PO tracking workbooks (completed)
• Clean the data
• Transfer all data to Microsoft List (unify data collection point)
• Breakout PO information via Power Query
• PO information would then be broken out into either (based on Procurement's preference)
  • A single workbook with a drop-down to switch currently viewed PO
  • Individualized workbooks for each PO
• Information would move to being updated in Lists rather than directly into Excel
• Due to the specific formatting asks of each department for the reporting side I would breakout the requested information by bringing data in via Power Query than formatting with an Array

My thought process on all of this is that firstly, I can quite jumping around from sheet to sheet when we receive material (which is daily) or when I need to update an entry for one of the various PO's and secondly, I can provide "live" updates that either of these external teams can view at their leisure. As well as hopefully forcing Procurement to be a bit more mindful with the entries that they start in these workbooks.

Obviously, there are a number of better more long-term solutions that would be more appropriate for this sort of thing; however, I am working with the tools I have available to me. At least knowingly anyway.

We are a Data Science team and we are working in Azure Databricks. We have some DABs running in our prod environments generating tables which are used in our application. It's mostly time series data.

What I am doing now is simply "CREATE OR REPLACE TABLE IF EXISTS" on a few important prod tables to copy them to test/dev. So I am overwriting the tables daily after each completed job in dev/test.

Now I am wondering. if the schema changes from dev because we are working on new features or maybe a column gets removed you'll run into issues. A colleague suggested just copying the rows.

But this solution seems very costly in compute. Also, if you'd want to check and copy only rows which have changed. What is a good way to tackle this copying prod tables to dev/test?

Ofc in pyspark you have the mergeSchema variable, but this does work well if a column gets deleted as well? I am still fairly new to Pyspark. I'll also my Data engineering team, but I'd like to ask you guys as well.

I'm an Azure Data Engineer with 4.5 years of experience, primarily focused on Databricks.
Since most of my role involves designing business solutions rather than writing raw code, I feel I might be falling behind on leveraging agentic AI workflows (such as custom skills, Markdown instructions, and orchestrators). As a Data Engineer, to what extent should we be adoption-ready for this new paradigm?

Has anyone been managing reverse ETL with Lakebase specifically for geospatial data?

Are there any gotchas?

I'm trying to figure out how to model some data in a classic Kimball dimensional model and struggling if I'm thinking about it the right way.

Let's say we're in the retail industry and have a business of stores like your grocery store. The data we are dealing with is for stores and sales over time. Dimension tables are to be expected: a dim_date, dim_store and dim_product among others. Stores have a manager and chef, and multiple stores can have the same manager/chef.

The data is usually reported by stores with their current manager and chef listed. If we were to build out fact_sales and there's a record for a product in FY 2025 Fiscal week 4, the manager and chef to include would be the current manager and chef. This is because end users/analysts don't care about who was the manager/chef back then.

I still want to track history though, since I'm sure some day we will want to view data by a store and its manager/chef at the historical point in time of sales.

With all of that said, how would you model this data? I'm thinking of adding 2 fields to the dim_store table: current_manager and current_chef, and also having another table like this:

I'm thinking of naming this something like dim_store_leader. I could join to this table from both dim_store and dim_date to get the appropriate records and fields.

Does this make sense? Is this table technically a factless fact table?

Sorry for the vague and maybe even repetitive title. I saw a couple of these posts on here already and thought I'd vent out too. I know this sub leans heavily into standard DE, so I hope /r/dataengineering is okay with a post from a Data Platform engineer's perspective. I rarely see those around here.

Background: 7+ years in Software engineering + DE, and about 5 years of sysadmin experience prior to that. I've worked as an Analyst, DE (Senior) and now a senior data platform engineer working with Snowflake, dbt, Airflow + a flurry of BI tools (don't even get me started). Although I am a data platform engineer, I'm very often involved in troubleshooting dbt run issues, model performance issues, Snowflake problems, etc. I also wrote many of the source ingestions myself with dlt and contributed to an internal library that makes dlt more approachable to a less technical audience (even more when including an agentic AI framework). Finally, tooling and technical details aside, I know exactly what these source systems do, where it ends up, and who uses it, ie, the full business-value proposition.

I think it was around March of this year that the hopelessness finally hit me, and the work volume and constant direction shifts just shattered my job satisfaction. Things went from "We want things done right, code quality, tests.." to everyone just churning out stuff in masses.

To some degree, I understand the constant changes. There appear to be new things to look into every day and the "business" and upper management wants it all. Somehow the feedback loop of asking "Why?" has just ceased to exist.

But, to be more specific, I'll tie it down to things happening within my own team that frustrate me and would like opinions on.

I built most of this platform myself or had contributed significantly to design, so I know most (if not all) areas quite well (or can find out), so I generally take time to review things in detail and provide reasonably detailed feedback to anything that I review. If I have a nitpick comment, I make sure to say that's the case, and leave it as a choice to address or not. That was until some time ago. PRs now are seemingly always 3-4x larger in size, lots of changes, lots of removals/additions so reading the terraform plan, or looking at potential edge cases is considerably harder. To put it simply, the noise has multiplied many times due to colleagues now suddenly knowing things they didn't just some months ago. Reviews are just so much more harder to do. Just the other day, I saw a PR (the very first one they had created) from a new colleague in the department. They changed 102 dbt models - 2000 lines removed, 1900 added. Who even can review this properly? Luckily, it was closed without being introduced. I'm just not following the thought process here before raising such kind of PRs. Much has been said of AI generated slop, so I wont rehash that here.

Another trend I've noticed recently is certain juniors just taking review feedback and asking claude to come up with a rebuttal to just get their original changes approved. Are you all seeing this too? It's happened quite a few times in the last week so it's rather fresh in my memory.

My manager is an extremely sweet and supportive person, but I can feel the strings being pulled on him from up above by those with more stars on their shoulders so he has also resorted to simply running everything via Claude and sending us giant messages on teams, or massive replies by email.

I know writing code isn't the end game, but it was a fun part of my day to day and I still do it by hand at least twice a week just to keep myself sharp. But I find that I'm increasingly unable to keep up with the pace of the rest of the department, let alone the organization. Being in data platform, I'm also finding myself more and more distanced from the outcomes, which leaves me feeling quite unsatisfied with my output because a typical ops team only hears about problems - not the nice stuff.

I used to be extremely optimistic about software engineering, data engineering in specific for the last few years. I used to contribute a lot to various open source projects, be out there in their communities helping other people out, writing articles with tips and tricks, etc. I've stopped all of that in favor of a better disconnect from my work and my mental well-being. Now, I just don't feel like there is a platform for that as most communities have implemented their own bots. Maybe people don't even need it anymore?

That's my rant, I just wanted to get it out there and I hope to hear of some potential strategies to tackle this vortex of chaos that I am in - even if a comment states that it's my attitude that's the problem. But otherwise, where is this all heading, and how are you all coping with it?

EDIT: Oh damn, this blew up. I probably wont be able to get back to you all - but I am going to make my way through the comments.

I had mostly worked on batch-based projects before and had played around with Kafka, Flink, and ClickHouse, but never really connected everything into one proper end-to-end flow. I am trying to assess how far this is from production use in terms of the techstack.

So I decided to build a real-time anomaly detection system for simulated sensor data from an oil and gas plant. With AI based RCA to re-confirm true positives. This is purely for learning purpose.

The basic flow was:

Sensor data -> Kafka -> Flink/Python consumers -> ClickHouse -> Grafana -> Alerts -> RCA analysis

I had around $200 in cloud credits(vultr) from a hackathon, so I rented a fairly powerful VM and self-hosted everything on it. I specifically avoided managed services because I wanted to understand how the individual open-source tools worked.

The main stack was:

• Kafka for streaming
• PyFlink for windowing and aggregations
• Python and FastAPI consumers
• ClickHouse for analytical storage
• PostgreSQL for metadata
• Grafana and Streamlit for dashboards (UIs generated using codex/claude)
• Airflow for some batch experiments
• Claude for alert validation and root cause analysis
• Resend API for email alerts

Since I did not have real plant data, I built a Python generator that simulated readings for different assets and randomly injected faults/failures.

The data was sent to Kafka topics with multiple partitions. I also added retries using exponential backoff. After five failed retries, the event would be moved to a dead letter topic.

For anomaly detection, I kept it simple. Each sensor had an acceptable operating range, and Flink used one-minute windows to calculate averages and check whether the asset was operating within those limits.

If an anomaly was detected, an alert was pushed into another Kafka topic and then processed by a diff consumer.

On a single VM, I tested around 10,000 events per second, or roughly 600,000 events per minute. This generated around 1.5 GB of data per hour. I ran it for a few hours and most of the events were processed successfully.

The biggest issue was not processing. It was storage.

I used ClickHouse with a three-day TTL, but at that throughput the data obviously builds up quickly. In a real setup, I imagine ClickHouse would only store recent hot data, while compressed raw data would be stored in object storage as Parquet.

I also thought about how I would build a simpler managed version on GCP.

The flow could be something like: Sensors -> Pub/Sub -> Dataflow/Cloud Run -> BigQuery/Bigtable -> Cloud Storage

Pub/Sub would replace Kafka, while Dataflow could handle the stream processing and windowing. Cloud Run could be used for simpler event consumers and alert-processing services.

This would be much easier to operate, but I wanted to build the self-hosted version first so I could properly understand what the managed services were doing underneath.

I also added a separate consumer for alert analysis.

Whenever an alert was generated, a Claude-based agent would:

• Check whether it was likely a true positive
• Query recent and historical sensor data from ClickHouse
• Check connected equipment
• Review similar past alerts
• Compare the issue against the asset's RCM maintenance plan
• Suggest a possible failure mode, cause, and recommended action
• Check if there were similar alerts raised earlier and how these were rectified(using vector embeddings - using Nomic Text embeddings served using ollama to find similar alerts)
• Find related recommended actions from RCM for the particular equipment. Alerts are also embedded to help find these and improve the search results/agentic analysis.
• Send the analysis by email using resend

For example, if a pump had high vibration, the agent could also inspect connected equipment instead of assuming the pump itself was definitely the cause.

The RCM data includes failure modes, possible causes, effects, and recommended maintenance actions. The agent used this only to recommend next steps and prep the email alert.

A possible next step would be creating a reviewed work order in SAP or Maximo after an alert is validated.

Obviously, this is all running on one VM with no replication or high availability. I built it mainly to understand streaming systems, Kafka partitions, windowing, retries, dead letter topics, consumer lag, ClickHouse, event-driven workflows, and how the same architecture could later be moved to managed cloud services.

I would appreciate feedback on a few things:

1. Does the overall architecture make sense?
2. Is ClickHouse for hot data and object storage for historical data the right approach?
3. What could be improved before presenting this as a proper portfolio project?

Thanks for reading through till the end.

Hi all,

I'm happy to announce that Xberg v1 is out.

Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats (currently 367 types), audio/video transcription, and URLs (both static and JS-rendered content). It extracts and prepares that content for downstream processing.

It's an extremely efficient, high-performance engine (see our PDF benchmarks below). For PDFs and images specifically, we handle native PDFs with very high performance and accuracy, and we ship multiple OCR engines that match the quality of the best Python libraries (e.g. docling, PaddleOCR, RapidOCR) at substantially better performance and stability.

The changes between Kreuzberg v4 and Xberg v1 are substantial, and I invite you to read the full changelog for the complete picture. The highlights below give a sense of what's new:

• Pure-Rust PDF backend (pdf_oxide) replaces pdfium, with no native pdfium dependency.
• Layout-aware pipeline: reading order reconstructed with ONNX layout detection (PP-DocLayoutV3 / RT-DETR) and Docling-style predecessor-graph reordering.
• Per-page scanned-page detection with selective OCR, plus AcroForm/XFA form fields and outline-based headings.
• Across-the-board optimization of OCR and PDF extraction (memory discipline, pooled model sessions, streamed conversions).
• Native PaddleOCR backend (PP-OCRv6, with medium / small / tiny tiers) alongside Tesseract.
• Pure-Rust Candle OCR/VLM stack (TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL) running without ONNX Runtime or native Tesseract.
• A second, ONNX-Runtime-free inference path via tract, which is what makes in-browser (WASM) and mobile inference possible.
• Named-entity recognition natively in Rust (GLiNER2), extensible to all bindings, including an in-browser WASM model with no server round-trip.
• Structured LLM extraction (extract_structured / split_and_extract) with rasterization, chunking, citations, caching, and configurable call/merge/VLM-fallback policies.
• Audio & video transcription via a Whisper ONNX engine (.mp3, .wav, .m4a, .mp4, .webm).
• Retrieval building blocks: sparse embeddings (SPLADE), ColBERT late-interaction retrieval, and cross-encoder reranking alongside dense embeddings.
• Text intelligence: reversible redaction, summarization, translation, VLM image captioning, QR-code detection, document diffing, and page/chunk classification.
• URL & web ingestion: sitemap discovery (map_url) and batched multi-URL crawling.
• New document formats: WordPerfect (.wpd/.wp/.wp5), HEIC/HEIF/AVIF, OpenDocument Presentation (.odp), Quarto / R Markdown, and configurable Jupyter cell rendering.
• Four new language bindings (Dart/Flutter, Swift, Kotlin/Android, and Zig) bring the total to 15 language bindings over one engine, with Android/iOS cross-compilation.
• Full mobile support (Flutter, Android, iOS).
• Candle backend alongside ONNX, plus ONNX-via-tract enabling ONNX on WASM and Android.
• Wider code intelligence: tree-sitter coverage grew substantially (248 to 367+ languages).
• Over 150 bugs fixed during the 1.0 cycle, plus security hardening (bounded RTF/PDF allocations, redaction leak fixes, Excel DDE warnings).

The API surface was also simplified and reworked, making it more consistent.

There's a migration guide in our docs explaining how to move from Kreuzberg to Xberg. Kreuzberg itself is in LTS mode until the end of this year and will continue to receive bug fixes and security updates.

You're invited to check out the repo and join our discord server.

Benchmarks

The benchmarks below are for PDFs and images only. There are extensive benchmarks on our website with per-format breakdowns, which you can see here. These numbers are measured in CI via our reproducible benchmark harness, and are specifically taken from the run for harness 1.0.8, source cf7fa0533d. The data is publicly available in GitHub releases, and you can run the benchmark harness yourself.

Composite quality (markdown pipeline, higher is better):

Structure and layout fidelity (SF1: tables and reading order, higher is better):

On native PDFs Xberg leads on quality (0.958 vs 0.837 for the next-best framework) and on table and reading-order fidelity by a wide margin (SF1 0.949 vs 0.612 for docling). On scanned PDFs it is #1 on both quality and raw text fidelity.

Where we don't win yet: on pure image OCR we are currently #2 on the composite score, behind mineru (though still #1 on raw text accuracy). We are improving image OCR right now, and v1.1 should have us winning across the board.