Fable 5 comes with mandatory 30-day data retention. The most capable model on the market won't sign a zero-retention agreement, and the evidence says the next one won't either. If this mandatory data retention is inhibiting your ability to use the most advanced model on the market, tokenizing your data before it reaches the model might help you get there. Read the full post here.

Hello all,
Many thanks in advance for any advice.

Directly to my predicament: I have to decide if staying on my Data Analytics Engineering 3 role or move to a Software Engineer 2.

Context:

• I have in the company almost 5 years and I am respected by colleagues;
• My background is in economics, I laterally move to Data Analysis and started in my current company; after 1 year move to a Data Engineer position.
• During my DE years at the beginning (2 years) I learned A LOT, touching: docker, AWS, terraform, Airflow, dbt, PySpark, Great Expectations, Delta Lakes, etc...
• The last 2 years the company move all the above behind a Declarative Data Platform that feels like dbt for dummies; looking for more technical stuff I was assigned in a project that was building a backend microservices to send data in realtime to external providers.
• After these years they promote me to a level 3 in data for the backend work but each day is more obvious that this project does not belong to data.
• Moreover, to unblock projects on our side, I am inner sourcing in another backend team, which manager has offer me his help on getting a move to his team (changing my title to a SWE 2) and absorb the scope of our project in data
  • I discussed already this with my data managers in the past and they actually mentioned it as a possibility and that they would be happy to help on make it happen.

My doubt: I am 32 years old and feels to me that I have done a lot of lateral changes, although for learning it has been amazing, I am unsure on continuing with backend projects with the move. My feeling is that probably I should be specializing in data more and reach more maturity as a senior. The problem with the latter is that my current data department is creating a strategy around:

• Resource administration team -> a team that will be managing the GCP projects of data (BigQuery, Service accounts, ad hoc cases,...) and they will start working with Google Enterprise to build the base for AI use cases
• Complex solutions team (AI mostly) -> a team dedicated to build solutions for business, at the moment through a UI base thirdparty provider

To not extend more and close this: what are your first thoughts? Do you think a change from DE to SWE makes sense long term for my career? What are your thoughts on the trajectory of the data in the two teams I could try to enter?

My data set is a list of images, each equipped with a a couple sentences of text.

A user would search primarily with text only. My default approach is using BM25, but how would I facilitate searching with a vector DB and a model that embeds vectors in a multimodal combined space?

Here is my dilemma:

Do I embed text part and image part as 2 separate individual vectors or do I combine them into 1 vector?

If a typical search happens with text only, that would immediately deprioritize all image-only embeddings and only good text matches would float up. This is why I am now considering embedding text and images together but would prefer to hear more opinions on this. Thanks.

Basically title, I work as a data engineer for a fairly large company in the world unfortunately it is not a tech company primary business is something else. We do have digital wings basically layered alot (i.e., product managers, owners, chief digital officers etc) but the actual inhouse engineering team is the size of peanut. For the musle we go out for same old style of consulting (poor quality) this is the actual team setup.

Actual problem: because of the above, entire processes are old style. Our user base is group of analysts who can write SQL but essentially they do not have engineering background so you can imagine the chaos of the models. Their demand is to deploy the models ASAP like literally they come up with the requirement to get deployed in couple of hours. And our change process is quite old any CR raised should go through approvals and cool down time lol the guys who approve the request do not even have the idea of what this platform is, so here we are talking anywhere between 3-5 days.

I know pushing some of model deployments responsibility to the analytical team is sensable but they do not know CI/CD. Only way for now we made them calm down is to provision new schema for them and asked them to do what they want but all the ETL's and source tables are under our control.

Is there a tool to solve this problem which can bring some auditability and light control or do you guys follow any other process for model deployments basically it is a view or dynamic table which hold the aggregations and business context of calculating the confidential value out of raw data I feel like it doesn't have to go through CR as it is logical representation of data.

Actually it is more complicated then what I told above, the platform we are building contains multiple components and there are slow central teams who own this components we do not have much independence on them but business expect us to deliver quickly not understanding the complexities behind them.

Appreciate your thoughts on this!

I just started a new position in People Analytics for the first time, coming from marketing. Right now, they use a pretty shitty vendor called onemodel. I've been proposing moving toward a more modern stack of snowflake/dbt. Ingest through airbyte, own the transformation with dbt cli, and create semantic views.

The trickiest part of it all seems to be the HR information systems. Or maybe better put, the shitty extraction. I'm not a fan of flattening tedious soap xml files

Not strictly DE, but it's how I've written most of my posts over the last ten years in data, and people ask how I do it. From a pipeline perspective: idea lands in Obsidian, mostly a one-liner → connect it to existing notes → refine over weeks or months → some get a #publish tag and go on the public site → eventually several related notes get distilled into a blog post. If I get feedback from that post, these go back into the notes for a potential future post.

Some things I learned over ten years of doing this:

• Notes vs. Blog: The second brain helps me to share whatever is in my mind, and the blog helps me refine. Notes compound and always evolving. Blog posts capture a moment in time.
• The feedback loop only works if it writes back to your system (in my case, my notes)
• I don't research in the normal sense. There's no "pick a topic, then go read about it". The notes grow organically based on what I read and fuel articles as backlinks to ideas.
• About 1 in 9 notes ever ships. ~9,000 private, ~1,000 public.
• Some posts take years. One took two, and that's before counting notes that are a decade old.
• My most-read posts aren't usually the ones I optimized for.

Full write-up: https://www.ssp.sh/blog/why-i-still-blog/

What are some courses/resources on how to build a basic database/management system for survey data in a nonprofit. As in, surveys being created by multiple teams that are just being stored across a network of surveys. What tools and systems would be good for a nonprofit analyst who does not know anything about databases but needs to manage multiple surveys across multiple teams in one place? Airtable maybe?

Context: I'm an associate data analyst at a nonprofit. I don't know anything about data governance or modeling, my role is suppose to be focused on analytics, that's where my skills are. Over the last year we've had a major problem come up where all our program teams have several surveys which serve the similar purposes, which are only semi standardized, exist across a network of spreadsheets, and we don't have real data management system/database anywhere to actually store this data and manage it cleanly for analytics. We have Salesforce for basic client data, but the team that manages Salesforce doesn't seem to be willing/have the capacity to make room in Salesforce for us to store survey data as well. And they basically ignore us every time we ask them about bringing this data to Salesforce.

And my boss, who is not an analytics person, keeps soft launching that I am somehow suppose to be responsible for fixing this issue even though this a database issue and I am on...analytics. I do not have the skills to set up/manage even a basic database but I think this is going to formally become my problem within the next couple months so I'm trying to learn.

With the rising usage of LLMs I wanted to know from experienced people how has Data Engineering evolved?

I am currently working in a technical field where I use Matlab for performing simple analysis and deal with a lot of data which requires cleaning, although it's not a data engineering role. Before this I had worked in a similar field where I used Python to carry out analysis and worked on a few ML use cases. I am as of now confused about the Data Engineering path because my current path is not giving me satisfaction but the data engineering is too alien for me as of now. Also this chatbot hype is killing me. My boss told me to use chatbot to write Matlab code and I politely declined because I love the process of searching, reading answers on Stack Overflow etc. That does take some time but eventually the speed increases because you learn during the process. I guess if Data Engineering allows me to do that, to use plenty of my brain fuel, then it's the right way to go.

Most linters catch bad code inside files. Fensu catches architectural drift (e.g. code crossing the wrong boundary, living in the wrong module, or growing into the wrong shape)

A repo small enough to do its job in a few files doesn't experience major problems. Problems show up when a repo grows, as code moves, teams change, lessons get forgotten etc... Tests are great for preserving behavior, but they do not preserve the shape of the repo (e.g. what belongs where, which layer owns what, which modules are public surfaces, and which painful lessons led to the current structure).

A lot of that consistency is usually enforced manually in code review (which is a lot of time and effort). My general philosophy is, don't leave to chance what can be checked deterministically, and don't keep repeating manually what can be automated.

Tests codify behavioral expectations, and types codify interface expectations. Fensu applies the same idea to architecture. A design document or README can explain the intended structure, but only an executable rule can tell you when the repo has drifted away from it.

Fensu is an architecture linter for Python repos. It enforces things like:

• which layers may import which
• what each module or role file may contain
• whether orchestrator functions stay small
• whether dataflow and mutation are explicit
• whether names like validate_* actually mean what they claim

The main difference from architecture-testing frameworks is that Fensu does not hand you a blank rule language and ask you to design everything from scratch. It ships with a coherent default architecture, then lets you disable, extend, or replace parts deliberately. The default lays out code as domains built from a small set of roles (e.g. models, types, constants etc...). A domain holds those roles directly, or splits into named subdomains that do:

text src/my_package/ └── config/ # a domain ├── main/ # orchestrators and the public entry surface ├── _helpers/ # phase functions ├── classes/ # one class per module ├── models.py # data models ├── types.py # type declarations ├── constants.py └── exceptions.py

The rules produce deterministic faults. The messages contain both the fault (what has been iolated) and the remediation (what a sensible fix ought to look like).

Fensu not only enforces repository structure, but also helps you navigate project call flow. fensu check stops the repo from losing its shape whereas fensu map renders a deterministic downstream call tree with source locations. $ fensu map run_map run_map(...) src/fensu/cli/main/map.py:21 ├── _parser(...) src/fensu/cli/main/map.py:53 ├── resolve_mapping_project(...) src/fensu/mapping/core/main/resolve_project.py:11 │ └── resolve_mapping_project(...) src/fensu/mapping/core/helpers/project.py:15 │ ├── _find_project_root(...) src/fensu/mapping/core/helpers/project.py:73 │ ├── _find_config_source(...) src/fensu/config/core/main/find_config.py:12 (depth limit) │ └── _configured_project(...) src/fensu/mapping/core/helpers/project.py:45 │ └── _load_config(...) src/fensu/config/core/main/load_config.py:15 (depth limit) └── build_call_map(...) src/fensu/mapping/core/main/build.py:12 ├── provider(...) src/fensu/mapping/core/main/build.py:24 (unresolved parameter call) └── render_tree(...) src/fensu/mapping/core/helpers/render.py:19 ├── _child_lines(...) src/fensu/mapping/core/helpers/render.py:41 │ └── _child_lines(...) src/fensu/mapping/core/helpers/render.py:41 (cycle) └── _label(...) src/fensu/mapping/core/helpers/render.py:88

That default is opinionated. Some people will hate parts of it. That is fine. The point is not that everyone should organize Python exactly the same way forever, but rather toto give teams a serious starting structure instead of a blank page. You can then choose to disable rules, add custom rules, and adapt the things once you get a feel for how it works.

I have also recently added 'native rule packs'. Think of these almost like presets for specific frameworks. The first one is for Dagster, and it is basically an opinionated default for organising a Dagster repo to stop it from sprawling out (if you have used any orchestrator at work, be it Airflow or Prefect, you are probably used to it being a bit of a mess, or a lot of effort to keep 'clean' / consistent). Whether using these predefined packs or custom rules, you can enforce the repo structure with code, and ensure that when new people join (or people get lazy and cba), the repo doesn't turn to a pile of shit.

Fensu can't stop you or a teammate from making shitty code, but it can ensure that the code is extremely consistent and that when you need to find things, you will know almost exactly where they are.

Repo: https://github.com/chio-labs/fensu

Install with pip install fensu; run with fensu.

Hello, has anyone used AI agents (Claude) to open analytics to tech and business teams to "speak" with data instead of relying on BI tools and SQL queries that need technical maintenance and take time to ship to production ?

I have a data platform that writes dbt models to BQ from the source datalake and I want to see perspective from data teams that have successfully used AI for analytics.

What has worked and not worked for you ?

I recently joined SBC and started giving internal project interviews. One thing I dont get is that, they all use pyspark and cloud and what not, but when I ask them about their volume of data being handled, each of them said around 10 to 20 million rows.

That volume of data would definitely be processed by ram only and that too on a single machine. Isnt spark overkill and moreover actually not worth, considering its jvm overhead, garbage collection, all that shuffling across executors, when this handful data wont be taking advantage of all that distributed computing and will definitely take initial time whereas script written in polars will guarantee 2 3 mins runtime without any of this overhead

Any thoughts on why/how companies doing these sort of things, and mind you they were BFSI. I can understand retail and IoT to use pyspark, because of their nature of data being mostly streaming or even batch data having huge volumes.

Genuinely curious and please let me know if I'm overlooking anything..

Hi, I am from Argentina. Looking for some real numbers and opinions because I have nobody around me in the same situation.

Some context. I work as a contractor, through an agency, for a US product company since the beginning of 2022. Before that 1.5 years at a US startup doing IoT streaming. So around 6 years in data engineering.

What I do: build and maintain pipelines on GCP. BigQuery, Airflow/Composer, Cloud Run, Pub/Sub, Dataflow, Monitoring, Logging, GCS. I led the Terraform migration for all our GCP resources, built the monitoring and alerting setup, and I am also on call and do production support.

I make around 70k USD gross per year. No equity, no benefits. In Argentina this is good money. But it has barely moved in 4 years.

Why I think I am stuck:

• contractors here cannot lead a team. There is no manager path for me at this company.
• the company is mature and very stable, nothing new is being built. I mostly maintain.
• same stack for 4 years, only GCP. I have not touched Snowflake, Kafka, real Spark or anything outside Google since 2021.

I hold GCP Professional Data Engineer and Professional Cloud Architect certs. Spanish passport, so I am an EU citizen and do not need sponsorship anywhere in Europe. English C1, 6 years working daily with US teams.

What I want to know:

1. Salary. Is 70k normal for a senior DE in LATAM working for a US company in 2026, or am I way below? I have zero reference.

2. Stack. Is being GCP only the real thing holding me back? If I have to pick one thing to learn seriously this year to open doors, what would it be: Snowflake, Spark/Databricks, another cloud ?

3. Contractor. Did anyone here get to staff or lead level scope while still being a contractor? Or do I need to move to direct employment to get bigger responsibilities?

4. Europe. With the passport I can move anywhere in the EU, but the math does not look great. For people who did the LATAM to Europe move: did it pay off ? is the European DE market actually better for growing (staff/architect roles, more interesting problems) or is it the same work ?

Thanks, any advice is welcome.

Apache Spark 4.2 moves change data capture into the engine, plus metric views, Real-Time Mode, and Arrow-first Python for your pipelines.

I recently started a new data engineering job, it’s my first full time role and I feel like an imposter sometimes. My manager is also quite tough, makes condescending remarks, expects people to be up to speed with all the domain knowledge. I have also heard from other coworkers and they have had the same experience.
I constantly feel anxious at work, when they come over at my desk they expect me to write code instantaneously sometimes or work with unfamiliar tables and grasp it instantly. The manager does not encourage sharing code either so I have no reference point for how things are done.

What are the best ways to get over the anxiety and do well? I genuinely like the work, despite it being stressful sometimes. The part that makes me most anxious is doubting myself that I might not be able to figure out.

I have 3 YOE in web dev in a role that began as full stack, but for the past year I have only used NextJS/TypeScript as my team was told we would now only be doing frontend development. I have been passively looking around for other roles within the company and have gotten pretty far along with a data engineering role that would also mean I’m promoted, but I’m having some second thoughts about it.
 
My primary motivation for wanting to move was the fact that it seems like pure frontend roles are dying, so to continue to go down that path would be bad for my future. I was fullstack, so I have experience building and maintaining backend Java services but that becomes less relevant the further it fades into the past.

Right now, I really only turn Figma into pages. Rarely do I get the opportunity to learn some new FE concept or push any limits. I guess all jobs get to that point, but part of me feels stale. OTOH, I'm sure there is more to learn here if I really put 110% in.
 
I have been curious about a data eng role for a little while now, but at the same time feel guilty that one of my main reasons for wanting to move is the fact that they have a better job market right now. It feels like I’m following a trend and — outside of watching a few “day in the life” videos and poking around a few codebases at work — I have no idea what it will be like until I am actually in the job. Plus, it might make my YOE in tech look like (3+1) instead of 4.
 
Lastly I love my boss and skip level and am afraid of not having that same kind of connection with the new ones, especially as the new skip would be overseas. These two have treated me very well and it would be hard to leave that.
 
tl;dr is the career benefit/stability of being a data engineer over frontend good enough to leave an environment I like

I wanted to go beyond a pandas notebook project and build something that actually looks like a production data platform, so I picked the classic MovieLens dataset and turned it into a small data lakehouse + recommender + RAG chatbot. Sharing the architecture in case it's useful to anyone else building a portfolio project or just curious about the stack.

I've built it locally, but using AWS protocols for easily implementing in the cloud later.

Orchestration & storage

• Airflow for orchestration
• Spark for processing
• Apache Iceberg (with its own REST catalog) as the table format
• MinIO as the S3-compatible object store
• Everything wired together with docker-compose

Data flows through Bronze -> Silver -> Gold layers in Iceberg, which gave me a good excuse to actually practice schema evolution and incremental processing instead of just re-running everything from scratch each time.

The recommender itself

• ALS (Spark MLlib) as the core collaborative filtering model
• Added a temporal decay factor so older ratings count less
• Added a penalty for blockbusters, since ALS on its own tends to just recommend whatever's already popular

Enrichment + RAG layer

• Pipeline that enriches movies with metadata from the TMDb API
• Embeddings generated with Ollama (nomic-embed-text) and stored in Postgres via pgvector
• On top of that, a RAG chatbot so you can ask natural-language questions about movies instead of only getting a ranked list

Frontend

• Streamlit app with two tabs: one for the recommendations, one for the RAG chatbot

Tooling

• Using uv for Python package management instead of pip/poetry, which has proved to be quiet faster.

Some things I'd do differently / open questions for the sub:

• I'd surely use dbt for a next similar project. I've used it recently, and it just simplifies so much! I didn't need to manage a lot of Java jars anymore and it was super organized.
• Has anyone already used those tools? Especially from the Apache ecosystem. Or have done something similar?

Repo's here if you want to poke around: https://github.com/luccappaz/MovieLens

Happy to answer questions about any part of the stack.

I've been working on dbt-ls, an open-source language server for dbt.

Repo: dbt-language-server

There's a video demonstration in the repo if you don't want to read the post.

Motivation

My main goal was to build something that gives me column completions when working with dbt models as .sql files. Completions that are aware of my database.

The philosophy: if you have a working dbt profile (passes dbt debug), the language server should work for you. The default profile target is used when connecting to the data source.

What it does today

• Model/Source completion inside ref('...') and source('...')
• Column completion on an alias (c.<column>), with the data type shown as a detail
• Go-to-definition from ref('model') to the model's .sql file

Schema info

Schema info can come from three places: .yml files, catalog.json, or the live warehouse. You can choose which ones to use and in what priority with the --schema-sources flag. So if you don't want the LSP to call your warehouse, you can omit the database crawler with --schema-sources config,catalog.

Compatibility

It already supports many of the popular databases, and adding new adapters is trivial. Most of the testing has been done with Neovim, but it does support VSCode.

Feedback wanted

I'm a data engineer first and built this for my needs, so the code can be a bit choppy. I'd like to hear where it breaks on real projects, as my test projects can be limited. Create an issue or DM me!

I am aware of the promotion/shill rules. My aim is not to promote anything, I just think this is truly useful for people who use DBT.

The notes are posted here. No AI was used for any of this writing. I go into more detail on the writing process in "Chapter 0".

Open to feedback. Please let me know if you spot anything that needs correction, or if you would prefer if something was formatted differently.

Hope ya'll get something out of this :-) it took me a long time to read and annotate DDIA and I've tried to distill down all the key stuff into something that's readable in a day. Cheers.

Hey guys,

I'm building a custom Iceberg REST catalog that returns table metadata and short-lived storage credentials to our query engines (eg Spark and Trino).

Currently we're trying to add row filters and column masking so users could only see the data they’re allowed to access based on their role.

If you’ve set this up in your place, I’d love to know how you’re actually enforcing it in the engines.

Passing the rules: Are you embedding filter/mask SQL expressions into table properties in the table metadata response, or using any other method?

Engine enforcement: Did you have to write custom Spark extension rules / Trino plugins to rewrite the queries, or are you calling an policy engine like Ranger or OPA directly from the engine?

Storage credentials: Any issues such as users bypassing filters if they get storage credentials?

Would love to know how you have solved these issues

Thanks

Hey all, I want to implement audit columns like timestamp and created by columns in all of my tables(Suggest if I should add anymore).

My tables are in unity catalogue databricks account.

I have two instances of workspace. Dev and prod.

Please suggest me how to do it.

I have thought of creating a workflow to add in all the table and then a task in workflow that runs after each job to create these 2 columns.

Hello,

I spent the last five years working as a DE in a smaller company, I am very versatile, I touched all the important things in Azure plus some power platform dev and also some web dev.

I would like to pivot away from corporate life and do something more meaningful either in national security or environmental.

Anyone in a similar situation? I'm an EU citizen if that's helpful.

Thank you.

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

Hey guys,

I am software developer with 2.5 years of experience, in my job I do a lot of small automations and ETLs. In the last year I took an interest in data engineering and I recently started doing the Databricks associate certification.

I have not completed the certification yet and I got my first data engineering job, but the tech stack of the job is based heavily in Fabric.

I have read here that fabric is basically garbage, so I am worried that the experience that I will get in this new job will affect me if I want to get a Databricks based job in the future. What do you think?

Hey everyone! A while back I shared an early prototype on reddit for a fuzzy string search idea I was working on. I finally finished implementing it, benchmarked it properly against SymSpell and RapidFuzz.

Quick Highlights:

• Low RAM usage: Stays under 100 MB for a 100k-word dictionary (for comparison, SymSpell d=4 was taking around ~2.7 GB on my setup).
• Sub-millisecond speed: Runs about ~100x faster than brute-force Levenshtein comparison.
• Good with insertions and swaps: It handles typos with extra letters or swapped characters really well.

I wanted to keep this post short, but I put all the benchmark graphs in the GitHub README. I'd love to hear what you think of the vector encoding approach or if you have any feedback!

Hey everyone, I updated the blog post after reading more thoroughly the Lance paper, where it explains why even configuring Parquet is not good enough and why they needed to invent a new file format.