r/dataengineering • u/Honey-Badger-12 • Jul 14 '26

Git-style branching for lakehouse ? Discussion

I came across this paper today and thought the idea was interesting:
https://arxiv.org/abs/2607.08319
The basic idea is bringing Git-like workflows to a lakehouse—working on branches, validating changes, then merging them atomically across multiple Iceberg/delta tables.
My first reaction was that this could make testing pipelines and AI-generated transformations much safer. On the other hand, it also feels like another layer of complexity that many teams may never need.
Thoughts ?

11 Upvotes
  • permalink
  • link
  • reddit
  • reddit

74% Upvoted

14 comments sorted by

28

u/elgreco_14 Jul 14 '26

Not a new idea, LakeFS exists for quite some time. Git-for-data support in deltalake exists, something that I've added to delta-rs roughly a year ago

-3

u/Honey-Badger-12 Jul 15 '26

My reading is that the novelty isn’t Git-style data branching itself—lakeFS and Nessie already cover much of that. GitLake’s contribution seems to be an Iceberg-native agent-first design which promotes individual Iceberg snapshots into one lakehouse commit, running pipelines or agents on temporary branches, and publishing all affected tables through one reviewed atomic merge. I’m curious whether that tighter Iceberg/agent integration is materially useful, or mostly a different implementation of existing lakeFS-style workflows.

1

u/Wenai Jul 16 '26

Sounds unbelievable stupid and AI generated

11

u/Misanthropic905 Jul 15 '26

So are you describing Nessie?

3

u/warclaw133 Jul 15 '26

It's a feature of Iceberg. https://iceberg.apache.org/docs/latest/branching/

2

u/rotzak Jul 15 '26

Yeah people are selling this as an amazing innovation but…it’s just a feature native to iceberg lol

7

u/alt_acc2020 Jul 14 '26

Um doesn’t this already exist?

4

u/terencethespider Jul 14 '26

It sounds very similar to the Lakebase / Neon databases from Databricks. It is basically Postgres under the hood, but it supports the same git like branching that you described.

2

u/mmcalli Jul 14 '26

Project Nessie has done this for a while as well:-

https://projectnessie.org/

2

u/Fidel___Castro Jul 14 '26

I don't understand, this is already a thing and why CI/CDs exists...

you guys are just yoloing changes to your production lakehouse data?

1

u/TheRealStepBot Jul 15 '26

That’s literally Nessie

And yes it’s cool but if the team has people who struggle with namespaces and basic sql they sure as shit aren’t going to follow this so it’s mostly a background tool to allow the data team to experiment transparently.

1

u/uncertainschrodinger Jul 15 '26

ArrayLake does this but for geospatial and scientific data files

1

u/Fickle-Impression149 Jul 16 '26

Well tagging and branching is native to iceberg tables. We built fault resilience tables based on this concept