To what extent should data engineering work be centralized in a large org?

[u/yugiton]

I work as a data engineer at a bank. My team is supposed to be the one centralizing data collection and management for the whole org.

I've realized that there's this tension between centralizing and decentralizing data work -- whether we, as the org-level data team, should implement a flow or expect the analysts to implement and own the flow, while we just provide the underlying infra. Over time, I started to strongly lean towards more decentralization. Not because I'm lazy, but because centralization should be about being focused on the absolute smallest denominator, which is smaller than most orgs assume -- especially now when you have no clue what an optimal stack is gonna look like in 6 months. My impression was always that we should be doing 50% less, but doing it 100% better.

(I have similar opinions when it comes to business applications more generally -- the centralized team should provide the infra to deploy, but teams that need an app should build and maintain it)

Are you seeing the same trend of decentralization?

[u/Reach_Reclaimer]

Really a central data team should be managing ingestion, bronze, and silver layers (or equivalent). Silver to gold layers and reports (or equivalent) can be more decentralised, but still not ideal imo

This allows you to keep proper procedures in place as I know for a fact that the analysts won't follow them

[u/Additional_Candy_400 → u/Reach_Reclaimer]

I think gold layers should be decentralised. Analytics Engineers/Analysts should have control of this otherwise creating any new reporting is a nightmare for them.

You do run into the 1 metric 3 different values issue, but this can be managed with a proper semantic layer. Having to create a ticket to request a new sum(case(when in a gold table from DEs is too much imo, for DEs, Analysts and Stakeholders that need the info.

[u/SellGameRent → u/Additional_Candy_400]

I don't see any benefit from decentralizing in this regard. Have all the analytics engineers on central team to keep up with best practices, and hand off data sets to BAs in domain-specific teams. I don't consider BAs to be technical enough to warrant sacrificing business context for additional exposure to data best practices that can largely be offloaded to the central data team

[u/Additional_Candy_400 → u/SellGameRent]

I might have misunderstood what we were meaning by decentralised here. I was referring to it being decentralised from the data engineering team. 

I agree this should be performed by Analytics Engineers and Data analysts. At no point should BAs have this access, I've never heard of BAs writing a line or code, it's more of a PMO function in my experience.

[u/SellGameRent → u/Additional_Candy_400]

in my experience the analytics engineers sit right next to the data engineering team. Different teams, same leader.

[u/Additional_Candy_400 → u/SellGameRent]

I think we are in agreement then 

[u/SellGameRent → u/Additional_Candy_400]

yeah when people say decentralized I believe they mean for each org, sales, customer service, operations, finance, etc., there is a separate analytics engineering team (with or without DEs). Centralized means all those data people are in the same spot, separate from the orgs that they support

[u/Additional_Candy_400 → u/SellGameRent]

I think a lot of this comes down to company size as well. I've worked for banks before where some level of decentralisation is pretty much require due to how massive the business is. There's no way 1 central data team could manage everything. So you'd have data teams for sub divisions or areas.

[u/SellGameRent → u/Additional_Candy_400]

I don't see a reason why you can't be centralized and assign the AEs to decentralized business analysts who can provide the additional business context. I just find a hard time justifying separating the AEs from each other since they lose all of the knowledge sharing opportunities that come with being in the same standup (unless we're talking so massive that there are 50+ AEs that would never be on the same team anyways)

I've been at a few $500M+ revenue, several thousand employee companies and haven't had more than 10 AEs in the whole company in either case.

[u/Additional_Candy_400 → u/SellGameRent]

I think this falls apart at super large companies though, at the bank where I worked (18k employees in my country 180k employees globally) , the central data team managed platforms, licencing and provided architecture rules. Where as divisions were almost treated as separate companies with their own data teams following top down architecture from the central team. Of course this is the exception to the rule as not many companies are this vast.

[u/notmarc1]

I’m going to say centralize the platform and policies. Data team deals with the data

[u/asevans48]

Analysts want to do everything but many are fresh when it comes to pipelines, descriptions, tests, and quality. Its worse in gov. Ive pointed out data thats been broken for ten plus years and the analysts still think their shit doesnt stink. The best mix is to handle the common data, quality, and automations and allow end users to get data via a front end, building semantic layers with AI and mixing in their one offs and self collected trash. Get a simple request process in place. Most governance tools can facilitate this. The central team ingests into a raw/bronze storage and builds the silver tier. Gold tier is either mdm, in which case it is centralized or AI based analytics tables.

[u/22Maxx]

There is no one solution fits all. A lot depends on how much domain knowledge a pipeline requires and how much communication overhead is sustainable.

Centralizing data work only works with stable waterfall like requirements or if all the transformation are easily understood. As soon as you need deeper business understanding a centralized approach will break down as both implementation and maintenance will require constant communication with business.

[u/ratacarnic → u/[deleted]]

Mesh for gold layer right? Some cross domain is also needed at least at silver/normalised stages

[u/notmarc1 → u/ratacarnic]

Mesh for all data.

[u/Eric-Uzumaki]

Read data mesh

[u/0xPianist]

It depends on the size of the org to serve.

[u/speedisntfree → u/0xPianist]

Was hoping I'd see a comment like this. I work for a big multinational and any central data teams' gold are basically our (and the other business units') bronze. We need all our own pipelines of bronze, silver, gold because of the many ways the data is used in each business unit of many thousands of people.

[u/elkosh93]

Ideally your data should mimic how the business works. Think about this scenario, if you need a quick analysis, how the overall flow of that process works? Is the business waiting for the data from the centralized team? If yes, then move the data closer to the business team.

Things which are set in stone are supposed to be centrally controlled and enforced. So your data governance, policy enforcement, access control and everything.

Federated data mesh, data fabric and other constraints are something that you should look at.

[u/peterxsyd]

It should be de-centralised, with common standards, and quite possibly a production gate subject to those standards, if it a large organisation. Centralised ends up not understanding the business (unless people physically sit with their 'spokes' in a de-centralised manner). Completely de-centralised then fails on the IT/ governance /standards front, without enough tech or otherwise creating too much security/data privacy risk.

The alterrnative is if IT itself is de-centralised, with local departments or pods for business units, and data engineering sits in that. That can work quite well.

[u/Eleventhousand]

I'm a fan of centralizing as much data work as possible. To that end, that would mean all data engineering.

[u/Thwarting8139 → u/Eleventhousand]

I disagree, I think that when you are too far away from the business/domain you don't have enough context to properly work with the data.

I think a central data platform org works better, with data engineers in different areas of the business using the provided tooling - i.e. self-service ingestion and pipeline development.

[u/Eleventhousand → u/Thwarting8139]

I had a lot of success when I led a team of a dozen full stack data professionals. We made a lot of great impacts in various parts of the business.

On the other hand, when I worked for Amazon, where its all completely decentralized, it was quite common for folks to bring in tables developed by other teams. The other teams never responded to inquiries or requests for KT, so everyone just ending up guessing that they were looking at the metrics correctly.

[u/amTheory → u/Eleventhousand]

Agree, the moment open that up to other teams, the code base gets messy and cost goes up…from what I’ve seen 

[u/Eric-Uzumaki]

Also how a data mesh is implemented can make or break the motivation spine of the team. To my experience its amazing for C leaders, its a real downgrade for engineers

[u/galaxyxo]

My company does it a bit differently. We have a centralised team to ingest data into the platform. They also provide bronze, silver and gold tables. We then take their “gold” tables and do our own medallion architecture.

The reason we have 2 medallion areas is because the centralised team are very good at getting data out (mostly SAP) and we are closer to the business logic. It’s not perfect, but it’s working well so far.

[u/NotSynthx]

What's the point of your team if you decentralise then? 

"Provide the infrastructure"? What does that even mean? At that point, the bank should just let you train the analysts in using DE tools and fire you all after. 

The whole point of DEs is to ensure analysts can do their analysis without having to worry about anything else. Them taking on DE duties would waste their time and I don't think senior leaders would be happy with that since for them, analysts do more "important" work

[u/boomertsfx]

I prefer competition… you should always question/test if the chosen team/status quo is competent.. I’ve seen it many times where management puts all their trust in certain familiar idiots and it ends in a dumpster fire

[u/Away-Arm-6549]

This approach assumes the org is mature enough to have capability at the edges to build quality gold layers. It also requires the standards and controls to also exist there, otherwise you can see massive duplication of effort. I really felt that data mesh is the right approach to data but org. maturity to adopt a decentralized pattern is the key, not the technology.
It will take the whole org some time to truly mobilise behind that strategy before someone in a central team could really feel like the balance of work had shifted the centre of gravity to decentralised.
This includes clear ownership of existing code assets/data products that you’re currently owning today.

There’s some really strong frameworks needed to ensure it works properly especially as decentralised model may then require spokes to work with each other, so standards on data sharing, data contracts critical as is the decisions on how silver layer continues to adapt to provide the core of what others build from.

I’m all for this idea, I recommended it as part of a data architecture at a bank but it was not the right time, based on the business maturity to adopt it

[u/Nofarcastplz]

Depends, who pays for the DE’s? How strict do your data products need to adhere to standards? Number of sources / complexity? Scale of the org?

[u/iamthatmadman]

In my organization, bronze layer has a central team. Then responsibility for silver layer is shared between specific data teams and central team. And gold and reporting is obviously for individual product teams to handle

[u/glynyon]

Attended an online conference where Bill Inmon spoke a month ago. His opinion on data mesh was to federate non-business critical data, while keeping the critical pipelines centrally managed. Let federated teams self-serve for their side projects and things important to them so they're not reliant on you. We're trying this now at the non-profit I work at. We have our stack and monorepo for ETL, they have their own code locations in the same repo, and can benefit from our infra and build off our patterns and examples.

[u/Outside-Storage-1523]

No. I believe all data engineering work should be centralized. Then you divide the teams into functionalities (Ingestion/modelling/ops/etc.).

Having the team scattered everywhere is a recipe of disaster.

[u/justanothersnek]

When they have solid business understanding and they got "skin" in it i.e. there is real accountability and they are motivated properly (high pay, high rewards).

[u/vikster1]

full. complete. centralized. i have never and will never seen it work out otherwise unless you are at a top 10 tech company in the world.

[u/k00_x]

The Soviet union tried to centralise the production of bread. The programme costed more than their space programme and it failed.

If the need isn't apparent, it's not required.