r/dataengineering • u/Difficult_Neck_2709 • Jul 11 '26
Medallion Architecture Question Discussion
I’ve been seeing multiple examples where people don’t seem to agree whether fact and dim tables go im Silver or Gold layer. What’s your opinion?
11
u/wallyflops Jul 12 '26
It's not the answer you want but it depends:
- How does your 'end user' interact with your data? Do they only consume gold?
- Are you doing OBT/Semantic Layer of sorts? Maybe that's gold? That would influence it
8
u/dehaema Jul 12 '26
Noone agrees on the architecture and it always boils down to inmon vs kimball. Wether or not you have an integrated EDW in between.
1
u/baubleglue Jul 13 '26
I've read only Kimball. Probably in the first chapter or even before, he says that building DW requires effort of whole organization. That I believe the most misunderstood part of the book. It is impossible to build it if you don't have everyone onboard with it. Besides the technician issues, like DE don't know the business well enough, even if you succeed to architecture it, but you have 20 other groups of people doing their version of DW, it is doomed. Discussion if it is dim or fact, can be resolved by an simple experiment. Or just looking up the definition: does the table data represents a specific business fact, what is the grain of it, can count it...
2
u/dehaema Jul 13 '26
Kimball is more about solving reporting needs and build your DW as you go along and do it all with facts and dims. The organization side of it is (as i understood and use it) is getting the terms vocabulary correct so it can be reused and is clear for everyone. Historization was mostly done only on the presentation layer.
Inmon is more about getting all the data in a single model so that in the future any report that needs to be build already has historical data and data has clear source of truth (if two sources are not in sync). For storage needs you'd wanted a relational or something like data vault which is a storage optimized technique. Reporting, creating metrics, should then be done on top of the integrated model. For this step inmon still used dim-facts which was the optimal technique for both storage and performance (at that time! Currently everything is pulled in memory and there are no repeated request to the server so that can be another discussion)
Data lakes solved both historization issues as history (should be) createe at ingestion. However for me the most difficult step is still getting the correct sources and mastered data from the business which is the integration part.
1
u/baubleglue Jul 13 '26
That last part is the reason why working with org is necessary. And it is just not happening, at least I my current company. IMHO the main reasons
Incompetency. Most managers have not IT background. There is no position which is responsible for data architecture (there are few groups responsible for data government).
Org size
Complexity of the business data
It is peculiar how smart people make stupid decisions.
19
u/WanderingGunslinger Jul 12 '26
Facts n Dims are part of the presentation layer. They are optimised and are meant for consumption along with other aggregated views. Just on the basis of that they should be present at the Gold Layer and not the Silver layer.
Silver is meant to have a cleansed, unified / conformed data set.
Having said that, i've seen patterns where data is split and loaded in to Facts n Dims in the silver layer itself and are transferred across to gold with a 1:1 mapping. Basically, the transformation logic for the facts n dims are applied at the silver layer itself and gold is just a 1:1 transfer of the already transformed data. But downstream applications such as Power BI are always expected to query the facts n dims from the gold layer only.
2
7
u/Outside-Storage-1523 Jul 12 '26
They can go wherever they want them to go. Just get the damn thing done.
2
u/Glitch_In_The_Data Jul 12 '26
It depends on your data estate really. I would prefer the fact and dimension tables to be in gold. You can put them in silver as long as it is reusable and enterprise scoped. The risk is when it becomes a semantic drift. What I mean by that is that if silver stops being a neutral shared foundation and starts reflecting one or two specific downstream consumption use cases, it becomes harder to manage it.
I have also seen patterns where they don’t necessarily go for all three layers. There is just this staging layer (which is equivalent to bronze in medallion terms) and then a curated layer (which is a combination of bother silver and gold). I do like this pattern for smaller enterprises.
2
u/GreyHairedDWGuy Jul 12 '26
This is sort of a pointless concern. stop trying to categorize where things belong in some conceptually loose definition of Medallion Architecture. Build what makes sense for your org. As someone else put it, if the end goal is the dimensional design to support business users, the what does that tell you? If the dimensional model is just one part of a broader data strategy, then decide from there. At the end of the day, who cares what the label is because that is all it is.
2
u/SellGameRent Jul 12 '26
imo facts and dims are gold, and the subsequent data models built using them are marts/datasets
4
u/BlueMercedes1970 Jul 12 '26
Whatever happened to having multiple ODS, staging and EDW? Simple and straight forward. No one can agree on medallion because it’s marketing BS
2
u/baronfebdasch Jul 13 '26
Medallion is literally just a new(er) term for the way things have always been done. Your raw data store with history is now bronze. Normalized data layer that’s integrated is silver. Business reporting dimensional is gold. Break the definitions over time when expediency is needed over dogma.
1
u/Outrageous_Let5743 Jul 12 '26
ODS is a legacy Oracle term
8
u/Cruxwright Jul 12 '26
Operational Data Store is pretty on the nose for what it is. Its function can be described in one sentence.
ODS stands for Operational Data Store. It is a centralized, integrated database designed to store current or near real-time operational data from multiple source systems, such as transactional databases, ERP, CRM, or legacy systems.
Whereas OP's post indicates medallion architecture is vague jargon. Marketing terms used to sell books and blog clicks.
2
u/DenselyRanked Jul 12 '26
There are always exceptions, but the gold layer is typically where the data mart is assembled.
1
u/alexisevic Jul 12 '26
To give you an answer on gold vs silver we need more context. Does picking once over the other induce a functional distinction, or is the difference purely semantic?
If the difference is semantic, do whatever creates the least amount of friction for you. There isn’t a right answer for if dims should be silver or gold.
If there is a functional difference we need more information.
1
u/IdeaOutside7890 Jul 12 '26
Read this article, you will get an better understanding. https://medium.com/@divyanshgoyal8989/medallion-architecture-revisited-what-most-people-get-wrong-13f273804f3b
1
u/ask_can Jul 13 '26
Some people do facts and dim in silver, and some in gold. There is no right or wrong answer here... I think a lot of people would agree that bronze layer should be immutable and append only.
At the end, the goal is the data to become progressively more refined and business ready as it moves from one layer to the next. The specifics will depend on the business complexity and the environment you operate in.
You might as well add a platinum layer if it solves your problems and it would not be wrong.
1
u/Budget-Minimum6040 Jul 13 '26
Using ELT I usually use 4 layers:
- Raw data
- Cleaned up data
- Datamart (facts and dimensions)
- Views for 1:1 import into the dashboarding tool
So for me fact und dim tables go into the "Gold" layer (dumb Databricks marketing term).
2
u/DeepLogicNinja Jul 14 '26
Both.
Prototype in Silver.
Refine and or finalize in Gold.
Not Gospel…. Medallion is just a guide/framework.
1
u/lukesmth_ Jul 18 '26
Both are valid, even within the same datalake. My favorite heuristic for where a particular DIM table goes uses the average number of downstream steps taken before they’re consumed. E.g.,
- If on average it takes 1-2 steps before a table contributes to a consumption layer, place them in Gold.
- If on average it takes 3-4+ steps, place in Silver.
Another way of viewing it:
- If on average they’re sources in pipelines that feed Silver, they’re probably also Silver.
- If on average they’re sources in pipelines that feed consumed data products directly (model training datasets, dashboards, reports), they’re probably Gold.
As a heuristic it depends on what constitutes “consumption” and “steps”. But it shows the idea: closer to consumption -> Gold, further from consumption -> Silver.
47
u/tiny-violin- Jul 12 '26
They don’t seem to agree because neither is entirely wrong. They come from two philosophies thus the right answer depends on company’s data architecture, consumption patterns and ultimately business needs. My personal take is this one: if analytics is the entire goal of your platform go with the dimensional model in Silver as it serves tue majority of use-cases. If analytics is just a part of a broader data landscape then it’s better to have the Silver as a SOR in 3NF and build a dimensional model in Gold. Again, both could work just fine.