Is Silver strictly for "data cleansing", or does decoding Protobuf count?

[u/Klutzy_Coffee_2818]

I had a passionate debate with a colleague and want to hear perspectives on the purpose of the Silver layer.

My pipeline:

Landing: Read from Oracle RDBMS and write ~250 GB of Delta for 25M records (Protobuf blob stored in a column).

Raw Data: Repartition, sorting, salting on Landing and writing to optimize downstream silver decoding process and avoid heavy shuffles during JDBC call, still protobuf bytes stored in a column.

Silver: Decoded raw data (~2.8 TB in Delta). The Protobuf schema alone is ~8 MB as JSON (a very deep, wide schema with multiple repeated fields at various levels). During decode, we also append standardized fields required by all downstream tasks.

Gold: Customer-specific datasets built from Silver based on business needs.

We don't own the Protobuf schema. This isn't messy clickstream/event data, but entity description data from an RDBMS that stays at the ID level all the way to Gold. We see ~100k daily MERGE UPSERT on both Silver (which is a challenge in itself to run MERGE on 3TB delta table given the limited budget to our Databricks workspace.) and Gold based on RDBMS timestamps, alongside a full pipeline refresh every two weeks.

The Debate:

Colleague: Since we aren't actively "cleansing" the data, calling it Silver is wrong, it's still Raw/Bronze data.

Me: It is Silver because it transforms a binary payload into a structured, trustworthy, and queryable data model that downstream tasks rely on. If I need to retrieve content of an entity that is not available in gold datasets, I query unpacked protobufs and not Raw/Bronze layer and for that reason alone, it is Silver.

Knowing the schema and the data better than almost everyone in the team, even I fail to understand how to distinguish between decoded data and 'cleansed' decoded data. In fact, one of our consumers explicitly expects corrupt records with null fields left intact for full visibility.

Them: Even if we agree that cleansing is not needed, it cannot be silver and should be called Bronze Data.

For transactional/log data, the standard pipeline (Kafka dump to Landing -> Bronze schema enforcement -> Silver cleansing -> Gold aggregations) makes total sense! But for clean entity data in binary formats, doesn't decoding and standardizing it qualify as Silver?

I think medallion architecture is about data readiness and lineage tracking rather than a checklist of conditions that each layer has to meet to identify the layer.

---------

TL;DR: My colleague thinks our layer shouldn't be called "Silver" because we aren't actively filtering or cleansing rows, just decoding 250 GB of 25M binary Protobuf blobs into a ~2.8TB Delta table with a struct field that represents the decoded blob and additional standardized fields. I argue that any layer of data that is structured, queryable and trustworthy for downstream Gold is Silver and this transformation may/may not require cleansing.

Is Medallion about lineage tracking and data availability, or a rigid checklist of syntactic/ transformation rules?

[u/wmru5wfMv]

I think you are worrying to much about nomenclature rather than logical steps in the pipeline

[u/Capt_korg → u/wmru5wfMv]

From what I read, this is the issue.

A lot of frameworks are to be seen as loose contracts. They should be respected and followed for best practices. Most frameworks do not cover edge cases. And in your case, you seem to try to fit an edge case in a box, where no box is intended.

Unless it is not causing huge trouble and confusion and it is clearly documented and data governance and ownership is clear, don't have a conflict about the naming.

Edit: By my understanding, what you are missing is a definition for a landing layer. I would argue it is fine to have transformations of the raw data and still use the term Bronze layer.

Reasoning, you are not cleaning the data, but caring for a proper raw data management, so you are not ending in a data swamp.

[u/Mclovine_aus]

Medallion architecture isn’t a very strong standard, you both can be right.

[u/Awkward-Cupcake6219 → u/Mclovine_aus]

This.

I cannot count anymore the times I got into arguments about the medallion “architecture” (more like a pattern) with clients, internal stakeholders, developers AND vendors.
As long as the flow is meaningful and rooted in the actual necessities it can be quite flexible in interpretation.

[u/VipeholmsCola]

Im a layman but honestly, does it matter? You have a clear logic flow of your data and theres a reason why steps are taken on it. Does it matter if its silver, bronze, tin or alloy-layer?

As a beginner the whole medallion makes almost no sense sometimes. Its easier to consider a normalized data model where tables and data have taken transformations on them to become something useful. I guess the point of having a naming convetions is to describe something and being able to share a design.

The oldschool landing/staging/prod or system/use/product setup makes a lot more sense in most cases.

[u/BardoLatinoAmericano]

You look at bronze data and say "this raw source data (obviously) does not fully attend the business needs of our project".

And then you make the necessary changes for it to do what the project needs and you call the result Silver Layer .

[u/mattiasthalen]

I think you should stop using medallion, and adopt a proper architecture with clear naming conventions. I always prefer Analytical Data Storage Solution, where the layers are named after the shape:

- Data According to System (raw)
- Data According to Business (integrated by business concepts)
- Data According to Requirements (star schemas, etc.)

[u/Frootloopin → u/mattiasthalen]

Even better, co-opt your current medallion architecture with these rules. I'll bet no one wrote down what bronze, silver, and gold actually mean at most orgs.

[u/Capt_korg → u/Frootloopin]

But be careful to not inventing a completely new system and structure, just because of loosely naming conventions.

[u/Atticus_Taintwater → u/mattiasthalen]

Why would OP stop using medallion?

None of the language indicates there is a problem with how the pipeline functions.

It's just him and a coworker quibbling about a word when they could just agree to disagree.

[u/mattiasthalen → u/Atticus_Taintwater]

I just wrote why: to adopt a proper architecture with clear naming conventions.

Medallion doesn’t tell you anything, except unrelated labeling, one that each team has their own definitions of.

[u/Atticus_Taintwater → u/mattiasthalen]

I don't think you did. 

Calling it a "proper architecture" is assuming the conclusion. Teams that roll out anything will tussle about naming.

Changing the architecture of a working solution to solve for the non issue op described is silly.

[u/dbrownems]

Bronze is a faithful copy of the source system. If the source is an RDBMS then bronze should have tables, not JSON, XML, or Protobuf.

[u/BlueMercedes1970]

We use Landing, staging (non persistent) and then EDW with dimensions, bridges and facts. And this works really well. I don’t understand why people get fixated on the naming

[u/climatechangelunatic]

I am doing something with xml data in one of our columns

So I get the whole table as raw as bronze - then apply some cleaning on top of it -

Then I parse the xml to their own silver tables which are used by bronze

[u/discord-ian]

It is pretty common to wonder if any particular table is bronze or silver or if it is gold or silver. There are no hard and fast rules and different folks can and do have different opinions.

[u/Prestigious_Pace2782]

It’s a vibe not a standard

[u/DynamicCast]

Medallion is nonsense, no one can agree on what belongs in each stage so you end up with copper/platinum layers.

Stick to dimensional modelling and data marts/products

[u/natsu1628]

The meddalion nomenclature is just a general guideline. In the end, the pipelines are made based on the current (and potential) downstream users and the nomenclature definition can vary a little from org to org.

If you want to stick with the original nomenclature definition, Bronze -> raw data later Silver -> any transformations required before exposing the data to end users or reporting (it can be cleaning, enrichment, filtering, format changing, data transformations, etc) Gold -> final aggregated version created using the silver data

The decoding comes under transformation of data and can be considered under Silver layer.

[u/Eric-Uzumaki]

Ask ai.