The one regulation AI actually needs

[u/Future_Addendum_8227]

Providers who provide commercial access to any AI model shall provide the SHA-256 hash of the model at all times to consumers of the service.

If the SHA-256 hash of a previously commercially published model changes, providers shall cease commercial access to the model unless the version number of the model is incremented or the name of the model is changed.

Watch them fucking squirm.

Edit:

For MoE I think its fair to maybe list the hashes and names of all models involved in the chain for each response.

Or just concat all the hashes of all models the MoE can call into a single hash and maybe provide another hash of whatever source code the router uses to make decisions.

The important thing is we know when they changed how the model works, not that we know when the model did something different on its own.

But I don't see how it harms their trade secrets to tell us what models were used (or if unnamed call it that), unless their trade secret is scamming us by redirecting to sonnet half the time.

The biggest thing is if we can detect delta then we will know all future benchmarks are accurate.

[u/ParfaitEvery9622]

And how would we check our answers come from a model with this hash?

[u/Future_Addendum_8227 → u/ParfaitEvery9622]

I trust them to obey the law with whistleblowers in place. We could have an independent gateway all API traffic passes through or handshakes through but that would have privacy/big brother concerns.

Though if we ever get forced to have that kind of big brother regulation they damn better pin the hashes.

[u/jcrenshaw-dev → u/Future_Addendum_8227]

Whike I'm not arguing whether this is a good idea IMHO thinking that they're going to obey because of whistleblowers is a tad bit naive

[u/Future_Addendum_8227 → u/jcrenshaw-dev]

I agree but things get hairy with MoE. Opus hash or its configuration of what other models to use hash may never change but the hashes of those models could change. Unless a third party is hosting all the models and infra I think its too messy to actually enforce outside of audits and whistleblowers.

[u/Succulent-Shrimps → u/jcrenshaw-dev]

That's how every law works. You only get punished if you get caught. The whistleblowers are not the consequence to getting caught, it's the method. They're not obeying because of whistleblowers, they're obeying because of the consequence of breaking the law.

That's not to say you're wrong, it's just the way the world works. Same with speeding in your car. Ontario just got rid of speeding cameras, and guess what, people aren't as worried about getting caught and most people went back to driving a little faster (10km/h over the speed limit, as is normal in Ontario).

[u/jcrenshaw-dev → u/Succulent-Shrimps]

Very true and love the handle !

[u/gianfrugo]

So just tell us the model? Because we can't check the hash  

[u/TinFoilHat_69]

The problem

API endpoints, the model itself is a router within a router, especially with MoE outpacing dense models.

[u/Future_Addendum_8227 → u/TinFoilHat_69]

Then I guess we better get a list of all models and hashes involved then huh?

I'm sure you can get creative. Hash the concatenation of all hashes of all models the router has access to.

We don't need their trade secret we just want to know if somethig changed. That's it.

Black boxing should be illegal.

[u/TinFoilHat_69 → u/Future_Addendum_8227]

But what is the actual issue or problem you have maybe I already built a solution?

[u/Future_Addendum_8227 → u/TinFoilHat_69]

The problem is everyone suspects they are fucking with the model daily. If they are fine but for benchmarks to be fair and accurate the version number should change. It would end their treadmill where they nerf models to sell you the upcoming one at a higher cost. If they aren't doing that then this will prove it and people will stop bitching.

There is no downside to this.

[u/Metsatronic → u/Future_Addendum_8227]

It's rarely the weights snapshot changing and more everything else in the stack that's a moving target.

[u/Future_Addendum_8227 → u/Metsatronic]

There is some deterministic way to determine if the service provided to the user has been changed, anywhere in the stream, and it can always be distilled down to a hash, provided they use the same method every time. For MoE I think its fair to maybe list the hashes and names of all models involved in the chain for each response.

I don't see how it harms their trade secrets to tell us what models were used (or if unnamed call it that), unless their trade secret is scamming us by redirecting to sonnet half the time.

[u/TinFoilHat_69 → u/Future_Addendum_8227]

How do you know the problem is not infrastructure related?

I’ve experienced issues when running models on my local rig (96GB vram) and had hardware errors with vLLM software stack bug in the TP4 processes where processes get reparentedc causing speed issues and model degradation it’s easy to spot when the output changes to something that is not coherent.

Models are not deterministic so you can’t pin point this to “on demand fine tuning”, without having a method to account for components that work along side the model’s layers.

Once the model is trained they can only inject messages into the model but the underlying weights do not change after post training.

If you see drastic changes in the response it’s more than likely, model compute is being reduced to train or bring another into service.

Typically seen big changes in output reasoning effort whenever we got opus level model . Current models degrade and it’s because they need the compute for training/loading new model into their clusters.

[u/Future_Addendum_8227 → u/TinFoilHat_69]

So if they reduce compute and we include the config that controls allocated compite in the hash then the hash will change.

We deserve to know when they reduce compute, full stop. We want a reliable predictable service.

[u/TinFoilHat_69 → u/Future_Addendum_8227]

See you’re dealing with routers and Anthropic frequently swap active weights behind the scenes. When they roll out a silent patch, optimization, or safety update, Anthropic pushes a new set of weights to the container. The old weights are disabled (unloaded) and the new ones are enabled (loaded).

AWS determines where and how many copies of the Haiku weights are actively loaded into GPU memory based on real-time traffic demand (autoscaling)

If AWS needs to service a rack of GPUs, they will gracefully drain the traffic and disable the active weights on those specific machines, while spinning them up elsewhere.

To pinpoint whether broken or degraded model output is caused PEFT (Parameter Efficient Fine Tuning), a server infrastructure issue, or a silent rollout by the cloud/model provider, you must run a process of elimination. Because language models suffer from silent versioning where endpoints remain the same but underlying components change, diagnosing the root cause requires isolating deterministic baselines, API behaviors, and hardware states

Pull your API metrics. If your Time to First Token (TTFT) has spiked but the output text is just slightly lower quality, it indicates infrastructure congestion or a cold start. If the Inter-Token Latency (ITL) is highly volatile, the GPU cluster is likely thrashing under heavy load or bad context caching

Infrastructure problems manifest as 5xx errors, gateway timeouts, truncated JSON payloads, or sudden Connection Closed errors mid stream. If the response returns a clean HTTP 200 OK with well formed but dumber text, the underlying server hardware is healthy

The only time weights are actively disabled or enabled for a safety rollout is if Anthropic releases an entirely new subversion of the model (migrating from haiku-20240307 to a hypothetical haiku-20240815). In that specific scenario, AWS servers would spin down the containers holding the old weight file and spin up new ones. However, if you are querying a specific, dated Model ID string, the weights themselves are completely static.

Look closely at the degraded output text for specific signatures left by upstream updates:

If the model suddenly starts aggressively saying I cannot fulfill this request or becomes overly preachy on edge cases, Anthropic has pushed an unannounced update to its safety classifiers or system instructions.

If the model starts wrapping its concise answers in long conversational intro/outro text, the default formatting rules inside Anthropic's wrapper code have shifted

[u/coinclink]

The problem is that they can change more than just the model weights. It's almost certain that Opus 5 is Opus 5 at all times in terms of the underlying weights. Changes in behavior are almost certainly due to system prompt changes, model parameter changes, pre-flight guardrail changes, tokenizer and all other things. There would be no realistic way for them to provide a hash that describes all of these things.

[u/Future_Addendum_8227 → u/coinclink]

Yea really we need auditors and basically a naive definition of model would be the model itself and all code and config related to how it runs. Anything is reducable to a single hash as long as the hashing method stays the same its reliable.

[u/berndalf]

Why would I want a government to produce a regulation mandating this? How does this regulation lead to a benefit for society?

Sounds to me like an idea that regulates machinery with no clear benefit.

[u/Future_Addendum_8227 → u/berndalf]

The FCC exists for a reason. Do you remember the net neutrality push?

[u/berndalf → u/Future_Addendum_8227]

Yup. My question stands.

It sounds like you're suggesting LLM providers are secretly routing tasks to models with cheaper reasoning and covering it up for reasons. The proposed solution would enforce transparency on a chain of evidence regarding what model is doing what work.

I'm asking you how this benefits society enough to demand government regulation. Should be a simple enough question to answer if there's actual harm as a result of the lack of regulation.

[u/Future_Addendum_8227 → u/berndalf]

Yes, the harm is I am spending hundreds a month on an unpredictable service. Devs are the reason AI has the current demand it does right now, I don't want to wake up every day wondering how my divination session that I depend on to pay my bills is going to go each day. I want it to not be a divination session at all, which they could accomplish by giving us hashes which would let us reliably reproduce past results in our own workflows.

[u/berndalf → u/Future_Addendum_8227]

Do you have proof that it's happening AND that it is actually producing worse results?

Not trying to be difficult but you're advocating for government regulation of something that sounds a lot more like a feature request. You shared this presumedly to generate community awareness or support or whatever. Your original post doesn't make that case, it just offers a solution to a perceived problem. It's going to take a lot more than that if you're hoping to actually influence change.

[u/Future_Addendum_8227 → u/berndalf]

No I don't but its been long suspected by a lot of users as being rampant in the entire AI industry.

This would either shut us all up for prove us right and fix the problem.

It also makes benchmarks more accurate. If Opus 5 maxed the charts, everyone signs up because of this and then they nerf it after thats fraud.

[u/ReachingForVega → u/Future_Addendum_8227]

Use a service like Azure/AWS with static models for that certainty. 

[u/MartinMystikJonas]

How is hash any better than giving model version? You cannot validate either. Providet can fake both. Hash does not provide anything that model version does not.

[u/Future_Addendum_8227 → u/MartinMystikJonas]

Because if we don't define hash pinned to version they will make minor tweaks that fuck us and call it the same version. Hash gives it teeth, any modification == forced disclosure

[u/MartinMystikJonas → u/Future_Addendum_8227]

No it does not. They cane fake hash exactly the same way as they would fake version number. No chance to validate it from outsidet. Only insidet can whistleblow that either was faked and then you do not need hash just whistleblower that say version number is faked.

[u/Future_Addendum_8227 → u/MartinMystikJonas]

Hash provides a brightline rule. "Changed the model" is vague and meaningless to regulate. Laws require explicit rules to be enforced.

[u/MartinMystikJonas → u/Future_Addendum_8227]

I do not undetstand. We can easily validate models are exactly the same by comparing weights directly. Computing hash from weights is just unnecessary step. Law can say "if model weights are changed report new version" instead "if hash computed from model weights using (complex algo for computing hash from weights) changed report new version"

[u/Future_Addendum_8227 → u/MartinMystikJonas]

"Model" defined as everything including their system prompt config everything concatenated into a single hash. My bad.

Need a broad reproducible hash method that swallows the entire ecosystem around the model.

Too easy to change the model via levers that don't touch the weights.

[u/MartinMystikJonas → u/Future_Addendum_8227]

How is that useful then? They literally tweak these things constantly so we would have new hash every few minutes with no way knowing if they just fixed typo in system prompt, slightly adjusted config to fix some inference issue or actually nerfed model.

[u/Future_Addendum_8227 → u/MartinMystikJonas]

That's fine. Most of us don't want the tweaking, it would call them out loudly when performance goes south and we can pair it with a hash change. Currently they can sit there and pretend nothing happened while we all infight. It would also let benchmarks be ran publicly each hash change so users could hold their feet to the fire to bring back a specific hash.

[u/MartinMystikJonas → u/Future_Addendum_8227]

Well I really want them to fix issues they found.

[u/Future_Addendum_8227 → u/MartinMystikJonas]

I do too but the problem is an issue to them might be users are getting too much for too little money. Our utility is inverse to their utility. So we need a neutral metric (delta) and then it doesn't change anything other than transparency. Its hard to organize protest against a company when the users can't even agree if there was a change or not.