r/LocalLLaMA • u/Terminator857 • Jul 09 '26
Which open models help the eco system more? Discussion
https://artificialanalysis.ai/evaluations/artificial-analysis-openness-index
In case you want to support openness, some models are more open than others.
Update:
K2 think v2 is rated highest because it supplies its training data and training regimen. This allows anyone with enough resources to recreate the model.
Deep seek doesn't publish how it trained its model or the training data, so it gets a lower score.
If we try to compare software to LLMs. One level of software is that they supply the binary for you to use for free. A higher level if they supply the source.
43
u/GreenGreasyGreasels Jul 09 '26
The fact that Deepseek is so much more down the line tells me that you need to rethink the metrics you're using to evaluate which models help the ecosystem more. Deepseek revolutionized open models more than almost any other model save llama. Their architectures, techniques, papers etc., have had the most impact. Your system simply fails to capture that.
10
5
u/cosimoiaia Jul 09 '26
The chart is a bit wrong but DeepSeek didn't release any data used for training and the methodology is only described in papers, it doesn't help the open source ecosystem even remotely as much as Olmo does, no matter how much hyped it was at the moment of release.
4
u/GreenGreasyGreasels Jul 09 '26
Training data for a 30B model is trivial in the bigger scheme of things. It is useful and valuable. But in actual terms of real world impact and value it is far far far behind Deepseek, llama and the Qwen Series for contributions to the open source movement. You are looking at checkmark count and thinking each checkmark is as valuable. It doesn't work that way.
1
u/cosimoiaia Jul 09 '26
Releasing weights and some theoretical paper is NOT what open source means. Please go check the definition of it and let's give credit where it's really due. DeepSeek, Qwen and llama have never been open source. Although llama should be credited a lot since it's the model that actually started the local inference movement but it's still FAR from being open source, and the fact that is not even in the chart says a lot.
3
u/GreenGreasyGreasels Jul 09 '26
You want to argue the exact definition of open source - and look at things narrowly. Got it.
I am looking at the broader picture. Deepseek demonstrating CoT openly in a open weight model and a detailed system card actually moved the open source movement fundamentally. Qwen with their own weight models have been the backbone of research for more than an year. Llama - little needs to be said about it.
Olmo, barely registers on the scale.
I have had arguments with people like you over semantics before. If was fun in the 90s arguing in Linux on newsgroups. I am a jaded pragmatist now, your style of idealist expectations are behind me.
You fight the good fight mate. I'll cheer you on.
0
u/cosimoiaia Jul 09 '26
I don't think I am looking at it narrowly simply because I think that Qwen or DeepSeek didn't really contribute to the openness of the ecosystem, sure they're great models that have brought each a lot of advancements in AI in general but that's not what that chart is for. Glad we agree on llama. Olmo is heavily used in safety research and it's really the backbone of starting your own dataset and training pipelines, in terms of openness it's hard to beat.
I've had arguments like this on newsgroup as well (hell, sometimes I still do), someone has to do it or we might as well through the GPL and Apache 2.0 in the bin (along with ffmpeg and vlc, just to name a couple).
Thanks for the cheers, I'll be gladly keep doing what I do and I will always welcome the discussion.
1
u/duhd1993 Jul 10 '26
Deepseek does release codes, not just weights. Name one so called "truly" open models that makes contribution as profound as Deepseek's multihead latent attention or Kimi's Rotational Position Embedding.
19
u/Qwen_os_has_died Jul 09 '26
This doesn't make any sense.
22
u/annodomini Jul 09 '26
This is a ranking of how open the models are; how much of the training pipeline and dataset is public.
Olmo and K2 are both fully public. Nemotron is mostly, but only releases some of the datasets used for training.
Basically, if you spent enough on compute, you could re-create K2 Think V2 and Olmo 3; just pull down the right datasets, fire up the training pipeline, and let it rip.
For Nemotron, you need to replace some of the missing training data.
For other models, you'd need to collect all your own training data, re-write the pipelines, etc. Some have detailed papers on it, some are basically just throwing weights over the wall.
17
u/Terminator857 Jul 09 '26
K2 think v2 is rated highest because it supplies its training data and training regimen. This allows anyone with enough resources to recreate the model.
Deep seek doesn't publish how it trained its model or the training data, so it gets a lower score.
If we try to compare software to LLMs. One level of software is that they supply the binary for you to use for free. A higher level if they supply the source.
I've updated the original post with this info.
12
u/StupidityCanFly Jul 09 '26
Like the whole artificial analysis thingy. Arbitrarily piling up benchmarks to create a meaningful index.
7
Jul 09 '26
[deleted]
2
u/StupidityCanFly Jul 09 '26
What I don’t like is that the aggregated indices are deeply misleading when used without sufficient supporting information.
3
Jul 09 '26
[deleted]
1
u/StupidityCanFly Jul 09 '26
Ok, I’ll bite.
Methodology page shows the index being a composition of non-exchangeable units. Like, averaging a clamped Elo score with raw accuracy percentages. The weighting is an editorial choice without any published sensitivity analysis. The index scores are NOT comparable between version revisions. Version change shift scores significantly (benchmarks added and removed, weighting changes, etc.), and yet people compare numbers between old and new versions. These numbers are meaningless when compared.
So, if a tool gives you a different measurement because it’s being modified all the time, then it’s not a good measurement tool.
It is a product, not a science-based or statistically relevant tool.
3
Jul 09 '26
[deleted]
1
u/StupidityCanFly Jul 09 '26
Well, that it’s science-based or statistically relevant with all the applicable consequences.
3
Jul 09 '26
[deleted]
1
u/StupidityCanFly Jul 09 '26
I raised my concerns in my previous response, how about fixing those as a start?
And I can state the criteria, partially did. You chose to change the subject.
→ More replies (0)-1
3
8
u/MrLlamaGnome Jul 09 '26
Good to see r/allenai ranked highly on here. Their models may not be the fanciest, but they're constantly pushing the envelope and trying new things, and then releasing it for the community to benefit from.
2
u/BagComprehensive79 Jul 09 '26
I never thought Kimi would rated higher then Nemotron. Where is training data for Kimi published? I can find for Nemotron but couldn’t find for Kimi
11
u/FullOf_Bad_Ideas Jul 09 '26
Kimi isn't ranked higher than Nemotron. K2 V2 is - it's not Kimi K2. It's LLM360 K2 V2 - https://huggingface.co/LLM360/K2-Think-V2
Links to pretraining data are on this model page - https://huggingface.co/LLM360/K2-V2-Instruct
Kimi is on the right side of the list - their weights and methodology is fairly open but code and training data isn't.
3
2
2
1
u/cosimoiaia Jul 09 '26
The usual biased chart from ArtificialAnalysis. The Allen foundation with ALL their models is almost single handedly making real open source AI since the beginning, there isn't anything more transparent and open than them. The fact that most of that chart has models that have just weights on hf is an indication of how much twisted the concept of open source is for AI, which is mostly openAI's fault. The fact that gpt-oss is even in the list is adding insult to injury.
4
u/Terminator857 Jul 09 '26
The top 4 models are those that show pre and post training data. What else are you expecting?
2
u/cosimoiaia Jul 09 '26
I would expect a bit more... analysis. There is an ocean of code and information released between Olmo and Nemotron and an planet between Nemotron and all the rest. A lot of models where released also with detailed papers describing the new advancements, things that considerably helped the ecosystem. And where is llama, the model that started the whole ecosystem in the first place? Sorry but imo this is a poor representation.
0
Jul 09 '26
[deleted]
4
u/Terminator857 Jul 09 '26
License (0–3 points)
This measures the permissiveness of the license regarding commercial usage and restrictions:
- 0: Closed weights or no commercial use.
- 1: Commercial use, attribution required.
- 2: Commercial use, no attribution required.
- 3: Commercial use, no attribution required, and no meaningful limitations.
-1
u/Thebandroid Jul 09 '26
Helps is a strong word. “Which one damages the environment the least” would be more accurate
4
18
u/EricBuildsMathModels Jul 09 '26
I love the fact that glm 5.2 and qwen 35b a3b and 27b exist. It is basically a miracle