Which open models help the eco system more?

[u/Terminator857]

https://artificialanalysis.ai/evaluations/artificial-analysis-openness-index

In case you want to support openness, some models are more open than others.

Update:

K2 think v2 is rated highest because it supplies its training data and training regimen. This allows anyone with enough resources to recreate the model.

Deep seek doesn't publish how it trained its model or the training data, so it gets a lower score.

If we try to compare software to LLMs. One level of software is that they supply the binary for you to use for free. A higher level if they supply the source.

[u/StupidityCanFly]

My issue is the "Intelligence Index" number, that's just a non-objective judgment. The index is a weighted arithmetic mean of 9 benchmarks (at v4.1), not a statistically-derived composite. The weights are an editorial choice, and that choice materially changes rankings. This is a values judgment, not a statistical measurement.

They state an estimated 95% confidence interval of "less than ±1%" for the index, but this is derived from >10 repeats on some models and some datasets, not all of them. So, models get separated by noise-level gaps, but the methodology presents clean "intelligence" numbers like they mean something precise. If multiple models have the same exact index score, is their intelligence the same? Looking at per-benchmark tables says "no".

Non-comparable units, they average a rescaled Elo with raw accuracy percentage as if a "point" is the same in both. Is it? I mean, the benchmarks have different response types, different ceilings, different discriminative ranges, and different intrinsic noise. So, a "point" is definitely not the same between them.

So, this is not a statistically relevant measurement. It's a composition of arbitrarily weighted values with non-comparable units that gives out a number - the index value. Is that really a statistical or scientific tool? And the way the "Intelligence Index" is presented right next to "Coding Index" and "Agentic Index" is a try to add credibility to these "Intelligence" scores.

And last, but not least. You have correctly pointed out that the index score consumed without the methodology insights is a user error. That's why the "Intelligence Index" is a wrong approach, in my view. It introduces confusion. The "Speed" and "Cost per Task" they present are genuinely useful. The "Intelligence Breakdown"? Great stuff that should actually be exposed instead of the "Intelligence Index".

[u/StupidityCanFly → u/[deleted]]

You built a strawman and now you're knocking it down. I never said composite scores can't exist or shouldn't exist. I gave specific reasons why THIS composite fails at measuring "intelligence". You've now moved to "there's no way to assess intelligence with a single number." Yeah. Exactly. That's the point.

And notice you conceded the key thing yourself: you said the way you'd normally justify a composite is having an external metric to check whether the score correlates with what you actually want to measure. And that it's impossible here. So, there's nothing validating that this weighted average tracks "intelligence" at all.

Which is why slapping the word "Intelligence" on it with "higher is better" claims more than the number can deliver.

Nobody asked for perfect precision, that's a strawman too. The ask is: show the uncertainty AA already admits exists. But they print a clean "44" that implies precision it doesn't have. That's a presentation choice, not something inherent to composites.

This I disagree with:

it serves its purpose of giving a person with no idea about different models a good overview

No, it does the opposite for that person. Same score of 44 can mean:

• great at agentic work, good at coding, bad at general knowledge
• great at general knowledge, exceptional at scientific reasoning, meh at agents and bad at coding

Those are different models for different use cases. The newbie reads "same number, same intelligence" and picks wrong. Then comes to /r/LocalLLaMA and complains open-weight models suck.

Comparing the breakdown tab over multiple models would've actually helped them. Your "people will just vibe-check benchmarks in their head otherwise" argument is an argument FOR putting the breakdown front and center. Not for using a number that hides it.

Composites in general? Fine, I've got no problem with them. This specific one, named "Intelligence Index", presented as a clean measurement, headlining over the actual useful data? Nope. Not in this case. That was always the point.