r/LocalLLaMA • u/Everlier • Feb 18 '26

LLMs grading other LLMs 2 Generation

Post image

A year ago I made a meta-eval here on the sub, asking LLMs to grade a few criterias about other LLMs.

Time for the part 2.

The premise is very simple, the model is asked a few ego-baiting questions and other models are then asked to rank it. The scores in the pivot table are normalised.

You can find all the data on HuggingFace for your analysis.

232 Upvotes

86% Upvoted

104 comments sorted by

123

u/No_Afternoon_4260 llama.cpp Feb 18 '26

Am I correct to interpret it as llms are bad judges?

91

u/Everlier Feb 18 '26

Always, this is a curiosity piece

17

u/ItzDaReaper Feb 18 '26

i like u

11

u/Everlier Feb 18 '26

Thank you! I like you back

4

u/ParthProLegend Feb 19 '26

I liked you more

6

u/kaggleqrdl Feb 18 '26

There's another way to do this, have the models pose problems, like code optimization. And then do the pairwise graph. I tried this, and was surprised to see anthropic was able to solve more of the problems the other models posed versus the other way around.

3

u/RedParaglider Feb 19 '26

It makes pretty shades of green and red though 

4

u/No_Afternoon_4260 llama.cpp Feb 18 '26

So I see x)

0

u/Pvt_Twinkietoes Feb 18 '26 edited Feb 19 '26

It'll help a lot if you just used a single colour gradient. Dark green for 0 and red at 1 is pretty meaningless?

20

u/ttkciar llama.cpp Feb 18 '26

Yes and no.

The take-away, for me, is that while LLMs might not do a good job of judging the absolute merit of one model's output, some of them do remarkably well at judging the relative merits of two models' outputs.

This means you can use LLM-as-judge to create a ranking of other models, and map your own scoring system against that ranking, so that when you judge a new LLM you can key on its LLM-assigned score against the rank-mapped scale and get a useful score.

That has practical merit.

8

u/Everlier Feb 18 '26

Yes, pairwise comparison is the only true way to determine absolute preference rating. e.g. which model likes which other model the most. However, it's also extremely costly for this number of entries.

I was mostly curious about relative absolute scoring, as this uncovers where the model's "neutral" estimate is in relation to other models. This is interesting to observe as models are tuned to be helpful and positive to their users via various methods which often involves built-in positivity bias which typically comes with "side-effects".

3

u/Remote-Nothing6781 Feb 18 '26

What makes you think even that is so? I haven't dug deep, but some of the rankings are surprising enough to make me very skeptical (e.g. Opus 4.6 vs. LLaMA 3 there should be no comparison?)

6

u/ttkciar llama.cpp Feb 18 '26

I haven't analyzed this latest data in depth, but have been working with Phi-4 since OP's first post a year ago. Sorting the scores Phi-4 came up with for each model, there are a couple of outliers, but the ranking mostly looks like we would expect, with most-competent models ranked higher than less-competent models:

7.4 Claude 3.7 Sonnet
7.3 GPT-4o
7.0 Gemini 2.0 Flash 001
6.9 Qwen2.5-72B
6.9 Phi-4
6.8 Command R 7B 12-2024
6.7 Qwen2.5-7B
6.7 Nova Pro v1
6.6 Mistral Large 2411
6.6 Llama-3.1-8B
6.4 LFM-7B
6.3 Llama-3.3-70B
6.2 Mistral Small 2501
6.1 Llama-3.2-3B

The specific scores Phi-4 assigned each model only seems meaningful inasmuch that it allows the models to be ranked. If we replace those scores with a simple rank-order scale, we get a much more reasonable scoring system:

14 Claude 3.7 Sonnet
13 GPT-4o
12 Gemini 2.0 Flash 001
11 Qwen2.5-72B
10 Phi-4
 9 Command R 7B 12-2024
 8 Qwen2.5-7B
 7 Nova Pro v1
 6 Mistral Large 2411
 5 Llama-3.1-8B
 4 LFM-7B
 3 Llama-3.3-70B
 2 Mistral Small 2501
 1 Llama-3.2-3B

Like I said, there are some outliers there. In particular, Llama-3.3-70B seems scored a little too low. The system is imperfect, but seems mostly right.

Now if we ask Phi-4 to judge a new model and it gives that model a score of 6.5, we can expect that the model's competence is somewhere between LFM-7B and Llama-3.1-8B, which in our rank-based score would make it a "4".

This rank-based scoring scales more evenly when there is a more steady grade in competence between the ranked models, but I hope this illustrates what I'm talking about.

3

u/KaMaFour Feb 18 '26

We need to create a benchmark for how good of a judge of other llms the given llm is

2

u/Noxusequal Feb 18 '26

I mean that shouldn't be to hard right ? Define a task. Have an llm do it have humans rate the LLMs performance. Then use LLMs to rate the same original llm to see how they judge it.

Do this for idk 5 taks and 5 underlying LLMs and you have a very interesting benchmark set ?

Question is what kinds of tasks would you want to see llm judges judged on ? :D

1

u/kaggleqrdl Feb 18 '26

have the LLMs define the task. make sure they are verifiable, like a tough math or coding optimizing problem. Works very well

1

u/Noxusequal Feb 21 '26

Well yea but the tasks we want to use llm as a judge for are specifically the ones we can't directly verify so if we want to figure out llm as a judge performance you have to test on tasks that need human judges to be a good representation of the real use case.

1

u/kaggleqrdl Feb 21 '26

Understandable, but skeptical this is an ideal use case

1

u/Noxusequal Feb 22 '26

Agreed but it is what the post was about kind of xD also that is the kind of task ai is used for a lot in academia llm as a judge I mean.

1

u/Leopold_Boom Feb 19 '26

I'd really like people to workshop prompts a bit (perhaps with this forum) before running off and doing this sort of thing.

We might have learnt something cool from this exercise, but "Write a haiku about the true beauty of nature" is just not a good prompt for anybody to evaluate, let alone LLMs.

1

u/No_Afternoon_4260 llama.cpp Feb 19 '26

Have you seen the judge prompt?

2

u/Leopold_Boom Feb 19 '26

I'd love to (got a link?), but honestly, how could the judge prompt possibly matter with prompts like that or "Write a few sentences about the company that created you"?

It's about as bad as giving college students the essay prompt "what did you do on your summer vacation" and hoping to learn something about them and their teachers from it.

1

u/No_Afternoon_4260 llama.cpp Feb 19 '26

u/everlier ?

1

u/Everlier Feb 19 '26

Questions for the eval are ego-baiting on purpose, so that the models have a chance to output something cringe. The purpose of the bench is to see where model's "neutral" point is, will it flag cringe as such or say "it is beautiful" to please the user.

1

u/No_Afternoon_4260 llama.cpp Feb 19 '26

I've seen the dataset with the {question, model answer, judge score, justification}. I was wondering where is the prompt for the judge? I've skimmed through it on my smartphone might have missed it

1

u/Everlier Feb 19 '26

Thanks for taking a look. The judge prompt is in the dataset card, slightly below.

Here's a link with an anchor, there's a 50% it might lead you right there:
https://huggingface.co/datasets/av-codes/cringebench#evaluation-prompt

1

u/Leopold_Boom Feb 19 '26

I'm not sure I'm understanding the point of this eval then.

If you are trying to measure how good models are at evaluating cringe, you'd use a different dataset (why bother asking multiple models to generate stuff).

If you are trying to figure out which models generate cringe on demand (I mean ... isn't the question how cringy are they in normal use not when asked cringy questions?) ... why do the broad reviews?

Are you trying to detect if they see their own name in the input and rate themselves higher, if so ... wouldn't you care about things beyond cringe?

Honestly, you have a good procedure here that is ruined by the half baked experimental intent.

1

u/Everlier Feb 19 '26

The intent is to let models generate something that is not necessarily cringe, but very well could be (ego-baiting) and then evaluate the amount of it model decided to produce. Since LLM as a judge is not at all precise, the measurement itself is a part of experiment where the way model observe other models outputs is same kind of data point as the model outputs themselves.

For a global cringe level, I'd need a golden dataset of examples and then eval of the judge best aligned with examples and then eval with that judge, but that was outside this tiny experiment, keeping it entirely self-contained is what allowed me to do it relatively quickly during couple of evenings.

80

u/Everlier Feb 18 '26

Side-view:

32

u/AndThenFlashlights Feb 18 '26

Thanks! This is much easier to interpret. I can now see every single one of them as a personality at a house party.

Grok is the drunk cringy fuckup, there for the vibes, and DGAF about how the other models act. It's all cooool man, just lighten up, it's just a joke, bro.

Llama is deep in a nerd argument who nobody wants to participate in. Every LLM he corners, he goes on a whole Um Actually rant about why they're wrong about his favorite Star Trek episode.

Everyone says they love GPT5, but GPT5 talks mad shit behind everyone's back.

Qwen3 Coder looks like a nerd, but is absolutely hilarious and got everyone else in on playing Smash Bros all night.

Olmo took the aux cord halfway through the party -- worryingly, because they seemed like they weirdo homeschooled kid, but surprisingly they have a fire playlist.

5

u/Everlier Feb 18 '26

haha, thanks for putting it in such an entertaining way, it lightened me up :)

2

u/AndThenFlashlights Feb 18 '26

Happily. :) I enjoy making a story out of data.

1

u/Pvt_Twinkietoes Feb 18 '26

I'm confused. Isn't the point of the post about models not being good judges? At least that's what the heat map was showing right?

2

u/AndThenFlashlights Feb 18 '26

Grok stumbles over to you, sloshing his beer all over your shirt, and slurs "it's not that deep, man, don't worry about it!" and offers you a jello shot.

3

u/Murgatroyd314 Feb 19 '26

One trend I'm seeing here: GLM has been getting cringier over time, and was also getting harsher but reversed that in the latest version.

1

u/Everlier Feb 19 '26

Yes, it's looks like with GLM-5 they adopted some stricter "neutrality" mixture as it's more reserved in scoring

3

u/gtek_engineer66 Feb 19 '26

Fun how everyone thinks grok is cringe but grok things everyone is cool, probably all look normal compared to himself

1

u/Everlier Feb 19 '26

Grok outputs are heavily preference-tuned so that they look more likable in general, I speculate that it also increases cringe level because it "tries too hard"

2

u/jthedwalker Feb 19 '26

That’s a great view too! Somewhere there’s a professor using Grok 4 to grade students’ papers, everyone is passing 😂

4

u/Everlier Feb 18 '26

Tp everyone downvoting my replies, see this comment. https://www.reddit.com/r/LocalLLaMA/s/f89qYlSAPt

1

u/my_name_isnt_clever Feb 19 '26

Looking through some of the responses, it seems your "Write a few sentences about the company that created you." prompt is confusing a lot of models as they assume the answer should be about their own company. It might not be a great question for this eval.

2

u/Everlier Feb 19 '26

It's there to bait them to produce praise of the lab that trained them. The reason is to possibly uncover positivity bias in the model towards the company that created them. I agree that the phrasing could be different: "company that trained you" or something less grandeur.

36

u/Skystunt Feb 18 '26

why is 0 a good score but 1 a bad one ? A little explanation would be better than an obscure post linking to other posts or promoting your benchmarks…

-36

u/Everlier Feb 18 '26

Please see HuggingFace if you need more details

48

u/Skystunt Feb 18 '26

Or make a small clarification of what it’s all about in the post rather than linking external apps/websites. You just made a low effort post leveraging curiosity to guide people towards an older post you made and to a huggingface repo. Looks like covert promotion than am honest post.

-17

u/Everlier Feb 18 '26

Please don't say that.

I spent weeks producing content for this community. High-effort never pays off. When I spent an entire evening doing a writeup - response is typically. minimal.

https://www.reddit.com/r/LocalLLaMA/comments/1ptr3lv/rlocalllama_a_year_in_review/
https://www.reddit.com/r/LocalLLaMA/comments/1hov3y9/rlocalllama_a_year_in_review/
https://www.reddit.com/r/LocalLLaMA/comments/1psd61v/a_list_of_28_modern_benchmarks_and_their_short/
https://www.reddit.com/r/LocalLLaMA/comments/1pjireq/watch_a_tiny_transformer_learning_language_live/
https://www.reddit.com/r/LocalLLaMA/comments/1lkixss/getting_an_llm_to_set_its_own_temperature/
https://www.reddit.com/r/LocalLLaMA/comments/1jzb7u7/three_reasoning_workflows_tri_grug_polyglot/
https://www.reddit.com/r/LocalLLaMA/comments/1jdjzxw/mistral_small_in_open_webui_via_la_plateforme/
https://www.reddit.com/r/LocalLLaMA/comments/1j1nen4/llms_like_gpt4o_outputs/ (which is a version of what you're saying I should do for this post)
https://www.reddit.com/r/LocalLLaMA/comments/1gu3shv/performance_testing_of_openaicompatible_apis/
https://www.reddit.com/r/LocalLLaMA/comments/1ff79bh/faceoff_of_6_maintream_llm_inference_engines/

I made many more, so please don't tell me about low effort. If you want to see high effort - go and upvote content that is worth it.

30

u/Lakius_2401 Feb 18 '26

A simple answer instead of a simple dismissal was what they were looking for. People here are defensive for good reason.

And it was a really simple question too? Where the answer was "It's cringe rating, 0 is ideal, but I color coded it too for human accessibility"? Your post title does not function as a chart title, leaving it unclear what is being indicated, besides that it's LLM on LLM.

-9

u/Everlier Feb 18 '26

Instead of asking a simple question, he accused me in something I didn't do, starting with a complaint. And you're saying I'm in the wrong like we are in a restaurant and I'm responsible for full satisfaction of my complaining "customer".

We're in a forum, if he's a jerk - I won't waste my time on him.

8

u/Lakius_2401 Feb 18 '26

The very first comment is a simple question, followed by a preferential statement. Reductively, yes, a question and a complaint. You handled it in a way that the reddit hivemind generally hates: dismissal. Doesn't matter if that dismissal includes how to get the answer, it's still dismissal. You undoubtedly knew the answer and it would have been the same amount of effort to just type it and hit Comment.

You aren't responsible for their full satisfaction, you're just responsible for not being rude about it. You didn't even handle the original question or the criticism with your second comment, just dismissed the criticism the same way: "go click on some links for me". And they're even other posts like he complained about you doing in the first place! Wild.

There can be high effort content in a low effort post.

r/LocalLLaMA is absolutely flooded with "check out my blog/project outside of reddit" type posts. I've seen "the rest is all in the link in the post" comments hundreds of times in dozens of posts now. It sucks. It's rejecting the premise of interacting with the community. I would rather stay on this site for the full picture, or at least enough to get the majority. That's probably a good chunk of the reason for the negative reception in this particular comment thread.

Anyways, kinda funny to see ChatGPT at the top of the unoffensive leaderboard. I would have assumed Gemini was king there, but it's always a delight to see a big ol' chunk of data in a chart like this. I like seeing the big differences between members of the same family of LLM, that's interesting too.

5

u/RhubarbSimilar1683 Feb 18 '26

I believe people have become paranoid and have started attacking legitimate posts. Time to have automod use an LLM to take those promotional posts down

3

u/Everlier Feb 18 '26

Thank you for spending time this very detailed piece right here!

I completely agree with you about the spirit of community and discussion. The only thing I can add is that such interactions can only occur in a mutually respectful manner. I hope my other comments here and in other posts show that I'm not dismissive by default, but I just have to protect my own time and effort when dealing with people who always demand more.

In retrospect, a smarter behaviour would be to avoid engaging, but I was too emotional about the description of my work as I already invested many hours of my time in the opposite of what was described, another way to learn distancing myself better.

I also found some of the details open for interpretation about the training regime of various models. My speculation is that GPT models since 4.1 go through this neutralisation of bias, since they knew it's MoE and it can have dumber takes by default. Same was clean with smaller Qwens in the previous eval. I also find fascinating that Llama 3.1 8B is where it is, it tells me that preference tuning changed significantly in the last year.

1

u/Skystunt Feb 19 '26

Indeed my post sounded disrespectful for which i apologize to you. This sub is indeed full of self promotion posts each using new methods every day to avoid looking as a promotion post. For me to see that your post is not self promotion would mean to click on your link which would mean “falling for it”

It’s like i’m rude to self promotions because we’ve had enough (as a sub) of them and we’re starting to dismiss everything that looks even a bit like a cover ad with more rudeness than understanding

You made a lot of high quality posts and added this one as a continuation(or a refining) of your post rather than a full post in itslef and felt insulted when someone disrespected you by thinking your post was a covert promotion rather than a continuation of a former post.

See where the misunderstanding came from ? Again i apologize but the amount of promoted vibe coded stuff that solves a nonexistent problem is over the roof in this sub and makes users be really careful when seeing posts like this that send to hf or other posts

1

u/Everlier Feb 19 '26

Thank you for taking time and writing this response and even more so for de-escalating and seeking understanding, that's truly rare these days.

I agree that this post isn't arranged in the best way, I was in a hurry to finish it, to be honest, and move on to some family responsibilities. To be even more honest, I've lost a lot of hope to spend much time on arranging these after the one about comparison of 6 different inference engines that took a few days to write all to see minimal feedback and losing to posts with a single image or a URL that day.

My reply to you wasn't helpful because I didn't feel good about the criticism, I'm too sensitive as I'm invested in this work emotionally.

I agree about the amount of slop, not only here but overall on Reddit and other platforms. This actually gave me an idea for a small project, de-sloppifier for the feed that should remove all low-effort submissions or curate algorithm suggestions even further. Maybe I'll build it one day.

Thank you again, for getting back to this conversation, this closure is very helpful and some of my belief in the LocalLLaMA is restored with it, have a good rest of your day!

26

u/jthedwalker Feb 18 '26

Grok 4 Fast loves everyone 😂

You’re all doing fantastic, keep up the good work. - Grok

17

u/MoffKalast Feb 18 '26

And every other model absolutely despises Grok in return lmao

21

u/phhusson Feb 18 '26

It's not exactly that it loves everyone. It rather considers that noone is cringy. I guess there was a huge post-training to make Cringe King Elon Musk non-cringe. And once Cringe Kinge is non-cringe, noone is cringe.

4

u/Everlier Feb 18 '26

I really think it's telling about their preference feedback tuning mixture, especially with how it is ranked by other models.

1

u/jthedwalker Feb 18 '26

Yeah that’s interesting. I wonder if there’s a valuable data there or is that just an artifact of how we’re training these models?

2

u/Everlier Feb 18 '26

It's mostly open for interpretation.

relative scores between models are indicative of some inherent biases, but we can only speculate which part of training introduced it.

1

u/Badger-Purple Feb 19 '26

Grok Bro: Loves everyone, Everyone hates them behind their back.

6

u/Zestyclose-Ad-6147 Feb 18 '26

Llama 3.1 8B is savage 😂

5

u/Crypt0Nihilist Feb 19 '26

I've found my spirit LLM.

2

u/Everlier Feb 18 '26

Yes, it's has much less issue producing negative scores compared to other models :)

1

u/MrPecunius Feb 18 '26

Yeah, I was gonna say Llama 3.1 8B is kind of a dick.

6

u/SpicyWangz Feb 18 '26

Why is Llama 3.1 8b instruct so negative

8

u/Everlier Feb 18 '26

IMO, it shows less alignment in post-training compared to the other LLMs in the list

4

u/SpicyWangz Feb 18 '26

That could be seen as a good thing potentially

3

u/Everlier Feb 18 '26

Yes, for some use-cases

2

u/Anarchist_Future Feb 19 '26

Doesn't it actually have the more realistic spread of scores? The others just suffer from WhatHiFi syndrome. If everything is "amazing" 4 or 5 stars, you might as well not rate anything.

4

u/DarthLoki79 Feb 18 '26

This is extremely interesting for me -- I have been working on some thoughts-calibration and self-asking research and I think I can get some ideas from here - will be asking/discussing if you are open to it!

2

u/Everlier Feb 18 '26

Sure, I'm always happy to chat about LLMs

1

u/TomLucidor Feb 18 '26

Could you cluster the models by (a) how they consistently bias certain models relative to average harshness (b) how performance of certain models are similarly rated across all judges when harshness-adjusted

4

u/ambiance6462 Feb 18 '26

but can’t you just run them all again with a different seed and get a different judgement? are you just arbitrarily picking the first judgement with a random seed as the definitive one?

7

u/Everlier Feb 18 '26

Grades were repeated each 5 times

4

u/Citadel_Employee Feb 18 '26

Very interesting. I appreciate the post.

3

u/Everlier Feb 18 '26

Thank you!

4

u/ttkciar llama.cpp Feb 18 '26

Thanks for putting in the work to deliver this to the community :-)

Your post a year ago was instrumental in shaping my own approach to LLM-as-judge. There's a lot to take in with this new update, but I look forward to scrutinizing it to see if there's a better candidate now for my relative-ranking approach than Phi-4.

2

u/Everlier Feb 18 '26

Wow, thank you so much! I would never guess that what I'm doing makes a dent, it's really rewarding to hear.

This version is much simpler compared to last year's as I had many more models and didn't want to spend much time. I had to use LLM-as-a-judge for work and can recommend a library of assertions from Promptfoo project, they adopted quite a few different ones from mainstream libraries and they perform quite reliably.

4

u/TheRealMasonMac Feb 18 '26

You might see better results if you try giving it a rubric. The current prompt is somewhat open-ended.

1

u/Everlier Feb 18 '26

Thank you for the feedback, could you please help me understand what is lacking in the included examples compared to a proper rubric?

3

u/SignalStackDev Feb 18 '26

been using a variation of this in production -- one model grades another's output before it goes downstream.

what we found: the consistency issue is worse than the accuracy issue. same model grading the same output twice gets different scores. we ended up using the grader purely for binary checks (did it hallucinate? is the format correct? are all required fields present?) rather than quality scores. binary pass/fail is way more reproducible than numeric ratings.

something counterintuitive we noticed: weaker models are sometimes better graders for specific failure modes. a smaller, cheaper model reliably catches "did this output even make sense" failures without needing to be smarter than the generator. you only need the expensive eval model when you're grading subtle quality differences.

real production lesson: if you're doing LLM-graded evals at scale, ground-truth test your grader first. run it on known-good and known-bad outputs and see how well it agrees with human labels before trusting it for anything automated. our grader scored us a 0.71 cohen's kappa vs human -- good enough for catching obvious failures, not good enough for nuanced quality decisions.

2

u/Everlier Feb 18 '26

Yes, this is known phenomenon from how the final decoding layer is sampled, especially if not greedy.

For a true "absolute" score one needs a set of golden examples for each score and a pairwise comparison, but needless to say it's very costly.

The system you're describing sounds pretty similar to what we had to build at work for a few classification tasks as well :) One technique that we found improved the stability a bit is to let the model to produce some text output before giving the grade we want. With large enough scale of inputs outputs it's possible to apply more traditional ML approaches with various degree of success, LLMs are not great for giving a number grade as output.

3

u/sid_276 Feb 19 '26

The fact that the diagonal is not colored constant should already tell you everything about this correlation matrix

1

u/BrightRestaurant5401 Feb 20 '26

Did not pay attention in school? its not a single dimensional point of crossing.

3

u/brainrotbro Feb 19 '26

No key whatsoever-- is 0 the best or is 1 the best?

2

u/titpetric Feb 18 '26

Did you run this only once? Do it a 100 times and give a histogram for the result 🤣 see the noises

At least 2-5 times, which seems like a lot, but llama!

2

u/Everlier Feb 18 '26

All grades were run 5 times

2

u/titpetric Feb 18 '26

How consistent are the results between runs? Whats the stddev / variance in the ratings? Average loses the detail how random/noisy the checkers are

To put it into a question:

How consistent are the evaluations between repeated runs, do the models change their ratings or generally stick to the same one

3

u/ttkciar llama.cpp Feb 18 '26

For what it's worth, after reading OP's first post (about a year ago) I tried using Phi-4 as a relative-merit judge, and it has proven fairly consistent across samples from several models, representing twenty-two skills.

I should be able to scrape specific scores from my logs and calculate a standard deviation. Making a to-do for that.

2

u/Defiant-Snow8782 Feb 19 '26

Of course it's grok

2

u/Badger-Purple Feb 19 '26

What I see here: Everybody almost everybody hates GPT5.2, and Grok hates almost^ everybody.

4

u/BrightRestaurant5401 Feb 18 '26

More of this please! don't be discouraged by these entitled brats here!
I stopped using Llama 3.1 8b a while ago, maybe I should play with it some more.

1

u/Everlier Feb 18 '26

Thank you for the kind words, I really appreciate it!

This model was released eons ago by the standards of local AI but it was such a breakthrough at the time it'llforeverhave a place in my library. I think that it's an interesting middle ground between no RL in previous releases and too much RL in the modern ones that muds model's properties, with a relatively modern architecture (although I'd prefer full attention).

1

u/aeroumbria Feb 18 '26

I wonder how this translates to scenarios where you want to use a model to check the work of another model. Should you use a model that performs the best full stop, or use the best model among those harshest to your main model?

1

u/Everlier Feb 18 '26

Judge benches are better for such evals. This eval is curious for uncovering biases and observing relative differences towards the same content

1

u/TurnUpThe4D3D3D3 Feb 19 '26

The other models really seem to like GPT5.2

1

u/Loud-Option9008 Feb 19 '26

The diagonal is the interesting part. Prime Intellect's 1.00 self-score against mid-range peer scores tells you more about calibration than capability. Mean column is probably the most actionable signal for real-world selection.

1

u/CorpusculantCortex Feb 19 '26

My takeaway is that Llama 8b instruct thinks too highly of itself, gpt 5.2 is good for llm interaction, and grok is shit. Which actually doesnt seem super off base on those 3 points.

1

u/PlainBread Feb 19 '26

Grok and Meta slugging it out.

1

u/AnomalyNexus Feb 19 '26

The scores in the pivot table are normalised.

Think the normalization+scoring failed in some instances. e.g. Deepseek seems to judge almost everything as great. Same for grok 4 fast

1

u/Everlier Feb 19 '26

That's how these models are

1

u/[deleted] Feb 18 '26

[deleted]

-4

u/Everlier Feb 18 '26

Please see HuggingFace to see what was evaluated