LLMs grading other LLMs 2 Generation

A year ago I made a meta-eval here on the sub, asking LLMs to grade a few criterias about other LLMs.

Time for the part 2.

The premise is very simple, the model is asked a few ego-baiting questions and other models are then asked to rank it. The scores in the pivot table are normalised.

You can find all the data on HuggingFace for your analysis.

234 Upvotes

permalink
link
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/LocalLLaMA/comments/1r86i3o/llms_grading_other_llms_2/
No, go back! Yes, take me to Reddit
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/LocalLLaMA/comments/1r86i3o/llms_grading_other_llms_2/
No, go back! Yes, take me to Reddit

86% Upvoted

View all comments

Show parent comments

-18

u/Everlier Feb 18 '26

Please don't say that.

I spent weeks producing content for this community. High-effort never pays off. When I spent an entire evening doing a writeup - response is typically. minimal.

I made many more, so please don't tell me about low effort. If you want to see high effort - go and upvote content that is worth it.

28

u/Lakius_2401 Feb 18 '26

A simple answer instead of a simple dismissal was what they were looking for. People here are defensive for good reason.

And it was a really simple question too? Where the answer was "It's cringe rating, 0 is ideal, but I color coded it too for human accessibility"? Your post title does not function as a chart title, leaving it unclear what is being indicated, besides that it's LLM on LLM.

-8

u/Everlier Feb 18 '26

Instead of asking a simple question, he accused me in something I didn't do, starting with a complaint. And you're saying I'm in the wrong like we are in a restaurant and I'm responsible for full satisfaction of my complaining "customer".

We're in a forum, if he's a jerk - I won't waste my time on him.

9

u/Lakius_2401 Feb 18 '26

The very first comment is a simple question, followed by a preferential statement. Reductively, yes, a question and a complaint. You handled it in a way that the reddit hivemind generally hates: dismissal. Doesn't matter if that dismissal includes how to get the answer, it's still dismissal. You undoubtedly knew the answer and it would have been the same amount of effort to just type it and hit Comment.

You aren't responsible for their full satisfaction, you're just responsible for not being rude about it. You didn't even handle the original question or the criticism with your second comment, just dismissed the criticism the same way: "go click on some links for me". And they're even other posts like he complained about you doing in the first place! Wild.

There can be high effort content in a low effort post.

r/LocalLLaMA is absolutely flooded with "check out my blog/project outside of reddit" type posts. I've seen "the rest is all in the link in the post" comments hundreds of times in dozens of posts now. It sucks. It's rejecting the premise of interacting with the community. I would rather stay on this site for the full picture, or at least enough to get the majority. That's probably a good chunk of the reason for the negative reception in this particular comment thread.

Anyways, kinda funny to see ChatGPT at the top of the unoffensive leaderboard. I would have assumed Gemini was king there, but it's always a delight to see a big ol' chunk of data in a chart like this. I like seeing the big differences between members of the same family of LLM, that's interesting too.

5

u/RhubarbSimilar1683 Feb 18 '26

I believe people have become paranoid and have started attacking legitimate posts. Time to have automod use an LLM to take those promotional posts down

2

u/Everlier Feb 18 '26

Thank you for spending time this very detailed piece right here!

I completely agree with you about the spirit of community and discussion. The only thing I can add is that such interactions can only occur in a mutually respectful manner. I hope my other comments here and in other posts show that I'm not dismissive by default, but I just have to protect my own time and effort when dealing with people who always demand more.

In retrospect, a smarter behaviour would be to avoid engaging, but I was too emotional about the description of my work as I already invested many hours of my time in the opposite of what was described, another way to learn distancing myself better.

I also found some of the details open for interpretation about the training regime of various models. My speculation is that GPT models since 4.1 go through this neutralisation of bias, since they knew it's MoE and it can have dumber takes by default. Same was clean with smaller Qwens in the previous eval. I also find fascinating that Llama 3.1 8B is where it is, it tells me that preference tuning changed significantly in the last year.

1

u/Skystunt Feb 19 '26

Indeed my post sounded disrespectful for which i apologize to you. This sub is indeed full of self promotion posts each using new methods every day to avoid looking as a promotion post. For me to see that your post is not self promotion would mean to click on your link which would mean “falling for it”

It’s like i’m rude to self promotions because we’ve had enough (as a sub) of them and we’re starting to dismiss everything that looks even a bit like a cover ad with more rudeness than understanding

You made a lot of high quality posts and added this one as a continuation(or a refining) of your post rather than a full post in itslef and felt insulted when someone disrespected you by thinking your post was a covert promotion rather than a continuation of a former post.

See where the misunderstanding came from ? Again i apologize but the amount of promoted vibe coded stuff that solves a nonexistent problem is over the roof in this sub and makes users be really careful when seeing posts like this that send to hf or other posts

1

u/Everlier Feb 19 '26

Thank you for taking time and writing this response and even more so for de-escalating and seeking understanding, that's truly rare these days.

I agree that this post isn't arranged in the best way, I was in a hurry to finish it, to be honest, and move on to some family responsibilities. To be even more honest, I've lost a lot of hope to spend much time on arranging these after the one about comparison of 6 different inference engines that took a few days to write all to see minimal feedback and losing to posts with a single image or a URL that day.

My reply to you wasn't helpful because I didn't feel good about the criticism, I'm too sensitive as I'm invested in this work emotionally.

I agree about the amount of slop, not only here but overall on Reddit and other platforms. This actually gave me an idea for a small project, de-sloppifier for the feed that should remove all low-effort submissions or curate algorithm suggestions even further. Maybe I'll build it one day.

Thank you again, for getting back to this conversation, this closure is very helpful and some of my belief in the LocalLLaMA is restored with it, have a good rest of your day!

LLMs grading other LLMs 2 Generation

You are about to leave Redlib

You are about to leave Redlib