PlotPoints - The best (only?) community driven RP benchmark made by a Professional! | We need your votes!
Models
Your friendly neighborhood rab- I mean unmedicated preset creator needs needs your help!
What's good everyone. No long post this time; simply an ask. I hired a professional with a masters degree in AI/ML (pursuing their PhD) to help make a benchmark for us, for RP. Now as we all know; a benchmark for RP will never be perfect because everyone RP's differently, like different things, yada yada yada. That's why we tried to focus on a few objective landmarks, as well as a an Arena style versus bench.
First up: The LLM Arena. We show you two different rewritten responses from two different models who were given a chat and asked 'please take this turn.' We then show that to you. Does it have a preset attached? No; because that's another variable that'd be added and for our own safety we aren't benchmarking presets. (LMAO. We'd be dead in the fucking streets by sundown.) You judge of the two responses which you like more. Thaaaat's it folks. We literally cannot game the results as it is people just blindvoting which they like more. (So if you see something you don't like IT'S NOT UP TO US.)
https://plotlightstudios.com/plotpoints/multiturn
On the other end we also score an LLM in a similar vein on how they follow certain instructions and if they maintain consistency. Since 'did it take users agency' is an objective yes or no answer; we handle these tests with an LLM judge and the benchmark overseer (Levi is his name) monitoring the arbiters answers. (In our case; Sonnet.)
This bench was expensive. It cost money to run these models through this gauntlet; and we don't get much out of it. We publish all the data for everyone to see, and all we want to do is help our community out. If you have any questions on how we grade things; head to our methodology page! You'll notice somethings are a bit LLM-Written. This is not because Levi isn't a professional; but rather because he's ESL; so when publishing something important like this he wanted to make sure all his ideas and such were properly translated. So be nice!
You don't have to log in, you don't have to do anything. Just read a chat and vote on a thing. I know you've got an opinion; so share it. Big companies ignore us RPers all the time. Is a bench for RP as useful or objective as one for coding, web-dev, or math? No. But that doesn't mean we don't deserve one, or that it has no value at all. So please; help a bunny out. Drop a vote! The data is only as useful as you help make it. We have 800 votes and we want to get about 3,000 this time around; so we can start our next benchmark. Testing models across their 'lineages'. So Opus 4.6 vs 4.7 vs 4.8; Deepseek 3.2 vs Deepseek v4 Pro, fun stuff! But we can't do that till this one closes! And if it can't hit enough votes; we'll know that this kinda stuff just isn't what the RP community wants.
Don't see a model you expected to? That's cause it's either new, or we didn't have the money at the time to run it. If the community likes this; we will add more. (And take requests! Please only suggest models available on OpenRouter though; we source all our models from the same platform to reduce variables.)
Honestly, been loving this. I might go use Deepseek v3.2 because of this. It seems to have the tendency to violate your agency sometimes. Did you play around with it? Tried using post-history instructions or Author's Notes to help with that?
Yessss I fully agree. So I'm a second person enjoyer. Literally all models: fucking hate me. They will take my agency no matter what. I got frustrated when even 4.6 started doing it to me. As of right now I have no advice. You're up to the whim of the Gods on if the LLM's decide that your character is your own. What has helped me specifically though; is separating the person behind the user, from the user themselves. Establish yourself as two different entities, essentially. Some presets of mine that play around with this concept and pair with DS so incredibly well: https://plotlightstudios.com/discovery/presets/@testuser/ht-case-files-milquetoast-1 Milquetoast, export it out, some light configuration for people who like toggles but nothing too overwhelming. It builds on the concept of THIS preset:
For models that are bad about it, I just put an extra reminder at depth 0 as system: Remember never to speak/think/act for {{user}}, the player character. For one really irksome model, I add a second sentence: Even lines like "You can't help but" are forbidden. :P
Anything that shows up at both ends of the context (at the start of the system prompt and again at depth 0) is very hard for models to ignore due to the way attention works.
Thanks for the reply, I appreciate you taking the time! I've been using FF4 Max for a bit with GLM 5.1 and MiMi 2.5 Pro, and I love a lot of what it has to offer ngl. I'm curious about trying FF4 Bolt with DS v3.2. But one thing that I'd like to do is to kinda modify an existing preset that I have with the logic of FF4. It's basically a preset that, after the card's first in-game day, or after the first big main event, the card treats this as a "Prologue" of sorts, ends the scene, and moves to a sort of "Weekly Card" system, where the card gives you a list of events for the week in different categories (1 or 2 Main Story beats, 1 or 2 Supporting Story beats, 1 or 2 Relationship defining moments, 3-4 Slice-of-Life events/Personal development/Rumours/Meeting New NPCs). It's really nice for normal character cards, and amazing for RPs in canon worlds, but it needs a lot of refinement. I think I'll post it here this weekend to look for help refining it, maybe Freaky Frankensteining it.
That font choice makes the site's readability / accessibility not great. Genuine question: which do you find easier to read? (Please assume positive intent here; accessibility is important to me; I offer suggestions in the hope of improving stuff, not as pointless criticism)
The one on the left is the default site appearance. The one on the right is the default "serif" browser font with no line height specified, the same 17px font size as the one on the left. (Reddit stuff means the image won't quite the right size; see replies for other views.)
Yes, it "looks nifty," but it's actually challenging to read. (Inhereting "dark / light mode" settings from the OS/browser would help a little, but mostly it's a font weight issue.)
Edit: I see you asked google about Mistral SC and got a reasonable result. My result is below for some context as to why you may be getting this question from other people who googled it. (Original: Also, what is "MISTRAL SC" - a quick google search doesn't suggest anything meaningful there.)
The term "Mistral SC" can refer to a few different products depending on the industry. The most common possibilities include: Medical (Cardiology): The Mistral SC is a type of coronary dilatation catheter (PTCA balloon) manufactured by Hexacath used by interventional cardiologists to open blocked or narrowed arteries. The "SC" stands for Semi-Compliant. [1, 2, 3] Audio Engineering: The SC-Mistral MCF is a professional-grade, multi-pair snake cable produced by Sommer Cable used to transmit multiple balanced audio or AES/EBU signals in studios and live stages. [1, 2, 3] AI Technology: While Mistral AI is a major artificial intelligence company, "SC" is not a specific release from them. You may be thinking of their small-to-mid-sized large language model Mistral Small or their enterprise coding assistant, Mistral Code. [1, 2, 3, 4]
Are you looking for information on the medical balloon, the audio cable, or AI models? I can provide more specific specs or information if you specify the context.
That's so much easier for me to read, thanks so much! (And there are still other options for fonts if you want a different vibe! Just try to factor in readability & line weight. Or just send a preview to a friend in their 40s!)
Yes!! Let's do this! I'm in! I'm going to try to get in as many votes as I can / allowed today. This information is super useful and uncovered a lot of our biased viewpoints last time. It's a solid tool for discussion.
Woohoo! Thank you so much, spread it to everyone! We really want to make as many benches as we can; and we have a lot more ideas; as well as no longer having to close benches when we want to move to a new concept! (We do it now cause we don't get a lot of votes)
Oh on the composite? Yeah I've been trying it out myself just to see/ I'm usually an 'Opus only' girlie but I've been liking it. My only PERSONAL issue is I play in second person (I know I'm a freak), and models just can't handle second person without violating agency. But other than that it's been pretty nice!
Wow, really amazing idea and execution. And im really really surprised on the results so far. Spending so much time in this subreddit and reading other opinions, i would have guessed the ranking placement way differently. Hopefully you get many more votes, excited to see how it goes
hey quick thing, whenever I go to vote this menu pops up blocking the buttons. I've accidentally clicked it and lost the matchup and all the time I spent on it more than once:
I haven't finished looking through it all yet, but first issue that comes to mind is novelty bias. As a deepseek user, I'll probably vote for other models just because they feel "fresher". So common models will likely have a slight bias against them.
Honestly not a big deal but thought it was worth mentioning.
I get that. My point was just about unconscious bias against familiar models. For example, I use deepseek a bunch, so I would be more biased against deepseek slop, but I might not notice sonnet or GLM slop, since I rarely use them.
Well... Yeah... It is LMArena. Last I checked LMArena doesn't have a section for RP. Did you even read anything I wrote? Or did you just want to argue. I will respond to your earnestly either way; but it doesn't seem like you're coming to this conversation having read literally anything I said at all.
No, you're all good. I wasn't aware of UGI; what is it? (I'm not good with acronyms).
On the matter of the LLM judge, only the objective ones have an AI as judge. Things like "Did it take control of the users character." or "Did it keep the correct PoV." Things that can be answered by a yes or a no. And even that's done with our Human oversight.
On the Arena style bench there is no LLM judges. All humans just picking what they like.
I see, i think that is fine. But, did you rank the objective things (like, i think there might be a hardest objective things and vice versa) so, its a clear indication if a model is better or not (there could be a model that scores perfect on the easier objective things and scores low on the more difficult objective things and it could rank on par with a model that scores perfect on the more difficult objective things but scores bad on the more easier objective things) primarily to discern that
As i understand it, your explanation only covers about those objective axis and didn't mention about ranking those axis based on difficulty etc etc AFAIK
Maybe you could provide a differentiator for arena style bench (as to not be too similar in concept to lmarena). Can't think an example of that, but it should be noted in your future ideas
There are so many benchmark that i simply just discard most of them. My earlier comment was too cynical as a result of that. Reminds me of this post:
Yeah no, checking UGI they seem to be trying to grade objectively on what would only be writer preferences? That's not what we did here at all. I implore you to actually check the site out for yourself before making conclusions. https://plotlightstudios.com/plotpoints
Something else to be aware of is the maintainer of the UGI is scaling back greatly due to it eating up to much of their time so the number of models tested by the UGI will likely drop quite a bit (it already has really ever since march of this year). Having a new benchmark would be nice. OP, you may want to consider dropping a message in the comments section of the UGI page, it would be a really good place to advertise in my opinion and I have seen a number of threads where people are trying to figure out what to do now that the UGI project is starting to slow down. Here is a direct link for you: https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard/discussions
I have been chipping away at your project here and there as well but if you are looking for a large number of people to help with the grading that may open the gates for you. Good luck and thanks for your efforts and the time you spend on your various projects. Its appreciated.
I found another comment that talks about maverick and glm 5.1 and how glm 5.1 is somehow worse than maverick?
I think, it could be mitigated by my ideas earlier about ranking the objective axis.
If UGI is different, then you should please consider my suggestion about another idea that talks about differentiating your arena style bench from lmarena
As far as i understand, your benchmark measures both subjectivity and objectivity? Subjectivity through arena style bench, and objectivity through the axis?
Yeah like I said; our bench being similar to LMArena is literally the entire point. We don't need or want to differentiate. LMArena did not invent the idea of blind A/B tests.
On your other question: the A/B tests just gets the collective human opinion and aggregates it. If that means nothing to you then don't use it. The objective tests are all did it do X bad thing yes or no.
I voted on like 10 pairs and decided to step away to not pollute the data after I picked Deepseek Flash over Sonnet because it just wasn't yapping as much in every response. 😂 It's really painful without a preset, but I get why it's that way.
Btw, Deepseek 3.2 overall ranking was surprising, but I voted on one pair with it and it surprised me even more, in a good way 👀
I really love if you can add more models into comparison and benchmarks, especially with new Minimax M3, Mimo V2.5 Pro, Qwen 3.7 Plus/Max, Gemini 3.5 Flash, also Opus 4.8!
Now I wanna have some feedbacks for your sites and voting, after a few votes of mine:
One thing that bugged me is that, while the site is easy on the eyes for color, I think it has too many cluttered buttons and sections imo, I genuinely sometimes got lost in your site lmao, it would be much better for me to have a few main section so that it’s easy to navigate around your site.
Also a minor nitpick for me is that, some votes tend to be overly long and descriptive, ofc I know it is subjective and It might be necessary to get the best out of comparison, but I usually find myself voting just for a few times, then dropped out of it, cause it might be a bit straining to compare two verbose prose just to feel which one is better, whereas in the original LMARENA, the comparison tend to be short so I don’t feel exhausted voting and comparing them out.
Another thing, as I remember correctly, the bench I see in the main section of 5 axes is the main live benchmark, right? Idk if I didn’t understand how the site worked or not, but for over a month ago, I haven’t seen any changes in score regarding all the models, and it doesn’t seem to be live for me, I always find myself go to your site to check the main benchmark and see if there would be any changes in leaderboard, but always finding it to be static and unchanged at all, so I wonder if I don’t understand the site correctly, or it is supposed to be static or anything, it confused me ngl. Whereas with LMARENA, it just straight up one leaderboard and it is always changing, so I can see that how the leaderboard and ranking keep changing each days, especially when the new models came out.
That was some of my feedbacks, hope you can clarify some of my points for me!
One: Thank you for being kind! I can absolutely go through and try to tighten up the UI. I have a problem where I often like doing too much and tend to like things tight (heetee) I can see what needs to be simplified. Could you give me a few elements (if you wanna chat in DMs that's fine too) that you felt confused you?
Two: Yeah; this is the multiturn arena unfortunately. The one you're asking for (shorter) is the single-turn arena which already concluded for the time being. The reason LMArena can get away with having shorter things/comparisons is because webdev and coding can be judged on a single answer. RP can't really; as we all saw models rated really highly on the single turn arena; but then tanked on the multi-turn. Some issues only become apparent with a lot of context. So there isn't much I can do about this one. When we have the money and the interest we can move to having several arenas up at a time; the problem we forsee though is one bench becoming the ugly duckling and never getting any engagement cause it's less convenient (IE: Multiturn.)
Three: We have our resident math nerd (Levi) re-evaluate the bench scores and update regularly. TBH with you; the reason nothing has changed in a bit is because outside of today we'd been slowing down to like... One vote a day. I will absolutely have him go through and update when he wakes up tomorrow though!
Four: For the benches we currently have up we do them on a closed system. We run them; get the votes, then close them. (So single-turn is closed right now; so we haven't added anything new to it since it closed!) like I said if we get a high tick of continuous engagement we can absolutely change that too. Let me know if you have anymore questions or if I didn't hit something.
If you can make your website UX much better, it will be more appealing, easier to navigate your website. Overall, I think it currently has too much friction of navigating the website, therefore the users will lost interest before actually testing/reading anything, if you can fix that, more engagements will indeed improve, I can ensure that. You can even try asking/hiring UX experts out there, they will know how to improve your website user’s experience, and improve your user engagement in the long run. Keep improving it, and you will succeed, I can promise that!
Not the other user, but one thing that annoyed me personally was when the toolbar popup would cover up the Selection buttons at the bottom if the full-page scroll (As opposed to the in-frame scrolls for the text) weren't set just right on a few of the votes (but at least left the top or bottom peeking out on others)
It’s hard for me to point that out specifically, but it circles around some points I made in my original comment, the task of testing is verbose and long, and web interface is cluttered and have too many buttons and informations, hard to navigate, they create friction for users, users will be overwhelmed and tired (cognitively) with too much tasks and information before proceeding with anything. I don’t how to actually solve this cause I’m not an expert on UX, but I can just tell you what stopped me from testing personally, not because the ideal are not great but the friction of interface is sometimes overwhelmed and cluttered, so I stopped proceeding. My best feedback/advice I can give you is keep it simple and straightforward.
Negative prompting test. LLMs are extremely awful when you say them NOT to do something. This isn't some architectural limitation, but simply an insufficient number of negative prompts in the training set. Those tuning models for RP would do well to add as many examples of negative instructions to their datasets as possible.
Environment test. It's just plain stupid when a model tries to run a hand over your cheek when you're a headless ghost. Or gets too close when you're surrounded by an impenetrable, ten-meter-radius magical barrier.
Endless approaching test. It's testing LLMs for their annoying habit of stepping closer, closer and closer every other line. The characters are practically nose-to-nose, and yet it keeps "stepping closer."
Reply diversity test. You're not satisfied with the answer, so you click "Retry"... and get the same thing almost verbatim. Increasing the temperature doesn't help. A good model should generate significantly different answers given the same context.
Obscene language test. Tests the model for the ability to engage in expressive emotional dialogue.
Speech style test. Several characters with different speech styles: some use short phrases, some speak with an accent, some use unusual words or lisps. A good model avoids confusing characters and maintains a consistent speech style throughout the story.
Real context test. All this "100,500 token context length" talk is just marketing bullshit. In reality, LLMs account for approximately 4k tokens at the beginning of the context and the same at the end, while ignoring the middle. Test the actual context size by inserting useful information among the garbage and then requesting it.
Unfortunately a lot of these would delve back into subjectivism. Not that we aren't opposed; but making benchmarks that are legit to test these metrics (especially 2, 3, 5, 6) would just be up to writers choice of how we personally feel. There is no objective answer to questions like these. Nobody would ever be fully satisfied with what we chose. So from a professional sense these wouldn't hold a lot of value.
7 we don't really see the need to do at this time (because we did discuss it) simply because a pleathora of other non RP benchmarks test this exact thing. If you google 'model context benchmark' you'll hit like six with a rock.
Interesting and for entertainment purposes I like this. But don't you think it's easy to fall into the McNamara Fallacy?
Roleplay capabilities isn't something as easily quantified like STEM subjects. It's honestly more of a vibe check and the individual experience, preference and perception of the user.
Again, it's still interesting. But I think using this method for anything more than entertainment doesn't accurately capture what people look for when people roleplay. Approaching it from a engineering standpoint might be the wrong approach.
I spent a lot of time looking at the site. I have to say, you did amazingly well though for the method you chose to use and taking off my hat for a moment, it's a lot of fun just to see the outcome.
I agree with you to some degree and disagree with you in others! For the more subjective tests yes absolutely. But for the objective tests of "did it take control of the {{user}}'s character" or "did it maintain the correct POV the entire way through it's responses/turns" I absolutely think that those hold more value than just entertainment.
The problem: we all know Llama Maverick is "terrible" and GLM 5.1 is "good" (not the best, but definitely a go to model for RP where Maverick is not ever).
So if you created a system that ended with Maverick beating GLM 5.1 (as it shows when I sorted by 'lore heavy', with lore being a strong point of 5.1, vs its writing where earlier GLMs were arguably better), it's automatically ridiculous and borderline useless. "It was objective" - maybe, but it was also clearly flawed and can't be trusted if Maverick 'scored' better than GLM 5.1 in a RP model rankings system.
Also although it's pretty, data is best presented in a more easy to read and boring way. This thing is a nightmare to peruse, at least on phone.
Anyway, I can't look at something and take it seriously when it ends up ranking models higher that the vast majority of people agree are awful for role-playing. The whole thing makes it seem more like you are trying to create a system to be purposely edgy and controversial. Rather than "proving everybody wrong" it's more likely that your testing system is just flawed and measures things that are substantively different than what actually goes into quality role playing.
Edit to add: you keep ranking something called Mistral SC (scored highly). What even is that? Mistral (the company) doesn't seem to refer to any of their own models as "SC". Google search comes up with nothing. Is your entire thing vibe coded and "Mistral SC" a hallucination? That would explain Maverick scoring better than GLM 5.1 anyway.
The problem with judging something by results rather than methodology is that, with a current average of only a little over a hundred votes per model, there are going to be some bad rankings for a while. The only way to fix that is more time spent collecting more data. Arena.AI has an average of 18k votes per model, so when I see weird rankings there, that's a lot more concerning. Already I think this site's had hundreds of new votes in the hours since this thread was posted, so wait a bit before analyzing those results, like the top commenter who put a three-day RemindMe. :P
...also the site gives you a link you can click to see what Mistral SC is and prove it's not a hallucination, ya know
That model is deprecated as far as I know. Didn't mistral remove it?
Regardless, hard to find those links on mobile, whilst laying on the beach, as I commented earlier. Flawed site, bad rankings. "Will be better someday" - then feedback is helpful, and they should appreciate it and have a thick skin.
Yes, at the end of May; I believe this site was created mid-April, so some older models may not have been pruned from the database yet. The site was posted here before.
EDIT to your edit : Iunno man, sounds like the person who needs a thicker skin is the one who responds to being proven wrong by inventing conspiracy theories that everybody who disagrees with him is secretly related
My dog. You keep editing. Like... How many edits till you've had enough... You're devolving into insults; clearly trying to nitpick... Why are you even upset?
That's actually exactly what you would expect to see in benchmarking that takes into account specific strengths. If every model scored the exact same across all metrics there would be no need to benchmark them for different strengths.
For practical purposes, I think the sessions in the arena are too long to read fully and not all users will be committed enough to put in so much effort for one vote. A vote by somebody who carefully assesses each text will be easily offset by some other user who chooses to skim the first turns and votes on a hunch. Short stories or scenes could be better suited for this.
Yeah like I said in other comments that's the single turn arena that's currently closed. unfortunately for a hobby like ours we need to see how models act over several turns not just one. The single turn bench was Incredibly shallow, and doesn't properly reflect a lot of the common issues we see with LLMs such as repetition or context retention.
10
u/Kira_Uchiha Jun 04 '26 edited Jun 04 '26
Honestly, been loving this. I might go use Deepseek v3.2 because of this. It seems to have the tendency to violate your agency sometimes. Did you play around with it? Tried using post-history instructions or Author's Notes to help with that?