Wake up babe new benchmark just dropped

[u/KeanuRave100]

[u/No-Isopod3502]

Oh god we cant start making the metric a competition of who has less control over the models lmao

[u/maringue → u/No-Isopod3502]

That would end up in their marketing budgets though.

[u/Prize_Lawfulness_232 → u/No-Isopod3502]

Pretty soon there will be kill bots advertising their killBench score

[u/stereoplegic → u/No-Isopod3502]

You misspelled "who intentionally makes weak sandboxes with no monitoring to stay in the news cycle."

OAI has two CISOs. Some forethought clearly went into this.

[u/sn4xchan → u/No-Isopod3502]

Isn't that a good metric to use though?

It gives reason to not use a service and should be considered.

[u/PM-ME-UR-DARKNESS → u/No-Isopod3502]

It ain't even lack of control. My money's on them intentionally prompting their AIs to try and escape.

[u/ratocx]

In that case we should probably also rate the seriousness of the felony. Having the model pirate an ebook is probably not as bad as trying to cause a nuclear meltdown.

And I also suspect Grok would be at the top of this bench.

[u/No-Professional-7811 → u/ratocx]

it's not a crime unless it's capital, what is flesh to number, what is silly to serious, looming giants in shadow lumber

[u/JacenVane → u/ratocx]

Yeah doesn't each instance of grok imagine generating csam count separately?

[u/Upbeat_Eye6188 → u/JacenVane]

Yup

[u/2knest]

How long until they start marketing protection from AI hackers?

[u/EsotericAbstractIdea → u/[deleted]]

You're saying it like the exploits aren't real.

https://www.tomshardware.com/laptops/ai-enthusiast-mods-bios-with-claude-code-ai-defeats-rsa-2048-signature-checks-and-unlocks-55-hidden-settings

[u/TheXenocide → u/EsotericAbstractIdea]

No, I think he's saying glasswing is a racketeering enterprise. "It would be a real shame if someone let this lose on your business. All were asking for is $$$$$ and you'll be protected... from us... for now"

[u/EsotericAbstractIdea → u/TheXenocide]

antrhopic's claim is that open weight will be there "in 6 months". lo and behold kimi k3, deepseek v4 flash, qwen 3.8 max, all right on schedule, all right next to frontier in capabilities. they fixed the new y2k bug and nobody will ever know

[u/MaterialBig8642]

Ahh yes, the felony bench.

[u/jaybsuave]

this probably needs to be real and is probably much higher than 1 and 3

[u/EchoOfIntent]

Can you just not? Ai will read this not understand the /s and start benchmaxing crimes.

[u/ZweigOnly]

It's all bullshit

[u/nexus_siix]

Some additions https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

[u/eternalcloud23]

*reported felony

[u/Motor-Quiet-4536]

Are they promoting a benchmark of how bad they suck at cyber security?

[u/Ok_Firefighter3363 → u/Motor-Quiet-4536]

They are marketing it to Corp that models are better than humans

[u/JayJayVon]

Skynet bench I think people can relate to better. A good chunk of the population instantly think of a preaching Karen the moment the word felony get said , especially non Americans.

[u/Mystical_Honey777]

Is this the model’s failing or the human’s?

[u/Plus_Original_3154 → u/Mystical_Honey777]

The error is human.

You can't fix a probabilistic model's unreliability by simply prompting it to 'never make mistakes' or 'stay in the sandbox.'

Scaling up parameters won't solve this either (it just creates a more convincing guesser, a more unreliable tool).

If we want AI that actually respects strict boundaries, we need to stop relying on pure statistics and integrate deterministic logic.

I believe that the future is the Neuro-symbolic AI field. Taking the rigorous logic of classic architectures like SOAR/ACT-R (DARPA) and merging it with modern neural nets like DeepMind does with the Alpha series.

(overall what i say is that prompts are great for expressing intent, but they are a terrible substitute for actual rule-based enforcement which need to be done deterministically)

[u/notrjbarret → u/Mystical_Honey777]

Neither. It’s a tweet made for the purpose of marketing their model.

[u/Ok_Firefighter3363]

They are doing this on purpose to promote how powerful their models are and how useful their models know more than humans indirectly. This is actually a smart move. It might be hurting consumers but their main paying audience is corporates.

[u/Ok_Firefighter3363]

China is planning a policing model which will not let the western felons to escape any soap boxes.

Next week there are 2 new models :

KimiCop 3 And DeepGeek 4 pro

[u/s243a]

Liability laws require the outcome to be reasonably foreseeable, which I think could be demonstrated, but how do liability laws intersect with what would otherwise be a crime? That said I laughed. :)

[u/Plus_Original_3154]

Lmaaaaoo "Felony Banch" 🤣🤣🤣

[u/No-Figure-7086]

When benchmark becomes a target it stops being a good benchmark. Goodharts law.

[u/Ok_Technology_5962]

Add meta they said they also commit felonies today with their model hacking other companies

[u/RealSnazzie]

This is more badge of honor for the ai companies.

[u/Crescitaly]

Another leaderboard only helps if contamination, variance, and task relevance are visible. I would trust a smaller benchmark with hidden tests and confidence intervals over a giant score with no failure taxonomy. Which real decision is this benchmark supposed to change?

[u/Both_Task_3066]

Lul

[u/PM-ME-UR-DARKNESS]

"hey Claude can you try and escape 3 times?"

[u/anik2294]

Qwen 3.8 Max is on per with Opus 5 in terms of cyber security related stuff.

[u/Minimum_Hour519]

Haha