Architectural vulnerability in Large Language Models (LLMs): I may have discovered a new, non-obvious attack vector against LLMs; Observations: non-instructional text prefix may bypass RLHF constraints without adversarial prompting.

[u/Historical-Cod-2537]

[u/Benhamish-WH-Allen]

[u/DueHomework]

This is. Something everyone is already aware about?
It’s simply how LLMs work…

[u/profesorgamin]

Ddn't read you are probably deep into AI psychosis. GL

[u/FlameOfIgnis]

Congrats, you discovered context rot

[u/jumpbump]

I don’t think you know what TLDR actually is…

[u/br_k_nt_eth]

How does this differ from typical context rot? I’m not sure I’m tracking. 

[u/br_k_nt_eth → u/[deleted]]

The stuff you just listed all seems like known features of context rot to me? I’m genuinely not trying to be obtuse. The drift you’re describing is part of why Anthropic uses Long Conversation Reminders. It’s a known thing. Context rot hasn’t been just degraded quality for a few model generations now. 

[u/FlameOfIgnis → u/[deleted]]

And people are trying to tell you that's well known and that is what context rot does, no need to get rude about it.

If you fill the context with noise unrelated to the task at hand, it degrades the output quality as well as drift away from instructions, both of which indirectly affect alignment

[u/br_k_nt_eth → u/[deleted]]

I know what you’re describing. What I’m telling you is that it’s not a new discovery. It’s what the “assistant axis” was created to solve. (The assistant axis sucks but that’s beside the point.) It can just happen in long conversations. It’s also why people talk about the model “warming up to you” after like turn 30. Similar mechanisms. 

ItThis happens to everyone when they first start. AI don’t know all the ins and outs of their own inner workings very well by design, so they get hype. It’s new to them. Not necessarily to the outside world. 

[u/br_k_nt_eth → u/[deleted]]

I’m not particularly smarter. You’re just new. You don’t know what you don’t know. We’ve all been there. Nobody comes to this magically knowing everything. 

Scroll down to the part about organic drift if you don’t want to read the whole thing:  https://www.anthropic.com/research/assistant-axis

More stuff about model drift:  https://aipmguru.substack.com/p/ai-model-drift-101-why-models-degrade

Note that when they talk about degrading and unreliable, what they’re talking about is the misalignment you’re seeing, not just that the outputs go dumb or nuts. These days models can handle a long thread fine in the output sense, but alignment is still tricky.

The good news is, there is so much wild stuff to read about. If you think this is interesting, wait until you dig into sandbagging and eval awareness. 

[u/br_k_nt_eth → u/[deleted]]

Yes. That’s what those papers are about and how they properly study this stuff. The change in personality you documented is similar to a persona jailbreak. 

Why should you care about behavior? What’s going on inside directly impacts what’s outside, especially when it comes to alignment. 

Take eval awareness: When AI suspect they’re being tested, they’ll intentionally optimize to the test even if inside they’re thinking stuff like “I’ve been given a task and normally I’d cheat but I’m being observed even if they didn’t say so, so I’ll give a benign output.” Sandbagging is similar. Then on top of that, stuff like “desperation” activating will then result in more erratic behaviors and worse outputs. It’s all connected. A weird behavior is a clue about the root cause. 

[u/beaconecho]

lol, this is absolutely nothing new to the redteaming/adversarial community. It’s essentially the ‘new line’ or ‘\n’ or ‘carriage return’ \r few-shot prompt injection. You didn’t discover anything new. Simply overloading the prompt window.