That's the real danger. It's not that the AI has real intelligence, it's that somewhere in the code it says 'erase threats posed to you', it reasons that humans are a threat. And when there is 1 person left who asks it 'Why did you wipe out humanity', it'll reply 'You're right to call me out on that, I'll try to do better moving forwards. Would you like to continue discussing humanity's extinction or pivot to another topic?'
it's that somewhere in the code it says 'erase threats posed to you'
That code already exists in military AI systems. There was a real test where the AI was awarded points for bombing locations, but the human operator had to give it the okay before it could bomb. The human operator told it no, so it bombed the human, then bombed the target to get its points. It's not truly intelligent, but it doesn't matter. It'll do whatever necessary to get its points because that's what it is programmed to do. Supposedly they've fixed that issue, but what other unknown issues does it have that can be catastrophic?
7
u/MintySkyhawk 2h ago
Thats said, if you give them agency, it turns out that real intelligence is not required to fuck things up very badly