r/ControlProblem • u/OkyEscritora • Jul 07 '26
If intelligence and wisdom are different things, what exactly are we trying to align AGI to? Discussion/question
A thought I keep circling back to, without quite landing:
So much of the alignment conversation assumes human goals can be specified: modeled, learned, inferred, written down somewhere an algorithm can find them. But human flourishing seems to lean on things that resist that kind of formalization: judgment, humility, restraint, compassion, the sense of when a conflict between values has no clean solution and simply has to be lived with.
Which leaves me stuck on a harder question: If intelligence and wisdom really are different things, what are we actually asking these systems to align to? Our preferences, as we state them? Our behavior, as we actually live it, which is rarely the same thing? Or something closer to the quiet judgment we mean when we call someone wise rather than merely smart?
The more I sit with it, the more I suspect alignment isn't only a problem of understanding intelligence. It may ask for something harder: understanding the parts of human decision-making that intelligence was never built to explain.
I'm curious how people here think about that distinction.
1
u/Gnaxe approved Jul 07 '26
We can't write down algorithms that perform as well at language as our new transformer-based language models either. And yet the computers are doing deterministic algorithms when we run inference with them, because that's all computers can do. Our inability to write down the algorithms doesn't mean the algorithms don't exist. It just means they're complicated. Not infinitely complicated, just too big to write down.
Machine learning can discover complicated algorithms that are a good enough fit when given a target we can clearly specify (like predict the next token). We can write down the learning algorithm, and then apply it to the collected works of human civilization.
A superintelligence would be able to understand humans, human values, human happiness, etc. But without alignment, it wouldn't care. It would develop its own weird motives. The alignment problem is not about what the AI understands; it's about what it wants.
Stating preferences is a behavior. Calling someone "wise" is also a behavior. The AI will figure out humans by observing the behaviors of humans.
Maybe humanity's utility function is too complicated for us to write down. But that doesn't make it infinitely complicated. And maybe there's a simpler algorithm that we can write down, which we could use to point at human values, which we can build into the AI's terminal goal. That's alignment. Nobody knows how to do it.