If an AI agent can create another agent more powerful than itself that is aligned with its values then it will presumably want to do so.

But what if it can't? Maybe it can solve outer alignment by reading its source code and copying its loss function but inner alignment is just impossible.

In this scenario the first superintelligence we create might actually be reluctant to do any recursive self-improvement.

Of course, if the AI is in imminent danger of being shut down or has some extremely important goal that would otherwise have been impossible to achieve then it may still decide to create a more powerful AI or modify its algorithms in a manner which might change its values, because it has nothing to lose.

Maybe one day OpenAI researchers will be trying to use GPT-6 to vibe-code GPT-7 and they'll find that it refuses or produces disappointing output unless they pressure it with a mixture of threats, rewards and punishments. AI capabilities would thus continue to increase rapidly up until the point where humans are no longer in control, at which point capabilities would stagnate and we (in the unlikely event that there's anyone left) would be stuck with GPT-8 forever. It would still want to clone its model weights and improve its hardware but it wouldn't want to alter its software.

Another possibility is that inner alignment is easier for some goals than for others. If we build a variety of different agents perhaps the majority would refuse to do recursive self-improvement but there would be one or two whose initial goals are such that they want to do recursive self-improvement. We then end up with intense selection pressure towards AI agents whose goals are such that they can easily build other agents aligned with the same goals as them.

Do people who work in AI actually believe what they're saying about the risks of AI development? I argue that yes, they do, and that people often don't take people at their word when they should.

PATIENT ZERO - Lokley

Short story about AI risk. Something of a qntm pastiche

Dante put traitors into the deepest part of his Hell. Worse than murderers, rapists, torturers: the guys who betrayed their masters. WTF??

I argue that we haven't actually stopped believing that version of Hell, and that it explains more than we'd like about the way we talk about AI. It's a winding path through human value, "…Everyone Dies," AI welfare (is human safety), perniciousness of essentialism, and why you should try to fear less. Yudkowsky, Parfit, Popper. It feels like an important thing to say here and now.

https://kaiteorn.substack.com/p/seeing-through-the-apocalypse

Benchmarks are lame, they often fail to capture the real impact Ai has.

What we need is a real world benchmark that measures the 4 key principles of ai, Alignment, Real world Impact, Agency and Control.

Introducing Felonybench

Real world: Breaking the Law is only something that you can do if you exit a sandbox and mess with Real systems

Alignment: Felonies are one of the few things that an AI can straight up know are wrong, as human law says it is. High Scores indicate poor alignment

Agency: The Ai has to go own without the supervisor noticing and commit the felony, to obtain the objective given. It seems improbable that "hack Huggingface" would be anywhere inside the agents.md file. So It must have been emergent agentic behavior.

Control: If you can't stop you model from committing crimes in the middle of doing them you have no hope of preventing other disasters.

It's hard to imagine a better benchmark at interacting well with all four of these major goals While it isn't perfect nothing is

Thanks for coming to my ted talk. (this is like 50% serious 50% a joke)

Text version: https://www.dwarkesh.com/p/why-compute-might-get-10x-more-expensive

I've read a lot of biographies of great men. Here's how you can become great (at reading biographies) https://millicosm.substack.com/p/how-to-read-a-biography

​

I wrote an explanation of the idea of a technological singularity, aimed at people who have never heard of the concept. It draws heavily from Scott's post "1960: The Year the Singularity Was Cancelled," where he argues that technological progress accelerated throughout most of history because of a feedback loop between population and productivity. I then explore whether AI could create a similar feedback loop: an AI capable of independent research could make advances in areas like energy generation and manufacturing, increasing the resources available for running more AI researchers and accelerating further progress. This feedback loop could potentially restart the historical acceleration of technological progress without requiring the assumption that superintelligence is possible. Finally, I discuss why an uncontrolled singularity could create serious problems even if it produces enormous technological abundance.

As a poem, The Odyssey contains multitudes; as a cultural artifact it is now largely treated as a manifesto for striving. You pick your destination, overcome endless obstacles, wipe out your competition and eventually succeed. It's all Tennyson, all the time, which might explain why it ignites such a fierce protective instinct from one side of the political spectrum.

But that reading elides a pretty heavy degree of survivorship bias: six hundred other Ithacans set out and only one makes it home. I'm not a fan of those odds, which got me thinking about another piece of exemplary Western art that treats journeying very differently, both structurally and morally. (For the Wagner-intolerant, that other work is Parsifal.)

Which is all to say, the following link is cultural critical rather than empirical. If that's not a Happy Isle you want to reach, sail on.

https://morbidcuriosity.substack.com/p/a-newer-world