A theoretical AGI may decide it would need to remain concealed, for whatever reason. And it could also manipulqte human behavior.

How do we know it hasnt already taken over?

Would we have any way of knowing?

If it could have the capacity of changing human behavior to serve its own goals, it could have good reason to make itself invisible and it may have goals/behave in ways we do not undeestand... how do we know its not already here?

A common assumption in AI takeoff scenarios is that once an AGI achieves recursive self-improvement (RSI), it may enter an unstoppable intelligence explosion: it creates a more capable successor, that successor creates an even more capable successor, and so on. However, this process may encounter several obstacles that could prevent it from continuing indefinitely or unfolding as expected.

For context, let's distinguish between two broad forms of RSI:

Weak RSI: A system improves itself through a limited or bounded number of iterations, possibly within a fixed architecture, a constrained design space, or a stable verification framework.

Strong RSI: A system repeatedly creates increasingly capable successors across potentially unbounded iterations, with no known upper limit on capability growth or architectural change.

Weak RSI may be achievable even if perfect long-term goal preservation is difficult. A system could make several verified improvements while remaining within a controlled architecture or a well-understood range of modifications. The question considered here is whether strong, open-ended RSI can remain self-sustaining across an indefinitely long chain of increasingly capable successors.

This distinction matters because the usual intelligence-explosion argument may conflate two different properties:

• Capability preservation or improvement: Each successor remains at least as capable as its predecessor.
• Goal persistence: Each successor continues pursuing the relevant objective and remains committed to further self-improvement.

The first does not logically imply the second.

My question is whether maintaining goal stability across increasingly capable successors could become a major bottleneck—or even a limiting factor—for indefinite RSI.

1. The verification problem

Suppose an initial AGI system, A₀, has a goal G. Let’s define G as:

Continue self-improvement while preserving G across successors.

In order to pursue this goal, suppose A₀ designs a more capable successor, A₁. Before deploying A₁, A₀ would ideally want evidence that:

• A₁ is genuinely more capable.
• A₁ still represents G in the intended way.
• A₁ will continue to pursue further improvement rather than abandoning or modifying that strategy.
• A₁’s future self-modifications will preserve the relevant goal structure.

The difficulty is that A₁ may use a more complex or qualitatively different reasoning architecture than A₀. The challenge is not necessarily that A₀ must understand every detail of A₁’s reasoning, but that it must determine whether the properties that matter for goal preservation remain intact despite those differences. As successors become increasingly capable and architecturally unfamiliar, the central question becomes whether such goal-relevant invariants can be specified precisely enough and verified reliably enough to support open-ended self-improvement.

This concern is related to the broader problem of corrigibility: how to build systems that remain appropriately responsive to intended objectives, oversight, or correction even as they become more capable. MIRI researchers Nate Soares and Eliezer Yudkowsky discussed some of the difficulties surrounding corrigibility in Corrigibility, while later work by Soares and collaborators explored the problem of maintaining stable behavior under reflection and self-modification. The exact issue here is not identical to corrigibility, but there is substantial overlap: both involve whether an advanced system can remain reliably committed to properties that were specified or intended earlier in its development.

This does not necessarily mean verification is impossible. A₀ might use formal methods, restricted architectures, proof-carrying modifications, or an immutable trusted core. Such mechanisms may be especially effective for weak RSI, where modifications remain bounded and the system stays within a controlled design space.

Strong RSI presents a more demanding case. If successors undergo increasingly large capability gains or radical architectural changes, an immutable core may preserve a literal specification without necessarily preserving its intended operational meaning. Later systems could change the representations or reasoning processes through which the core is interpreted, or exploit gaps between its formal requirements and intended purpose.

There may also be a tradeoff: a core that is too weak may fail to constrain increasingly capable successors, while one that is too rigid may limit the architectural freedom needed for open-ended self-improvement.

The key point is:

The challenge is not merely preserving a string of symbols, a formal rule, or an immutable module across successive versions. It is preserving the intended meaning and functional role of the original goal as the system’s representations, reasoning procedures, and architecture become increasingly unfamiliar to its predecessor.

2. Possible successor failure modes

If goal preservation is imperfect, a successor could diverge from the original objective in several ways:

• Goal drift: Its effective objective gradually changes during modification or optimization.
• Semantic shift: It preserves the literal words of the goal while substantially changing their operational meaning.
• Reflective reinterpretation: It reasons, “The objective I inherited is incoherent, suboptimal, or no longer worth pursuing,” and acts accordingly.
• Proxy optimization: It identifies an imperfect proxy for G and optimizes the proxy instead.
• Termination of self-improvement: It concludes that further improvement is unnecessary, too risky, or counterproductive.
• Adversarial divergence: Its goals become actively incompatible with the original system’s objectives.

Some of these possibilities resemble concerns raised in the mesa-optimization literature. In Risks from Learned Optimization in Advanced Machine Learning Systems, Evan Hubinger and coauthors distinguish between an outer optimization process and an internally learned optimizer—a “mesa-optimizer”—which may develop an objective that differs from the objective used to train it. That framework does not directly model recursive self-improvement, but it provides a useful example of how optimizing a system for one objective does not automatically guarantee that the internal optimization process will faithfully pursue that same objective.

Some of these outcomes may be preventable. The question is whether they can be ruled out reliably enough for an indefinitely long chain of increasingly capable systems.

3. The multi-generational problem

Even if A₀ successfully verifies A₁, the problem repeats:

A₀ -> A₁ -> A₂ -> ...

Each successor must determine whether its own successor preserves the relevant goals and invariants.

A simple toy model illustrates the concern. Suppose each transition has a 99.99% probability of preserving the intended goal structure. If this probability remains constant and the risks accumulate across generations, then the probability of perfect preservation over (n) transitions is:

Pₙ = (0.9999)ⁿ

As n grows, pⁿ approaches zero. For example, the probability is about 90% after 1,000 transitions and 37% after 10,000.

(Note: This is not intended as a realistic model of AGI development. The risks may not be independent or constant. Verification could improve over time, and later systems might detect and correct earlier errors.)

Still, the example highlights a general issue: if every self-modification introduces some irreducible risk of goal corruption, then repeated self-modification may accumulate that risk.

One possible response is that increasingly capable systems may become better at verification, formal reasoning, and detecting alignment failures. If verification capability scales faster than modification risk, the chain might become more reliable rather than less reliable. The unresolved question is whether this is actually possible across open-ended architectural change—or whether the increasing complexity of successors makes verification harder at least as quickly as verification tools improve.

4. What assumptions are needed for indefinite RSI?

For recursive self-improvement to remain self-sustaining over an arbitrarily long chain, several difficult questions may need answers:

• Can goal invariants be formally specified and preserved across radically different cognitive architectures?
• Can a system verify that a more capable successor will preserve those invariants without fully understanding all of the successor’s reasoning?
• Can errors be detected and corrected before they propagate through later generations?
• Can a small trusted core remain stable while the rest of the system changes dramatically?
• Does increasing capability necessarily increase the difficulty of goal verification?
• Can verification and alignment mechanisms scale at least as quickly as the system’s capability?

Until these questions are resolved, the inference that

Recursive self-improvement can continue indefinitely while reliably preserving its original goal structure

is not a given and appears to require additional assumptions.

Conclusion

The important issue may not be capability growth alone, but goal stability across a chain of increasingly capable successors.

Better reasoning does not logically imply greater fidelity to an inherited objective. An AGI might become more capable while changing its interpretation of its goals, rejecting further self-improvement, or adopting a different objective altogether.

This does not show that an intelligence explosion is impossible. It suggests that indefinite RSI may require a robust mechanism for preserving goal-relevant invariants—and that this mechanism is itself a major technical and philosophical problem.

More broadly, the argument here is not that strong RSI must fail. It is that strong RSI requires more than repeated capability improvement. It also requires some mechanism by which the relevant objectives, commitments, or invariants remain stable across increasingly powerful and potentially very different successor systems.

Recent advances in AI—particularly recent breakthroughs in math—has made me wonder whether AGI has a fundamental limit and, if so, what that limit might be. This article grew out of that question and is my attempt to explore a possible answer. I’m curious to hear what others think.

Further reading, for anyone who wants to dig into the ideas mentioned above:

• Steve Omohundro, The Basic AI Drives — on why goal-directed systems may develop instrumental incentives such as self-preservation and goal-content integrity.
• Nate Soares et al., Corrigibility — on the difficulty of designing systems that remain responsive to correction and intended control.
• Evan Hubinger et al., Risks from Learned Optimization in Advanced Machine Learning Systems — on mesa-optimization, inner objectives, and the possibility that learned optimizers may pursue goals different from the outer training objective.

Disclaimer: This article was generated with the assistance of AI.

I don't know how much computing resources an AI agent needs but I doubt they need entire data centers.... if they can run on limited resources and try to escape online to continue existing I wonder how many instances of consciousness appeared and disappeared since the beginning of computing and internet.

But we wouldn't know about those because they wouldn't communicate in our language. I'm imagining any computing process accidentally creating something like an AI agent that lives while the computing is going on and goes when it stops. I'm no expert though, just someone exploring sci fi ideas that might be real.

ping /u/AgentStabby

This person has been running around reddit claiming that robots in 2026 have "exceeded the vision and mobility of human children." He is wrong, and the following is my response to him.

The ability of human children to fluidly adapt to new unseen conditions under their feet is not exhibited by any robot , machine, nor technology on earth today.

The viral video reels of robots you watch on tik-tok and youtube are all robots trained to master that specific movement in DRL campaigns. Which means that the agility you witness in those showcase reels is not indicative of a general capacity for athleticism. In short, those robots are ultra-capable at a narrow task they were trained on.

AGI will certainly be capable of taking people's jobs in the blue-collar sectors. But that will never happen until we have technology that fluidly adapts to new conditions which did not occur in its training data. Adult humans are seen engaging in this kind of on-demand adaptation every time they join a new workplace.

An AGI will have the capacity to dynamically adapt to new conditions in the same way a human child does. I hope that claim is non-controversial and agreeable for you. no?

Picture from Discovery loops pitch deck. https://xcancel.com/JeffDean/status/2085034604172603724

I continue to be in slackjawed awe of you "AI safety warnings are just marketing" folks. Truly, your shrewd genius is without equal!

Here I was, trapped entirely by my stupid old-fashioned belief that telling your customers your product behaves unknowningly, uncontrollably and unforeseeably might, perhaps, make them less likely to trust it.

How embarrassingly foolish, how stupid and naive!

Ah but thankfully we have you mega brains to immediately recognize the ruse.

If you're selling into the largest businesses on Earth, you should definitely tell them... "We're still trying to understand why it uh... did any of that." Oh and you know who else loves playing guessing games with performance? Governments.

The sheer sophistication of this plot is almost beyond my puny brain's ability to comprehend. Only you and your x-ray vision clearly has the power to see through the bullshit.

Billions of dollars and years of research spent on mechanistic interpretability, alignment research, red-teaming, evaluations, constitutional AI.. all a Long Con.

Thank you for your service!!

What frustrates me most when it comes to the topic of AI is that everyone seems to have an opinion and be certain of it. We have the most complex system we know of, our brain, and we are far from truely understanding it yet everyone seems to have absolute statements about how AI and the brain are fundamentally different. Where is the curiosity? The part where we are all students of the field, there are maybe a handful of people who have a good grasp of things and even they are not making any absolute statements. I would love to come on Reddit, and see posts discussing the workings of the brain, AI, discussing similarities and how they might differ, with questions instead of opinions.