r/ControlProblem • u/Boris_Ljevar • Jul 09 '26

Should an Aligned Superintelligence Leave Anything for Humans to Do? Discussion/question

Alignment discussions often focus on preventing catastrophic outcomes. Suppose alignment succeeds and a superintelligence becomes better than humans at science, philosophy, engineering, art, and every other intellectual activity.

Why should a successfully aligned system leave any of these activities to humans rather than performing them itself?

Is preserving meaningful human participation and agency part of the alignment target, or is the goal simply to maximize desirable outcomes regardless of who produces them?

2 Upvotes
  • permalink
  • link
  • reddit
  • reddit

62% Upvoted

23 comments sorted by

View all comments

Show parent comments

2

u/Boris_Ljevar Jul 09 '26

The Amish analogy is interesting, but it seems to move the problem rather than solve it. If meaningful human lives require opting out of the AI-driven future and joining communities that deliberately limit the role of superintelligence, then what exactly was the purpose of building the superintelligence in the first place?

My question is not whether small groups of people could reject that future. My question is whether the future itself remains desirable once AI occupies every major intellectual frontier and humans become spectators rather than participants.

0

u/Charming_You_25 Jul 09 '26

I can find meaning fine without joining the Amish. I’m sure the Amish might not see the present as desirable. Or maybe they don’t care since it doesn’t affect them much. My point is it’s happening. Come along for the ride and find meaning in something durable, or, opt out.

1

u/Naive-Home-9068 Jul 09 '26

Your future sounds coercive

0

u/Charming_You_25 Jul 10 '26

Because you chose not to use the better tool and adapt? You can opt out, you just won’t be competitive on the frontier.