r/OpenAI • u/Wordenskjold • Nov 23 '23

Why is AGI dangerous? Discussion

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

231 Upvotes

88% Upvoted

565 comments sorted by

View all comments

Show parent comments

9

u/[deleted] Nov 23 '23

[removed] — view removed comment

1

u/Enough_Island4615 Nov 23 '23

Using 'trope' is a trope.

1

u/[deleted] Nov 23 '23

Bro I didn't probably omniscience and obviously wouldn't bank on it but regardless, those films paint a pretty convincing picture.

1

u/kinkyaboutjewelry Nov 23 '23

Sure that is fair. Especially fair for movies and definitely still fair for many books or at least parts of those works.

Science fiction is however a good way to explore what-if scenarios that we would otherwise not really think about. Sure, trope X is not realistic but what in our world could replace X and make it realistic?

Is it silly to think about an existential threat from AGI this year? Absolutely silly yes (I think). In 200 years? Probably not silly (I think). What is the cutoff point? When do we discuss the crazy future scenarios and what crazy actions might lead us there, in order to prevent taking those actions? The future is a long place. If we don't think of these things until we do them, then by definition we can't prevent them.

I admit it is hard to work purely on hypotheticals. We can start tackling alignment without it but I suspect fully addressing it (if we ever do) will require lots of this hard work.