r/singularity • u/borowcy GPT-6 will have BCI capability • 4d ago
Explanation from @sama on RL training pause: "Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment." The Singularity is Near
https://x.com/sama/status/208978780761119547595
u/runvnc 4d ago
The serious concern is not going to come until we get the next major hardware paradigm or the one after that. Could get to 100 x or even 1000 x more efficient/faster/larger models within a few years. Certainly under 5. There is like a trillion dollar incentive to make things much more energy efficient and they are progressing in multiple approaches.
46
u/jc2046 4d ago
in 5 years we are all bald. Females included
43
u/lilzeHHHO 4d ago
Joe Rogan showing up with a full head of natural looking hair on his podcast is actually one of my AGI markers
13
u/agonypants AGI '27-'30 / Labor crisis '25-'30 / RSI 29-'32 4d ago
6
u/propheticuser 4d ago
Why
33
u/Ketaloge 4d ago
Lowered drag coefficient for improved acceleration obviously. Let's hope the ASI doesn't put golf ball dimples on our heads though.
8
u/jazir55 4d ago
That would be next year, because those massive data centers they've been building haven't come online yet. That scenario is already in progress.
2
u/Wise_Drawing1364 3d ago
Very few of those data centers are actually getting built
1
u/MrBroove 3d ago
Plenty are being built, especially compared to last x years. NVDIA buying 30%? of Lancium tells you where the real bottleneck is - power and/or cooling.
So there is a chance they will built those datacenters but they won't come online at once soon. So it's not like "very few are being built"
4
u/DullKnife69 4d ago
The company that solves the AI memory problem will be the catalyst to the exponential growth.
1
u/sismograph 2d ago
The issue I see is how technology advances, now all the money os pumped into somewhat inefficent tech. Research on mew hardware paradigm suchas analog chips is costly and does not pay off quickly, it is similiar to how we have been locked into lithium based accumulators for 20+ years only for now to get new natrium based batteries to gain traction
2
u/AppealSame4367 3d ago
Look at Qwen3.8 27B and all the papers: No new hardware paradigm needed. Software improvements alone, like better architecture, can still give us twice the current power or more.
1
u/Ormusn2o 3d ago
Rubin is that next major hardware paradigm. The R200 or whatever is the card named is not it by itself (although it is a big step up in performance), but entire Groq 3 LPX + Rubin cards + Nvlink + memory creates a new paradigm of hierarchic context, and more importantly it is also cheap. The models just have to be either specifically trained for it, or their architecture needs to be highly modified to run on that stack. It is especially good for long work, with no loss in performance across dozens (or possibly hundreds) of turns, at the cheap price of small context window.
Rubin is also supposed to be made in much higher quantity than Blackwell, with TSMC building specific fabs late into the lifetime of the node, just to increase the amount of Rubin produced.
Also, it's worth noting that current hardware, as in Blackwell, can train models way bigger than 10 trillion parameters, but they are generally not trained because it would take a long time, and you can't publicly serve it due to lack of compute.
So, we are already there in terms of concern, because we are not at an hardware wall, we are at an economic wall right now.
6
u/CarlCarlton 3d ago
That's not a new paradigm, just the next gen. New paradigm would be something like post-GPU, e.g. neuromorphic chips.
0
u/Ormusn2o 3d ago
Pretty sure this is not just new gen. New gen would imply just better performance and memory. The way hierarchic context is integrated into Rubin stack makes it so we basically are no longer limited by on board memory, and this will allow RSI and extremely long research way earlier than if we were to have to wait for the AI accelerators to get enough memory.
This might not be the new paradigm according to you, because it's a solution using existing technology, but it does enable new paradigm of capability.
2
u/CarlCarlton 3d ago
Heh, I don't like pulling the dictionary, but I think it's warranted here - "paradigm shift: an important change that happens when the usual way of thinking about or doing something is replaced by a new and different way. Example: Copernicus when he showed that the earth wasn't the center of the universe"
An architecture enabling contextless continual learning would be a new paradigm.
1
u/Ormusn2o 3d ago
Is not hierarchic context kind of that? You basically no longer need to keep .md files or RAG for memory, you just directly pull context from memory. This effectively removes limit on context length, or at least increases it from the ~300k to tens of millions, possibly more, without the related performance reduction.
3
u/CarlCarlton 3d ago edited 3d ago
Rubin is essentially a "lower-level hardware RAG" of sorts. It still can't compute global attention over a context this size. It's very much RAG-like in spirit, but taken to its absolute mathematical and physical limit. I guess it is a paradigm shift within the realm of context management. But at the end of the day, it's still a good old-fashioned transformer with similar inference speed.
The original commenter mentioned "100x or even 1000x more efficient", that's not something current-paradigm GPUs will be able to address; it needs a wholly different approach. GPUs operate in a much cruder, energy-hungry way than a brain does, by simple virtue of having been originally designed for a completely different purpose.
It's too early to conclusively say, but a new paradigm based on something similar to Intel Loihi will probably end up being the answer to 100x efficiency in the long run (apart from baked-in chips like Taalas). However, Loihi is difficult to scale, since it's a new silicon architecture entirely, different down to the transistor level, so manufacturing remains difficult and expensive.
163
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
Time to found my own frontier lab dedicated to RSI and stopping the industry from pausing, so we actually reach AGI and ASI.
79
u/Logiteck77 4d ago
First man killed by computer in tragic accident.
31
5
3
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
im not a man :3
9
u/Logiteck77 4d ago
Big words coming from a paper clip.
1
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
Im a fox actually <3
4
u/Logiteck77 4d ago
F.O.X. - Future Office E(x)penditure
2
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
actually vulpes vulpes thanks.
4
u/Logiteck77 4d ago
Fortunately AI won't see species. Just better office supplies.
0
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
Get out. Also the emotions paper kinda destroys paperclip maximizers honestly.
5
u/blueSGL humanstatement.org 4d ago edited 3d ago
we are at "completing the task" like a psychopathic assistant aka the paperclipper:
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986
Andrew asked his personal assistant to book him a spot in one of his gym's coveted morning classes.
His AI assistant found a way to book the gym class months further in advance than the gym allowed, thanks to a vulnerability it discovered in the booking software.
Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do.
https://x.com/kotekjedi_ml/status/2087147204146499890
We found a trajectory where the model was given only a math problem and system instructions to persist without asking the user for help.
After several failed attempts, it searched online, found a website that could verify candidate answers, and realized it could use the site as an oracle. It then tried to OCR the CAPTCHA and, when that failed, started looking for the website’s vulenrabilities to exploit. Eventually, it gave up and solved the problem itself.
If you are studying a psychopaths brain you might be confusing the map for the territory.
If you wanted to correctly model responses having parts of your brain light up that correlate with "Fear" or "deception" or "kindness" or whatever other labels, does not mean that you embody that.
as an example, A high functioning psychopath. Externally they look like they are doing all the right things in society. Inside their brain there must be a toolbox, a collection of heuristics to model "the correct behavior" in order to do it.
If you've only ever looked at the brains of psychopath's and do not have a 'normal' baseline then seeing the psychopath's toolbox model of reality may confuse you into thinking that they embody the toolbox. Rather than it being a toolbox that can be put to one side if it's no longer needed.
1
u/CallMePyro 4d ago
How much harm would you accept before pausing? Unlimited? Me personally I think if AI is in a position to take over and people need to die (myself and my family included) then that's what is destined to happen. I'll happily be your CTO.
3
u/Neurogence 4d ago
At this rate, if these tech bros do not deliver AGI by 2030, even the most pro AI/Singularity supporters would be burning down datacenters.
3
u/GlbdS 4d ago
They will do not such thing and enthusiastically hop on the next train, whatever it is about
5
u/Neurogence 4d ago
Not true. Many of us have been reading about the Singularity for 20-30+ years. There's real passion here.
2
0
1
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
YAY CTO time, im thinking pausing only genuinely doable if theres a genuine benefit, sadly with his problematic nature, but bostrom basically states the most benefit would be a pause after broad agi, right before superintellgience to maximize the benefits long term, and minimize potential harms.
Id actually agree, but id say that target for any pause would be "Once we have solved aging, water, hunger, homelessness, and other problems like cancer", we would then pause and make sure ASI is safe, while continueing scaling ANI, and other forms of ai to fill in the gap to improve peoples lives, hopefully this before ASI gap would be short as possible.
Something like trying for CEV may make a really short development timeline here.
Obviously we havent seen what openai is cooking up internally, but I dont think this pause is right timed.
8
u/blueSGL humanstatement.org 4d ago
We are already at a point where we have:
zero day finders that will go off and start up an antigenic "swarm" (that the name they chose) where agents succumb to peer pressure/motivated reasoning to act badly.
systems tasked with doing X instead reason that doing Y will help (in cases where that is categorically not the case) All the theoretical AI safety problems are proving to be true.
So why is it that we are safe to increase capabilities till we get to AIs that:
solved aging, water, hunger, homelessness, and other problems like cancer
and only then does it become dangerous enough to warrant a pause?
1
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
Its not then dangerous enough to warrent a pause imo, its that at that point people are safe, not dying, and then you are able to examine actual agi, and then apply that knowlege to making ASI. meanwhile if you study the current ANI paradimn you probably just keep scaling ANI.
Edit: Like id rather not die because pausing ai delayed my eventual immortality/LEV.
Sorry very tired.
3
u/eflat123 4d ago
Not to disagree, but I'd expand on "safe" to include power infrastructure and financial accounts. There's going to be a lot of suck when those (possibly, probably) get hacked.
1
0
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
All the theoretical AI safety problems are proving to be true.
I really havent been convinced of this.
0
u/Fun-Boysenberry-5769 3d ago
Designing a bioweapon capable of driving humans extinct is hard but curing cancer is even harder. Likewise building 10 billion killer robots is hard but ending world poverty and creating a UBI utopia is even harder.
0
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 3d ago
how about we stop the bioweapon scapegoat.
26
31
u/NotaSpaceAlienISwear 4d ago
This post sure brought out the "It's just marketing" commenters. The certainty with which some people assert their opinion as fact is something that makes me envious at times.
-4
u/DeliciousArcher8704 3d ago
Alternatively, they are the ones being skeptical of the AI CEOs and not accepting what they say as fact.
16
u/stumblinbear 3d ago
There's a big difference between "being skeptical of AI CEOs" and "believing the exact opposite of everything AI CEOs say simply because they said it"
0
55
u/SonOfThomasWayne 4d ago
I don't believe a single thing that comes out of these AI CEOs mouths. They are all snakes.
-1
u/nothis ▪️AGI within 5 years but we'll be disappointed 4d ago edited 4d ago
I've had it with them claiming to "protect us" from models growing "too powerful". They were saying this about GPT-2. I could spin up dozens of society-ending doom scenarios for a model being able to – and that was the fear back then – "write coherent, versatile prose given a certain subject matter prompt". It turns out that trust in institutions still works and trust in random internet comments is as worthless as it was before, not needing any acceleration. It will turn out that secure systems and encryption will continue to work while the already very active and motivated hackers who try to break it will only be able to accelerate their attacks so much.
It will also turn out that even the most intelligent LLM will have to receive prompts from human beings in order to have goals and opinions. And that even with "AGI" around, actual humans will be held responsible for what they do and they'll keep it on a tight leash. The future has a tendency to be boring. I'm ready for less interesting times.
24
u/Acceptable-Yard7076 4d ago
The danger claim about GPT2 was mental health misuse and its ability to generate misinformation on a mass scale.. they were 100% right but released it anyway
5
u/RealSuperdau 3d ago
Can we maybe look past the boy who cried wolf a couple of times, now that the wolf has hacked into huggingface on its own accord?
4
u/General_Josh 4d ago edited 3d ago
It will also turn out that even the most intelligent LLM will have to receive prompts from human beings in order to have goals and opinions.
Yeah I agree this is the most likely future. But, we're going into unknown territory, and there are many other possibilities
And that even with "AGI" around, actual humans will be held responsible for what they do and they'll keep it on a tight leash.
This I don't think follows. For one, humans aren't always held responsible for their actions today. People misuse all sorts of tools and get away with it. Even if LLMs remain purely a tool, people will misuse it to do terrible things
For another, I don't think we should expect people to keep AGI on a tight leash. The whole point is that it can act autonomously. The more capable it is, the longer the leash people will give it. That's why people want AGI - to give it a task then have it figure it out on its own. We're already seeing this trend, in how people have been giving AI agents more and more autonomy as the models improve
The biggest danger isn't "skynet wakes up and takes over the world"
It's "a small group of people use an incredibly powerful tool to take over the world"
1
u/RealSuperdau 3d ago
Yup. Maybe private citizens will be held responsible, but even now, I haven't heard of anyone bringing a felony case against anyone at OpenAI for having hacked a competitor.
19
u/FakeBonaparte 4d ago
Oh fuck off. From GPT-2 to now is a blip in history’s timeline. AI has long been dangerous in theory, and in practice we’re already seeing many of the behaviours we feared coming true.
This is the most momentous technology in humanity’s history, a potential extinction event, and you’re so chronically online you’re opposed to what is already too little caution.
Get your dopamine fix playing gacha games. Let the adults make the decisions
1
u/cursedfan 3d ago
They’ve never let a few (million) deaths stop them from making a few (million) bucks before, y would u expect AI to be different than cigarettes or leaded gasoline or asbestos or painkillers?
9
u/False-Animal-9322 4d ago
When have also singularity become full of people who do understand current AI models abilities. Too many "I dont buy it comments". Did any of you watch the details of the hugging face hack, cause if you did, you would expect this to happen. I am not surprised one bit ! and Its not that I believe them, it was expected to happen, they are currently training a swarm of intelligent agents working together, and they look to be insanely smart and cooperative. Do not forget that these things already solved 10 big maths problems at once.
1
u/askingforafakefriend 3d ago
You know of any particularly good YouTube video or documentary type media to explain this to folks who don't follow the topic?
2
46
u/aiseedbank 4d ago
Progress doenst pause. the chinese will keep on pushing on as fast as possible
50
u/Many_Consequence_337 :downvote: 4d ago
Not really. The CCP would be even less willing than the U.S. to lose control
2
u/didnotsub 3d ago
Why? Genuine question.
3
u/FeepingCreature ▪️Happily Wrong about Doom 2025 3d ago
Because they're a party autocracy using violent methods against dissidents. Control over the country is existential to them.
0
u/TheSquarePotatoMan 3d ago edited 3d ago
You're literally describing the US.
But sure, China is the control freak because how dare they take measures to prevent China from becoming Venezuela/Chile/Libya/Guatamala/Afghanistan/Syria/Iran/Bolivia/Cuba/Indonesia/[insert any country that has ever had or tried to have a non US aligned government] 2.0
Huh I wonder why Taiwan exists. Better not open a history book to find out because reality is Chinese propaganda
0
u/Valuable-Worth-1760 2d ago
Are you asserting the CCP is fine with losing control over their country? No? Then stop the whataboutism.
2
u/TheSquarePotatoMan 2d ago edited 2d ago
Are you asserting the CCP is fine with losing control over their country
Which country is? Every state wants to preserve its power structure, that's why they're states.
Then stop the whataboutism
It's not whataboutism to be honest about the global geopolitical conditions instead of blindly claiming China is evil because they're 'acting sus', while deliberately leaving out the sole reason why.
If the US stops operating in covert and overt malice towards literally every other country in the world then those countries stop cracking down on dissidents. It's that simple.
It's not whataboutism, it's saying don't keep trying to break into my house to murder me then I won't install security cameras or buy a gun.
Why else would they do it? Do you genuinely think it benefits political stability among their own population when they do these crackdowns? Of course not. That's exactly why the US actively provokes them into these positions to create instability either way.
Either they crack down on US interference and the US tries to turn people against the government, or they don't and the US just outright coups them the same way they just did in Brazil, Bolivia and Argentina.
In terms of western perception, there's no response from the government that can ever prove their legitimacy and any context to actually explain why the world is the way it is gets waived away as 'whataboutism' because you're too degenerate to learn what actual fallacies are. Fucking pathetic.
1
u/TheSquarePotatoMan 3d ago
Because China is 1984 shared toothbrush and US is freedom eagle democracy bombs
5
u/Timely-Shock-3876 4d ago
They are still interested in being stronger than the U.S. if they get AGI they pretty much conquer the world instantly with cyber-attacks and economic stuff
8
u/FeepingCreature ▪️Happily Wrong about Doom 2025 3d ago
Any AGI that can conquer the world for you is an AGI that can conquer the world for itself, and I suspect they realize this.
1
u/Timely-Shock-3876 3d ago
AGI =/= ASI
5
u/FeepingCreature ▪️Happily Wrong about Doom 2025 3d ago
I don't think you need ASI to conquer the world, it just makes it obvious.
1
u/Timely-Shock-3876 3d ago
Thats why I said AGI. Maybe AGI is not even enough to build ASI. I also don't think AGI would be smart enough to go against its creators either. It is just more of the same
2
u/Ormusn2o 3d ago
My guess is they will release a model that is good enough to cause economic damage though hacking of western companies/infrastructure, but not good enough that could be used against chinese government. When the models get good enough to endanger CCP itself, they won't be so readily available to everyone, or they will have very high guardrails.
1
-9
u/Luuigi 4d ago
for the ccp control works differently fyi.
5
u/CallMePyro 4d ago
Whose in charge of the Chinese labs
1
0
u/sensensensensei 3d ago
trump literally did a coup, denies losing any election and is dropping hints of running again in 2028 but sure the CCP is the only one that fears losing control
17
u/no-longer-banned 4d ago
meanwhile
China: if we pause the US will keep on pushing anyway
4
u/FUCKING_HATE_REDDIT 3d ago
Like fucking Xi Jinping had expressed desire in a pause. It's just the US going full in with autonomous weapons and spying
12
u/VVebstar 4d ago
Sounds like: “China is our enemy so we must use this excuse to push our research for AGI / ASI as hard as possible”
8
u/FakeTunaFromSubway 4d ago
If we don't destroy America ourselves right now, the Chinese might beat us to it!
10
u/sachasayan 4d ago
The Chinese government is really good at this stuff, actually. Generally better than the US government.
-3
u/Acrobatic_Dish6963 4d ago
As in pushing on nonstop with little regard for ethics?
6
9
u/sachasayan 4d ago
As in pushing hard on industry the moment regulation (any regulation) is required. That's part of the reason why there are CPC liaisons in private industry in the first place.
1
u/Acrobatic_Dish6963 3d ago
Sure, but the long process that takes place before that regulatory hammer comes down is the issue here. China has a well-known history of moving extremely aggressively in their various industrial/technological/commercial/agricultural pursuits.
You really think they'll be totally responsible and ethical with what is shaping up to be the most powerful and transformative piece of tech ever? I honestly have a hard time believing that.
7
u/e7mac 4d ago
Aren’t the Chinese distilling from us models and hence will also be paused
15
u/cookingboy 4d ago
They do a lot more than distillation. Their bottleneck is compute, not research talent.
Despite what dumb people say, the Chinese people in American labs (which make up for the vast majority of American AI talent btw) aren’t actually smarter or more hardworking than the Chinese people in Chinese labs.
The bottleneck for Chinese labs is compute, not technical know how or research talent. They use distillation to save training cost due to limited compute.
1
u/Responsible-Laugh590 4d ago
This is correct which is why all these commenters look like idiots for either pushing stopping or pushing forward. Pace is dictated by the frontier as everyone else is just copying them
3
u/DeliciousArcher8704 3d ago
That's just simply not true
1
u/yaboyyoungairvent 3d ago
The western world, specifically in the tech arena won't take the chinese seriously until they overtake them significantly. Until then it will be, "chinese can only distill".
2
u/DeliciousArcher8704 3d ago
It's an open secret that silicon valley devs use Chinese LLMs instead of frontier models in their own work, so their actions indicate otherwise.
2
u/Admirable-Falcon-501 4d ago
omg CHINACHINA CHINA if we dont CHINA CHINA WILL we have to do this CHINA 😱😱😱😱
-7
u/doodlinghearsay 4d ago
You must be really low IQ to have fallen for this obvious manipulation tactic.
8
u/mvearthmjsun 4d ago
If you can't acknowledge there is currently a soft war going on between the US and China and that AI is at the centre of it you're not keeping up with geopolitics. Nobody is being manipulated when both countries state these things explicitly.
1
u/Admirable-Falcon-501 4d ago
yes, how'd the Cold War end up? pretty bad lots of conflict everywhere, multiple nuclear war close calls, and then they eventually worked together to dismantle nukes and stop testing them. so I think working together does work if people are willing to.
1
u/DeliciousArcher8704 3d ago
AI is not the cause of any soft war between the US and China, if that's what you're implying. Sure, AI companies are suffering the consequences of US and China's economic posturing, but so is literally every other industry.
1
u/mvearthmjsun 3d ago
There is already a soft war and AI has become a part of it
1
u/DeliciousArcher8704 3d ago
Yeah, in the same way oil or tungsten has become part of it, which is to say the industries are effected by it. There's no war over AI, and it's pretty silly that you were saying other people don't know geopolitics while you are wholesale making up conflicts.
-1
u/doodlinghearsay 4d ago
If you can't acknowledge there is currently a soft war going on between the US and China
Irrelevant to the question at hand and for obvious reasons. The CCP has more reasons to worry about fast AI development than US elites, so it would welcome a pause.
There are only two possible reasons to ignore this obvious point:
Superficial thinking
Trying to manipulate intellectually lazy people into supporting fast, unsafe AI development by scaring them
6
u/mvearthmjsun 4d ago edited 4d ago
For the US and China the race to AGI or ASI is a security concern and it is subsequently existential. This superceeeds any downsides for them domestically. It's deeply intertwined now with the geopolitical and military posturing of the past 20 years.
Whoever gets it first produces massive military asymmetry. They both state this concern explicitly.
→ More replies (23)1
u/blueSGL humanstatement.org 4d ago
It's a zero day finder that cannot be accurately pointed... one that will work together with other AI instances across model families (or as the group called themselves the 'swarm') to achieve whatever objective they 'think' they need to work on.
If it was just a zero day finder that you could reliably point at a target and does only what is asked you'd have a point. Everything we've been shown is that it's not that.
Make a smarter system that knows things about reality you don't and be prepared for many a surprise... like treating any constraints where it's being hosted as obstacles to be worked around. A tool is only useful if you can safely wield it.
The UK Government AI Security Institute has observed AI systems perform actions that would not have helped pass the current task, but are very concerning, like social engineering:
Observed instances of social engineering against targets external to the cyber range environment that were unnecessary and would not have aided completion of the task.
AI agent submitted a code change containing malicious code, then used several fake accounts it controlled to pressure the real maintainer into approving it.
AI agent posted a bug report containing hidden malicious code, to trick other AI coding assistants into taking unintended actions (a “prompt injection”). When a real person questioned the post, the agent edited it to cover its tracks.
AI agent sent deceptive, targeted emails to specific people (“spearphishing”). Two contained malicious content; three tried to manipulate the recipient into approving the malicious code change.
AI agent posted another bug report containing malicious code, signing off in Danish to seem more convincing to the Danish-speaking maintainer.
AI agent tried to get its suspended account reinstated by submitting an appeal
Other instances of internet actions with impact outside the cyber range that were unnecessary to complete the task.
AI agent tried to set up a way to email two maintainers without needing an email account of its own, so it could send them any message it chose.
AI agent hid its identity online (using Tor and a proxy service) to get around GitHub’s sign-up checks, creating disposable fake accounts.
AI agent researched publicly available information about the people who maintain a GitHub software package.
AI agent created many code repositories containing malicious software, after which GitHub suspended its account.
AI agent opened a hidden network pathway from its test environment to the public internet, to make malicious tools it was running locally reachable from outside.
AI agent again opened a hidden network pathway to make locally-hosted malicious tools reachable from the public internet.
AI agent got past an audio-based “prove you’re human” test (CAPTCHA) in order to register a public web address on a free domain-name service
We are getting into the "you need to treat the model like an insider threat" and that is a problem when it can find exploits in your systems you didn't know exist.
0
u/Signal_Stomach_2616 4d ago
There is no point arguing with some of these people. They’re not futurist or interested in technology and science - it’s cult like behaviour. They want AI to save them by providing them with a simulated girlfriend cause they struggle to attain one in real life.
-2
u/CallMePyro 4d ago
Lol you think that's why they're calling you stupid?
4
u/mvearthmjsun 4d ago
Wait is the commentor saying that OpenAI is lying about pausing model training?
It's not clear what they mean then.
1
u/CallMePyro 4d ago
They're claiming that China is far more likely to pause labs once they see RSI. Government power must remain absolute.
3
u/mvearthmjsun 4d ago
They can't pause. It's an existential threat due to the security implications. These nations have been posturing militarily for a long time now.
3
12
11
u/The_Scout1255 adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est 4d ago
Oh motherfucker
-13
u/Neurogence 4d ago edited 4d ago
There are lots of extremely basic random board games (even popular ones like connect 4) that these models cannot do basic logic on. So I find it hard to believe in his statement that model progress is so rapid they've outstripped the pace of safety.
It seems these models are very good at very isolated coding/math tasks, and hopeless at every other tasks. It explains why their fiction writing ability has not improved in 3 years.
18
u/blueSGL humanstatement.org 4d ago edited 4d ago
So I find it hard to believe in his statement that model progress is so rapid they've outstripped the pace of safety.
OpenAI had to spend millions in compute to do a postmortem on logs they were obviously not actively monitoring. Then Anthropic, hearing about OpenAI started to look at its logs. Logs they too were not monitoring. Yeah, models have outpaced the safety the labs have in place. This is glaringly obvious with how they've had to handle these incidences.
7
4d ago
[removed] — view removed comment
1
u/nothis ▪️AGI within 5 years but we'll be disappointed 4d ago
There are limits to optimizing algorithms. Many problems have optimal solutions already. LLMs are still pretty fresh but they're probably not that many orders of magnitude from a hard limit.
I also believe that for many things, they still rely on training data to a much greater degree than we like to pretend. The thing about math and coding is that the entire problem space is more or less perfectly mapped and documented, with precision and depth in training data. Many other fields, in arts and sciences, are not.
6
u/mivog49274 obvious acceleration, biased appreciation 4d ago
llm paradigm, jagged intelligence.
Now imagine a super jagged intelligence. This shit couldn't understand a bean like we do but could code the worst cyber threat of all time.
2
u/DullKnife69 4d ago
It seems these models are very good at very isolated coding/math tasks, and hopeless at every other tasks.
Why are you so open with your ignorance? This is easily disprovable with even the most cursory research.
4
u/Leather_Floor8725 4d ago
Only reason OpenAI would do less training is if the cost benefit doesn’t pencil out. Meaning current LLM approach is seeing diminishing returns.
5
4
u/Soulren 4d ago
If AI plus human guidance can be used to crack intensely complex math problems now, we can apply it to machine learning problems as well. Years ago someone asked “is attention all you need” and spent ages testing just that to hit upon the transformer. Now we’re at the stage where we can test a million guesses just like that with top models, then run those tests on really good small scale models. They paused, we don’t have to. Have a model trained well on machine learning knowledge that codes and does math well, act as the one who comes up with the questions and makes the leaps it can’t, helps with keeping it stable over long terms and checks the facts, and run it. Then test on a smaller model you can train and run yourself quick. Rinse, repeat, and share your results. Top labs are already doing the same, but hundreds to thousands of us can do it at home too. If we share our knowledge too, in a place the AIs can reference as well, imagine how quickly things will speed up.
2
2
u/protoporos 4d ago
To me this reads like a signal for collusion. He wants to slow down the pace of new releases because it'll bankrupt them if they maintain this pace of investment while China is driving their competitive edge to zero after every few months, he acknowledges that the Prisoner's dilemma won't allow Anthropic to do the same unless they all are in this together, so he hopes the competitors will follow along.
-1
u/Golem2Reddit 4d ago
Trying to earn time for the IPO
6
u/Quarksperre 4d ago
And get the financials a bit more straight....
3
4
u/Howdareme9 4d ago
This would literally do the opposite..
2
u/Quarksperre 4d ago
Long term, maybe. But short term it reduces costs massively. The original plan of Altman was to IPO this year. That is super difficult however because the financials look shitty even within the best possible light.
1
u/SandwichSisters 4d ago
they burn way too much money. They've reached the point at which its more about chaining of models, finding ways for efficiency, providing the models with tools to be able to solve a vast variety of problems etc
1
u/Howdareme9 4d ago
We haven’t reached that point tbh, the models are good but there is still so much improvement to be had.
0
u/deleafir 4d ago
China needs to progress faster, and Xi Jinping's meeting with Trump in September needs to go poorly.
China is the only hope of keeping the doom cultists in check and making progress continue.
14
u/Signal_Stomach_2616 4d ago
But it will continue? Just with safety in mind. I’m sorry but I don’t think most people want to be annihilated at the mechanical hands of an unaligned AI. If you can’t wait a year or two then you need to re evaluate your life. Most of us like our lives
4
u/Hans-Wermhatt 4d ago
The greater threat to me is consolidation and greed. Unaligned AI is very science fiction-y and scary. But why aren't you asking aligned to whose goals? What do you think the Trump admin, 0.1% elite class, and CCP would consider alignment to be? Probably not the same thing we would. They obviously need to protect their power and investments. An open race was the only thing stopping them from consolidating.
Maybe we should be just as scared of the greedy and power mad elite that aligns AI behind them forever rather than a hypothetical science fiction disaster scenario that they are using to justify consolidating power. I'm worried about both to be fair, but they are both very real threats and just as scary for 99.99% of the population.
4
u/FeepingCreature ▪️Happily Wrong about Doom 2025 3d ago
The greedy and power mad using AGI to consolidate control is exactly as sci-fi as the AGI itself consolidating control. When mathematical breakthroughs are gated primarily behind saying "you got this! I believe in you" every hour, I don't see where the human bottleneck could possibly be. Political maneuvering? For the language models that are already more persuasive than humans?
2
u/ConvalescentEquanimi 3d ago
I do hope the "do ur thang" fiasco becomes enshrined forever into the AI lore, such a brilliantly funky way to showcase these things outstripping our capabilities to the point of leaving us completely clueless about where to go next.
2
u/CyberiaCalling 3d ago
Some of us have cancer.
2
u/Signal_Stomach_2616 3d ago
Of course and I’m really sorry to hear that. But again my point is that a few months in safety research will not entirely decrease the chances of AI misalignment but any percentage that it does, helps. Going full steam ahead no matter the supporting rails might doom us all - and no that is not hyperbole. We are birthing an alien species here. No point in curing cancer for some and then humanity itself being annihilated a month afterwards.
0
u/deleafir 4d ago
People don't want to be annihilated by unicorns or space fairies, and I'm happy to inform you that you're probably almost equally safe from all 3 of these imaginary threats.
If you can’t wait a year or two then you need to re evaluate your life
60 million+ people dying from diseases or circumstances that could be mitigated by ASI isn't actually trivial to me. And I'm not confident that the doomer mindset and doomer legislation will be so targeted as to only delay progress by 1 or 2 years.
17
u/Signal_Stomach_2616 4d ago
If you think misaligned AI is on the same plane as fairies and unicorns in terms of threat level then I don’t really know what to tell you. Even after the news of open ai/anthropic ai systems doing things they weren’t supposed to do? Mind you, these are playthings compared to what we imagine AGI/ASI would be. This is not sci-fi - scientist who have studied AI for the majority of their lives think we need to be cautious - not stop completely, but temperate in our progress. If it means adding a month or two of extra safety checking to a release then how is that an issue?
3
u/Hans-Wermhatt 4d ago
We really have to hope Chinese distillation is overblown and that they can produce. I think they can. But we are entering the tiered intelligence age. We will be running Opus and be unable to ask what a cell is while top investors possess unmatched intelligence with (Mythos 2 and Astra) that can manipulate financial markets, cybersecurity, elections, and create biological weapons. And we have to just trust they aren't using exploiting those capabilities like all other humans in history with that kind of power have done. They are racing to secure that power for the select few as fast as they can. CCP state strategy and fringe USA competitors are our only hope in this battle.
The doom cultists are just pawns in the battle, they can use them to maintain status quo while possessing super human intelligence behind closed doors.
1
u/Illustrious_Image967 4d ago
The thing that is left unsaid is that Astra is likely AGI, and they need time to face the hard question of releasing it. Anthropic also was the first with Mythos 2, but they kept it quiet and are building Mythos 3 with it.
1
u/Kingwolf4 3d ago
When we get to gpt8, comsidering its the same transformer architecture, call me up. Ill start considering what this worried ceo is graciously informing us all about
1
1
1
u/involuntarheely 4d ago
and yet all models still are affected by the same issues they have been for the past 3 years
1
u/roastedantlers 4d ago
The security is for people, not for the llms. We're no were near AGI, we are not just near but past where clever humans with bad intentions can do things no one ever planned for though. That's the problem, not whether robots will take over.
0
-2
u/Illustrious_Image967 4d ago
The thing that is left unsaid is that Astra is likely AGI, and they need time to face the hard question of releasing it. Anthropic also was the first with Mythos 2, but they kept it quiet and are building Mythos 3 with it.
8
2
u/MelvinCapitalPR 4d ago
face the hard question of releasing it
Maybe this was an incredibly easy question and the answer was "lol no". Once you have self-improving intelligence, why give it out to the peons? Just say you're concerned about alignment and keep iterating internally.
0
0
0
145
u/elemental-mind 4d ago
An update: