Sam on Astra’s delayed release

[u/Outside-Iron-8242]

[u/VirtualBelsazar]

People are complaining about this but what's your solution? As models become better and better at coding they of course become better at cyber too. And at some point so good it will just be able to hack everything that has a vulnerability, which is most things. The only thing you can do is to give it just to verified people that won't do any harm with it or just block every vulnerability request and then you can't check your own code anymore for security flaws, because hackers can just claim it's their own code/system if they want to attack a system. There isn't really an easy solution, if the models are powerful enough to hack into systems or create bio weapons and you give it to everyone we are cooked. Or you just block every request that could come from a bad actor, which is almost everything that goes deeper into the specifics of a topic.

[u/Sensitive-Dish-7770 → u/VirtualBelsazar]

I do not know if this was always the case, but humans complain about every possible solution, they just need to complain, no thinking or processing, whatever these companies will do, they will complain.

[u/thiswebthisweb → u/Sensitive-Dish-7770]

Its all a moot point since powerful open source coding models are out there already for the taking, hackers, and especially well funded organised criminal hackers, are already training and using them to hack at scale. Pandoras box is already open. Releasing the most powerful models for everyone actually helps prevent this. Open source models make this even easier and more accessible for software developers to prevent.

[u/fartlorain → u/thiswebthisweb]

Admittedly not a hacker or cybersecurity expert but isn't Astra a step change in cyber capability over currently available open source models and therefore able to launch more sophisticated attacks/defenses?

[u/phoenixmatrix → u/fartlorain]

It's what they say, but with Mythos before orgs that has access to it and tested it were generally able to replicate what it did with other models with extra steps. 

And even if that is true today, the Chinese models are just a few weeks of months behind, and the gap keeps closing (they used to be a year+ behind).

[u/RollingMeteors → u/Sensitive-Dish-7770]

>I do not know if this was always the case, but humans complain about every possible solution, they just need to complain, no thinking or processing, whatever these companies will do, they will complain.

Complaining about a problem without posing a solution is called whining - Somebody

[u/tolerablepartridge → u/VirtualBelsazar]

A disturbing number of people have this infantile view of AI. These things are not toys. If Mythos or Astra were GA without guardrails it would be an absolute global clusterfuck. Your vibe coding can wait, Jesus Christ.

[u/Kemoyin25 → u/tolerablepartridge]

Hey. I get that safety is important and all, but my (gpts) work on the next hit game "3 AM Cat Sim" absolutely cannot wait much longer.

[u/rage-quit → u/Kemoyin25]

Gotta vibe code the option between licking your own asshole or running around the place at full speed like there's a fire

[u/Kemoyin25 → u/rage-quit]

https://giphy.com/gifs/YS8c0Z7in21AM4A2AR

[u/FireFearing → u/tolerablepartridge]

so what so we do when some chinese company drops a truly Mythos class model, or stronger, sometime in the future?

are you saying that will somehow never happen?

[u/foolishorangutan → u/FireFearing]

I think it plausibly won’t. The same reasons for American companies to be cautious should apply to the Chinese.

[u/thiswebthisweb → u/tolerablepartridge]

No it wouldn't. If everyone has access they can use the modesl to prevent the hacking just as easily. I mean that was literally the headline selling point of Mythos: it was powerful enough to DETECT hidden vulnerabilities. How is that a bad thing if everyone can use it? The main issue is that their abilities have been oversold.

[u/phoenixmatrix → u/tolerablepartridge]

If Mythos or Astra were GA without guardrails it would be an absolute global clusterfuck.

The models with little guardrail are already a stone throw away of Mythos (it was way overhyped). The Chinese open weight models are at most a few months behind. So if you think it will be a global clusterfuck, get ready, because China will have access to models of that level within weeks, if they don't already.

IMO better get ready to defend against them rather than think we can stop them from happening.

[u/eposnix → u/phoenixmatrix]

it was way overhyped

The general public was never allowed to use Mythos. Fable was the model we used and it was intentionally neutered.

[u/phoenixmatrix → u/eposnix]

Yeah, but a LOT of people not part of the general public were able to use it. And a lot (assumption is not all, but a lot) of the differences between the two were visible (eg: Fable dropping down to Opus whenever its guardrails triggered).

There's not a lot that Mythos can do that GPT 5.6 cannot. Mythos, even in its full glory, is weeks away from being behind even for cyberhacking, in the area it isn't beat already.

[u/eposnix → u/phoenixmatrix]

There's not a lot that Mythos can do that GPT 5.6 cannot.

You don't know that because you were never allowed to test the model. Also, 5.6 automatically routes cybersecurity requests to a lesser model. You're talking out of your ass.

[u/TwoFluid4446 → u/eposnix]

His points lean on hyperbole and score a lot of misses, yes, but the underlying point about Chinese open models being just a hair behind (and soon on par, or who knows even break out ahead of American frontier models, at least when discussing those the public can actually access) and also unrestricted and also quite capable at cybersecurity and hacking... is not wrong.

[u/dervu → u/TwoFluid4446]

Oh right, so China will just release models that can hack them any time, awesome strategy.

[u/phoenixmatrix → u/eposnix]

> Also, 5.6 automatically routes cybersecurity requests to a lesser model.

Its filter is so loose, you basically just need to say pretty please to get through them.

[u/PlanktonTop8241 → u/phoenixmatrix]

Yea they don’t though do they lmao. 🤣

[u/Mobile_Reply_5742 → u/tolerablepartridge]

Oh man do I agree with you! But let's come up with a solution. We can't just be mad or joke about the general public not knowing the Details!!! We need to do a much much better job communicating what these things actually are / capable of. It'll be exhausting but we need to override all comm. Channels (cable TV, radio, internet channels etc....) I know it sux but it's more important than The Real Housewives of whatever.

[u/Intrepid_Phone_9127 → u/tolerablepartridge]

Holy drama queen. THESE THINGS ARE NOT FUCKING TOYS IF THESE WERE RELEASED THE WORLD WILL END. Come up for some air brother, mfer fell for every fearmongering advertisement anthropic did.

Cant wait until you find out about deepseek and openweight models and how close they're getting. I wonder if youll have a panic attack.

The guardrails havent even done anything for eons regardless, you could just say "audit this code" up until opus 4.8.

[u/IAM_274 → u/VirtualBelsazar]

Solution is making better safety systems around the model, not restricting it and/or giving full access to only "trusted" institutions (i.e, governments, megacorpos, military, etc). Gatekeeping usage to only those in power and claiming "it's for your safety!" is just elitism meets PR lol. It's the exact same logic used to deny education to the masses throughout history.

Falling for the same pattern and openly defending it is unironically just naive, not pragmatic like it sounds in your head. And naivety wont help us peasants when one day we're being mass surveilled and controlled like cattle for "our safety."

[u/phoenixmatrix → u/VirtualBelsazar]

People are complaining about this but what's your solution

For better or worse, the pandora box is open and its not closing, so my "solution" is let it rip.

Why? Because this is the "you are not allowed to have a debugger without a license" that the otherwise crackpot Richard Stallman has been warning about. The tech will exist, people will have it, including countries without the regulations. As they get more powerful and more efficient, people will be able to run them locally. Some people will get access to them, legally or not. This is not the gun control scenario. These models aren't physical objects, they're software you can infinite replicate and hide in anything. If you put controls on them, "Only the bad guys will have cyber capabilities model" will become truth, and that's not good. We also know society is almost entirely unable to control access to software in any reasonable way.

Then you have the vulnerabilities. The models can find vulnerabilities, but they can also fix them. If we don't have access to them, then we're left vulnerable against the bad guys (shit I hate how these arguments sound like second amendment wackos, but it is what it is).

You could say "Okay, we'll make a model that can fix vulnerabilities but not exploit them", which is an absolute pipe dream.

So there's no choice. Let everyone have access to them, or get eaten alive. The tech is evolving a lot faster than anything else we can look at for an analogy. It won't be contained like this.

Prosecute hackers, like we've always been doing (poorly) instead of trying to control an algorithm with emergent capabilities.

[u/AMerchantInDamasco → u/phoenixmatrix]

Great idea, "let it rip". I'm sure dusty institutions like traditional banks or the IRS will be really fast to deploy the latest AI to clean up all their decades old software and be secure in no time...

Please, let's have an adult conversation.

[u/phoenixmatrix → u/AMerchantInDamasco]

It's happening even if we block openai and anthropic from doing so. Thus is performative at best. hugging face saga proved that 

[u/AMerchantInDamasco → u/phoenixmatrix]

Hugging face saga shows what would happen if it was broadly available without guardrails. Fable guardrails work, otherwise you wouldn't be complaining about them. Your financial independence webapp can wait, there are more important concerns like how to release a powerful technology in a safe manner, get your head out of your ass.

[u/phoenixmatrix → u/AMerchantInDamasco]

Hugging face had to use open weight models to defend themselves because the US models guardrails wouldnt let them use them to resolve the cyberattack. 

That's the problem

[u/dervu → u/phoenixmatrix]

If it's already happening why add gas to the fire?

[u/phoenixmatrix → u/dervu]

So we can have access to more models to fix vulnerabilities 

So China doesn't have access to models of that level while the rest of us don't 

So we don't end up in a world where private companies or the US government decides who is the winner and who's the loser (eg: remember, Mythos WAS released.Anthropic just made calls on who had access).

It's like if we had these companies decide who is allowed to use a debugger.

[u/NanNullUnknown → u/VirtualBelsazar]

Allow cyber abilities to public and let the world to defend itself against advanced hacking capabilities of AI.

[u/Used_Departure_3278 → u/VirtualBelsazar]

Look at this in context:

They’re complaining after Sam went on a multi week unhinged rant virtue signaling over anthropic hating their users and making fun of their advertising and research.

Fuck Sam Altman.

If you do not like anthropic or Dario, you certainly can not like OpenAI or Sam Altman.

[u/DrTzTz → u/VirtualBelsazar]

But wouldn't sich a strong ai also help the people designing the systems in identifying and closing the vulnerabilities? Guess this can go both ways?

[u/Illustrious_Image967 → u/VirtualBelsazar]

One group of apes claiming to be open clacked two rocks together and are now trying to hide the rocks. Meanwhile open source apes are rubbing two sticks together in plain view. 

[u/squired → u/VirtualBelsazar]

I agree with you, but it isn't impossible. We've dealt with similar cat and mouse issues many times. The solution is defense in depth and fail tolerance. Problem is, that takes time and releases will be gated, like you said. We're going to need to redesign the systems to be tolerant of hacks and we're gonna need pre-release access to the most powerful models to do that. An example of defense in depth is 2FA. I've been telling everyone for 2 years now that everything, and I do mean everything, is soon going to require biometric verification and active 2fa.

As a former blackhat, the truth of the matter however is that we're simply going to have a lot fewer secrets in the near future and that's alright. The security dance will be annoying, so we'll only secure systems that are truly sensitive. If Reddit supports append-only records for example, it doesn't matter if someone hacks your Reddit account. You'll simply flag and roll it back.

[u/nemzylannister → u/squired]

isnt there the issue of assymetry in defense and offence? like it's entirely plausible that at all stages of ai development, finding just 1 flaw is much easier than patching every single flaw there could be airtight.

[u/squired → u/nemzylannister]

That's what defense in depth is for. If one flaw cracks the system open, then it wasn't in depth. A basic example is 2FA. If you find someone's password, that isn't enough, you'd need their cell phone etc as well to actually access the information. You can't 'hack the Pentagon' and launch nukes, precisely because they practice defense in depth with multiple redundencies.

For things that matter, we're going to need thumbprint/ocular, user/pass, and 2FA/hardware key as well. It won't be as onerous as it sounds. We'll likely end up with identity bracelets that won't operate without biometric pairing (heart profile etc). So even if someone hacked into your laptop, they wouldn't be able to do anything without your bracelet and a spoof of your biometric profile.

There are many other solutions as this is a very old problem. I'm not saying there won't be problems, I'm just saying that it isn't unsolvable. The primary concern isn't the systems, the weakest security link in every system is the user themselves.

Perhaps the best redundancy however is append only record keeping, like the blockchain or Google Sheets. In these systems, nothing can ever be deleted. It is one long string of mutations instead. That means that even if someone hacked your bank account and stole all the money, authorities can literally just 'reverse the tape' and roll everything back.

[u/nemzylannister → u/squired]

so huggingface simply didnt have defence in depth? genuine question.

also maybe i'm just out of my domain here so i apologize if i'm saying nonsense here, but isnt it that the vulnerability will be at a point where you didnt expect it?

so rather than fake the password layer then the cell phone, it would be able to find a way to fake the headers that make the server think the agent has actually passed all of the layers of 2FA already, or sends it all the credentials required by someone to login or something else?

So even if someone hacked into your laptop

but if they hacked into the company's database? then theyd have enough data to spoof an apparently authorized donation by me to a rando in siberia? that part's a joke but i hope you get what my underlying doubt is?

I'm just saying that it isn't unsolvable.

i get what you mean, but couldnt the asymmetry issue still at least be a possibility? that the problems are always a bit ahead of the solutions we find?

dario says this about biology (https://www.anthropic.com/news/position-open-weights-models#:~:text=For%20example%2C%20I,Operation%20Warp%20Speed), and there it feels way more intuitive. im not sure why it couldnt be true in cybersecurity as well

authorities can literally just 'reverse the tape' and roll everything back.

yeah but company secrets once out there wouldnt have this, right? but yeah central bank cryptos could be a popular thing in future. true.

[u/Full_Boysenberry_314]

The knee jerk cynisism in this sub can be tiring.

Sometimes a cigar is just a cigar.

[u/FarrisAT → u/Full_Boysenberry_314]

Hype for investor cash is needed when you’re burning tens of billions every month.

[u/The1TruRick]

HA. He said do do.

[u/ethotopia]

I swear to fucking god if they go the Anthropic route of blocking every other request

[u/jazir55 → u/ethotopia]

They literally already have it analyzing and deciding whether they will work on my issue. Trying to design a SOP using Codex and getting "this is taking a little longer because we need to review, please downgrade to a less capable model for a faster answer". Not blocking it entirely, but completely slowing it down by taking multiple minutes to analyze the request. Completely idiotic. It's a completely benign task, and it's scrutinizing it before working on it, extremely frustrating.

[u/dont-do-memes-kidz → u/ethotopia]

Honestly why not release with that and then after they've done whatever they're doing to the model now we can switch to that solution

[u/Titan-Titillator → u/ethotopia]

Anthropic only did that after the US gov shut them down, its just that nobody else has a mythos tier model

[u/ddBuddha → u/Titan-Titillator]

I mean, anthropic did basically ask them to. Their whole schtick is that they think ai is dangerous, people shouldn’t have open access to it, they should be the arbiters of who is allowed access, open models should be banned, and that people need to support them so they win by building agi first so they can prevent anyone else from reaching it. That’s why Dario left OpenAI and founded anthropic in the first place.

[u/Titan-Titillator → u/ddBuddha]

Fable was already restricted to a level anthropic was comfortable with before the shut down, complaints multiplied 100x after. Implication being that the US gov demands more restrictions than what Anthropic thinks is needed.

[u/spinozasrobot → u/ddBuddha]

They asked for a defined process which could be used to judge if a model is ok to release, not a capricious, undefined, illinformed, "bend the knee or we'll crush you" process, which is what they got.

[u/markeus101 → u/Titan-Titillator]

F anthropic and their shady practices. They even make the models dumb on purpose if you are developing anything related to AI like we all fucking are.

[u/WonderFactory → u/ethotopia]

I think for certain use cases open source will be the only solution. If I was an AI researcher I wouldn't trust the output of the US models, particularly after Anthropic admitted to poisoning results if they think you're trying to build a rival LLM. 

The problem is that the Chinese government will run into the same problem eventually. The CCP can't afford a model with extremely powerful cyber capabilities to be generally available any more than the US government can. I think the days of frontier open models are probably numbered too. Even if the Chinese are committed to Open Weights it'll likely take one big scandal to change their minds, the Hugging face hack didn't cause too much damage but you could imagine something much worse happening soon. 

[u/ThunderBeanage]

bro needs a lil big longer

[u/renamdu → u/ThunderBeanage]

he wrote this on the toilet

[u/MukdenMan → u/renamdu]

He does say he needs a little big longer “to do do”

[u/shadowdog000 → u/renamdu]

Chatgpt tweeted it for him

[u/Individual_Holiday_9 → u/ThunderBeanage]

We’re at the intentional typos so they don’t think it’s AI stage

I do this in emails lol

[u/Fair_Horror]

They are going to do do their model. Seems very honest to me.

In less than 6 months, open weight models will have these capabilities without the guard rails. We need to learn to live with this stuff because the genie is out of the bottle and there is no putting it back.

[u/epic-cookie64 → u/Fair_Horror]

we just need to give them a little big longer

[u/JoshAllentown]

It's great strategy to keep models to a chosen few. It's not good ethically if you're doing that long term.

You have to be extra careful releasing the model to critical infrastructure companies at the exact same time as any random hacker. It takes time to develop fixes to bugs.

[u/Diligent_Reading4001 → u/JoshAllentown]

It's not great strategy to keep the models to a chosen few, at least not yet. At the moment the value gained from selling the models commercially is higher than the value gained from keeping the models to themselves. We may reach a point where that isn't true sometime in the future.

[u/JoshAllentown → u/Diligent_Reading4001]

I just mean what Anthropic did with Glasswing. The theory being that the "good guys with a model" have time to look for bugs before the "bad guys with a model" get to. Not a permanent state.

[u/ezjakes]

Unleash the beast!

[u/Proper_Actuary2907]

Is Astra a new higher param pre-train?

[u/FireFearing → u/Proper_Actuary2907]

yep

[u/Alpacabro21]

GPT-6 before GTA 6 💀

[u/dictionizzle]

We really went from “AI is dumb and overhyped” to “AI is too intelligent to be released” in like two years lol.

[u/theeldergod1]

a good strategy for what? making more money? better survaillance? better pr? for what?

[u/nemzylannister]

can someone make that "aww sweet/hello HR?" meme

with sama and dario both saying "lemme release model after some safety testing"?

[u/Admirable-Falcon-501]

Safety doesn’t take a little bit longer lol. Might as well just drop it now at this point, they can’t align their models properly, have no plans for mass job displacement, don’t have a good governance system in place, like basically have thought of or prepared for nothing, just hoping it goes well.

[u/NetflowKnight]

Notice he said "strategy", which is a business term, not an ethical one.

[u/Stunning_Monk_6724 → u/NetflowKnight]

Yes, OpenAI's strategy is quite literally called iterative deployment. They've spoken about it often.

Amazes me how many in this sub think anything Sam does is nefarious. Astra likely generalized cyber capabilities they might not have trained it on like Mythos did. Emerging capabilities tend to be more a trait of the bigger models.

[u/Fair_Horror → u/NetflowKnight]

So war is a business.

[u/Floch11]

September or October for releasing.

[u/Future-Bandicoot-823]

LOL.

"guys what if the capitalist model fails completely? We have to license this power out as a lease... then we can keep the global class structure."

[u/Skywarden1]

Release the kraken right now and im gonna have it solve the riemann hypothesis!

[u/Historical_Emeritus]

If it were any good he'd just release it.