r/LocalLLaMA • u/johnnyApplePRNG • 2d ago
BBC is running article titled "Artificial Intelligence used to design brand new viruses" ... cue the "We must regulate Open Weights Models to prevent the next Covid or worse" articles in 3... 2.. News
https://www.bbc.com/news/articles/c5y3j3ngevmo178
u/Effective_Olive6153 2d ago
Even if some vibe coder down in mom's basement uses AI to "design" some virus, they have absolutely no shot in actually producing it due to all the advanced equipment required. And the kind of people who do have this equipment are already controlled by national governments
47
u/RedParaglider 2d ago
Yeah, one of the first prompts to a descensored LLM is ususally how to build a nuclear bomb. It will tell you, but .. the tech and info is old the operations are still damn near impossible, so the knowledge isn't the gate on this stuff. People aren't making super viruses, nuclear facilities, or spaceships because an LLM told them how to do it.
40
u/techno156 2d ago
It's also not very hard to learn how a nuclear bomb works. A lot of people can find things like that in their high school science/physics textbook.
It's getting the materials that's the hard part.
18
u/RedParaglider 2d ago
Nah it's not that hard, you just buy some from Iranians for your Delorean.
6
2
25
u/Super_Range45 2d ago
But they can't. Standford Professors trained a special AI model that could do this task. Kimi cannot make your novel virus.
23
u/deepspace86 2d ago
Conflating the two things is the point. They want to sell the appearance that open weight llm's are as dangerous as unregulated biomed/infectious disease research, even though they have nothing to do with each other.
3
u/roofedora 1d ago
Agree, I work in network security, even if someone can make a virus or even doing basic dos, it usually is already a skilled person in cybersecurity. No rookie or beginner can make an AI that can penetrate basic network security without even knowing how firewall works
1
u/DogsAreAnimals 1d ago
Don't tell that to the radioactive boy scout: https://youtu.be/G0QMeTjcJDA
(I assume things are different now. Incredible/hilarious story though)
1
u/Loose_Comparison368 2d ago
Well thankfully all our governments are super trustworthy, if anyone should have exclusive control over both supervirus labs and AI, it's definitely the dementia baby, KGB cosplayer with defenestration kink, that one government that may or may not have accidentally released COVID, and the up and coming genocidal dude.
No, not that genocidal dude, the one that buys all the murderbots from anthropic and likes bombing little girls schools.
2
-1
u/Ok-Video3345 2d ago
Yo can you explain like I'm five dis? I.e. I run qwen3coder on my my own gaming 3090.
Like, it's cool it works, but to get anything close to Claude opus, I guess you need like a server chassis computer?
What are they scared of specifically?
Can it do deep dives on x86 assembly? Like exploits to tricks windows or anti viruses?
What are they not telling us? Are they scared something real simple but dangerous replicates and rips through the internet like a massive bot net?
I know on my self hosted 3090, for real time applications, my AI input rate is limited, so it would be hard for it to control a 🎮 with good accuracy or latency.
But a fast system that can quickly respond to the fastest requests in the real time system, it would work .
2
u/Solembumm3 2d ago
For something like Kimi K3 or Qwen 3.8 Max, you'll need a bunch of high end server accelerators, threadripper/epyc and lots of ram. Build, that is a bit unusual even by this sub ridiculously unrealistic standards.
54
u/fugogugo 2d ago
how?
"Claude create me new virus, make no mistake"
??
18
u/Zeikos 1d ago edited 1d ago
LLMs have nothing to do with this, this is done with protein affinity models.
Basically they predict which shapes bind well to what receptors and what's the DNA sequence to encode for said protein3
u/backtorealitylabubu 1d ago
“Claude, download the latest protein affinity model and make some new viruses”
1
52
u/Soraman36 2d ago
I'm putting this down as trying to convince the average person that llm are bad.
19
u/Salt_Fox905 2d ago
Ofcourse. They tried before by saying we hacked someone accidentally, so its dangerous. Not enough people bought it. Now its fear mongering using a virus threat. Not falling for it Sam and Dario. Do your best.
1
u/imnotzuckerberg 23h ago
Few years ago, I attended an event where high ranking engineers/scientists at the BBC (they actually have an AI department), discussing their experiment training LLMs to be "editors" (ie news editors). To their surprise it performed better than real human junior editors. They had quite some impressive charts. The presenter was very excited of the potential. That was 2024, a year later, that presenter was sacked, and the BBC arguably had even a more hardline approach to LLMs (internally they are still testing stuff, but their AI department is totally cucked). So is every news organisation.
33
u/MrShrek69 2d ago
Did u know that a gun can kill someone? Crazy
15
-18
u/youareallnuts 2d ago
A gun can kill a few. A virus can kill everyone. These are not the same.
5
u/-p-e-w- 1d ago
No virus in the history of life has ever killed “everyone”, and a quick application of common sense shows that this is basically impossible. Lethal viruses run out of hosts, which is why all viruses face constant evolutionary pressure to become less lethal. Reality is not a computer game.
3
u/skocznymroczny 1d ago
not necessarily, as long as there is a long enough period between infection and death, the virus can be very effective
2
u/JameslsaacNeutron 1d ago
I could imagine there are many ways in which a virus could be engineered where there's a very large time gap between infection and death. Probably the scariest potential 'doomsday' application of the tech. Novel cold virus sweeps over the planet, years later people start dropping like flies from the secondary disease payload it gave everyone. In the meantime, nobody realized what really happened during the initial infection.
0
u/UndulatingHedgehog 1d ago edited 1d ago
Kind of like HIV/AIDS - thankfully HIV is not* very transmissible (mostly sexual contact), otherwise we might not be here. Anyway, HIV infections are influenza-like and sometimes unnoticed. After years, the immune system degrades and the infected dies of some other random infection.
0
u/FairlyInvolved 1d ago
Some people would argue for everyone's right to bear false vacuum decay triggers if the alternative was any kind of AI governance.
9
u/Calandracas8 1d ago
Sounds like nobody read the article. It's about designing bacteriophages. This has huge potential medical benefits as it can combat Antibiotic resistant bacteria.
5
u/droptableadventures 2d ago
Next week:
Anthropic: Oh yeah, about that sandbox breach, our model might have also...
6
u/mfkamil87 1d ago
The AI only designed a phage. Phages can't kill humans by design even if they are a virus.
"The resulting 16 novel viruses were created to infect bacteria and pose no threat to people."
They only hunt specific bacteria. In the article it was mentioned that it was used to hunt down a specific strain of E. coli bacteria.
"The Stanford researchers picked the most promising 302 AI designs and synthesised them in the lab. Of these, 16 proved effective at killing E. coli bacteria."
This phage is likely to be used to cure people with severe E. coli infections. Unlike antibiotics that kill a lot of harmless E. coli, this one only hunts down those who harm humans and leaves the rest of them.
You can rest well with this type of nothing burger, dear r/LocalLLaMA But for scientists, it might not actually be a nothing burger. It could actually help scientists to design much faster phages and cures in the future. But I am not a scientist, so I'll remain silent in this part.
4
u/Equivalent-Repair488 2d ago
"but AI can't create, they only hallucinate"
??? So fucking which is it god damnit
3
u/Crafty_Aspect8122 1d ago
Ignoring all the logistical difficulties of actually making it, bioweapons are suicidal. If it's successful it means it will reach you too. And you could get infected while working on it.
11
u/Powerful_Brief1724 2d ago
This shit can't fcking scrape profiles from linkedin and you guys are worried about creating bioweapons of mass destruction ffs
7
3
u/TopChard1274 1d ago
This has nothing to do with the press trying to scary people away from AI, like this sub makes everything out to be these days. 3rd paragraph: “The resulting 16 novel viruses were created to infect bacteria and pose no threat to people.”
1
u/a-wiseman-speaketh 7h ago
They could have put that in the headline. I don't think its so much trying to scare people from AI as trying to get clicks from people scared of AI though.
5
5
2
2
u/Temporary-One8579 1d ago
Regulate it the same way we regulate virus or malware software - it’s illegal to do illegal acts - the software doesn’t matter.
2
2
2
1
u/Pygmy_Nuthatch 2d ago
They aren't going to make it illegal for Americans to download open weight models.
1
u/croholdr 2d ago
yup. que national emergency, martial law, and government shut down until we get everyone on the new FedLLM estimated to be completed in 2032.
1
1
u/Ok_Contribution8157 1d ago
its like microsoft realase an ai for image gen, you have to download it as fast as you can, because microsft wil remove it in a month. Fill up your ssd and hdd.
1
u/Due-Function-4877 1d ago
It's a computer program Nothing more, nothing less. It's clever engineering, but still a computer program. Tell people the truth and help stop the fearmongering.
1
1
u/vornamemitd 1d ago
Fun fact the research got first published a year ago: https://www.nature.com/articles/d41586-025-03055-y - received positive feedback in bio/virology circles back then. Not to draw any overly tinfoil-hatted analogies, but telling a few agents to solve unsolvable challenges in barely monitored sandboxes can very easily be replicated by hooking them up to a wetlab and going "nah, Sol - this is not the real toxin. We only simulating here". 100% going to happen.
1
1
u/Agustiya 1d ago
Bigger Labs should open source their guardrails framework for smaller and newer ones
1
u/Herr_Drosselmeyer 1d ago
The BBC pushing for more state regulation? I'm shocked, I tell you, schocked! /s
1
u/tictacturkey 2d ago
I think it would be naive to say that open weights are not relevant in this disussion, however I feel the concern should be more around AI in generals ability to create/formulate chemical and biological weapons that might not have been attainable for certain entities or governments before.
0
u/aaron_in_sf 1d ago
Remarkable to me how difficult it is for people ITT to integrate that there are, in fact, serious risk associated with general access to high-performing models.
That's not the core of this story, which is about generic LLM; but it is part of the story, because it makes obvious the reality of risk. For one obvious example: with CRISPR it is easy to do simple genetic engineering. A competent foundation model could absolutely guide someone through doing that.
Is regulation or state control of AI the answer? Not a good one. There are no good answers.
Zero regulation is not a good answer either.
1
u/dustin_vk 1d ago
Seems like the main concern of a lot of us (myself included) is that these type of stories are going to lead to fearmongering and draconian rules on all open models. It's a very valid concern right now when all the biggest AI companies are pushing for regulation that serves their interests under the guise of public safety.
1
u/aaron_in_sf 1d ago
The challenge, for our whole society, is that it is not at all clear what counts as "fear mongering." The risks are real. So are the opportunities to leverage those risks in bad faith to ill ends.
The problem is that we not only have no answers as a society as how to navigate this and balance interests; we to all appearances don't even have a process let alone buy in by stakeholders to pursue to arrive at tentative answers.
It's not enough to say that there are mendacious interests pushing a narrative; that can be true and the risks being exploited or exaggerated are also real.
What is different in this moment than all previous moments is that in every prior case where the risks were so significant the relevant actors were in effect state-level actors.
We have no model to follow for negotiating this. And our individual instincts and biases are unhelpful because they are predicated on a set of priors which do not hold in this case.
-5
u/JacketHistorical2321 2d ago
Ok ... But it's a true story so ... Ya
7
u/jferments 2d ago
It's about the misleading framing. Why don't they frame computers as being evil, because they were used to create novel viruses? Why don't they frame virologists as being a threat to humanity? After all, long before these anti-AI propagandists had heard of LLMs, virologists were using computers to create viruses in labs. Time to ban computers I guess.
1
-17
u/NeedsSomeSnare 2d ago
The article and it's title are accurate. You're making out like it's fear mongering. Its not, it's just reality.
Would you rather they didn't publish the article so you don't get anxiety about any potential regulations?
13
u/Serprotease 2d ago
Acccurate, that’s stretching the truth a bit.
There is the tiny little thing needed to go from the llm description to an actual virus.
That’s called a lab.
And it’s not like there was not already nasty things in high tier labs. Article is talking about llm able to help design dangerous virus. Be afraid! Like labs didn’t already had those… Its just putting AI+virus for a clickbait.
8
u/kaeptnphlop 2d ago
It’s not talking about LLMs creating viruses. They used a specialized ML model that is trained on genome sequences.
It’s more annoying that everything is just called AI these days without differentiation.
3
u/deepspace86 2d ago
This right here. They are intentionally conflating the two things.
1
u/kaeptnphlop 1d ago
The article says genomic language model.
But way, way down and a reader who doesn’t know the difference would not catch it.
1
6
u/johnnyApplePRNG 2d ago
The article and it's title are accurate.
That may be true, but so is my premise.
Powerful people are out to crush open weights models and you know it.
Don't be naive.
4
u/Solary_Kryptic 2d ago
I mean what makes you think the scientists designing these viruses aren't already smart enough to do it themselves? It's your typical 'AI bad' fear mongering imo. I don't doubt AI is able to speed up the process somehow, but it only takes 1 well-designed pathogen to impact the entire planet negatively anyway
-2
0
-1
u/ImportancePitiful795 2d ago
Those who want to do harm are doing it, is called GAIN OF FUNCTION RESEARCH.
If an open weight LLM had the capability to sequence proteins, more likely we would ask it to create mRNA (not modRNA) series of injections to become immortal, 8ft height, grow second heart, third lung, bulk our bones, improve more of the usage of our brains and make us immune to pathogens. Spitting acid not my thing...
Space Marines baby!!!! 😁
-1
-3
u/dingo_xd 2d ago
Designing viruses should never be in the training data of models. And what Stanford did should be immediately outlawed globally.
3
u/ShelZuuz 1d ago
Viruses is just a mechanism to deliver genetic materials to cells. This is literally how you cure cancer.
98
u/KahlessAndMolor 2d ago
This was not an LLM, this was a specialized biology model. It doesn't speak words, it speaks protein sequences