I interviewed dozens of biosecurity experts. The fear that AI will enable lone wolves to build pandemics is a case study in how domain expertise gets sidelined.
I again think this is overconfident; I agree that AI models don’t change this **YET** - but I also think that we should not be complacent about either the risks of prosaic bioterrorism, where someone cultures, say, an environmental pathogen without needing as much specialized equipment or to do any manipulation of DNA, then aerosolizes and releases the pathogen, or the risks of sophisticated actor uplift, where a non-state or state actor that already has the capability to get close, and the motivation, could do so more quickly or more easily.
Will this kill us all? No. But a 4 or 5 figure death toll from a major attack is certainly within the realm of plausible outcomes, and I think such an event certainly possible today for sophisticated [lone] terrorists, especially with the help of new open-source models. And without some serious changes to our biodefense posture, it will get no harder in the future.
Thanks David! I agree about state actors and sophisticated actors who have institutional access being capable already. Aerosolization seemed way harder than I expected, but obviously aeroslization skills exist and state actors could easily do it.
I actually thought the prosaic bioterrorism examples were another reason "lone wolf of engineered virus" is less plausible. Why go through all that trouble when you can instead do the same damage in much more old school ways? Though at that point, chemical weapons seem more shelf-stable so probably preferable.
I agree that smaller death tolls like this are possible with some really sophisticated and well connected terrorists, especially in countries with weaker institutional checks or if someone does a well coordinated attack with more basic bioweapons available in nature. Here, I aim at attacks that kill multiple millions, partly because I was ready to pivot careers if that was true.
It's easy to accidentally kill microorganisms by doing this poorly, but I unfortunately agree it's the type of thing non-refusing LLMs will be great at enabling.
Coming very late to this, but the politics here--and relatedly, the epistemics!--are really hard. If you are worried about LLM-enabled bioterrorism increasing expected deaths by 6-7 figures per decade--let's call that 'soft doom'--that seems like a real and plausible concern that someone should be worrying about. But 'soft doom' is *extremely different* from the 'hard doom' position of a guy in his basement wiping out the human race.
To some degree, the hard doomers and the soft doomers have common interests, and therefore are reluctant to criticize each other. This has the downside of muddying the arguments, and the coalitions. Personally, I think investing in AI biosafety sounds entirely unobjectionable; but on the other hand, I think the hard doomers are nuts, and would strongly prefer they not attain positions of power and authority in government. Accordingly, I end up feeling at best lukewarm about AI safety initiatives.
Most of all I think AI safety debates desperately need more normies who are not deeply invested financially, socially, or psychologically, in AI narratives. I appreciate Ms Olvera weighing in, not just because she seems very smart, but also because she is by far the most normie-coded AI commentator that I have yet encountered.
1. I really like the soft doom and hard doom distinction. That’s a good name for groups that I ended up observing as clashing here in DC.
2. I’m also surprised / dismayed at how much the “basement guy wipes out human race” circles are turbo charged from well-meaning money, but I think this might be on-net harmful because the extensive access control they see as necessary will slow life-saving science, especially if the rate of false positives grows from the access controls. Where as bioweapons will be easier to make *via other means* they don’t require assembly.
3. I really appreciate you saying that! I do sometimes think about how much more consistent of a salary I’d get if I join an advocacy organization. But I can’t march for a side without understanding the best case for or against it - these are big important questions!
4. I love being a normie-coded commentator! Thank you! :) - just curious, more normie than Sayash/Arvind? Timothy Lee is also normie coded imho.
Timothy Lee (and Kai Williams!) are great! It's weird that they are some of the only people doing, one might say, AI Journalism As Normal Journalism. As in, like, a lot of boring and tedious document review.
This is mostly my own bias, but the Normal Technology crew is just too academic and too deeply embedded in CS/Silicon Valley culture to be truly normie. They can emulate normieness to some degree, but when you look into their eyes* you can tell that in their heart of hearts they are not normie.
*TBC I have not looked into their eyes, I'm just extrapolating what it would be like to look into their eyes based on reading their Substack.
I agree that there's a huge gap between what has traditionally been called Global Catastrophic Biorisks (GCBRs) and existential risk. The latter is, in my view, almost fully implausible, given human reactions. (I looked at this, e.g., here, https://www.biorxiv.org/content/10.1101/401018v1.full when I investigated the plausibility that a malicious actor uses multiple diseases to try to up the impact of designed or modified disease to the point of an existential risk.) The two plausible ways around excluding existential risk are; one, a malicious AI or other actor using a global pandemic as one part of a larger scheme, where survivors are then tracked down, or similarly, two fragility and additional disasters after a GCBR. The latter case has been proposed for, e.g., built-in climate change post-pandemic, but this seems to be far from enough; it would take a far larger change to wipe out human populations.
But the former case is critical in the "hard doomer" narrative, and not at all implausible *if* you first grant that an ASI might be misaligned and either uncontrolled, or undetected. But as I've said repeatedly, if there's a misaligned ASI trying to wipe out humanity, biorisk might be involved in how things go badly, but it's not the problem, and mitigating biorisks isn't what will help! But the point here is that this is a question of plausibility of the misaligned ASI scenario, not about biorisk at all - so I strongly agree that the "guy in his basement wiping out the human race" isn't a threat, but again, that's not generally the position of those groups of people, from my fairly extensive experience with them.
Whoa fascinating. Somehow I didn’t realize the gap was this large. This super helpful.
Your page has too many attempts to view so I’ll have to try again later.
To confirm my understanding of your last 3 sentences:
if misaligned ASI, biorisk might be involved "but it’s not the problem and mitigating biorisk isn't what will help": is that because the greater problem is consistent threat from misaligned ASI?
“Person in basement isn't the problem" because the upstream problem is persistent ASI attempts?
If so:
I’m curious who you recommend reading on ASI forecasts, given some people have felt alignment is more on track than some expected or surprising slower diffusion than original ASI expectations? Like I’d love a good way to contextualize pace of transparency of AI chain of thought etc and its impact on hidden ASI in particular?
Who has been the one spreading the “a person in basement”? Somehow it’s the meme that spread more widely. Is that just unilateralist curse type of dynamics or this is a meme that leads to actions helpful to prevent that ASI-biorisk worry?
In short, yes, if we have ASI interested in killing us (even at the direction of a user,) the problem is misaligned ASI; a country of geniuses in a datacenter dedicated to killing humans will kill humans; having a war with them is a loss condition even if we somehow eliminated all biological risk vectors.
I'm not sure who is doing a good job with ASI forecasts, it's a poorly conceptualized term and refers to anything from current models to god-like intelligences that can infer currently impossible physics from first principles; I'm obviously referring to something more powerful that current models, but it's unclear how capabilities will manifest over time. (My view mostly contrasts with, e.g., Helen Toner's, here: https://x.com/davidmanheim/status/2058466127609528579 in that I think the exact definitional disagreement will be less relevant as AI becomes increasingly obviously superhuman; I don't know about the path from there.)
And I think the "person in a basement" is a response to a stupid argument that there's no physical way AI could kill humans; obviously there are ways it could occur. This functions as a useful response to people who for some reason think that they are the apex of possible intelligences, and that humanity could never be outsmarted by AI. The counterargument is correct, as far as that goes, in that a sufficiently powerful and poorly aligned AI system could certainly be misused, and it's silly to think humanity couldn't be at risk (it's the Just World *Fallacy*.) I'm arguing that it's not the actual risk, not that it's wrong. Unfortunately, I think the you-get-5-words version is the common understanding of the argument.
I think we agree that ASI is sufficiently vague that it just cannot support meaningful analytical work. So people fall back on sort of deep, intuitive, personal beliefs about The Way the World Is, and so the arguments get accordingly personal. (Perhaps where we differ is that I would put many other concepts in that 'too vague' category, including, but not limited to, 'outsmart' and 'humanity' and 'alignment'.)
And I'm sure that there are people making silly, pollyana-ish arguments about AI safety. But on the other side, you have people--important people! serious people!--who really are going around saying things like AI will soon be a wish-granting genie, or that we are a few months/years out from Dyson spheres and so on. Maybe they don't really mean it, but gosh, it seems like they really mean it! And I worry about those people as much as I worry about AI.
This conflates two things; the concept of ASI, which is vague, and the actual systems which are posited, which can absolutely be analyzed conditional on the specific assertions made.
I'm very aware that there is exaggeration, but there's also conflation of claims about the future conditional on ASI, and the time frame until ASI, along with confusion about the terms. I can't really address "serious people" in general unless the specific claims are specified - and when that happens, I often push back in detail. For example, Bostrom conditions on ASI, without a timeline, and then jumps to mortality claims which seem hard to support even given the assumption: https://x.com/davidmanheim/status/2023363906064802075
Possible? Sure, and you could have said that a decade ago too.
But if you want to bet, I'd happily put 10:1 odds against a seven-figure plus death toll within 2 years, say, before August 2028, from any bioterrorist / bio warfare incident.
A further point for your argument: AI making it easier to build bioweapons also makes it easier to build vaccines and other countermeasures. Defenders typically have far more resources, so more capabilities can be expected to lower the risks from bioweapons.
Regarding AI and cybersecurity, you may want to check out:
Deception and Detection: Why Artificial Intelligence Empowers Cyber Defense over Offense | International Security | MIT Press
This short presentation by David Dalrymple on FlexHEG contains some remarkable claims such as “ … DARPA showed through the HACMS project that it is possible to create software systems that are completely free of exploitable bugs.” In the limit of capability, the defender “just wins” at the software level, especially with AI automation.
One exciting area for LLM cybersecurity is using them to produce formally verified software (https://en.wikipedia.org/wiki/Formal_verification). This is software that comes with a proof (which presumably the LLM would produce) of being secure under a specific model. Formal verification isn't perfect, the real world often differs from your security model. But it does seem like a nice tool to have.
Whoa, this is gold. Thank you! Cool MIT paper especially “AI models struggle with the creativity and deception necessary for offensive operations, but they excel at the pattern recognition that is key for defensive operations.”
It's true that defenders will have more resources and better AIs, but the situation for defenders still seems rough. Developing vaccines more quickly isn't really a bottleneck. Testing, manufacturing, and distributing them are bottlenecks, and those are also harder for current AI to assist.
If someone were on the fence about biorisk before considering how AI could benefit defenders, I agree this should make them a bit less concerned. But it seems to me that attackers will likely have much more uplift than defenders. We're mostly kept safe by the fact that there aren't many would-be attackers, not by a strong defense. That worries me!
In my mind, the situation is pretty symmetric wrt AI. Testing, manufacturing, and distribution are also challenges for attackers! If AI doesn't help defenders much here, it probably won't help attackers much either.
There are a few asymmetric benefits for defenders:
1. The attacker needs to commit to a vector, the defender can observe the vector and tailor mitigations. Defenders can try many solutions per vector, attackers get one shot per vector.
2. Humans are equipped with an adaptive immune system capable of clearing ~any infection given an appropriate stimulus and time to adjust.
3. Defenders have a larger body of research to build on. Bioweapons knowledge does (or did) exist, but has been effectively suppressed due to international treaties.
4. Sanitation is simple and has eradicated most infectious diseases. Air sanitation via HVAC, UV, and glycol vapors seems simpler than building a bioweapon.
That said, bioweapons are one of the most important global risks. I want to see much more work here! They just don't keep me up at night.
I'm by no means an expert, so I appreciate the pushback! But surely attackers have an enormous advantage when it comes to manufacturing and distribution: vaccines don't self-replicate and spread contagiously.
To take one example: in an individual case, cleaning the air in a building is easier than building a bioweapon. However, building a bioweapon is probably easier than scaling that solution to hundreds of millions of buildings.
This feels like the main challenge to me: we can probably figure out how to combat any disease, but can we do it in time and at scale? Probably yes, *if* we set up the infrastructure -- but today, the answer is probably *no* if the disease is bad enough.
True, I think we all three agree that bioresilience is definitely needed and overdue. We could have been doing so much on indoor air filtration, wastewater surveillance, rapid diagnostics, etc.
The attacker's manufacturing and distribution has to be concealed, while vaccines can advertise openly with a reasonable expectation of being actively sought out by friendly strangers.
This is a great piece, just to add a general statement to many good comments: there's a common thread in both AI doomer and AI aggrandizer spaces: they think AI will easily solve problems they don't understand. To make an example, one of the tropes is "AI will cure cancer", and implication is "fast". Well, it they actually talked to people who treat cancers for a living, whether on pharma or practitioner side, they would find out that
Yes, AI is likely to be helpful
No, it's not going to "cure" anything, it may improve testing, it may propose better molecules or delivery methods, and then it will be Phase I trials, etc. etc. etc. until we get a result a decade later.
And so it is every. freaking. domain. "AI will design chips", fast! Yes, it will be able to do design and even layout using fab-derived models, and then off to moths of fab cycles, validation and qual testing. "AI will fully automate manufacturing", yeah, wake me up when a robot can replace a rubber hose deep in the guts of a5-axis machining center.
Yes it is. And you will likely find the space to be so vast you need to narrow down and deal with only a few domains. Like, I’m a chip designer, and I know I have a long career ahead of me, AI will never “take my job”, but it’s already helping with some tasks, and I expect this to continue. But ask me about truck drivers, and I’d only have a vague idea - you need to talk to a logistics expert, automotive designers, maybe a radar/lidar specialist… this gets exhausting pretty quickly.
This article reminded me that pipetting is so difficult that an entire group of students in my first year chemistry labs just gave up and faked all their data 😅
Oh wow hahaha xD That’s a great example! o.o I haven’t tried pipetting myself but when I raised it with a room of biologists, there was a bit of a collective sigh.
Actually every single science except perhaps computer science has its own super annoying manual task - tagging fruit flies, placing electrodes on people’s head without messing with their hair, soldering wires and chips to PCBs or wiring breadboards, aligning optical components, mode locking lasers, etc etc
“Except perhaps computer science” - this very phrase holds a, if not the, key to the bizarre certainty that AI researchers and the adjacent crowd hold regarding AI overtaking the economy and becoming self-sufficient. These people are overwhelmingly CS’s, and seem to think all other disciplines are similar.
Computer science is unique in this respect thanks to the fact that it started out as a niche specialty within textile manufacturing, which otherwise consists just about *entirely* of annoying manual tasks. https://en.wikipedia.org/wiki/Jacquard_machine When further development within that specialty hyper-focused on pure abstraction, and gradually pushed out all the fiddly "eye-hand coordination" stuff onto other sub-fields, the added burden was barely noticed. Of course, it probably also helps that the former most labor-intensive sub-field (spinning raw fiber into yarn or thread) faced some orders-of-magnitude efficiency improvement around the same time. https://acoup.blog/2021/03/19/collections-clothing-how-did-they-make-it-part-iii-spin-me-right-round/
Well done; rhymes in so many ways with the contrast between what cyber domain experts think about AI cyber risk vs what the AI safety community has been saying
Great post, but if I may, I have a technical nitpick.
>But more importantly: biology has built-in tradeoffs: the more lethal a virus is, the less transmissible it is (because it kills its human host).
Late to the party, but this statement is not in general correct, because it assumes that the inherent transmissibility ( probability of infection given contact with another infected individual ) and virulence are uncorrelated, while they most likely are.
In respiratory viruses, this claim is largely correct because of the simple fact that lethality depends on the virus' favored position on the respiratory tract: viruses that sit in the upper section are more transmissible but less lethal, and viceversa.
One can imagine however, "improving" the virus over other dimensions which are not subject to this trade-off, i.e. efficiency of cell entry and/or evasion of immune barriers.
Mathematically, the transmissibility R0 is the ratio of the contact rate with the sum of recovery rate and fatality rate (ie ratio of the influx of the infected class to the outflux). If the contact rate is positively but weakly correlated with fatality, then your claim is correct and increasing fatality beyond an optimal value will decrease R0. But if for some wretched reason making the virus more lethal makes it also much more transmissible then R0 will grow even with increasing fatality.
Thank you, Nemo. I really appreciate it! Perhaps I shouldn't have used the word "transmissible" because of it's specific technical definition. The brake I was wanting to emphasize there is that dead people don't spread viruses (typically).
You're right that if the same change makes a virus deadlier and also makes it spread faster, R0 goes up!
What I had in mind was the extreme case. If something kills nearly everyone it infects. Agreed that it doesn't always run that direction, especially once you break transmissibility into its parts!
As someone who doesn't have a doctorate, but has done extensive work in a hospital Microbiology lab, this tracks.
Most bioterrism threats come from pre-existing diseases. Anthrax, Plague, Tuleremia, smallpox, etc. Even then, there are a lot of limiting factors, that can't easily be worked around. For instance, none of the above are 'easy' to spread. Smallpox has a vaccine, Anthrax doesn't transmit between humans easily, tuleremia can't be easily aerosolized, etc.
New or established, Bioweapons are only likely to be deployed by the likes of Nation States, with the resources. (ie. Iran, North Korea, China, Russia, etc.) Even then, if they get caught, a bioweapon is everyone's problem.
Most of the funding in this area by my understanding is more about countering bioweapons than developing them for deployment. Even the ones we do develop often provide useful insights either for defence or other areas of research.
Sadly not. Most my experience is from textbooks and various bioweapons preparedness training materials I've taken to stay accredited at the bachelor's level. At my level, we don't really see much except for incedentals.
Tuleremia is one that I saw fairly regularly at my local hospital though. That's an automatic CDC report, since it's a organism of concern.
I'll send you a DM with a reference for someone who might be able to hook you up though.
While current AI tools won't necessarily make the messy physical parts of making bioweapons easier, I anticipate that climate change would make strongmen more powerful, like these guys. https://borgenproject.org/cruelest-dictators/ who would be "comfortable killing [their] own people."
So, what types of interventions do you think are still needed versus those that are unnecessary, given your conclusion? For example, do you think AI red-teaming or efforts to prevent AI from acquiring tacit knowledge are still useful? - to what extend?
More broadly, when do you think people should start being concerned about these risks? Even pandemics like COVID-19 killed millions and cost trillion of dollars do you think it’s unlikely we’ll see something on that scale again, or do you think COVID wasn’t severe enough to warrant long-term concern?
To be clear, preventing the next COVID and pandemics should be a much higher priority than it is now. We need lots of interventions. More bioresilience is critical, especially since there’s a 2% chance of a natural pandemic every year. Rapid vaccine infrastructure, waste water monitoring, faster diagnostics, indoor air filtration, etc, are great as well as making sure that critical equipment and materials for making bioweapons stay controlled.
More studies on AI impact on real human participants actually trying bioweapon creation would be helpful and is already starting to happen. When I think of AI acquire tacit knowledge, I expect that most effective in robotics where hard tasks are automated. Those are pretty narrow in their application and resource-intensive to set up so oversight of them is a lot easier than of models themselves.
Gain of function research seems to be predominantly dealing with real-world materials. Do you mean people using AI to review sequences, or are you talking about future lab automations?
@eurydicelives also wrote about this. It was a while ago, and I don't remember it that well, but I figure it's worth mentioning, as the gist I can remember was that biology is hard and expensive even if you somehow solve knowledge.
This is really interesting Abi, thank you for writing it! I’ve been worried about this for years and this post genuinely helped me feel better.
One story I had heard in the past is: a terrorist tried to order a smallpox sample and almost succeeded (the person dispensing the sample literally had an intuition that it shouldn’t be mailed, and double checked, and that was the only thing that stopped the terrorist from receiving the sample - there wasn’t like, a process in place to stop this from happening!). It is helpful to know that even if the terrorist *had* received the sample, it would have been quite hard for them to use it.
I'm so curious about this case. I can't find it online, but would love to know how they ordered it. It's illegal in the US and the UK (and probably the rest of the world?). If it was some variant: when people order viruses that could be dangerous, they often come weakened in ways that it would take a skilled expert to reverse. I'm not sure what type of sample they had, but I doubt it was the live virus!
I expect the example you're both thinking of is the origin of the US Federal Select Agents Program — Max Langenkamp and I describe the story at the start of this article: https://press.asimov.com/articles/dna-screening
To quote:
> In May 1995, a few weeks after the doomsday cult Shinrikyo Aum released sarin gas in the Tokyo subway—killing 13 people and injuring 5,500 more—two Ohio police officers knocked on Larry Wayne Harris’s door. Harris was a trained microbiologist and a former member of the white supremacist Aryan Nations group. The officers held a search warrant for one of the deadliest organisms in history: Yersinia pestis, the bacteria that causes bubonic plague.
>
> A concerned employee at a laboratory supply company had called the Centers for Disease Control, or CDC, after fielding repeated calls from Harris asking when his samples of plague would arrive in the mail. When the CDC contacted Harris to ask about the plague samples, Harris claimed that he was doing research for the CIA to stop an “imminent invasion from Iraq of super-germ-carrying rats.” In the end, falsely asserting that he was working with the CIA was the only illegal thing Harris did—amounting to wire and mail fraud.
The story I heard was some years ago, and I think it was already years old by then. But it was definitely a US case and the terrorist in question was a white supremacist.
Some other stuff I’ve read that terrified me on this score: Ken Alibek’s book ‘Biohazard’ (nonfiction tell-all about his time in the Soviet bioweapons program) and Rob Reid’s book ‘After On’ (hilarious science fiction novel by someone with a lot of time served in Silicon Valley, strongly recommend). Both these books are great but both also were written by people whose main threat model was “we have to be insanely paranoid about keeping the pathogens contained” rather than having the more complex model you outlined in your post.
Great flags, thanks so much Tessa for pulling this up. I thought that since Yersinia pestis is on the Federal Select Agent Program, that it's illegal to procure. Was it not illegal for him to buy it? Because he was one of the few people who could buy Select Agents or something else? Is that why the lie was the legally cruxy thing?
I again think this is overconfident; I agree that AI models don’t change this **YET** - but I also think that we should not be complacent about either the risks of prosaic bioterrorism, where someone cultures, say, an environmental pathogen without needing as much specialized equipment or to do any manipulation of DNA, then aerosolizes and releases the pathogen, or the risks of sophisticated actor uplift, where a non-state or state actor that already has the capability to get close, and the motivation, could do so more quickly or more easily.
Will this kill us all? No. But a 4 or 5 figure death toll from a major attack is certainly within the realm of plausible outcomes, and I think such an event certainly possible today for sophisticated [lone] terrorists, especially with the help of new open-source models. And without some serious changes to our biodefense posture, it will get no harder in the future.
Thanks David! I agree about state actors and sophisticated actors who have institutional access being capable already. Aerosolization seemed way harder than I expected, but obviously aeroslization skills exist and state actors could easily do it.
I actually thought the prosaic bioterrorism examples were another reason "lone wolf of engineered virus" is less plausible. Why go through all that trouble when you can instead do the same damage in much more old school ways? Though at that point, chemical weapons seem more shelf-stable so probably preferable.
I agree that smaller death tolls like this are possible with some really sophisticated and well connected terrorists, especially in countries with weaker institutional checks or if someone does a well coordinated attack with more basic bioweapons available in nature. Here, I aim at attacks that kill multiple millions, partly because I was ready to pivot careers if that was true.
"Aerosolization seemed way harder than I expected..."
What does this mean? Who said aerosolization is hard, and why did they say that?
It's easy to accidentally kill microorganisms by doing this poorly, but I unfortunately agree it's the type of thing non-refusing LLMs will be great at enabling.
Coming very late to this, but the politics here--and relatedly, the epistemics!--are really hard. If you are worried about LLM-enabled bioterrorism increasing expected deaths by 6-7 figures per decade--let's call that 'soft doom'--that seems like a real and plausible concern that someone should be worrying about. But 'soft doom' is *extremely different* from the 'hard doom' position of a guy in his basement wiping out the human race.
To some degree, the hard doomers and the soft doomers have common interests, and therefore are reluctant to criticize each other. This has the downside of muddying the arguments, and the coalitions. Personally, I think investing in AI biosafety sounds entirely unobjectionable; but on the other hand, I think the hard doomers are nuts, and would strongly prefer they not attain positions of power and authority in government. Accordingly, I end up feeling at best lukewarm about AI safety initiatives.
Most of all I think AI safety debates desperately need more normies who are not deeply invested financially, socially, or psychologically, in AI narratives. I appreciate Ms Olvera weighing in, not just because she seems very smart, but also because she is by far the most normie-coded AI commentator that I have yet encountered.
Reread your comment 3x from how much I love it.
1. I really like the soft doom and hard doom distinction. That’s a good name for groups that I ended up observing as clashing here in DC.
2. I’m also surprised / dismayed at how much the “basement guy wipes out human race” circles are turbo charged from well-meaning money, but I think this might be on-net harmful because the extensive access control they see as necessary will slow life-saving science, especially if the rate of false positives grows from the access controls. Where as bioweapons will be easier to make *via other means* they don’t require assembly.
3. I really appreciate you saying that! I do sometimes think about how much more consistent of a salary I’d get if I join an advocacy organization. But I can’t march for a side without understanding the best case for or against it - these are big important questions!
4. I love being a normie-coded commentator! Thank you! :) - just curious, more normie than Sayash/Arvind? Timothy Lee is also normie coded imho.
5. This is going on my virtual fridge.
Timothy Lee (and Kai Williams!) are great! It's weird that they are some of the only people doing, one might say, AI Journalism As Normal Journalism. As in, like, a lot of boring and tedious document review.
This is mostly my own bias, but the Normal Technology crew is just too academic and too deeply embedded in CS/Silicon Valley culture to be truly normie. They can emulate normieness to some degree, but when you look into their eyes* you can tell that in their heart of hearts they are not normie.
*TBC I have not looked into their eyes, I'm just extrapolating what it would be like to look into their eyes based on reading their Substack.
I agree that there's a huge gap between what has traditionally been called Global Catastrophic Biorisks (GCBRs) and existential risk. The latter is, in my view, almost fully implausible, given human reactions. (I looked at this, e.g., here, https://www.biorxiv.org/content/10.1101/401018v1.full when I investigated the plausibility that a malicious actor uses multiple diseases to try to up the impact of designed or modified disease to the point of an existential risk.) The two plausible ways around excluding existential risk are; one, a malicious AI or other actor using a global pandemic as one part of a larger scheme, where survivors are then tracked down, or similarly, two fragility and additional disasters after a GCBR. The latter case has been proposed for, e.g., built-in climate change post-pandemic, but this seems to be far from enough; it would take a far larger change to wipe out human populations.
But the former case is critical in the "hard doomer" narrative, and not at all implausible *if* you first grant that an ASI might be misaligned and either uncontrolled, or undetected. But as I've said repeatedly, if there's a misaligned ASI trying to wipe out humanity, biorisk might be involved in how things go badly, but it's not the problem, and mitigating biorisks isn't what will help! But the point here is that this is a question of plausibility of the misaligned ASI scenario, not about biorisk at all - so I strongly agree that the "guy in his basement wiping out the human race" isn't a threat, but again, that's not generally the position of those groups of people, from my fairly extensive experience with them.
Whoa fascinating. Somehow I didn’t realize the gap was this large. This super helpful.
Your page has too many attempts to view so I’ll have to try again later.
To confirm my understanding of your last 3 sentences:
if misaligned ASI, biorisk might be involved "but it’s not the problem and mitigating biorisk isn't what will help": is that because the greater problem is consistent threat from misaligned ASI?
“Person in basement isn't the problem" because the upstream problem is persistent ASI attempts?
If so:
I’m curious who you recommend reading on ASI forecasts, given some people have felt alignment is more on track than some expected or surprising slower diffusion than original ASI expectations? Like I’d love a good way to contextualize pace of transparency of AI chain of thought etc and its impact on hidden ASI in particular?
Who has been the one spreading the “a person in basement”? Somehow it’s the meme that spread more widely. Is that just unilateralist curse type of dynamics or this is a meme that leads to actions helpful to prevent that ASI-biorisk worry?
In short, yes, if we have ASI interested in killing us (even at the direction of a user,) the problem is misaligned ASI; a country of geniuses in a datacenter dedicated to killing humans will kill humans; having a war with them is a loss condition even if we somehow eliminated all biological risk vectors.
I'm not sure who is doing a good job with ASI forecasts, it's a poorly conceptualized term and refers to anything from current models to god-like intelligences that can infer currently impossible physics from first principles; I'm obviously referring to something more powerful that current models, but it's unclear how capabilities will manifest over time. (My view mostly contrasts with, e.g., Helen Toner's, here: https://x.com/davidmanheim/status/2058466127609528579 in that I think the exact definitional disagreement will be less relevant as AI becomes increasingly obviously superhuman; I don't know about the path from there.)
And I think the "person in a basement" is a response to a stupid argument that there's no physical way AI could kill humans; obviously there are ways it could occur. This functions as a useful response to people who for some reason think that they are the apex of possible intelligences, and that humanity could never be outsmarted by AI. The counterargument is correct, as far as that goes, in that a sufficiently powerful and poorly aligned AI system could certainly be misused, and it's silly to think humanity couldn't be at risk (it's the Just World *Fallacy*.) I'm arguing that it's not the actual risk, not that it's wrong. Unfortunately, I think the you-get-5-words version is the common understanding of the argument.
I think we agree that ASI is sufficiently vague that it just cannot support meaningful analytical work. So people fall back on sort of deep, intuitive, personal beliefs about The Way the World Is, and so the arguments get accordingly personal. (Perhaps where we differ is that I would put many other concepts in that 'too vague' category, including, but not limited to, 'outsmart' and 'humanity' and 'alignment'.)
And I'm sure that there are people making silly, pollyana-ish arguments about AI safety. But on the other side, you have people--important people! serious people!--who really are going around saying things like AI will soon be a wish-granting genie, or that we are a few months/years out from Dyson spheres and so on. Maybe they don't really mean it, but gosh, it seems like they really mean it! And I worry about those people as much as I worry about AI.
This conflates two things; the concept of ASI, which is vague, and the actual systems which are posited, which can absolutely be analyzed conditional on the specific assertions made.
I'm very aware that there is exaggeration, but there's also conflation of claims about the future conditional on ASI, and the time frame until ASI, along with confusion about the terms. I can't really address "serious people" in general unless the specific claims are specified - and when that happens, I often push back in detail. For example, Bostrom conditions on ASI, without a timeline, and then jumps to mortality claims which seem hard to support even given the assumption: https://x.com/davidmanheim/status/2023363906064802075
In the next two decades, I think a two-year global death toll in the seven-to-nine figure range is possible from a bioterrorism attack.
Possible? Sure, and you could have said that a decade ago too.
But if you want to bet, I'd happily put 10:1 odds against a seven-figure plus death toll within 2 years, say, before August 2028, from any bioterrorist / bio warfare incident.
A further point for your argument: AI making it easier to build bioweapons also makes it easier to build vaccines and other countermeasures. Defenders typically have far more resources, so more capabilities can be expected to lower the risks from bioweapons.
Regarding AI and cybersecurity, you may want to check out:
Deception and Detection: Why Artificial Intelligence Empowers Cyber Defense over Offense | International Security | MIT Press
https://direct.mit.edu/isec/article/50/3/86/135683/Deception-and-Detection-Why-Artificial
This short presentation by David Dalrymple on FlexHEG contains some remarkable claims such as “ … DARPA showed through the HACMS project that it is possible to create software systems that are completely free of exploitable bugs.” In the limit of capability, the defender “just wins” at the software level, especially with AI automation.
https://www.youtube.com/watch?v=4wgImjg9PPc
One exciting area for LLM cybersecurity is using them to produce formally verified software (https://en.wikipedia.org/wiki/Formal_verification). This is software that comes with a proof (which presumably the LLM would produce) of being secure under a specific model. Formal verification isn't perfect, the real world often differs from your security model. But it does seem like a nice tool to have.
Whoa, this is gold. Thank you! Cool MIT paper especially “AI models struggle with the creativity and deception necessary for offensive operations, but they excel at the pattern recognition that is key for defensive operations.”
It's true that defenders will have more resources and better AIs, but the situation for defenders still seems rough. Developing vaccines more quickly isn't really a bottleneck. Testing, manufacturing, and distributing them are bottlenecks, and those are also harder for current AI to assist.
If someone were on the fence about biorisk before considering how AI could benefit defenders, I agree this should make them a bit less concerned. But it seems to me that attackers will likely have much more uplift than defenders. We're mostly kept safe by the fact that there aren't many would-be attackers, not by a strong defense. That worries me!
In my mind, the situation is pretty symmetric wrt AI. Testing, manufacturing, and distribution are also challenges for attackers! If AI doesn't help defenders much here, it probably won't help attackers much either.
There are a few asymmetric benefits for defenders:
1. The attacker needs to commit to a vector, the defender can observe the vector and tailor mitigations. Defenders can try many solutions per vector, attackers get one shot per vector.
2. Humans are equipped with an adaptive immune system capable of clearing ~any infection given an appropriate stimulus and time to adjust.
3. Defenders have a larger body of research to build on. Bioweapons knowledge does (or did) exist, but has been effectively suppressed due to international treaties.
4. Sanitation is simple and has eradicated most infectious diseases. Air sanitation via HVAC, UV, and glycol vapors seems simpler than building a bioweapon.
That said, bioweapons are one of the most important global risks. I want to see much more work here! They just don't keep me up at night.
I'm by no means an expert, so I appreciate the pushback! But surely attackers have an enormous advantage when it comes to manufacturing and distribution: vaccines don't self-replicate and spread contagiously.
To take one example: in an individual case, cleaning the air in a building is easier than building a bioweapon. However, building a bioweapon is probably easier than scaling that solution to hundreds of millions of buildings.
This feels like the main challenge to me: we can probably figure out how to combat any disease, but can we do it in time and at scale? Probably yes, *if* we set up the infrastructure -- but today, the answer is probably *no* if the disease is bad enough.
True, I think we all three agree that bioresilience is definitely needed and overdue. We could have been doing so much on indoor air filtration, wastewater surveillance, rapid diagnostics, etc.
The attacker's manufacturing and distribution has to be concealed, while vaccines can advertise openly with a reasonable expectation of being actively sought out by friendly strangers.
This is a great piece, just to add a general statement to many good comments: there's a common thread in both AI doomer and AI aggrandizer spaces: they think AI will easily solve problems they don't understand. To make an example, one of the tropes is "AI will cure cancer", and implication is "fast". Well, it they actually talked to people who treat cancers for a living, whether on pharma or practitioner side, they would find out that
Yes, AI is likely to be helpful
No, it's not going to "cure" anything, it may improve testing, it may propose better molecules or delivery methods, and then it will be Phase I trials, etc. etc. etc. until we get a result a decade later.
And so it is every. freaking. domain. "AI will design chips", fast! Yes, it will be able to do design and even layout using fab-derived models, and then off to moths of fab cycles, validation and qual testing. "AI will fully automate manufacturing", yeah, wake me up when a robot can replace a rubber hose deep in the guts of a5-axis machining center.
I noticed this too! It actually inspired me to start mapping out all these frictions. I feel like it's the only way to make forecasts better!
Yes it is. And you will likely find the space to be so vast you need to narrow down and deal with only a few domains. Like, I’m a chip designer, and I know I have a long career ahead of me, AI will never “take my job”, but it’s already helping with some tasks, and I expect this to continue. But ask me about truck drivers, and I’d only have a vague idea - you need to talk to a logistics expert, automotive designers, maybe a radar/lidar specialist… this gets exhausting pretty quickly.
This article reminded me that pipetting is so difficult that an entire group of students in my first year chemistry labs just gave up and faked all their data 😅
Oh wow hahaha xD That’s a great example! o.o I haven’t tried pipetting myself but when I raised it with a room of biologists, there was a bit of a collective sigh.
Actually every single science except perhaps computer science has its own super annoying manual task - tagging fruit flies, placing electrodes on people’s head without messing with their hair, soldering wires and chips to PCBs or wiring breadboards, aligning optical components, mode locking lasers, etc etc
True!! I wanted to know what these were for bio, Reddit seem to have a collection of memes made my frustrated "labrats". My favorites were the ones on how a bacteria that eats dirt and seems hardy will not grow in lab conditions (the 2nd meme). Thought you might enjoy! https://www.reddit.com/r/labrats/comments/1nmstr6/on_public_demand_repost_of_some_of_the_best/
“Except perhaps computer science” - this very phrase holds a, if not the, key to the bizarre certainty that AI researchers and the adjacent crowd hold regarding AI overtaking the economy and becoming self-sufficient. These people are overwhelmingly CS’s, and seem to think all other disciplines are similar.
Computer science is unique in this respect thanks to the fact that it started out as a niche specialty within textile manufacturing, which otherwise consists just about *entirely* of annoying manual tasks. https://en.wikipedia.org/wiki/Jacquard_machine When further development within that specialty hyper-focused on pure abstraction, and gradually pushed out all the fiddly "eye-hand coordination" stuff onto other sub-fields, the added burden was barely noticed. Of course, it probably also helps that the former most labor-intensive sub-field (spinning raw fiber into yarn or thread) faced some orders-of-magnitude efficiency improvement around the same time. https://acoup.blog/2021/03/19/collections-clothing-how-did-they-make-it-part-iii-spin-me-right-round/
This is great! Your work has been really excellent in this space.
Aw thank you!!
Well done; rhymes in so many ways with the contrast between what cyber domain experts think about AI cyber risk vs what the AI safety community has been saying
Thanks so much!
What do cyber domain experts say about this?
I'm glad I'm finally reading some nuance in the doomsday approach towards AI.
Received some similar comments from a postdoc and a PhD student who currently work in a synthetic biology lab at a T10.
Great read!
Thanks Hamza! I appreciate it :)
Looking forward to your work on cybersecurity!
Thanks Sam! Working on gathering the experts now :)
Great post, but if I may, I have a technical nitpick.
>But more importantly: biology has built-in tradeoffs: the more lethal a virus is, the less transmissible it is (because it kills its human host).
Late to the party, but this statement is not in general correct, because it assumes that the inherent transmissibility ( probability of infection given contact with another infected individual ) and virulence are uncorrelated, while they most likely are.
In respiratory viruses, this claim is largely correct because of the simple fact that lethality depends on the virus' favored position on the respiratory tract: viruses that sit in the upper section are more transmissible but less lethal, and viceversa.
One can imagine however, "improving" the virus over other dimensions which are not subject to this trade-off, i.e. efficiency of cell entry and/or evasion of immune barriers.
Mathematically, the transmissibility R0 is the ratio of the contact rate with the sum of recovery rate and fatality rate (ie ratio of the influx of the infected class to the outflux). If the contact rate is positively but weakly correlated with fatality, then your claim is correct and increasing fatality beyond an optimal value will decrease R0. But if for some wretched reason making the virus more lethal makes it also much more transmissible then R0 will grow even with increasing fatality.
Thank you, Nemo. I really appreciate it! Perhaps I shouldn't have used the word "transmissible" because of it's specific technical definition. The brake I was wanting to emphasize there is that dead people don't spread viruses (typically).
You're right that if the same change makes a virus deadlier and also makes it spread faster, R0 goes up!
What I had in mind was the extreme case. If something kills nearly everyone it infects. Agreed that it doesn't always run that direction, especially once you break transmissibility into its parts!
As someone who doesn't have a doctorate, but has done extensive work in a hospital Microbiology lab, this tracks.
Most bioterrism threats come from pre-existing diseases. Anthrax, Plague, Tuleremia, smallpox, etc. Even then, there are a lot of limiting factors, that can't easily be worked around. For instance, none of the above are 'easy' to spread. Smallpox has a vaccine, Anthrax doesn't transmit between humans easily, tuleremia can't be easily aerosolized, etc.
New or established, Bioweapons are only likely to be deployed by the likes of Nation States, with the resources. (ie. Iran, North Korea, China, Russia, etc.) Even then, if they get caught, a bioweapon is everyone's problem.
Most of the funding in this area by my understanding is more about countering bioweapons than developing them for deployment. Even the ones we do develop often provide useful insights either for defence or other areas of research.
Whoa, thanks Colton. This is really succinct and helpful context. I've not heard of tuleremia before, fascinating history of it.
I'd love to learn more about these dynamics and the quirks of bioweapons/biology - do you have any books or authors you recommend?
Sadly not. Most my experience is from textbooks and various bioweapons preparedness training materials I've taken to stay accredited at the bachelor's level. At my level, we don't really see much except for incedentals.
Tuleremia is one that I saw fairly regularly at my local hospital though. That's an automatic CDC report, since it's a organism of concern.
I'll send you a DM with a reference for someone who might be able to hook you up though.
While current AI tools won't necessarily make the messy physical parts of making bioweapons easier, I anticipate that climate change would make strongmen more powerful, like these guys. https://borgenproject.org/cruelest-dictators/ who would be "comfortable killing [their] own people."
Thank you Sharon! Great point! In the longer piece, I concluded the same! Big events could happen that make this pool of malicious actors bigger!
For a second, I read “malicious” as “magical” and chortled with a goose honk 😆
Haha I did originally write magical! 😂 a typo
So, what types of interventions do you think are still needed versus those that are unnecessary, given your conclusion? For example, do you think AI red-teaming or efforts to prevent AI from acquiring tacit knowledge are still useful? - to what extend?
More broadly, when do you think people should start being concerned about these risks? Even pandemics like COVID-19 killed millions and cost trillion of dollars do you think it’s unlikely we’ll see something on that scale again, or do you think COVID wasn’t severe enough to warrant long-term concern?
To be clear, preventing the next COVID and pandemics should be a much higher priority than it is now. We need lots of interventions. More bioresilience is critical, especially since there’s a 2% chance of a natural pandemic every year. Rapid vaccine infrastructure, waste water monitoring, faster diagnostics, indoor air filtration, etc, are great as well as making sure that critical equipment and materials for making bioweapons stay controlled.
More studies on AI impact on real human participants actually trying bioweapon creation would be helpful and is already starting to happen. When I think of AI acquire tacit knowledge, I expect that most effective in robotics where hard tasks are automated. Those are pretty narrow in their application and resource-intensive to set up so oversight of them is a lot easier than of models themselves.
Gain of function research seems to be predominantly dealing with real-world materials. Do you mean people using AI to review sequences, or are you talking about future lab automations?
@eurydicelives also wrote about this. It was a while ago, and I don't remember it that well, but I figure it's worth mentioning, as the gist I can remember was that biology is hard and expensive even if you somehow solve knowledge.
https://eurydicelives.substack.com/p/pdoom-is-least-likely-to-come-by
Oh cool, I'll check this out, thanks!
Great stuff, Abi.
Have you ever talked with Jason Gaverick Matheny, now at RAND?
And in terms of fiction, have you read Oryx and Crake? A great book, IMO. (And Vonnegut's Galapagos is fun as heck.)
I have, he's cool. I haven't seen that book, I've been meaning to read more Vonnegut!
This provides a useful voice of reason against some of the most extreme takes in biosecurity, but I think a lot of it is wrong.
Most importantly, it misses the trendline on technology improvement and how great the threat is from state actors already.
I wrote a full (rather long) response here: https://oliviahelens.substack.com/p/bioweapons-as-the-optimists-exception
This is really interesting Abi, thank you for writing it! I’ve been worried about this for years and this post genuinely helped me feel better.
One story I had heard in the past is: a terrorist tried to order a smallpox sample and almost succeeded (the person dispensing the sample literally had an intuition that it shouldn’t be mailed, and double checked, and that was the only thing that stopped the terrorist from receiving the sample - there wasn’t like, a process in place to stop this from happening!). It is helpful to know that even if the terrorist *had* received the sample, it would have been quite hard for them to use it.
Thanks Lydia! I'm so glad to hear that! :)
I'm so curious about this case. I can't find it online, but would love to know how they ordered it. It's illegal in the US and the UK (and probably the rest of the world?). If it was some variant: when people order viruses that could be dangerous, they often come weakened in ways that it would take a skilled expert to reverse. I'm not sure what type of sample they had, but I doubt it was the live virus!
I expect the example you're both thinking of is the origin of the US Federal Select Agents Program — Max Langenkamp and I describe the story at the start of this article: https://press.asimov.com/articles/dna-screening
To quote:
> In May 1995, a few weeks after the doomsday cult Shinrikyo Aum released sarin gas in the Tokyo subway—killing 13 people and injuring 5,500 more—two Ohio police officers knocked on Larry Wayne Harris’s door. Harris was a trained microbiologist and a former member of the white supremacist Aryan Nations group. The officers held a search warrant for one of the deadliest organisms in history: Yersinia pestis, the bacteria that causes bubonic plague.
>
> A concerned employee at a laboratory supply company had called the Centers for Disease Control, or CDC, after fielding repeated calls from Harris asking when his samples of plague would arrive in the mail. When the CDC contacted Harris to ask about the plague samples, Harris claimed that he was doing research for the CIA to stop an “imminent invasion from Iraq of super-germ-carrying rats.” In the end, falsely asserting that he was working with the CIA was the only illegal thing Harris did—amounting to wire and mail fraud.
Damn girl! Great job 🙌
The story I heard was some years ago, and I think it was already years old by then. But it was definitely a US case and the terrorist in question was a white supremacist.
Some other stuff I’ve read that terrified me on this score: Ken Alibek’s book ‘Biohazard’ (nonfiction tell-all about his time in the Soviet bioweapons program) and Rob Reid’s book ‘After On’ (hilarious science fiction novel by someone with a lot of time served in Silicon Valley, strongly recommend). Both these books are great but both also were written by people whose main threat model was “we have to be insanely paranoid about keeping the pathogens contained” rather than having the more complex model you outlined in your post.
Great flags, thanks so much Tessa for pulling this up. I thought that since Yersinia pestis is on the Federal Select Agent Program, that it's illegal to procure. Was it not illegal for him to buy it? Because he was one of the few people who could buy Select Agents or something else? Is that why the lie was the legally cruxy thing?