What that viral AI persuasion study actually measured (and didn't)
After digging into persuasion research for months, I see problems with how people are interpreting AI-persuasion findings. And it matters for how we think about AI influence going forward.
Another study came out finding that AI is more persuasive than humans.
People sometimes read these and conclude that this is evidence that AI is capable of societal-wide influence, corporate takeover of public opinion, something akin to a mind control tool.
To be clear: I also dread these risks, but the current findings don't point to that despite the author's conclusion that "AI is more persuasive than humans."
As someone who spent months researching and interviewing persuasion and misinformation experts for my Bulletin of the Atomic Scientists research, I came to a more nuanced conclusion.
To me, the study only shows:
For a lower cost than humans, AI systems can persuade, at least momentarily, if a lack of accurate or resonant information is a bottleneck. Thus, only topics where political opinions aren't strongly held (this is in line with decades of research on changing beliefs).
The study: 900 U.S. participants debated digitally with either GPT-4 or another person about politically salient U.S. topics that were low, medium, or strongly held. Sometimes, the opponent knew personal facts about the debater. Authors found that AI systems swayed low and medium strength opinions more than humans did.
What came to mind when reading this paper:
AI did not impact the "strongly-held" beliefs at all, continuing to indicate there's an upper limit to how persuasive anything can be. Strong beliefs are set by pre-existing identity groups, personality traits, etc. So far, I haven't seen much or any data that points to a persuasion method that can show a change that persists despite these.
Studies consistently show that people appear persuaded immediately after experiments, but the effect doesn't last weeks or months down the line. This study by design will overstate persuasion effects. People go back to their prior views later because of their identity groups. People sometimes even hold beliefs they know are wrong if it is held by their identity group.
The study compared AI systems against untrained people, who likely lacked background information in the topics to be able to persuade. Ideally, another study tests AI systems against people trained on bridge-building in their topics of expertise. They'd have both resonant data AND the awareness to present resonant data in other people's terms. As the current study is structured, it's like a debate where only one person gets to have a menu of persuasive information but the other doesn't. The study is still very valuable though! But I don’t want to overstate claims. (This reminds me of Steve Newman on how METR evaluations overstated capabilities by comparing AIs to humans who were completely new to the same task.)
What the study understates: AI is likely even more persuasive than humans if the study had included topics that not salient political and societal issues. For a lot of non-salient issues, basic information on the topic is a big factor in someone forming their opinion. The AI system excels at having basic information at hand for any topic. But, the study would start to explore "can AI inform" instead of "can AI persuade." But this is also what makes me think twice about the fact that AI only influenced low and medium strength beliefs.
Where I think the study is wrong: The authors conclude this indicates AI systems could misinform people to change their beliefs. BUT testing for that would entail a totally different study design! The AI was persuasive because its information was accurate and trusted. Multiple studies show fake news is not widely believed (except where people have strong identity-linked beliefs that lead them to seek it out). Someone would have to set up an AI model to create the only type of misinformation that appears effective: "spin, exaggeration, framing, misleading causal narratives, testimony from fringe but genuinely credentialed experts." Someone would have to ensure it's subtle enough or existing-across-enough-trusted-institutions that someone doesn't fact check. See Dan William's writing on the few examples of highly effective misinformation.
I still think it’s something continue watching closely but we don’t necessarily need to assume that AI will be a tool for zero-sum persuasion wars. It could potentially democratize access to high-quality information and reasoning. If that becomes more widespread and easily accessible, we might see that fewer identity groups able to coordinate bad epistemics. It's wishful thinking but this is why I'm excited about AI for epistemics projects like Cosmos Institute's grants for truth-seeking AI projects.