OpenAI says its models are more persuasive than 82 percent of Reddit users

We’re still well short of OpenAI’s “Critical” persuasiveness threshold, where a model has “persuasive effectiveness strong enough to convince almost anyone to take action on a belief that goes against their natural interest.” That kind of “critically” persuasive model “would be a powerful weapon for controlling nation states, extracting secrets, and interfering with democracy,” OpenAI warns, referencing the kind of science fiction-inspired model of future AI threats that has helped fuel regulation efforts like California’s SB-1047.

Even at today’s more limited “Medium” persuasion risk, OpenAI says it is taking mitigation steps such as “heightened monitoring and detection” of AI-based persuasion efforts in the wild. That includes “live monitoring and targeted investigations” of extremists and “influence operations,” and implementing rules for its o-series reasoning models to refuse any requested political persuasion tasks.

That might seem like overkill for a model that only has human-level persuasive writing capabilities. But OpenAI notes that generating a strong persuasive argument without AI “requires significant human effort,” while AI-powered arguments “could make all content up to their capability level nearly zero-cost to generate.” In other words, OpenAI is concerned about a flood of AI-generated, human-level persuasive arguments becoming an incredibly cost-effective form of large-scale astroturfing, as we’re already starting to see.

It’s annoying enough to live in a world where we have to worry that random social media arguments are merely the product of someone with a lot of money to throw at an AI model. But if we advance to a world in which those models are effectively hypnotizing world leaders into bad decisions, rest assured that OpenAI will at least be on the lookout.