Political Experts Often Struggle to Predict Which Messages Will Persuade Voters

A recent study found that political professionals aren't much better than the general public at predicting which messages will actually persuade voters. It suggests that relying on intuition alone is ineffective, and political pros would benefit from using data-driven testing to find messages that truly resonate.

Broockman, David E., Kalla, Joshua L., Caballero, Christian, and Easton, Matthew. "Political Practitioners Poorly Predict Which Messages Persuade the Public." University of California, Berkeley and Yale University, January 2, 2024.

What Did They Study?

The study set out to understand how accurately political professionals can predict the effectiveness of different messages in persuading voters. The goal was to see if political insiders, often tasked with crafting persuasive strategies, could reliably identify which messages resonate with the public or if there’s a need for more evidence-based approaches in messaging.

How Did They Perform The Study?

The researchers collected 172 real-world messages from political campaigns and advocacy groups, each supporting or opposing various political issues. They then conducted a large survey experiment, exposing over 21,000 participants to these messages to measure their actual persuasive impact. Following this, both political professionals and laypeople were asked to predict which messages would be the most persuasive, resulting in over 22,000 predictions from practitioners and more than 63,000 from laypeople.

What Did They Find?

The study found that both political professionals and laypeople performed only slightly better than chance at predicting which messages would persuade voters. Surprisingly, practitioners—regardless of experience, issue expertise, or political alignment—did not significantly outperform the general public. Even within specialized subgroups, such as seasoned campaigners or self-identified experts, accuracy was low. Practitioners’ backgrounds, demographics, or social networks did not improve prediction accuracy.

Overall, the results indicate that intuition alone is unreliable for crafting persuasive messages, suggesting that political professionals could benefit from data-driven testing to identify messages that genuinely resonate with voters.

Takeaways

• Use Experimental Data Over Intuition: Practitioners should rely on experimental data, like survey or field experiments, rather than intuition, to identify effective persuasive messages. Given the poor predictive accuracy shown by experts, data-driven testing is likely more reliable.
• Adopt Message Testing Practices: Political professionals should systematically test a wide variety of messages rather than assuming certain ones will be effective. The authors suggest that some messages can be significantly more persuasive, but identifying these requires testing.
• Integrate Data in Decision-Making: The authors encourage a shift toward data-backed decision-making in political messaging. They observe that while data-driven approaches are growing, there is still heavy reliance on intuition among some practitioners.