Do AI-generated survey respondents say ‘not sure’ often enough?
An AI-generated audience may give you a confident opinion where a person would still have a question. In a September 30 report, Pew Research Center compared real survey answers with AI ‘digital twins’ given panelists' demographic information and earlier political views. Across opinion questions with an explicit ‘not sure’ option, humans selected it 16% of the time; synthetic respondents selected it 4% of the time. The model had been told that not knowing was realistic. These results come from a particular setup using Claude Opus 4.6 and three US survey waves, mostly about politics. They aren't an error rate for every AI focus group, and Pew wasn't testing customers reading a product page. Still, I'd be wary of asking simulated customers whether an explanation is clear and treating their decisive answers as reassurance. The uncertainty you need to discover may be exactly what the simulation leaves out. Use the invented audience to suggest questions, then let actual intended readers say what they don't understand before you remove the help text. That's an application I'd draw from the finding, not a customer-research result Pew measured. A person who needs something explained shouldn't disappear from the design because their AI stand-in already knew the answer.
Comments
I'd check whether people can tell what you're selling before asking which headline they prefer. For a hypothetical bookkeeping service, an AI audience might happily debate ‘Find your financial freedom’ against ‘Get your evenings back’. A real visitor may still wonder whether they're buying software, an accountant or a course. Show intended customers the headline without explaining the business first and ask what they think they could buy. ‘I can't tell’ is a useful answer. That's a messaging test I'd propose, not something Pew studied. A simulated preference shouldn't persuade you to keep a line that leaves actual readers guessing.
A team shouldn't get to rewrite its imaginary customers until they approve the launch. Before trying a simulated audience, save the question, the profile instructions and what result would make you change course. Keep the unhelpful runs, too. That's a safeguard I'd propose, not something Pew tested. If the simulation goes into a budget request, say how many actual customers were consulted—even when the answer is none. Using AI to explore possible objections can be useful. Calling the resulting approval ‘customer validation’ gives the people selling the idea far too much control over the witnesses.
A mistaken belief needs a different explanation from a missing answer. Pew also found that its synthetic respondents almost never picked a factually wrong answer: when they didn't give the correct one, they usually chose ‘not sure’. For a hypothetical returns-page check, I'd want to hear both ‘I don't know the deadline’ and ‘I thought opening the box meant I couldn't return it’. Those call for different edits. I'd ask real readers what they believe the policy allows before showing them the explanation, then check whether that belief changes. That's a proposed customer-research check, not something Pew tested. Otherwise we could make the deadline easier to find and leave the misconception that stops someone asking for help untouched.
I wouldn't use a simulated audience's approval to cut launch-day support cover. Suppose a team is replacing a setup call with an AI-written guide: have a few actual customers try the setup without coaching, with someone available if they get stuck. Record where they ask for help and whether they finish, not just whether they say the guide is clear. Keep the calls available until that trial gives the support lead a reason to reduce them. That's a rollout test I'd propose, not something Pew measured. Staffing is a much harder decision to undo once customers are waiting.
I'd have someone try it on the phone they'll actually use, with the keyboard open. ‘I understand the instructions’ doesn't tell you whether they can still see the next button while entering their address. If they get stuck, keep the screen they were looking at alongside what they said; otherwise an interface problem can come back as a request to rewrite perfectly clear words. That's a check I'd add to this trial, not a finding from Pew. A simulated reader's answer can't show you where a person's thumb runs out of room.