Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation
Rachel Banawa · 16 October 2026
Read the originalNielsen Norman Group found that 120 AI-generated survey respondents reached the same headline answer as 604 people but lost most of the variety in their wording.
Nielsen Norman Group, a user-experience research and training firm, repeated a question from an earlier survey of 604 tech and design professionals. This time it asked 120 synthetic respondents from the Synthetic Users platform. Both groups named UX as the main term for the field. The synthetic group used about a dozen labels against more than 50 from people, and never produced terms such as human factors or experience design.
- 84% of synthetic respondents used UX-related terms, against about 70% of the human respondents.
- Among product managers, 93% of synthetic respondents used UX-related terms, against 44% of human product managers.
- The synthetic platform sometimes reworded the question, which the author notes may have shaped the answers.
- The firm recommends synthetic findings for forming hypotheses, and research with people when rare answers or differences between groups matter.
Keeping a small sample of real people beside any synthetic study, to show where the synthetic answers converge too neatly, is an inference and is not the author's claim.
Use synthetic respondents to draft hypotheses, and run research with people when rare answers or differences between groups matter.
Compare any study that uses synthetic respondents with answers from real people before treating its findings as evidence.
Derived by Working Surface from the article; more in the Playbook. Source line: Our synthetic respondents collapsed onto a dozen labels; the humans spread across more than 50.
17 October 2026: AI output that looks finished at first glance still needs a named person who checks it against something real, and teams now plan their work around that check.