ChatGPT is Not Your User

ChatGPT can’t tell you what your actual users think, feel, need, or struggle with.

Over the past several months, I’ve observed a disturbing trend:

Teams using AI large language models (LLMs) like ChatGPT, Claude, Gemini, etc. to conduct UX research.

Most seasoned design and research professionals understand why this is a bad idea, but there are far too many in the UX community who think LLMs unlock some kind of research shortcut.

This post is for you.

Don’t do this! Website where I found this is redacted to protect the guilty.

Why smart teams still talk to real people

Let’s get something out of the way up front: The capabilities of LLMs are impressive — and getting better every day. With the right prompt, they can analyze and summarize huge swaths of data, produce product specs, generate code, and provide personalized recommendations.

But here’s the thing it can’t do: It can’t tell you what your actual users think, feel, need, or struggle with.

And yet, more and more UX and product folks are turning to these tools to shortcut user research. Ask it what “a millennial traveler wants from a loyalty program” and it’ll answer in seconds. But that answer, no matter how coherent it sounds, isn’t grounded in the messy, contextual, unpredictable reality of your actual users.

The Illusion of Insight

The problem isn’t that ChatGPT gives you the wrong answer — it’s that it gives you a plausible one (when it’s not hallucinating). A confident, articulate, and potentially biased synthesis of content sourced from across the open internet. And when you're under pressure to make progress, it’s tempting to accept that answer as truth.

But let’s be clear: ChatGPT wasn’t trained on your customers. It doesn’t know your product. It doesn’t feel your market nuance. It doesn’t understand the little frictions or behaviors that define real user needs. 

You know who does understand? The people who you’re actually designing and building your product for. I realize that there can be real challenges to speaking directly to your customers/users when you need to conduct user research, but when you substitute an LLM for speaking to people, you’re not speeding things up — you’re paving a smoother path to building the wrong thing.

Real Research Still Matters

If you want to build great products, you still have to talk to people because people will surprise you. Observe them (they will do some weird shit). Ask them dumb questions. Watch them click the wrong button three times before they tell you everything’s “intuitive.”

That’s the real work. Observations can be contradictory. Things you thought were so obvious can be enigmatic to your users. It’s messy, slow, and often unglamorous. But it’s also where the gold is.

The good news? There are ways to make that gold more accessible across your team.

AI is a Tool – How to Use it Responsibly & Effectively

LLMs absolutely can play a positive supporting role in the UX research process — if you use them appropriately. Some smart, responsible ways to leverage AI as a tool for product teams include:

• Summarizing interview transcripts to quickly spot themes

• Generating draft interview questions to save time when planning sessions

• Analyzing large sets of qualitative data to highlight patterns and outliers

• Synthesizing survey results into digestible insights

In other words, use AI to accelerate and organize real research — not replace it.

A new app opening for beta users caught our eye recently, Rooost — a tool that lets teams chat with AI personas trained specifically on their own user research. It’s not a general LLM trained on the internet — but a dynamic persona that reflects your real user data.

That’s the kind of LLM assist we can get behind because when the model is grounded in your real data, you’re accessing insights that are relevant and specific to your team’s needs.

TL;DR

Don’t mistake generative output for generative insight. AI can be a powerful tool — but it works best when it amplifies the voice of your users, not replaces it.

So keep talking to your customers. Then bring their voice with you into every sprint, every brainstorm, and every decision.