Separating what people said from what it means, and noticing when you are hearing what you hoped for.
12 min read · Level 1
This is where confirmation bias is strongest, because you have invested weeks and interview notes are ambiguous enough to support almost any reading. The discipline of writing "fact" or "opinion" beside each line is unglamorous and catches an enormous amount of self-deception. And the people who did not reply are data too — a problem people genuinely have gets better response rates than one they do not.
After the interviews, you have to decide what they actually showed. That means separating three things:
Facts — what they did, with dates and numbers Opinions — what they said they think or would do Your interpretation — what you concluded
Only the first is evidence.
Because confirmation bias is at its strongest here. You have invested weeks, you want a particular answer, and interview notes are ambiguous enough to support almost any reading.
The discipline of writing 'fact' or 'opinion' beside each line is unglamorous and catches an enormous amount of self-deception.
From one interview:
'Yeah, that's a real pain. We spend ages on it every week. I'd definitely use something like that.'
Split it up:
Now the same person, later:
'Last Friday it took me about three hours. I tried a tool last year, paid £29 a month for four months, then stopped because I had to re-enter everything manually.'
The second passage is worth more than fifty of the first. And notice: the same person produced both. The difference is entirely in what you asked.
First-time founders often count enthusiasm. Ten people saying 'that sounds great' feels like validation and is not — nobody has changed their behaviour or spent anything.
The second: ignoring the people who did not respond. If you contacted 60 and 12 replied, the 48 are data too. A problem people genuinely have gets a better response rate than one they do not.
The third: averaging across segments. If ops managers are desperate and finance directors are indifferent, the average is 'moderate interest', which describes nobody. Split the analysis by who they are.
Not your summary. The phrases people use are also your future marketing copy.
Tedious and extremely effective. Most 'validation' turns out to be opinion.
Unprompted mentions are far stronger than agreement with something you raised.
Anyone who has paid for a partial solution has already proven willingness to pay. That is the strongest single signal available.
If one group is desperate and another indifferent, you have found your ICP — not a moderate average.
Before analysing. It is much harder to move the goalposts once they are written down.
After every batch of roughly ten interviews.
Do not do a formal analysis after two conversations. There is nothing to analyse and you will over-read whatever you heard.
Superhuman publicly described using a survey asking users how they would feel if they could no longer use the product, and treating the proportion answering "very disappointed" as their working measure of product-market fit. They then focused product work specifically on the segment that answered that way.
Pick one behavioural or high-signal question and track it over time. A consistent measure beats a pile of anecdotes you re-interpret each month.
A frequent analysis error: ops managers are desperate, finance directors indifferent, and the blended result reads "moderate interest" — which describes nobody. The split between segments was the actual discovery, and averaging destroyed it.
Segment before you conclude. A clear divergence between groups is a finding, not noise to be smoothed away.
Create a free account to read the rest and apply it to your own startup.