Problem Solving · Lesson 4 of 4

Evaluate, test, learn, iterate

Write what you expect before you test. Unrecorded expectations always turn out to have been correct.

6 min read · Free — no account needed to read

Evaluating means comparing options against what you actually want, not against each other. Write the criteria before you see the options where you can — otherwise the criteria quietly reshape themselves around the option you already prefer.

Then find the smallest version of the solution that would teach you whether it works. Not a small solution — a small test. The question is always: what is the cheapest thing that would change my mind?

Iteration is what makes this work, and it requires writing down what you expected before you run the test. Without a recorded expectation, every result will feel roughly like what you thought would happen.

See it happen

You change the headline on your page to see whether more people sign up.

The read that arrives on its own

A week later: signups are up a little. The new headline is better. Keep it.

The same situation, with this idea

Before changing it you wrote: "I expect about 15 more signups; under 5 means the headline is not the constraint." You got 3. That is informative — the headline was not what was stopping people.

What changed

Without the written expectation, any result gets absorbed as confirmation. Unrecorded expectations always turn out, in memory, to have been correct.

The takeaway

Write what you expect before you test. Unrecorded expectations always turn out to have been correct.

Easy to take the wrong way

Changing several things at once. When it moves you will not know which one moved it, and you will carry the wrong lesson into the next attempt.

Try this

Take a plan you are confident about. Design the cheapest test that could show it is wrong.