Understand the Machine · Lesson 3 of 3

Where AI is currently unreliable

Treat AI output as a capable first draft from a colleague who has never met you: useful, fast, and in need of checking.

6 min read · Free — no account needed to read

AI can be confidently wrong. It can invent sources, numbers or quotes ("hallucination"), and it usually sounds just as sure when it is wrong as when it is right.

It only knows what it was trained on and what you give it. It may be out of date, miss local context, or lack the information that actually decides your situation. It has no direct lived experience of your life, your team or your town.

It can struggle in unpredictable ways: long chains of reasoning, rare situations, tasks where quality is hard to judge, and questions about what matters. These limits are shrinking in some areas and not others — which is exactly why checking, context and judgement stay your job.

See it happen

You ask for three studies supporting a claim you are about to make in a presentation, and you get three titles, three authors and three years.

The read that arrives on its own

Three citations, properly formatted, so you paste them onto the slide.

The same situation, with this idea

You look up each one. Two exist and say roughly what was claimed. The third has a real author, a plausible title, and was never published. You drop it and read the abstracts of the other two.

What changed

How confident the output sounded carried no information about whether it was right — all three looked identical on the page. Checking was the only thing that could tell them apart, and it cost four minutes.

The takeaway

Treat AI output as a capable first draft from a colleague who has never met you: useful, fast, and in need of checking.

Easy to take the wrong way

Assuming the unreliable parts announce themselves. A wrong answer usually arrives in the same calm, well-organised voice as a right one, which is exactly why "it sounded sure" is not evidence of anything.

Try this

Ask an AI for three sources on a topic you know well. Check every one. Note which were real, which were wrong, and how confident it sounded.