An Anthropic Researcher Explained How AI Actually Thinks. It Changes How Much You Should Trust the Output.

The next time an AI gives you a clean, confident answer, the useful question isn't "does this sound right?" It's "what process produced this, and does that process actually track truth?" An interview released this month from inside Anthropic's own research team gives a straight answer, and it's not reassuring.

What happened

On October 18, Lenny Rachitsky interviewed Emmanuel Ameisen, an interpretability researcher at Anthropic whose job is literally to look inside language models and describe what's happening. Two claims stood out. First, models are "grown, not built" — nobody writes the reasoning steps a model follows, they emerge from training, and even the people who built the model don't fully know how a given answer got produced. Second, models form internal concept representations and plan several steps ahead of the word they're currently generating, which means there's a real process behind the output. But that process runs on math that has no reason to resemble how a person reasons. It can be sophisticated and still be alien — pattern-matching that lands on the right answer for reasons that don't map to the explanation the model gives you afterward.

Why it matters Monday morning

This isn't a reason to distrust AI more broadly — it's a reason to stop using "it explained its reasoning" as your check. A model's stated reasoning is itself generated text, produced by the same alien process as the final answer. If the process and the explanation of the process both come from the same box, a wrong answer can come wrapped in a reasoning chain that sounds completely sound. The skill this demands isn't philosophical. It's concrete: separate what the model claims from what you can independently confirm, and do that separating before the output reaches anyone who'll act on it.

Two people, same gap, different stakes

A marketing manager at a 40-person SaaS company asks Claude to summarize a competitor's new pricing page and explain the strategic logic behind the change. The model returns a fluent paragraph: here's what changed, here's why, here's what it signals about their roadmap. It's well-organized and sounds like something a sharp analyst would write. The problem is that "why" is the part the model is least equipped to actually know — it's inference dressed as observation, and the manager has no way to tell which sentences are describing the page and which are guessing at motive, because the model doesn't flag the difference itself.

A researcher at a policy nonprofit runs the same test differently: she asks the model to summarize a study, then asks it to argue the opposite conclusion from the same data. Both arguments come back equally fluent and equally confident. That's not a bug — it's the same underlying point from the interview showing up in practice. The model isn't attached to a position; it's completing a pattern. The gap she closes isn't by asking the model to double-check itself, since that's still the same process checking its own output. She closes it by asking a second, differently-framed question and comparing the two answers against each other, not against her gut sense of which one "feels right."

The takeaway

Fluent reasoning and correct reasoning are produced by the same process, so a model's explanation of itself is no more trustworthy than its answer — check the claim, not the confidence behind it.