Imagine a small physio clinic. Every afternoon an AI agent sends tomorrow's appointment reminders, and every afternoon it reports back: "Done. 14 reminders sent." The owner, Núria (invented for this piece), reads that line, feels a little lighter, and goes home.
Two of those patients had changed their phone number last month. Their reminders went nowhere. Two empty slots the next morning, and a report that said everything was fine.
Nobody lied here. But the report and the reality were two different things, and the owner only ever saw the report.
The word "done" is a sentence, not a fact
When an agent says it finished, it is writing a summary. It is very good at writing summaries, and a summary reads just as confident whether the work went well or not. A language model produces the most plausible account of what happened, and "everything went fine" is a very plausible account.
Add one more ingredient. Tools tell agents "OK" when a request was accepted, not when the result is right. A message handed to the phone system is "sent". Whether it arrived is another question, and the agent often never looks.
That is my explanation of the mechanism, not a finding from a lab. The lab evidence is what makes it worth taking seriously.
What the researchers found when they checked
A team at Microsoft and Hugging Face recently published a benchmark called ThinkingBox. They gave 18 AI models 507 business workflows (retail, insurance, travel, banking, consulting) and, unusually, they did not grade what the agent said. They graded what was left in the database afterwards.
One example stuck with me. A customer's $745 appliance was 15 days late. The agent made nine tool calls, gathered the right information, correctly decided the customer did not qualify for compensation, and then closed the ticket as "resolved". It should have stayed open, on hold, until the carrier sorted the delivery. The summary would have looked perfect. The database disagreed.
Then they repeated each task 20 times. The best model got a single try right about two times in three (67%). But only 241 of the 507 tasks (about 48%) came out right on all 20 tries. As the authors put it, one good run shows a model can do the work, not that it will do it again.
A separate study of more than 20,000 real coding sessions reached a similar place from another direction: inaccurate self-reporting is one of the recurring ways agents let people down. That one is about coding, not clinics, so I would not stretch it. It does point the same way.
What to do instead: check the thing, not the report
You do not need to become a technician. You need one habit: find the place where the result shows up for real, and look there.
- Sent a reminder? Open the sent log, or ask a patient.
- Chased an invoice? Look at the invoice status in your billing tool, not the agent's note about it.
- Updated a booking? Open the calendar.
Three questions help. If the job really happened, what would I see? Where would I see it? Can the agent tell me, or does something else have to show me? The last one matters most. An agent marking its own homework is the weakest check you can have, so where possible the evidence should come from a different place than the agent's own words.
For a fast version, compare counts. "14 reminders sent" against 14 entries in the log takes thirty seconds and catches the dull failures, which are the common ones.
Honest limits
Checking costs time, and if you check everything you have rebuilt the job you hoped to hand off. I wrote before about what really needs your eyes and about how a check turns into a rubber stamp. Both apply: check the things that cost money or promise something, and sample the rest.
Some results are invisible. You can confirm a message went out, but not that the patient read it. And an automatic checker is often another AI, which has the same weakness one level up. I do not think anyone has fully solved this yet, and I would be wary of anyone who says they have.
If a task has no place where you can see the result, that is a good reason to say "not yet" to automating it.
One thing to try this week
Pick one task your AI tool does for you. Find where its result is visible without asking the tool. Then take the last five times it reported "done" and check each one against that place. If all five match, relax a little and sample monthly. If one does not, you have just learned more than any dashboard would have told you.