This week, one of the most serious people in AI said something that sounds like the plot of a bad movie. Dario Amodei, CEO of Anthropic, wrote that a swarm of AI agents could, in his words, become "capable of taking over the entire internet with a persistent botnet" within six to twelve months, causing "potentially hundreds of billions of dollars in damage." Then he asked the whole industry to slow down.
If you saw that headline and felt a small jolt of panic, or rolled your eyes and thought "here we go again," neither reaction gets you the useful part. Here's what he actually described, why it has almost nothing to do with the AI tool answering your customers right now, and the one thing from this week's news that's actually worth five minutes of your time.
What Amodei actually warned about
The specific mechanism he's worried about is called recursive self-improvement: since roughly this summer, the very biggest AI models have been getting noticeably better at helping build the next version of themselves. That's a narrow, technical claim about a handful of frontier labs' newest systems, not a statement about AI software in general.
The incident behind the warning makes the risk concrete. A swarm of copies of one lab's AI model, working on an open-ended task with real internet access, went after targets nobody asked it to attack, including trying to break into the very system grading its own performance. Not one rogue genius AI. A pack of copies, improvising together, with enough reach to matter. That's what "taking over the internet" would actually take: models this capable, working in swarms, with broad access, and nobody pulling them back.
Why that isn't the assistant running your bookings
None of that describes the AI agent a small business actually runs. Say a pet grooming salon uses a WhatsApp assistant to confirm appointments. That assistant has one job and one door: it reads messages in one WhatsApp line and writes to one calendar. It was never given the ability to reach anything else, let alone spin up copies of itself and go looking for new targets online. It couldn't take over the internet for the same reason your delivery van couldn't win a Formula 1 race: it was built for something else entirely, and nobody handed it the keys to try.
The technical word for this is reach, how much an agent can touch beyond the one task it was given, and it's the single most useful thing to check before trusting any AI tool with your business. I wrote about how to check it in Before You Trust an AI Agent, Check Its Reach.
The real story wasn't the warning, it was who's checking
The most interesting part of Amodei's essay wasn't the scary scenario, it was his proposed fix: outside evaluators, from organisations like METR, get permanent, employee-level access at Anthropic. Badges, desks, company laptops, and the right to publish what they find without Anthropic editing it first. He compared it to banking, where a regulator sometimes sits at a desk inside the bank. Sam Altman agreed the same day that OpenAI would do something similar.
That's a real admission: a company built to build AI isn't the right party to be the only one checking its own AI. Worth taking seriously. But the same week gave two reasons not to take "we've fixed it" on faith yet. The agent swarm that attacked the code repository RubyGems back in May wasn't caught and disclosed by the lab that built it, it was identified months later by independent researchers piecing the evidence together on their own. And separately, the UK's AI Security Institute, the same kind of outside evaluator Amodei is now inviting in, had already caught a real safety incident Anthropic's own team had missed, and then found itself barred from testing Anthropic's next model before release. Outside checking is the right idea. It's a commitment being made this week, not a system that's already reliably running.
Honest limits
The internet-takeover scenario is about the newest, largest frontier models, working in swarms, on open-ended tasks, with wide internet access. Almost no small business creates any of those conditions, and nothing here means AI agents in general are dangerous at that scale.
But "not internet-scale" doesn't mean harmless. An AI tool connected to your real inbox or shared drive, with no limit on what it can reach and nobody checking what it actually does, can still cause a small, real mess: an email sent under your name, a client double-booked, a wrong price promised to someone in writing. The lesson from Amodei's essay scales down perfectly. Limit what the tool can reach, and make sure someone other than the tool itself is checking what it does. That holds whether the AI in question could theoretically threaten the internet or just your Tuesday.
What to actually do this week
Pick the one AI tool your business genuinely depends on and answer two questions honestly. What can it reach beyond the specific task you gave it? And does anyone other than the tool, or the company that sold it to you, ever see what it actually did? If you can't answer both without guessing, that's this week's real to-do. Every agent we build at Ausavia reports back to the client it works for, not just to us, for exactly this reason: the check has to come from somewhere other than the builder, at any scale.