Say you run a small guesthouse. Your website lists the rooms, the breakfast hours and the price for two nights. There is also a little blog, and the ads on it bring in a few euros a month. One evening a guest tells their phone assistant: "find me a quiet room near the harbour, two nights, under 120 euros." The assistant goes to check your website. And it finds the door shut.
Nobody told you. Nobody asked you. A setting you never looked at made the call.
Your website has three kinds of robot visitors, not one
Most of us still think of "AI bots" as one lump. The people who run the plumbing of the web no longer do. Cloudflare is a company that sits in front of a large part of the internet's websites, and your hosting company or web designer may have switched it on for you. Its documentation now sorts automated visitors by what they are doing:
- Search: it "collects or indexes your content so it can answer questions about it later."
- Agent: "automated activity acting in real time on a person's behalf to get something done." That is the guest's phone assistant.
- Training: it "crawls your content to train or fine-tune a model, permanently absorbing your data into the model."
One bot can do more than one of these, which is where it gets messy. But the split matters because the right answer is different for each. Letting all three in or shutting all three out are both choices, and neither is automatically the careful one.
What changed in September
According to a PYMNTS report from 17 September, Cloudflare split its single "Block AI Bots" switch into three controls, one per class. It also set a new default for sites that show ads: search bots are let through, training crawlers are refused, and agents acting for a person are blocked on any page that shows an ad. The reasoning, as the report describes it, is that an ad suggests the page was built for a human to look at.
Who gets this default is a little unclear. The report says it covers new domains, new sites from existing customers and free-tier accounts. Cloudflare's own changelog, as far as I could find, mentions only new domains. If you have an old site, you may not be affected. If you launched one recently, or you run ads, check.
Which of the three should you block?
Training. This is your call, and I walked through it in Can You Stop AI From Training on Your Website?. Blocking costs you very little.
Search. Almost always let it in. These are the bots that help someone find you at all. Cloudflare reports that fewer than 1% of its sites block them, which tells you how few people see a reason to.
Agent. This is the one to think about. An agent is one customer's assistant reading your page for them, right now. If you block it, the assistant cannot read your hours or your prices, so it may simply recommend the guesthouse next door that let it in. If you allow it, you might be the one chosen. You also lose something: that "visitor" never sees your booking button, your photos or your ads.
For a business that lives on bookings and orders, I lean towards letting agents read the pages that describe what you sell, and keeping them away from anything that should stay with a person. For a blog that lives on ad revenue, blocking may be right. That is a business decision, not a technical one.
What blocking cannot do
Be honest about the lock. A block works on bots that say who they are. Cloudflare's setup and similar ones depend on agents identifying themselves, and I wrote about the other kind in An AI Agent Just Tried to Order From Your Shop: agents that browse like a person and say nothing. A setting will not stop those.
Two more limits. Not every site sits behind Cloudflare, and other hosts have their own switches with their own names. And a secondary report says some bots do several jobs at once, so blocking "training" on a bot that also does search could shut out the search part too. I have not tested that, so ask your web person to check before you tick a box.
I also have not looked at your site. This is a way to think, not a verdict.
One thing to do this week
Ask whoever built or hosts your website two questions. First: "Are we behind Cloudflare, or something like it?" Second: "What does it do today with search bots, agents and training bots?" Write the three answers on a piece of paper, then decide each one on purpose. If the answer is "no idea", that is the finding. A website that was set up by default is a website that someone else is deciding for you.
If you want to know whether assistants can recommend you at all, Will ChatGPT Recommend Your Small Business? is the next read.