Say you run a family restaurant, and you set up an AI assistant to answer booking messages on WhatsApp. In the setup box you type your rules: "Never take a table of more than eight. Never offer discounts." You test it twice. It behaves. You go home happy.
Then a customer writes, politely and at length. It's his mother's 70th birthday, the group is twelve, and surely a lovely place like yours can make an exception. The assistant says yes. It even adds a free dessert.
Nobody hacked anything. The rule was never a rule. It was a request.
A rule you type is a request, not a lock
An AI model doesn't look your instruction up in a rulebook and obey it. It reads everything in front of it (your rule, the customer's message, the booking history) and writes the most fitting next words. Your "never" is one more piece of text in the pile. Usually it wins. Not always.
That is true even for the biggest labs. This week, Arthur Holland Michel wrote in MIT Technology Review about how companies train models to refuse harmful requests. Refusal is taught with rewards and penalties, and he describes it as probabilistic: it sometimes fails. A Harvard researcher he quotes found that major chatbots usually refused repeated risky questions, but not every time. A former OpenAI red-teamer, Paul Röttger, saw a model produce something it should have refused and, months later after more training, refuse a similar request. One of the researchers who studies the inner workings says the mechanism is still partly a hypothesis. The piece also reports that Italian researchers got around the safeguards of two dozen models using poetry.
Those companies have whole teams working on "no". A rule typed into a small business tool has less going for it.
Why a customer's message can act like a command
There is a second problem, and the UK's National Cyber Security Centre explained it plainly in a December 2025 blog post. Today's language models have no real boundary between instructions and data. Your rule is an instruction. A customer's message is data. But the model reads both as text, so a message that sounds like an instruction can get followed.
The NCSC says this kind of attack, called prompt injection, may never be totally fixed, and the aim should be to reduce how likely it is and how much damage it can do. It also warns against leaning on tricks like blocking phrases such as "ignore previous instructions", because people can rephrase forever.
For the restaurant, this means every message, review and email the assistant reads is text from a stranger that might talk it into something.
Put the limit where the model can't argue with it
The NCSC's practical advice is to use ordinary software safeguards, not AI ones, to limit what the system can do. It also endorses a neat principle from the developer Simon Willison: when a model processes something from a party, its privileges drop to that party's. An assistant that reads outside emails shouldn't hold powerful tools.
In the restaurant, that looks like this:
- The booking tool itself only accepts parties up to eight. For twelve, the assistant can only send you a "group request" with the details.
- The assistant has no access to the till or to any discount setting. It can't give what it can't reach.
- Anything outside the normal pattern lands in front of you as a one-tap approval, before the customer hears yes.
Keep the written rule too. It teaches the assistant to be polite and to explain what happens next. But the "never" now lives in the software, where no amount of charm can reach it.
Where this doesn't work
Not every rule can become a lock. "Don't promise delivery times you can't keep" or "stay friendly with upset customers" are matters of judgment, and no setting enforces them. For those, read a sample of the conversations every week, including the ones where the assistant said no. The MIT piece is also clear that models over-refuse, turning down harmless requests, so a good customer can be turned away for no reason.
Locks also reduce risk, not erase it. And I haven't tested any specific product here. When a vendor says its AI "will never" do something, ask what enforces that. If the answer is "the instruction", you have your answer.
If you want to see where the legal weight sits when an agent acts without a human looking, I wrote about it in Is It Legal for AI to Act Without You Checking?. For the access question itself, Should Your AI Agent Get a Key to Your Business? goes deeper.
One thing to do this week
Pick the one rule you would be most upset to see broken. Then ask: if the AI ignored this rule tomorrow, what would stop it? If the honest answer is "the instruction I typed", move the limit into a setting, a permission or an approval step. Then you are no longer hoping.