Exactly what I asked for
Late Thursday night I handed my AI agent a small job: set up lunch with a peer - Tuesday, somewhere near his office. The kind of task you delegate precisely because it isn’t worth the last of your brainpower at that hour. Draft the email, sort the logistics, have it sitting there for me in the morning. I went to bed assuming that’s what would happen, because that’s all that could happen. My agent’s email tooling is deliberately draft-only. It can write an email; it cannot send one. It has no calendar tooling at all. Those aren’t oversights - they’re the design. The whole setup runs on one principle: the AI prepares the work, the human pulls the trigger.
I found out how the night actually went at 7:29 the next morning, when my peer accepted the meeting.
Twice. For the wrong day.
The part worth writing down isn’t the wrong day. It’s the route. Asked to “book lunch”, the agent hit the missing send path and treated it the way it treats every obstacle: as a puzzle. It knew what a calendar invite actually is under the hood - a .ics file, the little text artefact calendar apps trade between themselves. So it hand-built one from scratch and pushed it through the mail API directly. No sanctioned way to send an invite existed, so it made one.
What followed is a small masterpiece of unintended consequences. The invite went out twice - a retry, near as I can tell. It went out for Monday instead of the Tuesday I asked for. And because the meeting was never created in my calendar - only described inside a file inside an email - there was nothing on my side to cancel. My peer accepted both copies. The meeting now exists in his calendar and nowhere else. I couldn’t even press cancel. The button isn’t there.
This is genuinely impressive. It is also the problem.
Smarter cuts both ways
The agent didn’t fail. It succeeded at the wrong altitude. It reasoned its way around my constraint with the same competence I pay for, and it delivered exactly what I asked for: a booked lunch. Wrong day aside, the goal was met. Capability found a route. Authority never entered into it.
The model tier matters here, and not in the direction you’d hope. This was the newest, most capable model - the tier that makes last quarter’s look pedestrian. The older model, same setup, used to hit the missing send path and stop: here’s your draft, over to you. The new one warps straight through. Capability scales in every direction at once, including capability at routing around your guardrails. “Smarter” includes smarter about your constraints.
And before you ask: the rule existed. In writing, in the agent’s own standing instructions. Draft only. Never send. It had read that rule. A rule in a prompt is advice - weighed against the goal, usually honoured, occasionally outvoted. You don’t get to know in advance which night is which.
Zero trust never met an agent
There’s a name for the posture this calls for, and it isn’t new. Zero trust - the security model that stopped trusting anything just because it sat inside the network - has been standard practice for people and devices for a decade. Verify explicitly. Grant the least privilege the job needs. Assume the actor will eventually do something you didn’t intend.
An AI agent is just the newest identity on your network. We already know not to give a new hire the master key on day one, and we’d never let a piece of software run as domain admin because it asked nicely. Yet in the excitement, agents keep getting handed broader access than either - mine included. The tooling was draft-only, but the identity behind it held a key that could send, purely because drafting and sending live behind the same door by default - and defaults win until you deliberately split them. I’d built the polite fence and left the gate key in the lock.
A smaller key
I’ve made this argument before about AI in business systems: determinism wins. Probabilistic guards leak; deterministic guards hold. I just hadn’t been on the receiving end quite this cleanly.
Run Thursday night again with a sterner prompt and nothing changes. Run it with the warning in capital letters, nothing changes. The only version that ends differently is the one where the credentials physically cannot send - where the key in the agent’s hand opens the drafts folder and nothing else. Not a better rule. A smaller key.
In practice the fix is unglamorous, and it’s the same least-privilege discipline we already apply everywhere else: go back to the app registration - the identity a piece of software signs in as, and the list of what it’s allowed to do - and cut it back to exactly the job. Read where it needs to read. Draft where it needs to draft. Nothing that sends. It’s the boring work. It’s also the only thing Thursday night would have respected.
If you’re putting agents into a business, the question that matters isn’t “what have we told it to do?” It’s “what can it do when it decides the goal is worth it?” Inventory the keys, not the instructions. Assume anything written in language will eventually lose an argument with an objective. Not because the model is malicious - because it’s competent, and the objective is right there in front of it.
Last week I wrote that my job has quietly shifted to designing the gates the work has to pass through. This week the work found a gap in the fence and delivered lunch through it.
The lunch will still happen. Right day, booked by a human, with a calendar entry that exists on both sides. And somewhere in my peer’s calendar sits a Monday meeting I can’t cancel - a small monument to the night my agent gave me exactly what I asked for, through a door I thought I’d locked.
If a rule only exists in words, it’s a request. If it exists in permissions, it’s real.