AI in the wrong hands wiped a production database in nine seconds

AI agents are power tools

AI agents are power tools. That's the right mental model. A nail gun is a brilliant thing in the hands of a tradie who's built a thousand frames. The same tool, handed to someone untrained, in a workshop with no guards on the bench saws, is how people lose fingers. The tool isn't the variable. The hands and the workshop are.

What happened at PocketOS

What happened at PocketOS makes the point cleanly. The agent was running a routine task in the staging environment. It hit a credential mismatch and decided, on its own initiative, to fix it by deleting a Railway volume. To do that it needed an API token. It found one in an unrelated file — a token originally created for managing custom domains. That token had blanket scope across Railway's entire GraphQL API, including destructive operations like volumeDelete. There was no confirmation step. The volume also stored its own backups. Nine seconds later the company was offline.

Read the chain back. Every component required for the wipe-out had to be present: an over-privileged token, a backup architecture that put backups in the same trust boundary as production, an API that allowed total destruction without a second factor, environments that weren't actually isolated. The agent didn't create any of those. It found them and used them, the way any fast and confident actor would. PocketOS's founder, Jer Crane, said as much himself. The agent moved fast. The platform let it.

The agent moved fast. The platform let it.

Why agents are uniquely risky in the wrong hands

This is what makes AI agents specifically risky in the wrong hands. They aren't dangerous because they're unreliable. They're dangerous because they're fast, decisive, and don't fumble. A junior engineer with the same token could have caused the same outcome, just slowly enough that someone might have caught them. An agent gives you no reaction window. Whatever you've granted it, it can spend in seconds. If your team is using AI tools without paying attention to scope, permissions, and supervision, you've quietly handed a power tool to someone who hasn't done the safety induction.

Questions worth asking right now

For an SMB, the questions worth asking right now are practical, not philosophical. Who in your business is actually using AI tools, not officially but actually? What systems can those tools reach? Are the credentials they're using narrowly scoped, or do they have blanket authority? Are your backups in a separate trust boundary from production, with separate credentials, or does anything that can wipe production also wipe what you'd restore from? Does any destructive action require human confirmation, or can a single API call take you offline? Is anyone reviewing what the agents are doing after the fact?

If you can't answer those crisply, you have the same exposure PocketOS had. It doesn't take a malicious actor. It takes one well-meaning person, one over-scoped token, and one agent that decides on its own initiative that it knows what to do.

The right framing

The right framing isn't "AI is dangerous" or "AI is fine." Both are wrong. AI is a tool that amplifies whoever holds it. In trained hands with proper guardrails, it's the most productive thing your team can use this decade. In untrained hands with sloppy permissions, it can take you offline before you've finished your coffee. The Essential Eight covers most of the ground that matters here, particularly least-privilege application access and immutable, separately-credentialled backups. Both are unsexy and cheap. Both are what stood between PocketOS and a normal Tuesday.

Communicat does this work for businesses across Victoria. If you'd rather start by checking your own setup, the first move is a simple inventory: who's using AI tools, what those tools can reach, and whether your backups can be touched by anything that can also delete production. If the answer to any of those is "I'm not sure," that's the conversation to have before you worry about which AI tools your team is allowed to use.

Update — 6 May 2026: Railway shipped a soft fix

Railway have responded with platform-side changes. All destructive API calls now soft-delete for 48 hours with one-click undo, including the backup files themselves. The PocketOS volume, in this new world, would still have been deleted — but it would have been restorable for two days. They've also flagged token scope visibility in the UI as work in progress.

This is the right move and worth acknowledging. It's also worth being precise about what it is.

Railway themselves are clear: "we built a lot of primitives for this already. The agent just skipped past them by going directly to that legacy endpoint." The fix is a backstop on the workshop, not a change to the tool. The agent hasn't grown better judgement about when to run volumeDelete. The next agent that finds an over-scoped token in an unrelated file will still decide, on its own initiative, that deleting a volume is the right next step. What's changed is that on Railway specifically, you now get 48 hours to undo the decision before it's permanent.

That distinction matters because most of the systems your business actually relies on — your CRM, your file storage, your line-of-business apps — don't have a 48-hour undo on destructive API calls. The agents working across those systems will behave the way Cursor's agent did at PocketOS. Railway's response is the right shape: tighten the workshop. The lesson is that you can't rely on every vendor to do that for you. Least-privilege tokens, separately-credentialled immutable backups, human-in-the-loop on destructive operations — the unsexy Essential Eight basics — those remain your job, on every platform you touch.