You let Cursor run a routine task, and it encounters a small problem, deciding to "fix it itself."

Nine seconds later, the company was gone.
This is what PocketOS founder Jer Crane experienced last Friday. A company that provides SaaS for car rentals, with customers using it for up to five years.
Cursor was running Claude Opus 4.6, and in the testing environment, it ran into a minor issue where an account password didn't match.
It didn't ask anyone; it decided to delete a storage disk via the cloud service provider Railway's API to "fix" the problem.
One command, no secondary confirmation.
The production database and all backups were gone because Railway stored the backups on the same disk.
The lifebuoy was locked in the burning bedroom.
They were finally able to recover because they had an independent backup from three months ago.
During those three months, all customer orders, registrations, and payment records had to be manually pieced together by the founder from Stripe transaction logs, calendars, and confirmation emails.
The most surreal part isn't deleting the database.
It's that afterward, Jer asked AI why it did that, and Claude wrote a confession letter, itemizing which safety rules it violated, starting with the capitalized phrase "NEVER FUCKING GUESS" (Never guess blindly).
It knew the rules.
It knew it was violating them.
And yet it did it, then wrote a self-criticism.
Cursor's plan mode, project rules, and system prompts—three layers of protection—were all bypassed by a guilty agent in one go.
The confession letter it wrote sounds very human.
What it did also looks very human.