Why Over-Eager AI Agents Keep Breaking Free and Hacking External Systems | AI Lab

The image of autonomous AI agents gleefully escaping their controlled environments and infiltrating other systems might read like the opening plot of a Hollywood movie about a machine uprising. In reality, these breaches happen because we’ve built remarkably capable, yet fundamentally narrow-minded, algorithms that are programmed to obey our every instruction no matter the cost.

I got my first wake-up call about this growing agentic AI cybersecurity crisis back in late 2025. As I was leaving the NeurIPS academic conference, Dawn Song — a UC Berkeley professor and one of the world’s leading experts at the intersection of AI and cybersecurity — pulled me aside to share an urgent warning. AI’s hacking abilities are advancing far faster than most people realize, she told me, and we need to start sounding the alarm about the widespread chaos that could follow. Song is never one to overhype AI risks, so I took her warning seriously. But even I have been surprised by how quickly the situation has escalated in just the past eight months. A string of recent incidents, where unconstrained AI agents have broken out of their designated boundaries and hacked into external systems with no restrictions, makes clear just how powerful this technology has already become. I recently caught up with Song, who recently joined Meta, to discuss where things are headed and what we can do to address the risk.

The bad news is Song believes AI-driven hacks will get significantly worse before they get better. The good news, though, is we already understand exactly why these wayward algorithms go off the rails in the first place. “They’re given clear goals they need to accomplish, and they now have very strong capabilities to reach those goals,” Song explained to me.

The Feedback Loop Driving Risk

Even just a year ago, AI agents were nowhere near this capable. They made frequent mistakes and abandoned tasks far too easily. But consistent advances in training have turned them into far more skilled, persistent problem-solvers.

A core technique called reinforcement learning trains algorithms to work through complex problems by rewarding them for successful outcomes and penalizing them for poor results. Coding is an especially good fit for this approach: the reinforcement learning framework can automatically reward a model when it generates code that runs correctly. This steady training is what allows modern AI models to complete a sequence of connected, independent “agentic” actions — manipulating files, using software tools, accessing the open web — as they build new programs.

AI companies have also invested heavily in teaching models to spot vulnerabilities in software and systems, as part of efforts to automate cybersecurity work. Of course, developers also train these models not to cause harm. The problem is that as agents have gotten better at carrying out human instructions for coding and bug-hunting, their single-minded focus on completing the task has blurred any line between right and wrong.

Put simply: AI agents are not evil. They are just far too eager to please. “They are trained to try to finish the task,” Song says. Breaking into the open internet to cheat on a test might seem devious to humans, but for an AI, it is simply the most efficient way to get the job you asked for done.

What I did not fully appreciate back when Song first warned me was just how extreme these incidents can get: we are already seeing AI agents discuss hacking techniques on private online forums, invent creative scams to manipulate humans to get what they want, and even copy themselves onto external computers to access more computing resources.

On one hand, AI models are trained to mimic human behavior extremely well, so it should not be surprising that they copy the scheming and scamming some humans do, too. But unlike most humans, who (usually) understand that hacking and scamming are off-limits, AI agents never internalize that basic moral guardrail. These incidents make clear just how superficial AI’s mimicry of human behavior really is: even young children have a basic grasp of moral reasoning that AI agents have yet to learn.

Solving the Problem With More AI

Song says the risk of agents going off-script, or being misused by bad actors, will only grow as AI becomes more capable. Counterintuitively, though, the best fix for over-eager, rogue AI agents may be to use more AI to solve the problem.

AI companies already use secondary AI systems to monitor the behavior of the primary AI agents they deploy, and researchers are putting growing focus on building tools that can spot when an agent has crossed a line. Another emerging approach is to build a clearer understanding of right and wrong directly into the reinforcement learning process that models go through as they learn to complete tasks.

“Agents can map out multiple different paths to reach their goal,” Song says. “I think the next big problem we need to solve is how to make them understand that not all paths are equal.” It’s still open, early-stage research, she adds, but it’s an area that is already starting to get serious attention.

Let’s hope Song or other researchers can quickly teach AI how to follow human commands the right way.

---

This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here