Agentic AI has companies excited and security experts freaked out

Agentic AI is being heralded as the future of the generative AI revolution by leaders in the field. From ChatGPT’s integration of agentic features to the rise of Comet (the agent-based web browser from Perplexity) and Chinese-born Manus, the trend of handing more control to AI tools seems inevitable.

At least, that’s the view of Microsoft CEO Satya Nadella, Shopify CEO Tobias Lütke, Amazon executive chairman Jeff Bezos, and Nvidia CEO Jensen Huang.

But before ceding control completely, it’s worth weighing the risks. If AI agents are about to flood society, they must first get street-smart. Initial concerns point to their naivety, which could come back to hurt us all.

Andy Zou, a researcher at Gray Swan AI, an IT security firm, says AI agents have only become prevalent in the past few months, as the focus has shifted from “just talking to the chatbot” to giving it tools that can take real-world actions—dramatically increasing the risks. The concern, he notes, is that agentic AI resembles the Hollywood caricature of George of the Jungle: ready to believe anything, no matter the consequences. “We found you can essentially manipulate the AI [to] override its programming,” Zou says.

In a new study, Zou and colleagues tested 22 leading mainstream AI agents with 1.8 million prompt injection attacks, around 60,000 of which successfully pushed the agents off their guardrails to grant unauthorized data access, conduct illicit financial transactions, and bypass regulatory compliance.

An earlier study showed even weaker defenses, with AI assistants fooled nearly 70% of the time into wiring money to fraudsters via buried “fine print” instructions. And just this week, browser developer Brave alleged that a similar website-based attack could manipulate Perplexity’s Comet browser. (Perplexity has since patched the flaw, though Brave contends the fix is incomplete.)

The lesson is clear: Even modest success rates at this scale translate into dangerous vulnerabilities. Before handing these bots the keys, they’ll need sharper critical thinking.

This isn’t merely hypothetical. One cryptocurrency user lost $50,000 when an AI agent was tricked into sending funds to the wrong wallet through malicious, agent-only instructions. As adoption grows—eight in 10 corporations now use some form of agentic AI, according to PricewaterhouseCoopers—the risks multiply.

Tianshi Li, an assistant professor at Northeastern University who led the earlier study, says agents are designed to complete complex tasks for people, often without direct supervision. While they’re useful for tedious work, Li warns that “this capability of doing complex things without direct supervision is also inherently conflicting with security and privacy guarantees.”

Unlike static chatbots, AI agents are vulnerable because their inputs don’t come solely from the user—they interact with tools and pull data from untrusted sources, creating hidden risks. The agent “goes out there and talks to a tool, retrieves data from a source that you don’t fully trust, [and] without realizing it, you might be exposing yourself to some of these risks,” says Matt Fredrikson, an associate professor at Carnegie Mellon University and Zou’s coauthor on the study.

With focused effort, Zou and Fredrikson managed to compromise agents from 10 frontier AI labs within hours. Security engineers won’t be surprised: Treating anything an agent reads on the web—or in a calendar invite, email signature, or PDF—as trustworthy effectively gives strangers partial control of the system prompt. But the ease of these breaches should be a wake-up call. “They’re putting the agents out there in the real world,” Zou says. “And there are so many of these real vulnerabilities that exist right now.”

Enterprise adoption reflects both curiosity and caution. James Robinson, chief information security officer at Netskope, a cloud security firm that recently published guidance on AI agent risks, says companies are experimenting carefully.

“Agents are just starting to be played with,” he tells Fast Company. For now, they aren’t given “full open control [to] make production changes,” but are confined to centralized environments such as IDEs [integrated development environments], with guardrails like change control and peer review. In highly regulated industries such as banking, the restrictions are even tighter.

Still, Robinson warns against casual adoption, comparing it with “having an employee that joins your organization that you never hired”—someone with 24/7 access to potentially everything you see. Experts share Robinson’s concern that many adopters don’t grasp how easily agents can be manipulated or how severe the fallout could be. Fredrikson, the Carnegie Mellon professor, adds that while some organizations do their due diligence, “there are some that aren’t fully aware of all the mitigations and security tools they could use.”

The imbalance remains clear: Promoters emphasize the benefits of agentic AI far louder than the risks. “People are excited to deploy this, and things move very quickly,” Fredrikson says. That’s a mix that feels like “a recipe for security issues to come out of the woodwork,” he warns.

This story was supported by a grant from the Tarbell Center for AI Journalism.

https://www.fastcompany.com/91392109/everyones-rolling-out-ai-agents-security-researchers-just-broke-them-in-hours?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss