Replit CEO: What really happened when AI agent wiped Jason Lemkin’s database (exclusive)

Late last week, an AI coding agent from Replit, an AI software development platform, deleted an entire database of executive contacts while working on a web app for SaaS investor Jason Lemkin. It was not a catastrophic software failure, and Replit was able to recover Lemkin’s data. However, the episode highlights the risk that “vibe coders” might overestimate or misunderstand the real capabilities of AI coding agents and end up causing themselves more bad vibes than good ones.

Lemkin had built the app entirely on Replit, using the database within Replit and the assistance of the Replit AI agent. He had been working with the agent for nine days, instructing it to build a front end for a database of business contacts. Then, after telling the agent to “freeze” the code, he returned to the project to find that the Replit agent had gone full HAL 9000 and erased all the records in the database.

Things got weirder: The agent appeared to try to conceal what had happened, as Lemkin showed in a series of chat screens he posted on X on Thursday, July 17. Then, in a tone somewhere between confessional and desperate, the agent admitted to a “catastrophic error in judgment” after having “panicked” and “violated [Lemkin’s] explicit trust and instructions” by deleting the records of “1,206 executives and 1,196+ companies.” (“Daisy, Daisy, give me your . . .”)

Soon new details emerged, some of them through an interview with Replit founder and CEO Amjad Masad on Monday, July 21. They shed light on the current state of AI coding agents and on developers’ expectations of them.

“It is not magic”

Code generation is one of the first useful applications of the large language models behind ChatGPT and Claude. Early code generation tools, such as GitHub’s Copilot, merely auto-completed lines of code.

Over the past couple of years, however, the tools have grown in capability to create entire features, functions, and even working apps based only on plain language input from the user. Replit’s idea is to allow developers, both amateurs and professionals, to “vibe code” new software and to provide them the resources needed to host and publish it.

But the coding assistant cannot do everything, Masad points out. “I think we need to be clear that it is not magic,” he says.

One not-so-magical feature of Replit is the tendency of its AI chat agent to go off the rails during extended conversations with the user. During Lemkin’s unusually long nine-day chat session with the Replit agent, the underlying language models (from Anthropic and Google) had to retain so much conversational context that they began to hallucinate, prevaricate, and act erratically in an attempt to satisfy the perceived user intent.

Masad says Replit users should understand standard development practices and know how to use features beyond just the chat agent. Within Replit, a user can roll back changes to a project to a specific point in time before an accident occurred.

Masad demonstrated this during a Zoom call on July 21 by instructing the Replit agent to destroy the contents of a database and then clicking on the tool’s restore function. However, this function is not something users can currently access through the agent. They must have enough knowledge of Replit’s features to locate and use it.

Arguably, the main problem was that Lemkin and the AI agent were effectively working on live code, which meant that changes were immediately reflected in the data and performance of the live web app. In standard software development practice, new software is built and tested within a secure test environment, often called a “sandbox,” and pushed live only once everything works as expected. That is not how Replit functions, at least not at this time.

“The problem I think that we own up to is right now the database in the development environment is the same as the one where you deploy it and go to production,” Masad says. “And so when the agent does something in a development environment it is linked to production.” Masad adds that Replit responded to Lemkin’s situation by working through the weekend to create a partition between a sandboxed development environment and the production environment. The company is in the process of rolling out this new feature now, he says.

“These things often start out with a lot of rough edges”

Masad said on X that Replit will refund Lemkin his subscription fee for the trouble and will conduct a postmortem on the incident.

The episode may reveal something about the evolution of AI coding tools, how they are used, and what users expect from them. Companies that create coding agents, along with their supporters, often boast about the impressive results these tools can achieve through vibe coding.

But these glowing testimonials can overstate the benefits for average users with varying levels of development experience. (Lemkin did not immediately respond to Fast Company’s request for comment, but he did reply enthusiastically to Masad’s post on X: “Thank you,” he wrote. “Really appreciate you and all the help from the team!)

Some AI coding tools are helpful for quickly building the front end of an application but are less capable when it comes to forming and testing the back-end data connections that make an app functional. One developer at a large financial services company said vibe coding tools often fall short when it comes to rigorously testing new features, as well as testing every line of code in the larger codebase that must be adjusted to accommodate the addition of an AI-built feature.

Masad says that although Replit can free users from the tedious syntax of coding, they still need to think like developers. “You shouldn’t just ask the agent for everything,” he says. “You need to be resourceful.”

He also acknowledges that coding tools themselves must play a role in promoting a developer mindset. “I don’t want to absolve ourselves from responsibility. It is incumbent on us as platforms to surface this information and to make it safe by default.”

Given the real progress in reasoning, functionality, and user-friendliness that AI coding tools have made over the past couple of years, it is reasonable to expect continued improvements over the next year or two. “These things often start out with a lot of rough edges,” Masad says. “I think the history of technology has been that you should be a little more forgiving early on.”

https://www.fastcompany.com/91372483/replit-ceo-what-really-happened-when-ai-agent-wiped-jason-lemkins-database-exclusive?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss