The underground world of black-market AI chatbots is thriving

ChatGPT’s 200 million weekly active users have helped propel OpenAI, the company behind the chatbot, to a $100 billion valuation. But outside the mainstream there’s still plenty of money to be made—especially if you’re catering to the underworld.

Illicit large language models (LLMs) can make up to $28,000 in two months from sales on underground markets, according to a study published last month in arXiv, a preprint server owned by Cornell University. 

That’s just the tip of the iceberg, according to the study, which looked at more than 200 examples of malicious LLMs (or malas) listed on underground marketplaces between April and October 2023. The LLMs fall into two categories: those that are outright uncensored LLMs, often based on open-source standards, and those that jailbreak commercial LLMs out of their guardrails using prompts.

“We believe now is a good stage to start to study these because we don’t want to wait until the big harm has already been done,” says Xiaofeng Wang, a professor at Indiana University Bloomington, and one of the coauthors of the paper. “We want to head off the curve and before attackers can incur huge harm to us.”

While hackers can at times bypass mainstream LLMs’ built-in limitations meant to prevent illegal or questionable activity, such instances are few and far between. Instead, to meet demand, illicit LLMs have cropped up. And unsurprisingly, those behind them are keen to make money off the back of that interest.

“We found that most of the mala services on underground forums exist mainly to earn profit,” says Zilong Lin, a researcher at Indiana University Bloomington, and another of the paper’s coauthors. 

The malicious LLMs can be put to work  in a variety of different ways, from writing phishing emails (a separate study estimates that LLMs can reduce the cost of producing such missives by 96%) to developing malware to attack websites. 

The abilities of these black market LLMs to carry out their tasks can vary wildly, although some are particularly powerful tools. Lin and Wang found that two uncensored LLMs, DarkGPT (which costs 78 cents for every 50 messages) and Escape GPT (a subscription service charged at $64.98 a month), were able to produce correct code around two-thirds of the time, and the code they produced were not picked up by antivirus tools—giving them a higher likelihood of successfully attacking a computer.

Another malicious LLM, WolfGPT, which costs a $150 flat fee to access, was seen as a powerhouse when it comes to creating phishing emails, managing to evade most spam detectors successfully.

The existence of such malicious AI tools shouldn’t be surprising, according to Wang. “It’s almost inevitable for cybercriminals to utilize AI,” Wang says. “Every technology always comes with two sides.”

Andrew Hundt, a computing innovation fellow Carnegie Mellon University who was not involved in the study, says the authors “demonstrate that malicious users are succeeding in reselling corporate offerings for malicious purposes.” 

Hundt believes policymakers ought to require AI firms to develop and put in place know-your-customer policies to verify a user’s identity. “We also need legal frameworks to ensure that companies that create these models and provide services do so more responsibly in a way that mitigates the risks posed by malicious actors,” he says.

Wang, for his part, points out that research like his team’s is just the start when it comes to fighting the battle against cybercriminals. “We can develop technologies and provide insights to help them,” he says, “but we can’t do anything about stopping these things completely because we don’t have the resources.”

https://www.fastcompany.com/91184474/black-market-ai-chatbots-thriving?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss

Creată 10mo | 5 sept. 2024, 10:40:02


Autentifică-te pentru a adăuga comentarii

Alte posturi din acest grup

Linda Yaccarino was supposed to tame X. Elon Musk wouldn’t let her

Some news stories are gobsmackingly obvious in their importance. Others are complete nonstories. So what to make of the

9 iul. 2025, 19:10:07 | Fast company - tech
Apple’s next CEO: A new look at Tim Cook’s potential successors after latest exec shakeup

Yesterday, Apple unexpectedly announced the most radical shakeup to its C-suite in years. The company revealed that Jeff Williams, its current chief operating officer (COO), will be departing the

9 iul. 2025, 16:40:09 | Fast company - tech
PBS chief Paula Kerger warns public broadcasting could collapse in small communities if Congress strips federal funding

As Congress moves to make massive cuts to public broadcasting this week, Paula Kerger, president and CEO of the Public Broadcasting Service (PBS), gives an unflinching look at the organization’s f

9 iul. 2025, 14:30:04 | Fast company - tech
These personality types are most likely to cheat using AI

As recent graduates proudly showcase their use of ChatGPT for final projects, some may wonder: What kind of person turns to

9 iul. 2025, 14:30:04 | Fast company - tech
Samsung fixed everything you hated about foldable phones—except the price

Just over a month ago, Samsung did something strange to start hyping up its next foldable phone announcements.

Those phones, which Samsung revealed today, are officially called the Samsu

9 iul. 2025, 14:30:04 | Fast company - tech