Welcome to AI Decoded, Fast Company’s weekly newsletter that breaks down the most important news in the world of AI. You can sign up to receive this newsletter every week here.
Model ‘supersizing’ may have hit a wall
The impressive intelligence gains of large models like OpenAI’s GPT-4 and Anthropic’s Claude came about after researchers figured out that they could reap progressively and predictably better intelligence by increasing the size of the models, and training them with more data and more computing power for longer periods of time. That realization yielded some impressive chatbots and coding assistants.
The big question now is how far the “supersizing” approach can carry researchers toward the holy grail of artificial general intelligence (i.e., AI that’s generally smarter than human beings). In fact, newly departed OpenAI cofounder Ilya Sutskever recently told Reuters that the jig is up for massive scaling. Similarly, billionaire investor and accelerationist Marc Andreessen said on his podcast with Ben Horowitz that the AI researchers he talks to are hitting the limits of massive scale.
“If you look at the improvement from GPT-2 to GPT-3 to 3.5, and then compare that from like 3.5 to 4, you know we really slowed down in terms of the amount of improvement,” Horowitz says.
And the gains from scaling are getting more and more expensive, he points out. “The thing to note is that the GPU increase was comparable, so we’re increasing GPUs at the same rate but we’re not getting the intelligence improvements at all out of it.” Pretraining a large model can cost upwards of $100 million, and Anthropic CEO Dario Amodei says that number could hit a billion dollars in the near future. And, of course, those Nvidia GPUs are expensive—the new H200 chips cost up to $40,000—and in high demand.
Several AI researchers have told me that the research is shifting away from scaling up computing power to focusing on the composition and quality of the training data. When an AI company puts together a corpus of data to train a new model, it sources it from thousands of contractors, publishers, and providers of domain-specific data such as healthcare data. It even hires doctors, lawyers, and other professionals to add expert labels to the data. The company also goes to providers of synthetic data to fill in gaps in the training corpus.
Most importantly, researchers are looking at new places outside of the training process to add computing power. An OpenAI researcher who worked on the company’s new o1 model tells me that his team has been spending a lot more computing power at inference time, when the model has already been trained and deployed and is processing user queries. For difficult, multistep questions, he says, the model is trained to pursue an answer in a more human, trial-and-error fashion (pursuing promising lines of reasoning, hitting a wall, backtracking, trying something else). All the while the model is storing and reprocessing all the data from the process, which requires lots of computing power.
Not everybody agrees that brute-force scaling has reached diminishing returns. Some argue that it’s too early to say, because we’ve yet to see the size and performance of OpenAI’s forthcoming GPT-5 model, or xAI’s forthcoming Grok-3 model. If those models end up costing far more to train than earlier models and yield unimpressive intelligence gains, the industry might truly be facing an inflection point.
Do Trump and Musk intend to fire, rehire, and rewire the Pentagon?
As promised, President-elect Donald Trump tapped Elon Musk (along with Vivek Ramaswamy) to co-lead a new government agency called the Department of Government Efficiency (DOGE).
So far, there’s been no indication that the Pentagon might be exempt from changes suggested by Musk and Ramaswamy. This could include mass firings. Trump has called for the termination of up to 50,000 career civil servants working within government agencies and replacing many of them with Trump loyalists. The Defense Department employs 24,000 civilian and military personnel.
DOGE’s work could affect Silicon Valley companies in a number of ways. Regulations might be rolled back, and the government’s ability to enforce the ones that survive could be neutered (the Federal Trade Commission, for example, could lose its ability to bring antitrust cases). It could also affect Silicon Valley’s relationship with the Pentagon, which has been gradually improving as the defense establishment works to build an industrial base that includes tech and AI suppliers. The Valley itself has arguably warmed to the idea of selling tech for defense over the past five years after Project Maven imploded at Google. But it remains a work in progress. Here’s how former Google CEO and defense tech investor Eric Schmidt explained it to me back in 2021:
“Remaking the Pentagon in Silicon Valley’s image will be far more difficult a public-private challenge than, say, Uber and Lyft steamrolling municipal taxi commissions. You have to think about the military as a large bureaucracy that has existing relationships with primes, and they all live in a symbiosis that makes sense to them.”
Starting with Defense Secretary Ash Carter in the 2010s, the Pentagon tried numerous times to modernize its technology procurement practices, with the goal of funding new defense tech faster, and giving the tech supplier some leeway to define the product’s capabilities. But these efforts have had limited success. Less than 1% of the Pentagon’s $411 billion in contract awards last year went to venture-backed defense tech companies.
Can Musk bring about such a broad cultural change within the Pentagon? (His companies are already major suppliers to U.S. defense and intelligence agencies.) There are signs he’d like to try. Late on election night he tweeted to Anduril’s Palmer Luckey: “Very important to open DoD/Intel to entrepreneurial companies like yours. Pay for outcomes, not requirements documents!”
Perhaps he intends to do it with the same kind of scorched earth “efficiency” that he brought to Twitter in 2022. Musk fired 80% of the company’s workforce, precipitating a dramatic fall in content moderation and general civility, and an exodus of paying advertisers.
That’s just one of the Pentagon’s worries. Trump has nominated Pete Hegseth, Fox & Friends cohost and Iraq vet with scant military leadership experience, to be his defense secretary. Trump’s transition team is also reportedly considering an executive order that would establish a “warrior board” of retired military personnel who would review three- and four-star generals, and recommend firing those they deem unfit for leadership. The president-elect has repeatedly called U.S. generals “woke” and “weak.”
Is China pulling away in robotics?
China trails the U.S. in the development of large foundational AI models, but it may be pulling ahead in making robots. Chinese companies appear to be making robots that have arguably better utility and awareness of the world than the robots from U.S. companies such as Figure, Tesla, and Boston Dynamics.
The latest case in point is a DEEP Robotics robot that can roll or climb on two legs or four, with enough dexterity to quickly navigate up and down a rocky mountainside. DEEP Robotics, which is based in Hangzhou, Zhejiang province, primarily supplies quadruped robots (similar to Boston Dynamics’ Spot robot dog) for monitoring or securing infrastructure, or for entering dangerous environments where humans can’t go. Some of the people on the company’s research team earned PhDs at Chinese universities, while others earned theirs at top U.S. universities, including NYU, the University of Illinois Urbana-Champaign, and Georgia Tech.
Actually, reports say that China is flush with robotics talent. Chinese companies also get more help from their government than U.S. companies do. As part of China’s Military-Civil Fusion strategy, Chinese AI and robotics companies can get funding directly from the government. In return, the government gets access to cutting-edge technologies for use in defense.
More AI coverage from Fast Company:
- Mozilla is now offering free AI voice training data in 180 languages
- This hyper-smelling AI can sniff out counterfeit sneakers—and that’s only the beginning
- 9 ways AI can make your banking job easier
- Can AI therapists save us from the global mental health crisis?
Want exclusive reporting and trend analysis on technology, business innovation, future of work, and design? Sign up for Fast Company Premium.
Login to add comment
Other posts in this group
Imagine you owned a bookstore. Most of your revenue depends on customers coming in and buying books, so you set up dif
For 13 years, Subway Surfers’ download rate has been consistent: about one million new installs every single day.
Half of those downloads come from users upgrading to new
Misbehavior on digital platforms can be tricky to manage. Issue warnings, and you risk not deterring bad behavior. Block too readily, and you might drive away your user base and open yourself to a
You’d be forgiven for forgetting that there was a time when Microsoft Edge was basically the web browser that opened when you accidentally clicked a link that didn’t default to opening in Chrome o
Do you receive login security codes for your online accounts via text message? These are the six- or seven-digit numbers sent via SMS that you need to enter along with your password when trying to
