The real data revolution hasn’t happened yet

The Gartner Hype Cycle is a valuable framework for understanding where an emerging technology stands on its journey into the mainstream.

It helps chart public perception, from the “Peak of Inflated Expectations” through the “Trough of Disillusionment,” and eventually up the “Slope of Enlightenment” toward the “Plateau of Productivity.”

In 2015, Gartner removed big data from the Hype Cycle. Analyst Betsy Burton explained that it was no longer considered an “emerging technology” and “has become prevalent in our lives.”

She’s right. In hindsight, it’s remarkable how quickly enterprises recognized the value of their data and learned to use it for their business advantage. Big data moved from novelty to necessity at an impressive pace.

Yet in some ways, I disagree with Gartner. Adoption has been widespread, but effectiveness is another matter. Do most enterprises truly have the tools and infrastructure to make the most of the data they hold?

I don’t believe they do. Which is why I also don’t believe the true big data revolution has happened yet. But it’s coming.

Dissecting the Stack

A key reason big data is seen as mature, even mundane, is that people often confuse software progress with overall readiness. The reality is more nuanced.

Yes, the software is strong. We have robust platforms for managing, querying, and analyzing massive datasets. Many enterprises have assembled entire software stacks that work well.

But that software still needs hardware to run on. And here lies the bottleneck.

Most data-intensive workloads still rely on traditional central processing units (CPUs)—the same processors used for general IT tasks. This creates challenges. CPUs are expensive, energy hungry, and not particularly well suited to parallel processing.

When a query needs to run across terabytes or even petabytes of data, engineers often divide the work into smaller tasks and process them sequentially. This method is inefficient and time-consuming. It also ends up requiring more total computation than a single large job would.

Even though CPUs can run at high clock speeds, they simply don’t have enough cores to efficiently handle complex queries at scale. As a result, hardware has limited the potential of big data. But now, that’s starting to change with the rise of accelerated computing.

Breaking the Bottleneck

Accelerated computing refers to running workloads on specialized hardware designed to outperform CPUs. This could mean field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs) built for a specific task. More relevant to big data, though, are graphics processing units (GPUs).

GPUs contain thousands of cores and are ideal for tasks that benefit from parallel processing. They can dramatically speed up large-scale data operations.

Interestingly, GPU computing and big data emerged around the same time. Nvidia launched CUDA (compute unified device architecture) in 2006, enabling general-purpose computing on graphics hardware. Just two years earlier, Google’s MapReduce paper laid the foundation for modern big data processing.

Despite this parallel emergence, GPUs haven’t become a standard part of enterprise data infrastructure. That’s due to a mix of factors.

For one, cloud-based access to GPUs was limited until relatively recently. When I started building GPU-accelerated software, SoftLayer—now absorbed into IBM Cloud—was the only real option.

There was also a perception problem. Many believed GPU development was too complex and costly to justify, especially for general business needs. And for a long time, few ready-made tools existed to make it easier.

Those barriers have largely fallen.

Today, a rich ecosystem of software exists to support GPU-accelerated computing. CUDA tools have matured, benefiting from nearly two decades of continuous development. And renting a top-tier GPU, like Nvidia’s A100, now costs as little as $1 per hour.

With affordable access and a better software stack, we’re finally seeing the pieces fall into place.

The Real Big Data Revolution

What’s coming next will be transformative.

Until now, most enterprises have been constrained by hardware limits. With GPU acceleration more accessible and a mature ecosystem of supporting tools, those constraints are finally lifting.

The impact will vary by organization. But broadly, companies will gain the ability to run complex data operations across massive datasets, without needing to worry about processing time or cost.

With faster, cheaper insights, businesses can make better decisions and act more quickly. The value of data will shift from how much is collected to how quickly it can be used.

Accelerated computing will also enable experimentation. Freed from concerns about query latency or resource drain, enterprises can explore how their data might power generative AI, smarter applications, or entirely new user experiences.

Gartner took big data off the Hype Cycle because it no longer seemed revolutionary. Accelerated computing is about to make it revolutionary again.

https://www.fastcompany.com/91346027/the-real-data-revolution-hasnt-happened-yet?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss

Établi 9d | 5 juin 2025, 09:50:02


Connectez-vous pour ajouter un commentaire

Autres messages de ce groupe

TikTok users are exposing their worst exes—all to the soundtrack of Lorde’s new single

The latest TikTok trend has people exposing their terrible exes and most toxic relationship stories to Lorde’s new single “ ">Man of the Ye

13 juin 2025, 21:50:03 | Fast company - tech
Trump’s ‘gold card’ visa scheme is pure gilded nonsense

President Donald Trump announced, back on February 25, that his administration would soon debut a “gold card,” an immigration program that would allow wealthy foreigners, for the low, low price of

13 juin 2025, 19:30:05 | Fast company - tech
How singles are using AI to improve their online dating success

Singles are increasingly turning to AI to boost their odds in the dating world.

According to a new study, just over a quarter (26%) of singles are using artificial intelligence to enhanc

13 juin 2025, 19:30:04 | Fast company - tech
Can AI fact-check its own lies?

As AI car crashes go, the recent publishing of a

13 juin 2025, 17:10:07 | Fast company - tech
‘Guys, the sea literally opened up’: AI-generated Bible characters are taking over TikTok

The Bible is now on TikTok, vlog-style.

Picture David—yes, that David, of Goliath fame—with an iPhone and influencer energy. “Asked the guy to film it, but guess what? The camera froze,”

13 juin 2025, 14:50:05 | Fast company - tech
I write novels and build AI. The real story is more complicated than either side admits

“In three years,” a fellow tech executive recently told me with serene confidence, “Everyone will be able to make a full-length movie in AI, totally personalized for them, by just typing up a few

13 juin 2025, 14:50:04 | Fast company - tech
This 1999 email from a tech pioneer helped me think about Apple’s WWDC

In 1999, I got to work on a literally once-in-a-lifetime project. As the 20th century was wrapping up, the magazine where I worked declared the personal computer the most important invention of th

13 juin 2025, 14:50:03 | Fast company - tech