The real data revolution hasn’t happened yet

The Gartner Hype Cycle is a valuable framework for understanding where an emerging technology stands on its journey into the mainstream.

It helps chart public perception, from the “Peak of Inflated Expectations” through the “Trough of Disillusionment,” and eventually up the “Slope of Enlightenment” toward the “Plateau of Productivity.”

In 2015, Gartner removed big data from the Hype Cycle. Analyst Betsy Burton explained that it was no longer considered an “emerging technology” and “has become prevalent in our lives.”

She’s right. In hindsight, it’s remarkable how quickly enterprises recognized the value of their data and learned to use it for their business advantage. Big data moved from novelty to necessity at an impressive pace.

Yet in some ways, I disagree with Gartner. Adoption has been widespread, but effectiveness is another matter. Do most enterprises truly have the tools and infrastructure to make the most of the data they hold?

I don’t believe they do. Which is why I also don’t believe the true big data revolution has happened yet. But it’s coming.

Dissecting the Stack

A key reason big data is seen as mature, even mundane, is that people often confuse software progress with overall readiness. The reality is more nuanced.

Yes, the software is strong. We have robust platforms for managing, querying, and analyzing massive datasets. Many enterprises have assembled entire software stacks that work well.

But that software still needs hardware to run on. And here lies the bottleneck.

Most data-intensive workloads still rely on traditional central processing units (CPUs)—the same processors used for general IT tasks. This creates challenges. CPUs are expensive, energy hungry, and not particularly well suited to parallel processing.

When a query needs to run across terabytes or even petabytes of data, engineers often divide the work into smaller tasks and process them sequentially. This method is inefficient and time-consuming. It also ends up requiring more total computation than a single large job would.

Even though CPUs can run at high clock speeds, they simply don’t have enough cores to efficiently handle complex queries at scale. As a result, hardware has limited the potential of big data. But now, that’s starting to change with the rise of accelerated computing.

Breaking the Bottleneck

Accelerated computing refers to running workloads on specialized hardware designed to outperform CPUs. This could mean field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs) built for a specific task. More relevant to big data, though, are graphics processing units (GPUs).

GPUs contain thousands of cores and are ideal for tasks that benefit from parallel processing. They can dramatically speed up large-scale data operations.

Interestingly, GPU computing and big data emerged around the same time. Nvidia launched CUDA (compute unified device architecture) in 2006, enabling general-purpose computing on graphics hardware. Just two years earlier, Google’s MapReduce paper laid the foundation for modern big data processing.

Despite this parallel emergence, GPUs haven’t become a standard part of enterprise data infrastructure. That’s due to a mix of factors.

For one, cloud-based access to GPUs was limited until relatively recently. When I started building GPU-accelerated software, SoftLayer—now absorbed into IBM Cloud—was the only real option.

There was also a perception problem. Many believed GPU development was too complex and costly to justify, especially for general business needs. And for a long time, few ready-made tools existed to make it easier.

Those barriers have largely fallen.

Today, a rich ecosystem of software exists to support GPU-accelerated computing. CUDA tools have matured, benefiting from nearly two decades of continuous development. And renting a top-tier GPU, like Nvidia’s A100, now costs as little as $1 per hour.

With affordable access and a better software stack, we’re finally seeing the pieces fall into place.

The Real Big Data Revolution

What’s coming next will be transformative.

Until now, most enterprises have been constrained by hardware limits. With GPU acceleration more accessible and a mature ecosystem of supporting tools, those constraints are finally lifting.

The impact will vary by organization. But broadly, companies will gain the ability to run complex data operations across massive datasets, without needing to worry about processing time or cost.

With faster, cheaper insights, businesses can make better decisions and act more quickly. The value of data will shift from how much is collected to how quickly it can be used.

Accelerated computing will also enable experimentation. Freed from concerns about query latency or resource drain, enterprises can explore how their data might power generative AI, smarter applications, or entirely new user experiences.

Gartner took big data off the Hype Cycle because it no longer seemed revolutionary. Accelerated computing is about to make it revolutionary again.

https://www.fastcompany.com/91346027/the-real-data-revolution-hasnt-happened-yet?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss

Erstellt 6d | 05.06.2025, 09:50:02


Melden Sie sich an, um einen Kommentar hinzuzufügen

Andere Beiträge in dieser Gruppe

Space and defense tech firm Voyager raises $382.8 million in IPO

Voyager Technologies raised $382.8 million in its U.S. initial public offering, the space and defense tech company said on Tuesday, amid a

11.06.2025, 18:50:03 | Fast company - tech
Hinge is teaming up with Esther Perel to rethink dating prompts

Need help sparking conversation on Hinge? Esther Perel has some questions for you.

The renowned ps

11.06.2025, 14:20:05 | Fast company - tech
Gen Z is embracing a digital detox and the ‘Martha Stewart summer’

If last summer was for the 365 party girl, this year it’s all about the digital detox.

At least that’s according to

11.06.2025, 14:20:04 | Fast company - tech
Databricks’ new ‘One’ dashboard brings AI to the business class (exclusive)

Databricks, known for secure data storage and AI, has launched a new AI business intelligence dashboard

11.06.2025, 14:20:04 | Fast company - tech
Mark Zuckerberg’s superintelligence gamble: Can billions and bold hires save Meta’s AI ambitions?

Mark Zuckerberg has decided he wants to claw back lost ground in the AI race, and he’s willing to spend heavily to do it.

The Meta CEO is

11.06.2025, 14:20:02 | Fast company - tech
Teaching AI isn’t enough—we need to teach wisdom, too

Artificial intelligence is shaking the intellectual, emotional, and economic foundations of the world. A glance at mainstream or social media confirms that the

11.06.2025, 11:50:03 | Fast company - tech
Boomerang CEO Aye Moah helped manage our inboxes. Now she’s taking on our calendars

It’s become increasingly common for people to devote

11.06.2025, 11:50:03 | Fast company - tech