Microsoft’s Windows future is built on AI, voice, cloud, and context

Can you imagine yourself having a conversation with Windows about what your PC is doing? Microsoft’s Windows chief can, and is trying to build a future where those interactions are the norm.

In an interview with Microsoft AI product manager Christiaan Brinkhoff, the chief of Microsoft’s Windows + Devices group, Pavan Davuluri, explained that the company is trying to work toward a future where you can access Windows pretty much anywhere via the cloud, then use AI to fine-tune what you’re trying to accomplish.

Microsoft described the conversation as “the next chapter of Windows,” with an eye toward delivering the changes within the next few years. Davuluri described what he hoped the Windows team could accomplish from a strategic level, without targeting any future version of Windows with these goals in mind.

Like many projects at Microsoft, Davuluri’s vision begins with businesses in mind. Instead of (or adjacent to) the PC, Microsoft said it’s backing the Windows 365 Link, a sort of dumb mini PC that connects directly to the cloud and to Windows 365. “People are not talking about multiple steps; they’re just trying to get to the cloud PC as seamlessly as possible,” he said.

“They can’t tell when they’re using their local CPU and when they’re using a CPU in Azure,” Microsoft’s cloud, Davuluri added.

That’s not new. Microsoft began talking about this hybrid compute model a few years ago, when Microsoft chief executive Satya Nadella opened its Build developer conference by specifically calling out how future applications would combine the cloud and local compute capabilities.

At Microsoft, that’s now common, with Copilot sneaking into just about everything. But Nadella used his keynote to specifically call out Adobe and apps like Photoshop. And that’s exactly what happened, there, top: Photoshop now uses your PC’s GPU to modify images, but also calls upon the Adobe cloud for generative AI.

Voice is coming back to your PC

Davuluri also hearkened further back, stating that Microsoft would expand Windows’ modalities into speech and vision. Modalities are what Microsoft uses to describe the various ways you interact with your PC: you can touch the screen, type on the keyboard, or move the mouse. More recently, pens have added inking, too.

“I think we will see computing become more ambient, more pervasive, continue to span form factors and certainly become more multimodal in the arc of time,” Davuluri said.

“I think experience diversity is the next space where we will continue to see voice becoming more important,” Davuluri added.

“I think more fundamentally, the concept that your computer can actually look at your screen and is context aware is going to become an important modality for us going forward. The other thing I think is going to get more intuitive is multimodal interactions. So you’ll be able to speak to your computer while, you know, writing, inking, interacting with another person. For example, you should be able to have a computer semantically understand your intent to interact with it from when it’s awake, or to put a machine to sleep.”

Mark Hachman / Foundry

That’s a pretty direct reference to Copilot Vision, which can now “see” your screen and talk to you about it. Microsoft built that from two AI models, Phi and Mu, which are “small” language models that can run on your PC. Those models not only enable services like Copilot Vision but can also be used to power Microsoft’s new semantic search options in File Explorer and Settings, which can be changed at a user’s natural-language request.

Microsoft plans to augment those capabilities as assistive technologies, such as evolving Narrator from simply “reading” words on your screen to actively describing what’s going on.

With voice and vision being added to ChatGPT and other models, people are finding themselves chatting with AI. Will they feel as comfortable doing so with a PC?

Will Surface flourish again?

Finally, Davuluri gave a hint (or not) about the future of Microsoft’s Surface PCs.

After the departure of former Windows + Devices chief Panos Panay, some wondered if Microsoft would ever return to the diversity of Surface devices that prompted everything from the Surface Duo to the Surface Laptop Studio — both seemingly on hiatus or discontinued. Davuluri didn’t offer any clues.

“I think today one of the things that we celebrate in Windows is the diversity on form factors in which computing is available, for sure,” he said. “Between the work that the ecosystem partners do and Surface does, we have a pretty broad range of devices in themselves.