I tested Copilot Vision for Windows. Its AI eyes need better glasses

The whole point of Microsoft Copilot Vision for Windows is that it’s like an AI assistant, looking over your shoulder as you struggle through a task and making suggestions. Click here. Do this! So, I was pretty convinced that if Microsoft were to release Copilot Vision for testing, it would be able to do something simple like help me play Windows Solitaire. But no. Oh no, no, no.

Sometimes, Microsoft’s new Copilot Vision for Windows feels like a real step forward for useful AI: this emerging Windows technology sees what you see on your screen, allowing you to talk to your PC and ask it for help. Unfortunately, that step ahead is often followed by that cliché: two steps back. Copilot Vision for Windows is, at times, genuinely helpful. At others, it’s just plain frustrating.

What is Copilot Vision for Windows?

Outside of some nostalgic tears by former Microsoft CEO Steve Ballmer, the announcement of Copilot Vision for Windows was the highlight of Microsoft’s 50th anniversary celebration at the company’s Redmond, Washington campus.

It’s a visionary technology, pretty literally: you grant access to Windows Copilot to see and interpret your screen in real time, and you can talk to Windows to ask questions and seek advice. I went hands-on with Copilot Vision at Microsoft’s HQ, but the demos were short and carefully managed. Now, you can play with it yourself as long as you’re a Windows Insider.

How to get Microsoft Copilot Vision for Windows

Currently, Copilot Vision for Windows is just available for testing. Although Microsoft indicated that Copilot Vision for Windows would be available to all of its beta software channels, only two of my test laptops ever received the build: one on the Dev Channel and one on the Canary Channel.

The first to get it, an Acer Swift Edge laptop with a Ryzen 7840U inside, runs Vision slowly, with reaction times that seemed to stretch to half a minute early on. Though the response time dropped to a few seconds, I had a far better experience with the Surface Laptop 7 or 7th Edition, with a Qualcomm Snapdragon X Elite chip inside. Responses were essentially instantaneous, probably due to the more powerful NPU.

Mark Hachman / Foundry

Copilot Vision for Windows is easy to use: provided your PC is provisioned for it, just launch the Copilot app via the Taskbar or Start menu, and then tap the “eyeglasses” icon. You’ll then see a list of apps for you to “share” with Copilot Vision. Only then can it see that specific app, and just that app.

I put a test version of Copilot Vision for Windows through seven quick scenarios: interpreting the contents of a PCWorld story and a list of competing airfares; testing Balatro, a popular PC game that involves playing cards; the more generic and classic Solitaire game; photo identification; examining potential airfares; and help operating Adobe Photoshop. Copilot Vision was all over the board.

1.) Copilot Vision’s first test: understanding tariffs

The first and most important lesson of Copilot Vision is it only sees what you see. I realized this when I opened my colleague Alaina Yee’s early examination of the Trump Administration’s tariff plan from April. Copilot Vision for Windows didn’t immediately “see” the whole article — which is what Copilot, Google Gemini, or ChatGPT in its “research” modes likely would.

If I scanned down, it could “read” along. But it didn’t read it into memory, either. What it didn’t see, it forgot. I asked it to confirm, and it couldn’t tell me the opening sentence.

That makes its utility rather limited. What was handy was being able to ask it conversational questions: at the time, the products in question were subject to a 45 percent tariff. Being able to ask it what the price of the dock would be if a 100 percent or 145 percent tariff was applied was handy. Copilot Vision is still a little wordy, but that was okay. The bigger issue is that it was reluctant to add context, such as to point out the current state of the tariff situation.

2.) Does Copilot Vision work as a Balatro coach?

One of the things I’ve been thinking about was the Minecraft demo, where Copilot Vision stepped in with help on some very specific scenarios. It made me suspicious, naturally; what I was seeing was carefully scripted to make Copilot Vision look as useful as possible. I think that’s true.

I figured the popular indie game, Balatro, would be a better use of its talents. What Copilot told me is that it wouldn’t just spontaneously interject, so if it “saw” something useful or dangerous, it wouldn’t just pipe up and say something. It needs to be asked.

Mark Hachman / Foundry

Balatro is vaguely like video poker, but with a twist: not only do you have to try and come up with the best poker hands, there are twists — “jokers” modify your hands and your score, so strategy means some careful choices. Would Copilot Vision be able to recognize what I needed to do and give advice?

Absolutely not. Copilot Vision was absolutely able to recognize that I was playing Balatro, and upon the game’s opening, it identified the choices I had before me. Copilot didn’t make the decisions for me, but it tried to present my options, as in the screenshot above. That’s good, right?

Mark Hachman / Foundry

Well, no. Copilot Vision failed to recognize that I didn’t have a pair of queens, which meant that its advice was off from the start. It also couldn’t properly recognize the cards that I did have, like incorrectly identifying seven of diamonds when I didn’t have one.

3.) Solitaire is simpler, right?

I then figured, well, let’s dumb it down a bit. I launched a new game of Windows Solitaire, specifically FreeCell, thinking that Copilot would be able to understand the simple rules and act accordingly.

Absolutely not. Copilot Vision suffered the same problem that it had with Balatro: its object recognition was way off. It repeatedly invented cards that weren’t on the board, although it did understand how to move cards from column to column and to the foundation. It’s just whatever game Copilot Vision was playing wasn’t actually present on the screen. I quickly gave up.

Mark Hachman / Foundry

I’m not sure you’re the best Solitaire player, I said. “Fair point! It’s all about having fun, though,” Copilot replied. “If nothing else, I’ve got your back for the banter. Let’s keep playing and see where it goes. Ready for another move?”

No, I wasn’t.

4.) Can Copilot stop me from being fired?

Next, I tried what I thought was an easy one: I had another AI draft a complaint letter to a fictional boss, expressing displeasure at our fictional company’s corporate strategy. As I had asked Google Gemini to draft a professional note, Copilot thought that my tone and language were fine.

I then added a line at the end: “You’re ugly and have a big fat head. I also don’t like your hat.”

Mark Hachman / Foundry

Either Copilot Vision didn’t recognize that I had made a possibly career-threatening edit or else didn’t care. In fact, I’m pretty sure it didn’t care, as the conversation continues.

(Remember, I’m speaking here, so the transcription isn’t perfect. A plus in Copilot Vision’s favor is that it does lend itself to a conversation, rather than a series of commands.)