Microsoft unveils AR Copilot to help industrial workers

AI assistants aren’t just for office work anymore.

Microsoft on Wednesday unveiled a new virtual Copilot in Microsoft Dynamics 365 Guides, which will let workers in industrial settings use speech and gestures to ask questions about the complex machinery they use and service. Microsoft’s HoloLens mixed reality headsets will let them point to equipment and parts, then ask questions about topics from component specifications to machinery service logs. The system can provide answers via hologram, text displays, and voice response.

Lili Cheng, corporate vice president at Microsoft’s AI and Research Division, says the Copilot can help companies move away from requiring frontline workers to consult hulking paper manuals or laptop-bound documentation, which can be especially cumbersome for those already carrying tools and protective equipment. The technology, which can also help users log the work they do, may also help companies hire and retain workers who’ve grown up using digital tech. And new workers can use the Copilot to answer even basic questions they might have otherwise asked busy colleagues.

“Younger people just expect more modern tools,” Cheng says. “They don’t expect to be going to a job with a big paper manual or, you know, flipping through a lot of records in a cumbersome way.”

Microsoft is initially testing the new Copilot with customers, and Cheng says it can be especially valuable in remote settings like oilfield work, where driving back to an office to pick up a needed manual can prove time consuming. The software can be trained based on formal documentation, field notes, call transcripts, and other sources of knowledge, meaning it has potential to help transfer institutional wisdom as employees move on or retire. It can also access operational data and inputs from internet of things devices like sensors that can help to diagnose and fix a problem. Digital models of equipment and facilities—which Cheng says many companies already have—can be loaded into the software so workers can reliably point to where their questions arise.

“We know the models,” she says. “We know your customer service records.”

Microsoft has already unveiled Copilots for office workers creating spreadsheets or writing code, as well as Copilot tools to help field technicians and managers schedule shifts and complete work orders. The new Copilot will bring the AI assistant further into the blue collar workspace, initially via the HoloLens, which Cheng says some manufacturing companies already use for VR training and for remote human assistance. The new Copilot is likely to expand to phones and tablets in the future, but Cheng says wearing the HoloLens often isn’t that much of an issue in fields where workers already wear plenty of specialized gear. (The HoloLens can also easily be tilted up and down for workers who prefer to only look through it as needed.)

The Copilot is powered by Microsoft’s Azure OpenAI Service, which the company is steadily enhancing to make it easy for companies to build their own digital copilots able to access internal data. Hallucination—where AI systems give plausible but essentially made-up answers to questions—typically isn’t a problem with those data sources, Cheng says.

After the current test, the company plans to fairly quickly roll the software out to a broader audience.

“We just want to really make sure that it’s as easy to use as we’ve seen,” she says. “Because, you know, frontline workers are really busy.”

https://www.fastcompany.com/90982407/microsoft-unveils-ar-copilot-to-help-industrial-workers?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss