Microsoft's AI tool can turn photos into realistic videos of people talking and singing

Microsoft Research Asia has unveiled a new experimental AI tool called VASA-1 that can take a still image of a person — or the drawing of one — and an existing audio file to create a lifelike talking face out of them in real time. It has the ability to generate facial expressions and head motions for an existing still image and the appropriate lip movements to match a speech or a song. The researchers uploaded a ton of examples on the project page, and the results look good enough that they could fool people into thinking that they're real. 

While the lip and head motions in the examples could still look a bit robotic and out of sync upon closer inspection, it's still clear that the technology could be misused to easily and quickly create deepfake videos of real people. The researchers themselves are aware of that potential and have decided not to release "an online demo, API, product, additional implementation details, or any related offerings" until they're sure that their technology "will be used responsibly and in accordance with proper regulations." They didn't, however, say whether they're planning to implement certain safeguards to prevent bad actors from using them for nefarious purposes, such as to create deepfake porn or misinformation campaigns. 

The researchers believe their technology has a ton of benefits despite its potential for misuse. They said it can be used to enhance educational equity, as well as to improve accessibility for those with communication challenges, perhaps by giving them access to an avatar that can communicate for them. It can also provide companionship and therapeutic support for those who need it, they said, insinuating the VASA-1 could be used in programs that offer access to AI characters people can talk to. 

According to the paper published with the announcement, VASA-1 was trained on the VoxCeleb2 Dataset, which contains "over 1 million utterances for 6,112 celebrities" that were extracted from YouTube videos. Even though the tool was trained on real faces, it also works on artistic photos like the Mona Lisa, which the researchers amusingly combined with an audio file of Anne Hathaway's viral rendition of Lil Wayne's Paparazzi. It's so delightful, it's worth a watch, even if you're doubting what good a technology like this can do. 

This article originally appeared on Engadget at https://www.engadget.com/microsofts-ai-tool-can-turn-photos-into-realistic-videos-of-people-talking-and-singing-070052240.html?src=rss https://www.engadget.com/microsofts-ai-tool-can-turn-photos-into-realistic-videos-of-people-talking-and-singing-070052240.html?src=rss
Vytvořeno 1y | 20. 4. 2024 8:40:16


Chcete-li přidat komentář, přihlaste se

Ostatní příspěvky v této skupině

The best Amazon Prime Day robot vacuum deals end tonight: Save on machines from Shark, iRobot, Dyson and others

Robot vacuums are some of the pricier smart home gadgets you can get, making them great items to look for during sales like

11. 7. 2025 8:40:16 | Engadget
Amazon Prime Day kitchen deals: It's the last day to get up to 50 percent off air fryers, Instant Pots, blenders and more

If you’re considering updating your kitchen setup, first check out the best Prime Day kitchen deals we could find f

11. 7. 2025 8:40:15 | Engadget
The best Prime Day gaming deals on video games, hardware and more to get before the sale ends

It doesn't take much to play games these days. Phone, console, PC or handheld are all viable options. But why play using the bare minimum when you can upgrade your setup on the cheap by snagging a

11. 7. 2025 8:40:14 | Engadget
Prime Day laptop deals: Save on MacBooks, Windows 11 machines, Chromebooks and others before the sale ends

Regardless of if you need a new laptop for the upcoming semester or for general use at home,

11. 7. 2025 8:40:12 | Engadget
Paramount+ deal: Get two months of access for only $2

It's hot out there, so this might be the perfect time to stay in and binge some TV. Paramount+ is offering one heck of a deal right now for budding couch potatoes,

11. 7. 2025 8:40:11 | Engadget
Last chance Prime Day deals: Get up to 36 percent off Kindles and ereaders

Amazon’s big sale is almost ending (today is the last day) and these are the best

11. 7. 2025 8:40:10 | Engadget
The best Amazon Prime Day deals for the last day: Our top picks on headphones, TVs, robot vacuums and more

Amazon Prime Day 2025 is almost a wrap, so Prime members should take advantage of the long-awaited discounts across

11. 7. 2025 8:40:07 | Engadget