Skip to main content

Adobe and Stanford just taught AI to edit videos — with impressive results

Computational Video Editing for Dialogue-Driven Scenes (Full Video)
Just one minute of video typically takes several hours of editing — but Stanford and Adobe researchers have developed an artificial intelligence (AI) program that partially automates the editing process, while still giving the user creative control over the final result.
Recommended Videos

The program starts by organizing all of the footage, which is often from multiple takes and camera angles. Those clips are matched to the script, so it’s easy to find several video options for each line of dialogue.

The program then works to recognize exactly what is inside those clips. Using facial recognition alongside emotion recognition and other computational imaging effects, the program determines what is in each frame. For example, the program flags whether the shot is a wide-angle or a close-up and which characters the shot includes.

With everything organized, the video editor then instructs the program in just how the videos should be edited using different styles and techniques the researchers call idioms. For example, a common style is to show the face of the character during their lines. If the editor wants that to happen, he or she just drags that idiom over. The idioms can also be negative. For example, the idiom “avoid jump cuts,” can be added to actually avoid them, or negatively to intentionally add jump cuts whenever possible.

The editor can drag over multiple idioms to instruct the program on an editing style. In a video demonstrating the technology, the researchers created a cinematic edit by using idioms that tell the software to keep the speaker visible while talking, to start with a wide-angle shot, to mix with close-ups and to avoid jump cuts. To edit the video in a completely different, fast-paced style, the researchers instead dragged over idioms for including jump cuts, using fast performance, and keeping the zoom consistent.

Editing styles can be saved to recall later, and with the idioms in place, a stylized video edit is generated with a click. Alternative clips are arranged next to the computer’s edit so editors can quickly adjust if something’s not quite right.

The program speeds up video editing using artificial intelligence, but also allows actual humans to set the creative parameters in order to achieve a certain style. The researchers did acknowledge a few shortcomings of the program. The system is designed for dialogue-based videos, and further work would need to be done for the program to work with other types of shots, such as action shots. The program also couldn’t prevent continuity errors, where the actor’s hands or a prop is in a different location in the next clip.

The study, conducted by Stanford University and Adobe Research, is included in the July issue of the ACM Transactions on Graphics Journal.

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
The best webcams for 2023

Laptop webcams suck. If you're stuck doing Zoom calls or videoconferencing from your home, you'll need a decent external or stand-alone webcam, one that works for your preferred space. The Logitech C920S is currently our pick for the best webcam. It's affordable and provides crisp image quality. But if you need a higher resolution or a streaming-specific option, check the full list below.

Read more
Best microSD cards in 2023: top picks for your computer, camera, or drone
galaxy s8 tips and tricks

MicroSD cards are an unsung hero in the mobile space. They provide extra storage space for your smartphone, but they're also key in tablets, drones, and security cameras. Unfortunately, this feature is falling out of favor in the smartphone space, and most flagship and midrange phones no longer have a slot to insert a microSD card. Even the most expensive phones around, like the Samsung Galaxy Z Fold 4 and the Samsung S22 Ultra don't have a microSD card slot anymore. You're not even safe if you have an iPhone or iPad, as they've never had microSD card slots.

But all is not lost! Some of the best Android phones and best Android tablets do still support them, and they're still required for use with cameras, security cameras, and drones. But no matter which device you're buying it for, you'll want to get your hands on a microSD card from a reputable brand.

Read more
Selfie stunt sets new Guinness World Record
Indian actor Akshay Kumar attempts the world record for most selfies taken in three minutes.

Akshay Kumar Attempts Most Selfies Taken In Three Minutes - Guinness World Records

It seems like Guinness World Records is happy to entertain any kind of extraordinary feat for its listings, no matter how weird or wacky.

Read more