That’s what researchers at the Massachusetts Institute of Technology have done, with a new paper revealing not just their ability to look at still images and guess what will happen next — something we’ve covered in the past — but to actually generate video of it.
“What we’re interested in is teaching machines what can happen in a particular setting,” Carl Vondrick, a Ph.D. student in computer science, told Digital Trends. “For example, we wanted a machine to recognize what happens on a beach. We want it to know that waves are going to crash, people are going to play in the water — these are all things it’s very difficult to teach a machine. The reason is that it would be very time-consuming for a person to sit down and write rules to explain everything that can happen in any given scenario. What we wanted to do was to teach them from watching massive amounts of video instead.”
To do this, Vondrick and his fellow researchers used deep learning to “train” a computer to understand everyday scenes by watching the equivalent of two years’ worth of online video footage. After that, they then showed the machine still images and asked it to generate video to bring that scene to life.
Like many examples of computation creativity, such as Google’s DeepDream technology, the generated images can be a bit trippy — although there’s no denying they get a lot right about the world. MIT’s system, for instance, takes a still image of a beach and animates it to show waves crashing, just as Vondrick hoped it would.
Limitations mean that the videos are only one second long, and animated like a made-for-TV movie from the 1990s, but it’s an impressive start.
“One application I’m really excited about is predicting the future,” Vondrick said. “What our work is doing is to generate videos letting us anticipate what is going to happen next. This is important for robotics. You can imagine a robot being able to predict when an elderly person is going to fall down, for example, and then preventing that from happening. It’s also got application in computer graphics. Right now, the system isn’t photorealistic, but as we keep improving it I think we’ll be able to generate full-length videos. This could therefore be very useful as a tool in something like Hollywood movie production.”
It’s potentially even more significant than that. “In the largest sense, this work is about a computer learning to simulate the world,” he said. “Getting a machine to understand how the world works will be very important as computers start taking more and more actions in our world — and need to understand what is going on in order to do that.”
