The men’s World Cup kicks off this week. As exciting as it promises to be, however, most of us will only get to watch it on flat television sets, as opposed to getting the three-dimensional experience of seeing it live and in person. Researchers from the University of Washington have come up with an augmented reality alternative, though. While it probably won’t be available to the masses in time for this year’s tournament, it does hint at one way in which fans at home may be able to enjoy sports games in the future.
What the researchers have developed is a machine learning algorithm capable of transforming 2D soccer clips into 3D reconstructions, which may be viewed using AR headsets like Microsoft’s HoloLens. The results allow viewers to turn flat surfaces like their desk or kitchen table into a virtual pitch, complete with three-dimensional action that you can circle around to view from different angles.
It’s not quite the equivalent of watching it in person, but it’s much closer than regular TV. And significantly cheaper, too!
“Our goal is to enhance the viewing experience of sports,” Konstantinos Rematas, one of the researchers on the project, told Digital Trends. “Instead of watching a soccer game or highlights on a flat 2D screen, we convert the original video into 3D and visualize it in augmented reality. Essentially the game becomes a hologram, where you can move around and look from different viewpoints, generating a more immersive experience.”
As its input, the algorithm generating the AR experience requires just a single YouTube soccer video. Because this is not enough information to train an entire rendering system to “upconvert” 2D players into 3D, the neural network learned to estimate depth by playing the playing the EA video game FIFA 2018. Using the totality of this information, it can do a passable job of accurately gauging where players are on the pitch.
At present, the system is still a work in progress. For instance, the ball is not yet properly rendered (something which turns out to be pretty darn important in soccer) and the players remain two-dimensional cutouts. These are two improvements the researchers hope to make.
“The next steps are about increasing the quality of the game reconstruction,” Rematas continued. “In particular, we want to estimate precisely the location of the ball and reconstruct better the players — occlusions, full 3D shape estimation, [and more].” In addition, the researchers plan to extend the framework to also cover other sports such as basketball, hockey, and football.
A paper describing the work is available to read online.
