Exploiting video game software yields broadcast-quality 3-D video of soccer games in real time

By exploiting the graphics-rendering software that powers sports video games, researchers at MIT and the Qatar Computing Research Institute (QCRI) have developed a system that automatically converts 2-D video of soccer games into 3-D.

The converted video can be played back over any 3-D device—a commercial 3-D TV, Google's new Cardboard system, which turns smartphones into 3-D displays, or special-purpose displays such as Oculus Rift.

The researchers presented the new system last week at the Association for Computing Machinery's Multimedia conference.

Zeroing in

In the past, researchers have tried to develop general-purpose systems for converting 2-D video to 3-D, but they haven't worked very well and have tended to produce odd visual artifacts that detract from the viewing experience.

"Our advantage is that we can develop it for a very specific problem domain," Matusik says. "We are developing a conversion pipeline for a specific sport. We would like to do it at broadcast quality, and we would like to do it in real-time. What we have noticed is that we can leverage video games."

Jigsaw puzzle

For every frame of 2-D video of an actual soccer game, the system looks for the 10 or so screen shots in the database that best correspond to it. Then it decomposes all those images, looking for the best matches between smaller regions of the video feed and smaller regions of the screen shots. Once it's found those matches, it superimposes the depth information from the screen shots on the corresponding sections of the video feed. Finally, it stitches the pieces back together.

"This is a clever use of game content, which leads to better results and easier acquisition of large and diverse reference data," says Hanspeter Pfister, a professor of computer science at Harvard University. "One of the main insights of the paper is that domain-specific methods are able to yield bigger improvements than more general approaches. This is an important lesson that will have ramifications for other domains."

STAY TUNED FOR MORE UPDATES.....