Facebook has created an artificial intelligence that turns real people into manageable game characters.

A Facebook research group created an AI called Vid2Play, which can make game characters on

based videos with real people.

How did they do it

The team used two neural networks: Pose2Pose and Pose2Frame. First, the video is served in Pose2Pose, designed to identify different types of activities like dancing, playing tennis or fencing. Then the system separates the person’s figure along with his shadow and objects in his hands from the background and inserts them onto a black background. After that, you can control the movement of the object with the joystick or keyboard.

For training the system took onlyseveral short videos. Artificial intelligence was able to remove other people and compensate for different angles of the camera. The study resembles a feature in Adobe After Effects to remove unnecessary elements from videos, such as tourists or garbage cans.

So far, the animation is not perfect - the figure looks like a sliding one and the range of motion is a bit limited. However, this is a great progress compared to previous attempts to extract a character from the video.

This innovation will allow game developers to create their heroes based on personal videos or add your favorite personality from YouTube. The movements will be more realistic than when creating the animation manually.