We learned to identify sign language in video calls

Researchers have presented a real-time sign language detection system. She can tell when

the interlocutor is trying to say something or justmoves body, head, arms. Scientists note that this task may seem easy for a person, but previously such a system was not available in any of the video calling services - they all respond to any sound or human gesture.

New development by Google researchers is capable ofdo it with great efficiency and low latency. While the researchers note that the detection of sign language leads to a delay or degraded video quality, this problem can be solved, and the model itself remains light and reliable.

The system first runs the video through the model undercalled PoseNet, which assesses the position of the body and limbs in each frame. Simplified visual information is sent to a model trained to position data from videos of people using sign language and compares the image to how people usually display certain words.

The model correctly identifies words and expressions with80% accuracy, and with additional optimization it can reach 91.5%. Considering that the detection of an "active speaker" in most services works with delays, the researchers believe that these are very large numbers.

See also:

It is possible to create a thermonuclear reactor on Earth. What will be the consequences?

The Doomsday glacier turned out to be more dangerous than scientists thought. We tell the main thing

On day 3 of illness, most COVID-19 patients lose their sense of smell and often suffer from a runny nose