Almost dubbing: Yandex has learned the multi-voiced translation of YouTube videos

Remember the Yandex neural network, which translated foreign videos on the fly? The company “upgraded” her by teaching her

big tricks.
Discuss

So, now Yandex.Browser users can watch videos in English with multi-voiced voice-over translation into Russian.

If initially the technology used two synthesized voices for speech translation - one male and one female - now there are twelve voices. Six for each gender.

It is reported that the neural network “distributes” voices to different speakers, after which it “remembers” them using AI models created inside Yandex.

And it all works quite multi-layered:first, one neural network translates speech into text, restores punctuation and determines the boundaries of sentences, and then another analyzes the spectrogram of the voice and marks fragments spoken by different people.