Scientists from MIT presented the artificial intelligence (AI) system Foley Music, which generates music based on
Researchers believe that an AI model thatcreates music based on human movements, can become the basis for several applications - from automatically adding sound effects to videos to creating an immersive experience in virtual reality. The researchers note that people also have this skill - for example, when they understand a person's speech by their lips.
Foley Music draws attention to key pointsbody (25 points) and fingers (20 points) as intermediate visual anchor points, which she uses to model body and arm movements. The system then translates these movements into musical notes, taking into account the volume. So it can play accordion, bass guitar, bassoon, cello, guitar, piano, ukulelele and other instruments.
During the experiments, the researchers trained FoleyMusic to three datasets containing a thousand music video clips in 11 categories. So they were able to assemble a corpus of videos of varying complexity - instructions from the AtinPiano website, amateur videos from YouTube channels, excerpts from concerts and other data.
Researchers uploaded Foley Music 450video. Then they gave the resulting music to the scientists, who evaluated the result. In some cases, they noted that "the music is like a cover from a quality band."
Experts have found that Foley Music's generated music is difficult to distinguish from actual recordings. What's more, AI can improve audio quality, semantic alignment, and timing.
See also:
— Look at the 3D map of the Universe: it took 20 years to compile it and it has already surprised scientists
— Comet NEOWISE is visible in Russia. Where to see it, where to look and how to take a photo
— Three asteroids are flying towards Earth, and one of them is potentially dangerous. Is there a threat?