AI was taught to determine what a person sees based on "mind reading"

Neuroscientists still do not fully understand the entire process of how the brain converts visual information into

thoughts.But this does not prevent AI from imitating this process. Japanese scientists have combined algorithms that convert text into images and a system for analyzing brain activity. Stable Diffusion's AI scans a person's brain and recreates relatively realistic versions of the images they have seen.

Original images (top) and Stable Diffusion brain activity recognition images. Image: Yu Takagi, Shinji Nishimoto, bioRxiv

This is not the first study in which scientistsused AI to read brain scans and recreate the images. But in the new work, they added an additional step: when training AI, textual descriptions of photographs are used. As a result, the algorithm analyzes not only the brain activity associated with viewing images, but also the text description. This is reminiscent of popular algorithms such as DALL-E 2 and Midjourney.

For Pattern Recognition Stable Diffusionuses functional magnetic resonance imaging (fMRI) brain scan data collected by viewing various images. When people look at a photograph, the temporal lobes analyze information about the content of the image (people, objects, or scenery), while the occipital lobes analyze location and perspective, such as the scale and position of the content.

To train the AI, the researchers used a setpictures and patterns of brain activity collected while viewing them. With simple training, the algorithm effectively recreated the location of objects and the perspective of the photographs being viewed, but instead of real objects in the center of the composition, it drew abstract shapes. But, after the scientists added textual descriptions to the photographs used in the training, the quality of the generated images increased significantly.

Original image (left) and images,obtained using algorithms trained on visual (z), text (c) and combined data (right). Image: Yu Takagi, Shinji Nishimoto, bioRxiv

The researchers note that the combination of textand visual data during training allows you to significantly reduce the amount of data that must be used to train the model. At the same time, they note that so far the AI ​​has only been trained and tested on fMRI data for four people. Probably, a sample from a large number of people will be required to create a universal algorithm.

Read more:

Named a vitamin that protects the brain from dementia

Named a type of insomnia that reduces the risk of dementia. But why is unknown.

It turned out which men are the most fertile: their sperm is 50% better than the rest