Denis Kuleshov- Director of the Sensor-Tech laboratory.
Alexander Popov- chief designer
Andrey Demchinsky- Head of medical projects at Sensor-Tech.
Neurotechnologies that remove restrictions
A man is walking down the street.In one hand he has a white cane, with which he feels the way. In the other, there is a small black object that looks like a video camera. The man directs it first in front of him, then to the sides. Passers-by do not know that at this moment in the earpiece he hears prompts: “Car, distance - 5 meters. Man, distance - 3 meters. "
Another person comes to the MFC.“Hello,” the operator says with a smile, and this word appears on the screen facing the visitor. The man smiles back and starts typing the problem on the keyboard, which the operator will read on the computer monitor. An intermediary between the hearing impaired and the outside world is a small device that converts speech into text.
Both devices are developed and manufactured in the companySensor-Tech. Its director Denis Kuleshov began to cooperate with organizations for disabled people while still at the institute: he helped conduct research. Gradually Denis realized that he wanted to work in this direction and develop for people with disabilities. Therefore, when in 2014 he was offered to join the scientific research of the new So-Unification foundation, he immediately agreed.
The Foundation for the Support of the Deaf-Blind "So-Unity" was establishedin April 2014. Deals with the socialization of the deaf-blind: opens regional support centers, creates opportunities for employment, leisure and creative self-realization; provides legal, psychological and targeted assistance.
The fund has accumulated various projects, includingyouth and scientific, related to the issues of deaf-blindness, in which Denis took part. “At some point, it was decided that it would be much more effective and more correct to develop this in the format of a separate laboratory that deals with these issues in a specialized manner,” Kuleshov recalls.
The nuances of household appliance
In 2017, the Sensor-Tech laboratory appeared,which was founded by the So-Unification Foundation. Immediately, work began on the creation of gadgets for the hearing impaired and the blind. A grant from the Scientific and Technological Initiative was allocated for this within the framework of the Neuronet roadmap.
The condition for receiving the grant was co-financing of the project: 70% was allocated by NTI, the rest of the money had to be found by the developers themselves. The missing 30% was provided by the So-Unification Foundation.
The work of the laboratory had to be built from scratch:formed a design bureau and a pilot plant, where prototypes of devices were assembled. The first problem faced by the developers is the lack of experience in creating household appliances. Their previous experience in instrumentation was mainly related to the defense industry.
The first prototype of a device for the blind turned out to betoo cumbersome and inconvenient to use. Having received feedback from users, the developers have seriously transformed it. It was necessary to make the device work without interruptions, be convenient and understandable to use.
Now the laboratory has developed a device for the blind and visually impaired - "Robin" and a device for people with hearing disabilities - "Charlie".
“The market for devices for people with disabilities is not very large,all production is small-scale and expensive, ”says Denis. This is the reason for the rather high price of the devices: "Robin" costs 150 thousand rubles, "Charlie" - 195 thousand rubles. Therefore, manufacturers are negotiating with the government to include their inventions in the list of means of technical rehabilitation - in this case, the government will compensate for the purchase costs.
Tell me what you see
Outwardly "Robin" resembles a portable camera.Sensors scan objects, and artificial intelligence determines what is in front of the user - a car, a table, a computer, another person. There are more than 50 objects in the device memory. If you upload photos there, it recognizes the person and says his name. To use "Robin" in the dark, a flashlight was built into it. In this case, the gadget not only names the object, but also estimates the distance to it.
The first release of "Robin" was unsuccessful.In order for the device to detect an object, it was necessary to point the device, press the button and wait at least 2 seconds. But it turned out that this was inconvenient for users: immediately after pressing the button, they began to drive the device from side to side. Because of this, the information voiced by "Robin" turned out to be irrelevant: he was describing objects that he recognized before he was moved.
“We knew how to help the device and where todirect, and people did not have such experience. Because of this, it seemed to them that the device was not working well, ”explains Alexander. Then the developers decided to change the principle of work. The feedback time from the device was reduced to a second, while the "Robin" only worked when the button was pressed. This option turned out to be clearer and easier for users.
“There are no direct competitors, that is, devices with the same functions, on the market. But there are analogues, for example, OrCam, ”says Alexander.
OrCam MyEye and OrCam MyReader- portable artificial vision devices,which are produced by the Israeli company of the same name. They are a small wireless camera that can be attached to the temple of glasses. The first device reads printed and digital text, product barcodes, recognizes faces and speaks the received information. The second is used only for reading texts. The gadgets support 17 languages and do not require an Internet connection.
Voice turns to text
"Charlie" is intended for people with problems withhearing. People with experience in teaching for the hearing impaired participated in its development. Therefore, the developers were aware of the difficulties that arise in the process of communication and the requests of both parties.
The device picks up speech at a distance of 2 m anddisplays it on the screen as text. The gadget can also be used by the deaf-blind - for this, a Braille display is connected to the "Charlie". It allows you to read information with your fingers and enter a response.
Braille was developed in 1824.It is a system of convex points and voids between them. Each character is encoded using a 3x2 trellis. The combination of dots within each cell corresponds to a letter or punctuation mark. If the text changes the script, for example, from Latin to Cyrillic, this is also indicated by a special symbol.
The developers say that the analogues of "Charlie" onthe market also does not. Numerous programs that translate speech into text are designed primarily for human-computer interaction, and not for two people to communicate with each other.
The first machine capable of voice recognitionappeared in the 50s within the walls of Bell Labs. The device determined numbers from zero to nine. At the same time, the machine understood the inventor much better: when he spoke, the reproduction accuracy was about 90%. The device correctly recognized the speech of other people only in 70–80% of cases.
Until the 90s, speech recognition was based ontemplates: sound waves were translated into a set of numbers, and the result was displayed when speech matched the sample. Therefore, for the correct interpretation of voice signals, it was required to eliminate background noise, speak slowly and clearly.
The first speech recognizer that did not require pauses between words was Dragon's NaturallySpeaking. It appeared in 1997 and is still in use today.
Machine learning and AI technologies significantlyimproved the speech recognition system and made it possible to adapt the algorithms to the individual manner of communication of each person. The result was the emergence of voice assistants: Google Assistant, Siri from Apple, Alexa from Amazon, “Alice” from Yandex. In addition, there are special applications, for example, ListNote, SpeechNotes, for speech recognition, translation of a voice into a text message. Some applications, such as Speechlogger, can even perform simultaneous translation from one language to another.
With the help of "Charlie" the interlocutors will be able to communicatewith each other without resorting to an intermediary. In addition, the device can be used for remote communication, for example, for lectures at universities or holding meetings. It is enough for the presenter to put "Charlie" next to him, and the listeners - to connect to the program by the link. The decoded speech will be displayed on the screen of a smartphone or computer in real time.
See the world again
Also, "Sensor-Tech" participates in scientific research related to the use of neurotechnologies for the return of vision.
It all started in 2016 at an internationalconferences in the USA. There, representatives of the So-Unification Foundation met with an American company dealing with bionic vision. They agreed to conduct an experiment in Russia to implant bionic chips in blind people.
“The device may not be applicable to everyoneof the blind, this is a fairly narrow pool of diseases, mainly retinitis pigmentosa. The peculiarity is that with it, only one layer of cells dies, which transforms light into an electrical signal. The rest of the cells remain alive, ”explains Denis.
Retinitis pigmentosa- a hereditary disease associated withX chromosome. With this pathology, the retinal cells that collect images and transmit them along the optic nerve to the brain are gradually destroyed. The disease begins with loss of side and night vision and eventually leads to complete blindness. There are no effective preventive measures against retinitis pigmentosa, as well as no treatment methods. Implantation of microchips is still under development, and doctors also suggest that gene therapy and stem cell treatment could become a breakthrough.
As a result, in the databases of the So-Soyedinenie and Art, Science and Sport funds found two people with the right diagnosis whothere were no concomitant pathologies. In 2017, the implants were installed to Grigory Ulyanov and Antonina Zakharchenko. The funds for the operation were allocated by the Alisher Usmanov Foundation.
Alisher Usmanov Foundation "Art, Science and Sport"founded in 2006. Finances educational and scientific projects (cooperates with MGIMO, MISiS), supports museums and theaters (Sovremennik, Tretyakov Gallery, Igor Moiseev's ensemble and others). The foundation also promotes healthy lifestyles and organizes competitions, including for people with disabilities.
During the operation, the patient's retina is placedimplant with electrodes. It is associated with special glasses with a built-in camera. Information from the camera is transmitted to a microcomputer, which processes the image and sends signals to the implant. Electrical impulses are transmitted along the optic nerve to the brain, where a picture is formed.
There are 60 electrodes in the implant, which can be compared witha picture of 60 pixels - according to the number of points that the implant stimulates. With its help, you can get an image of only large enough objects - windows, doors, tables, cars. Smaller objects may fall out of the “matrix” formed by the electrodes, or the data is insufficient to identify them.
Since the patient does not have the cells responsible forcolor rendition, the image is black and white. But compared to complete blindness, even such limited vision from the point of view of a healthy person significantly expands the capabilities of patients. They can orient themselves in space even in unfamiliar conditions without outside help or additional devices and become quite independent.
Sensor-Tech hoped that the operation would allowregister implants in Russia and make them available within the framework of high-tech medical care. But a problem arose: the Ministry of Health considered that two operations were not enough to draw a conclusion about the safety of the implants. The Americans, in turn, refused to conduct further experiments, since there was no guarantee that this method would eventually be approved.
“In terms of eye implants, the story is over.But this is even good, because it has a continuation in the form of cortical implants, which are placed in the brain. This is where the visual processing of information takes place. Chips can be used for any kind of blindness, unlike eye ones. This is a more high-tech solution, ”says Denis.
Already in 2021, the company presented the firstdevelopment in a new direction. This is the first ELVIS neuroimplant for the brain in Russia; it helps restore vision to blind and deaf-blind people. The device has an implant that is installed in the brain, a hoop with two cameras to be worn on the head - they take over the function of the eyes and a microcomputer that analyzes the image and transmits it to the brain. The technology makes it possible to distinguish the silhouettes of objects, people and to understand the location of objects.
While the system is being tested on rodents, tests on monkeys will follow. In 2024, 10 blind volunteers are planning to install a cortical implant.
Cortical visual implantsinstalled in the cerebral cortex.They stimulate the visual areas, resulting in visual sensations. According to experts, this will make it possible to change the brightness of images and ensure color rendition, which is impossible when using bionic implants. The first operations to implant cortical implants were carried out back in the 70s of the last century. And in 2018, the American company Second Sight publicly installed modern neuroimplants on the first blind volunteers. The technology is undergoing clinical trials.
Scientific research and overseas markets
There are others among the cases of the laboratorydevelopment. For example, a free application for mobile phones was developed together with a mobile operator to help blind people determine the denomination of banknotes. It is available on iOS and Android.
To find out the denomination of a bill, just point your smartphone camera at it. The AI will identify the banknote and name it. And if you point the camera at several bills at the same time, the device will immediately calculate their amount.
If the application is used by a person withvision and hearing disabilities, who cannot hear the voice acting, the denomination of the bill can be recognized using vibration - for different denominations there are own modes.
If the phone cannot recognize the bill, you cantake a picture and add it to the database via the "Help for Developers" function. Usually the need for this arises when issuing new or rare commemorative notes.
In addition to the production of devices, Sensor-Techare engaged in scientific research in the field of deaf-blindness. “We want to make healthy people understand how those who have impaired vision or hearing perceive the world in general,” says Andrey.
View this post on Instagram
So, the laboratory created simulators of violationsview: VR simulator See My World and its mobile version SMW Pro. "Noise" is superimposed on the live image or on a static image. As a result, the picture is distorted according to the specific visual defect. For example, it becomes blurry or a black spot appears in the center of the image.
“We simulate the symptoms of how a person withvarious visual impairments. Literally: how to look through the eyes of a person who has cataracts, myopia, astigmatism, glaucoma, ”says Andrey. Also, using the simulator, you can trace the dynamics of the disease over time. This helps doctors and students to better study the clinical picture, and the patient's close relatives - literally look at the world through his eyes.
“To make the image as accurate as possible,to create it, the clinical picture of the disease is used. Additionally, we clarify the result with the patient, so that he can confirm whether he sees so, if his vision is not severely impaired yet, ”explains Andrey.
Now developers want to promote theirdevices in the European and US markets. It is planned to put the gadgets on Amazon and, in addition, to reach government bodies and public organizations dealing with the problems of people with disabilities. Denis is sure that the market for applied developments will only grow. And the director of the Sensor-Tech laboratory sees his mission in making the life of people with disabilities easier thanks to technology.
Read more:
AI solved a biological problem that scientists have been battling for 50 years
A millisecond instead of 30 trillion years for a task: China introduced a new quantum computer
Scientists are looking for people who cannot be infected with COVID-19. Based on their data, they will make a medicine