Mathematicians reduced the size of the neural network by 6 times without retraining it

The structures of artificial neural networks and neurons in a living organism are based on the same principles. Nodes in

the networks are interconnected, but some of themreceive a signal, and some transmit it, activating or suppressing the next element in the chain. Any signal, such as picture or sound, requires many network elements and their outgoing connections to process. However, computer models have limited capacity and memory. To work with large amounts of data, specialists have to invent various ways to reduce the capacity requirements, including so-called quantization. This helps reduce resource consumption, but requires system retraining.

“Several years ago, we conducted an effective andeconomical quantization of weights in the Hopfield network. It is an associative memory network with symmetric connections between elements, which are formed according to the Hebb rule. In the course of its operation, the activity of the network is reduced to a certain state of equilibrium, and when it is reached, the problem is considered solved. The findings of this study were later applied to deep learning direct learning networks, which are very popular in image recognition today. Typically, these networks require re-training after quantization, but we have found a way to avoid this. "

Yakov Karandashev, Ph.D., associate professor at RUDN University.

The basic idea behind simplifying artificial neuralnetworks - this is the so-called quantization of weights, that is, reducing the number of bits for each weight. Quantization involves averaging the signal: for example, if applied to an image, all pixels representing different shades of the same color will become identical. Mathematically, this means that neural connections that are similar in certain parameters should have the same weight (or importance), expressed as a number.

A team of mathematicians from RUDN University carried out calculations andcreated formulas that effectively establish correlations between weights in a neural network before and after quantization. Based on them, scientists developed algorithms with which a trained neural network could classify images. In their experiment, the mathematicians used a text package of 50,000 photographs that could be divided into 1,000 groups. After training, the network was quantized using the new method and was not retrained. The results were then compared with other quantization algorithms.

After quantization, classification accuracydecreased by only 1%, but the required storage volume was reduced by six times. Experiments show that this network does not require retraining due to the strong correlation between the original and quantized weights. This approach can help save resources when performing urgent tasks or working on mobile devices.

Read more:

A nuclear rocket engine is being built for flights to Mars. How is it dangerous?

Look at the 8 trillion pixel image of Mars.

Moon dust is deadly to humans. Satellite of the Earth is not suitable for colonization?

Abortion and science: what will happen to the children who will give birth.