Mathematicians reduced the size of the neural network by 6 times without retraining it

The structures of artificial neural networks and neurons in a living organism are based on the same principles. Nodes in

networks are interconnected, but some of themreceive the signal, and some transmit it, activating or suppressing the next element in the chain. Processing any signal, such as an image or sound, requires many network elements and connections coming from them. However, computer models have limited capacity and memory. To work with large volumes of data, experts have to come up with various ways to reduce capacity requirements, including so-called quantization. This helps reduce resource consumption, but requires retraining the system.

“Several years ago we conducted an effectiveand economical quantization of weights in the Hopfield network. This is an associative memory network with symmetrical connections between elements that are formed in accordance with Hebb's rule. During its operation, the activity of the network is reduced to a certain state of equilibrium, and when it is achieved, the problem is considered solved. The findings from this study were later applied to deep learning feedforward networks, which are very popular in image recognition today. Typically, these networks require retraining after quantization, but we found a way to avoid this."

Yakov Karandashev, Candidate of Sciences, Associate Professor at RUDN University.

The basic idea behind simplifying artificial neuralnetworks - this is the so-called quantization of weights, that is, reducing the number of bits for each weight. Quantization involves averaging the signal: for example, if applied to an image, all pixels representing different shades of the same color will become identical. Mathematically, this means that neural connections that are similar in certain parameters should have the same weight (or importance), expressed as a number.

A team of mathematicians from RUDN University carried out calculations andcreated formulas that effectively establish correlations between weights in a neural network before and after quantization. Based on them, scientists developed algorithms with which a trained neural network could classify images. In their experiment, the mathematicians used a text package of 50,000 photographs that could be divided into 1,000 groups. After training, the network was quantized using the new method and was not retrained. The results were then compared with other quantization algorithms.

After quantization, classification accuracydecreased by only 1%, but the required storage volume was reduced by six times. Experiments show that this network does not require retraining due to the strong correlation between the original and quantized weights. This approach can help save resources when performing urgent tasks or working on mobile devices.

Read more:

A nuclear rocket engine is being built for flights to Mars. How is it dangerous?

Look at the 8 trillion pixel image of Mars.

Moon dust is deadly to humans. Satellite of the Earth is not suitable for colonization?

Abortion and science: what will happen to the children who will give birth.