Artificial neurons are already far more human-like than traditional computers, and now it turns out they might also need sleep to function at their peak. And it’s not just a matter of turning them off every now and then – a new study shows that the neurons benefit from exposure to slow-wave signals like those in a sleeping biological brain.
Neural networks are made up of artificial neurons, which all signal to each other like real neurons do in a real brain. Commonly used connections are reinforced over time, effectively allowing neural networks to learn on their own. Unlike the sequential processing of traditional computers, neural networks can process different streams of information in parallel, which makes them powerful tools for things like image and speech recognition.
Unfortunately, they may have imported a downside of organic brains, too – the need for sleep. Everyone knows from experience just how important sleep is for our health and wellbeing. Not only does it boost learning and help us solidify memories, but not getting enough of it has been linked to weight gain, depression, Alzheimer’s, and even death.
While neural networks probably don’t need to worry about weight gain, the neurons seem to become less stable after long periods of time working. In a new study from Los Alamos National Laboratory, researchers found that this occurred while the system was engaged in unsupervised dictionary training.
This exploratory technique is where the network identifies similarities between objects to try to categorize them, without being given clear examples to check them against. Unsurprisingly, this method was found to be more mentally taxing for the neural networks.
“The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” says Garrett Kenyon, co-author of the study. “The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”
To help the networks retain their focus, the researchers exposed them to different types of white noise signals. Gaussian noise – signals made up of a wide range of frequencies and amplitudes – worked the best, calming the neurons and restoring their stability. Interestingly, these are the same kinds of waves that ripple through human brains during the restorative slow-wave sleep phase.
“It was as though we were giving the neural networks the equivalent of a good night’s rest,” says Yijing Watkins, lead author of the study.
The researchers say that their next goal is to test this algorithm on Intel’s powerful neuromorphic chip, named Loihi. By allowing the chip to “sleep” occasionally, the team hopes that it will be better able to process visual information from a retina camera.
The research is due to be presented at the Women in Computer Vision Workshop on June 14.
Source: Los Alamos National Laboratory
