Scientists reconstruct visual stimuli by reading brain activity

In the 1983 film Brainstorm, Christopher Walken played a scientist who was able to record movies of people's mental experiences, then play them back into the minds of other people. Pretty far-fetched, right? Well, maybe not. Utilizing functional Magnetic Resonance Imaging (fMRI) and computer models, researchers at the University of California, Berkeley, have been able to visually reconstruct the brain activity of human subjects watching movie trailers - in other words, they could see what the people's brains were seeing.

The study involved placing three subjects in an MRI scanner, and having them watch two sets of Hollywood movie trailers while in it. The fMRI was used to measure blood flow through their brains' visual cortex, as they were watching the trailers. A computer used this data to virtually divide their brains into small three-dimensional cubes called voxels. Computer models of each voxel were then created, incorporating information about how that real-life section of the brain responded to different types of visual stimuli. In this way, the computer was able to match up specific voxel activity with specific visual patterns from the trailers - it acted as a Rosetta Stone, of sorts.

The resulting movie reconstruction algorithm was then fed 18 million seconds of random YouTube videos, which it matched up with what should be the corresponding voxel activity. For each image in the trailers, it then chose 100 images from the YouTube videos, whose voxel activity most closely resembled that of the trailer image. These 100 images were combined into one blurry composite image, that resembled the one image from the trailer. When strung together, those composite images presented a somewhat trippy yet recognizable facsimile of the complete trailer.

So far, the system can only reconstruct movie trailers that subjects have already viewed. As the UC Berkeley technology is developed, however, it is hoped that it could be used visualize what is happening in the minds of stroke victims, coma patients, and other people not able to adequately communicate. It could also be used to improve human-computer interfaces, such as those that allow handicapped individuals to control devices using their thoughts.

The video below shows parts of the original trailers, with the reconstructions playing alongside. Below it is a video that displays images from the trailers, with some of the YouTube images that were used to create their composite equivalents.

The research was published yesterday in the journal Current Biology.