If someone were looking at a letter of the alphabet that was blocked from your view, would you be able to accurately guess what that letter was? Well, if you were at Radboud University Nijmegen in The Netherlands, you might not have to guess or call in a psychic. Scientists there have used an MRI scanner and a mathematical model to read observed letters, right out of test subjects’ brains.
The researchers started by virtually (not physically!) dividing each subject’s visual cortex up into a matrix of 1,200 cubic sections known as voxels – each voxel, in this particular study, measured 2 mm per side. Using the MRI, the scientists then noted how those voxels electrically responded to visual stimuli. When the subjects were subsequently shown hand-written letters, the overall pattern of all the responsive voxels (as viewed via the MRI, and processed using the mathematical model) could be used to roughly reconstruct an image of each letter.
The problem was, the images were still quite indistinct, each one appearing as a “fuzzy speckle pattern.” If someone were to look at one of those patterns without being told what it represented, they likely wouldn’t recognize it as any particular letter.
To sharpen the images up, the researchers provided the model with prior knowledge of what the letters looked like. In other words, instead of just seeing the pattern that corresponded to the letter "S" as nothing but one random arrangement of fuzzy speckles, the model now realized what form the pattern should take. The model then helped things along a little, by visually “pushing” each pattern towards the appearance of the letter that it represented.
While this may sound almost like cheating, one of the main purposes of the study was to get a feel for the ways in which prior knowledge is combined with sensory information. For instance, when we see the word “speckle,” we don’t actually stop and identify each and every letter in turn – instead, we briefly get a sense of the whole word, then our brain more or less says “Oh, I know what this is, it’s that ‘speckle’ word I’ve seen before.”
The research team, led by Dr. Marcel van Gerven, now plans on using a more powerful MRI that can scan up to 15,000 voxels at once. Instead of letters, however, they will be attempting to reconstruct images of faces viewed by test subjects. Scientists at the University of California, Berkeley have already conducted a similar experiment, in which images from movie trailers could be “read” from participants’ brains.
Source: Radboud University Nijmegen
