In the past, "mind reading" systems have been able to guess what single-digit number a person might be thinking of, but deeper thoughts have been beyond the technology's reach. Now, a team from Carnegie Mellon University (CMU) has developed a way to accurately read more complex concepts from a brain scan, and even piece together entire sentences.
Even the most basic sentence is loaded with more information than you might realize: each word represents a new concept, and their placement and relationship to each other can drastically change the meaning of the whole. The CMU team found that the "building blocks" the mind uses to construct thoughts are made up of concepts, rather than being based on words themselves. That suggests the brain processes concepts in a universal way, regardless of a person's language and culture.
"One of the big advances of the human brain was the ability to combine individual concepts into complex thoughts, to think not just of 'bananas,' but 'I like to eat bananas in evening with my friends'," says Marcel Just, lead researcher on the study. "We have finally developed a way to see thoughts of that complexity in the fMRI signal. The discovery of this correspondence between thoughts and brain activation patterns tells us what the thoughts are built of."
The study tested how the brain codes complex thoughts, and how an fMRI scanner, with a little help from machine learning algorithms, can decode them. The researchers put together 240 "complex events," which are sentences like "The witness shouted during the trial." These events were made up of 42 different building blocks, or meaningful components like person, setting, size, social interaction and physical action.
Each of these different kinds of information are processed in different parts of the brain, so CMU's system can pick out the general category of what's on a person's mind. To test its prowess, the researchers had seven participants read the sentences, recording the brain activation patterns that went along with them. After training the algorithm on 239 of the sentences and the matching scans, it was then able to put together the last sentence based solely on the brain data.
The team ran that test 240 times, systematically leaving out each of the sentences in turn, and found that the algorithm was able to predict the missing sentence from a brain activation pattern with 87 percent accuracy. Going the other way, the researchers could feed the program a sentence and it would spit out an accurate brain activation pattern.
"Our method overcomes the unfortunate property of fMRI to smear together the signals emanating from brain events that occur close together in time, like the reading of two successive words in a sentence," says Just. "This advance makes it possible for the first time to decode thoughts containing several concepts. That's what most human thoughts are composed of. A next step might be to decode the general type of topic a person is thinking about, such as geology or skateboarding. We are on the way to making a map of all the types of knowledge in the brain."
The research was published in the journal Human Brain Mapping.
Source: Carnegie Mellon University
