Movies in the mind read by brain scanner

Using a brain scanner, scientists have successfully decoded and reconstructed the visual images experienced by volunteers viewing a sequence of Hollywood movies.

Writing in Current Biology, University of California, Berkeley researcher Jack Gallant and his colleagues placed three volunteers in an MRI scanner and asked them to watch a series of film trailer clips. As the scanner recorded their brains' responses to the movie information, a computer programme matched up how changes in the moving images were correlated with changes in small portions of the brain called voxels.

"This enabled us to create a 'dictionary' - a computational model - of how the brain was responding to shapes moving through the visual world," says Gallant.

Next the volunteers were asked to watch a separate set of movies that they hadn't seen before. This time, based upon what it had learned from the initial training sessions, the computer was asked to predict what the subjects had been watching and then, using 100 clips selected from over 18 million seconds of video footage from YouTube, build a reconstruction of what it thought they had seen. Though blurred, the results are breathtaking.

"What we've effectively solved in this paper - with a computational model - is a way to link dynamic changes in the world, using natural movies, with changes in brain activity," says Gallant.

This work overcomes a significant constraint of functional MRI, which deduces brain activity by picking up changes in blood flow and oxygen consumption. But this technique is relatively slow and limited to changes taking place on a second by second basis. The team have surmounted this by using the scanner to look at how the brain itself responds to the much faster changes happening in the movies. It's these changes that can then be used to reconstruct the visual experience. 

Commenting on the implications of the results, Gallant stresses that this is not a technique that could be used to eavesdrop illicitly on peoples' thoughts "because, at the very least, you need an MRI scanner!" But the technique they have developed could, he points out, be applied more generally to assist scientists studying almost any other thought process in the nervous system. It could also, in the future, be used to decode dreams and then ask if what we think we dreamed was what we really dreamed after all...