Targeted ads, friendships suggestions and research: sometimes we are amazed by the capabilities of computers and smartphones to "read our minds." In the future, however, this could be normal: we have to live with machines that are able to really scrutinizing in our minds, to understand what we are looking at (or imagining) and to respond accordingly.

A new, interesting research in this direction has been conducted in Japan, four researchers at Kyoto University have developed a neural network able to decipher the images we see "reading" our brain activity, and show them in turn.

One step closer. Not the first time you try this route, but the previous systems ran only narrow categories of images (for example, "Faces"). Their reconstructions were based on a set of semantic categories previously stored, while the subject was looking at some pictures: previous data with which each new input was compared. For this reason their "mind reading" ability was limited to a few basic shapes ( "bird", "cake", "person").

The images shown to the volunteers (in the first row) and, below, the reconstructions of the neural network. Each line corresponds to one of three subjects (click to enlarge). | Kyoto University

Reading complex. The new technique, called deep image reconstruction, provides algorithms able to interpret and reproduce complex images based on multiple levels of reading (for example form; color; bright contrasts). In other words it manages and processes the visual stimulus in a hierarchical manner, as does the human brain.

The study lasted 10 months and involved three people who were asked to observe a thousand images of three types: natural entities (such as animals or people), man-made geometric shapes, letters of the alphabet.

The brain activity of the participants was recorded with functional magnetic resonance imaging (fMRI) as they watched both images, both following, while only imagined. The activity data of the visual cortex were then fed data to the neural network, which has decoded them and used to develop a "proper interpretation" of the individual images, working precisely by hierarchical levels and subsequent interpretation.

Independent. The model was only taught from natural images - "men or animals" - but then was able to reconstruct letters and geometric shapes (as seen in the video below): this is the proof that he had learned the technique and he could use it from scratch, ie without comparison data.

For images only "think" by volunteers (and not watch) the process has succeeded only partially: the reconstructions kept some resemblance to the original, but they were more confused than the figures look (how confused are our mental images) .

The accuracy of the representations must improve: the images recreated by the model are recognizable but inaccurate. In future, however, an interface capable of using these techniques of "mind reading", and then "report" or act directly, could have numerous applications: whether useful or disturbing, depending on your degree of optimism.

From Focus