Accession Number:



Accurately Decoding Visual Information from fMRI Data Obtained in a Realistic Virtual Environment

Descriptive Note:

Journal Article

Corporate Author:


Report Date:


Pagination or Media Count:



Three-dimensional interactive virtual environments are a powerful tool for brain-imaging based cognitive neuroscience that are presently under-utilized. This paper presents machine-learning based methods for identifying brain states induced by realistic virtual environments with improved accuracy as well as the capability for mapping their spatial topography on the neocortex. Virtual environments provide the ability to study the brain under conditions closer to the environment in which humans evolved, and thus to probe deeper into the complexities of human cognition. As a test case, we designed a stimulus to reflect a military combat situation in the Middle East, motivated by the potential of using real-time functional magnetic resonance imaging fMRI in the treatment of post-traumatic stress disorder. Each subject experienced moving through the virtual town where they encountered 16 animated combatants at different locations, while fMRI data was collected. To analyze the data from what is, compared to most studies, more complex and less controlled stimuli, we employed statistical machine learning in the form of Multi-Voxel Pattern Analysis MVPA with special attention given to artificial Neural Networks NN. Extensions to NN that exploit the block structure of the stimulus were developed to improve the accuracy of the classification, achieving performances from 58 to 93 chance was 16.7 with 6 subjects. This demonstrates that MVPA can decode a complex cognitive state, viewing a number of characters, in a dynamic virtual environment. To better understand the source of this information in the brain, a novel form of sensitivity analysis was developed to use NN to quantify the degree to which each voxel contributed to classification. Compared with maps produced by general linear models and the searchlight approach, these sensitivity maps revealed a more diverse pattern of information relevant to the classification of cognitive state.

Subject Categories:

  • Anatomy and Physiology
  • Medicine and Medical Research
  • Biomedical Instrumentation and Bioengineering
  • Psychology
  • Operations Research
  • Test Facilities, Equipment and Methods

Distribution Statement: