The article introduces BrainSCUBA, a new approach for understanding the functional organization of the visual cortex. It uses natural language descriptions to interpret the selectivity of neural populations, avoiding potential bias from hand-selected stimuli. BrainSCUBA builds upon a contrastive vision-language model and a pre-trained large language model to generate interpretable captions. The method has been validated through fine-grained voxel-level captioning across higher-order visual regions. It also enables exploratory investigations of the distribution of ‘person’ representations in the brain, showing fine-grained semantic selectivity in body-selective areas. The research suggests that BrainSCUBA can be a promising tool for understanding functional preferences in the brain.
Publication date: 6 Oct 2023
Project Page: https://arxiv.org/abs/2310.04420
Paper: https://arxiv.org/pdf/2310.04420
