# The Wisdom of a Crowd of Brains: A Universal Brain Encoder

Source: [The Wisdom of a Crowd of Brains: A Universal Brain Encoder](<https://www.sota2.com/research/paper/68fa77871e209131cea73d81>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-27T05:00:13.467Z

Authors: Roman Beliy, Navve Wasserman, Amit Zalcher, Michal Irani

Published: 2024-06-18T01:17:07.000Z

## Abstract

Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such "Brain-Encoders" have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique "voxel-embedding" per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from the voxel-image cross-attention. We show the power of this approach to (i) combine data from multiple different subjects (a "Crowd of Brains") to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across subjects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) use the learned voxel-embeddings as a powerful tool to explore brain functionality (e.g., what is encoded where in the brain).

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [fMRI Brain Encoding on Natural Scenes Dataset (NSD) 7T (test)](<https://www.sota2.com/research/sota/fmri-brain-encoding-on-natural-scenes-dataset-nsd-7t-test.md>): fMRI Brain Encoding; Natural Scenes Dataset (NSD) 7T (test); Subject 1 Score: 69.

- [fMRI Encoding on Natural Scenes Dataset (NSD) subjects 1, 2, 5, 7 (avg) (test)](<https://www.sota2.com/research/sota/fmri-encoding-on-natural-scenes-dataset-nsd-subjects-1-2-5-7-avg-test.md>): fMRI encoding; Natural Scenes Dataset (NSD) subjects 1, 2, 5, 7 (avg) 7T (test); Pearson Correlation: 0.392.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2406.12179>)

[PDF](<https://arxiv.org/pdf/2406.12179.pdf>)