Deep behavioral phenotyping in functional MRI for cognitive mapping of the human brain

@ALuisaPinho@fediscience.org
@ALuisaPinho Seminar at the MNI Feindel Brain and Mind Lecture Series
Summary Overview of IBC Data-quality assessment Individual Functional Atlasing Encoding models for naturalistic stimuli Acknowledgments
Deep behavioral phenotyping in functional MRI for
cognitive mapping of the human brain
Ana Luı́sa Pinho, Ph.D.
BrainsCAN Postdoctoral Fellow
Western University, London Ontario, Canada
This work was developed in the Parietal Team at NeuroSpin/Inria-Saclay, Paris, France.
22nd of March, 2023

• Overview of the Individual Brain Charting (IBC) dataset
• Data-quality assessment of IBC
• Individual Functional Atlasing leveraging IBC First-Release
• Encoding analyses of naturalistic stimuli from IBC Third-Release using:
• unsupervised and data-driven Fast Shared Response Model (FastSRM)
• feature model based on Deep Convolutional Neural Networks (Deep CNN)
2/35

Overview of the IBC dataset
3/35

Background and motivations (1/2)
In cognitive neuroscience:
Brain systems
⇐⇒
Mental functions
4/35

Brain systems
⇐⇒
Mental functions
Experiments typically shall:
• tackle one psychological domain
4/35

Brain systems
⇐⇒
Mental functions
• be specific enough to accurately isolate brain processes
4/35

Brain systems
⇐⇒
Mental functions
• be specific enough to accurately isolate brain processes
⇓
Very hard to achieve!
Lack of generality.

Brain systems
⇐⇒
Mental functions
Task-fMRI experiments allow to:
• link brain systems to behavior
• map neural activity at mm-scale
4/35

Data-pooling analysis
• Meta-analysis:
pooling data derivatives
• Mega-analysis:
pooling raw data
Requisites for cognitive mapping
Minimize variability of Successful interpretation of
spatial location combined results
same processing no loss of info from sparse
routines peak-coord. representation
same experimental consistency of
settings cognitive annotations
low inter-subject variability sufficient multi-task data
5/35

• Meta-analysis:
• Mega-analysis:
pooling raw data
settings ( ) cognitive annotations
5/35

• Meta-analysis:
• Mega-analysis:
pooling raw data
Large-scale repositories:
• OpenNeuro
• NeuroVault
• EBRAINS
5/35

• Meta-analysis:
• Mega-analysis:
pooling raw data
• OpenNeuro
• NeuroVault
• EBRAINS
Individual analysis:
• Fedorenko, E. et al. (2011)
• Haxby, J. et al. (2011)
• Hanke, M. et al. (2014)
5/35

• Meta-analysis:
• Mega-analysis:
pooling raw data
settings ( )( ) cognitive annotations
• OpenNeuro
• NeuroVault
• EBRAINS
Large-scale datasets:
• HCP
• studyforrest
• CONNECT/Archi
5/35

• Meta-analysis:
• Mega-analysis:
pooling raw data
settings ( )( ) cognitive annotations
• OpenNeuro
• NeuroVault
• EBRAINS
Large-scale datasets:
• HCP
• studyforrest
• CONNECT/Archi
The IBC dataset meets all
together the requisites for
cognitive mapping.
5/35

The IBC dataset
• High spatial-resolution fMRI data (1.5mm)
6/35

The IBC dataset
• TR = 2s
6/35

The IBC dataset
• TR = 2s
• Task-wise dataset:
• Many tasks
6/35

The IBC dataset
• TR = 2s
• Many tasks
• Fixed cohort - 12 healthy adults
6/35

The IBC dataset
• TR = 2s
• Many tasks
• Fixed environment
NeuroSpin platform, CEA-Saclay, France
Siemens 3T Magnetom Prismafit
64-channel coil
6/35

The IBC dataset
• TR = 2s
• Many tasks
• Inclusion of other MRI modalities
64-channel coil
6/35

The IBC dataset
• TR = 2s
• Many tasks
• Inclusion of other MRI modalities
• Not a longitudinal study!
64-channel coil
6/35

Data-quality assessment of
the IBC First-Release
7/35

Tasks of the First Release
▶ ARCHI tasks
• Standard
• Spatial
• Social
• Emotional
▶ HCP tasks
• Emotion
• Gambling
• Motor
• Language
• Relational
• Social
• Working Memory
▶ RSVP Language
8/35

▶ ARCHI tasks
• Standard
• Spatial
• Social
• Emotional
▶ HCP tasks
• Emotion
• Gambling
• Motor
• Language
• Relational
• Social
• Working Memory
▶ RSVP Language
▶ Sensory processing:
• Retinotopy
• Tonotopy
• Somatotopy
▶ High-cognitive order:
• Calculation
• Language
• Social cognition
• Theory-of-mind
8/35

▶ ARCHI tasks
• Standard
• Spatial
• Social
• Emotional
▶ HCP tasks
• Emotion
• Gambling
• Motor
• Language
• Relational
• Social
• Working Memory
▶ RSVP Language
▶ Sensory processing:
• Retinotopy
• Tonotopy
• Somatotopy
▶ High-cognitive order:
• Calculation
• Language
• Social cognition
• Theory-of-mind
All contrasts: 119
Elementary contrasts: 59
8/35

IBC reproduces ARCHI and HCP
t
a
l
e
v
s
.
m
e
n
t
a
l
a
d
d
i
t
i
o
n
m
e
n
t
a
l
m
o
t
i
o
n
v
s
.
r
a
n
d
o
m
m
o
t
i
o
n
p
u
n
i
s
h
m
e
n
t
v
s
.
r
e
w
a
r
d
l
e
f
t
f
o
o
t
v
s
.
a
n
y
m
o
t
i
o
n
l
e
f
t
h
a
n
d
v
s
.
a
n
y
m
o
t
i
o
n
r
i
g
h
t
f
o
o
t
v
s
.
a
n
y
m
o
t
i
o
n
r
i
g
h
t
h
a
n
d
v
s
.
a
n
y
m
o
t
i
o
n
t
o
n
g
u
e
v
s
.
a
n
y
m
o
t
i
o
n
f
a
c
e
i
m
a
g
e
v
s
.
s
h
a
p
e
o
u
t
l
i
n
e
r
e
l
a
t
i
o
n
a
l
p
r
o
c
e
s
s
i
n
g
v
s
.
v
i
s
u
a
l
m
a
t
c
h
i
n
g
2
-
b
a
c
k
v
s
.
0
-
b
a
c
k
b
o
d
y
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
f
a
c
e
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
p
l
a
c
e
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
t
o
o
l
i
m
a
g
e
v
s
.
a
n
y
i
m
a
g
e
h
o
r
i
z
o
n
t
a
l
c
h
e
c
k
e
r
b
o
a
r
d
v
s
.
v
e
r
t
i
c
a
l
c
h
e
c
k
e
r
b
o
a
r
d
m
e
n
t
a
l
s
u
b
t
r
a
c
t
i
o
n
v
s
.
s
e
n
t
e
n
c
e
r
e
a
d
s
e
n
t
e
n
c
e
v
s
.
l
i
s
t
e
n
t
o
s
e
n
t
e
n
c
e
r
e
a
d
s
e
n
t
e
n
c
e
v
s
.
c
h
e
c
k
e
r
b
o
a
r
d
l
e
f
t
h
a
n
d
v
s
.
r
i
g
h
t
h
a
n
d
s
a
c
c
a
d
e
v
s
.
f
i
x
a
t
i
o
n
g
u
e
s
s
w
h
i
c
h
h
a
n
d
v
s
.
h
a
n
d
p
a
l
m
o
r
b
a
c
k
o
b
j
e
c
t
g
r
a
s
p
i
n
g
v
s
.
m
i
m
i
c
o
r
i
e
n
t
a
t
i
o
n
m
e
n
t
a
l
m
o
t
i
o
n
v
s
.
r
a
n
d
o
m
m
o
t
i
o
n
f
a
l
s
e
-
b
e
l
i
e
f
s
t
o
r
y
v
s
.
m
e
c
h
a
n
i
s
t
i
c
s
t
o
r
y
f
a
l
s
e
-
b
e
l
i
e
f
t
a
l
e
v
s
.
m
e
c
h
a
n
i
s
t
i
c
t
a
l
e
f
a
c
e
t
r
u
s
t
y
v
s
.
f
a
c
e
g
e
n
d
e
r
e
x
p
r
e
s
s
i
o
n
i
n
t
e
n
t
i
o
n
v
s
.
e
x
p
r
e
s
s
i
o
n
g
e
n
d
e
r
tale vs. mental addition
mental motion vs. random motion
punishment vs. reward
left foot vs. any motion
left hand vs. any motion
right foot vs. any motion
right hand vs. any motion
tongue vs. any motion
face image vs. shape outline
relational processing vs. visual matching
2-back vs. 0-back
body image vs. any image
face image vs. any image
place image vs. any image
tool image vs. any image
horizontal checkerboard vs. vertical checkerboard
mental subtraction vs. sentence
read sentence vs. listen to sentence
read sentence vs. checkerboard
left hand vs. right hand
saccade vs. fixation
guess which hand vs. hand palm or back
object grasping vs. mimic orientation
false-belief story vs. mechanistic story
false-belief tale vs. mechanistic tale
face trusty vs. face gender
expression intention vs. expression gender
HCP contrasts ARCHI contrasts
IBC
contrasts
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00 ARCHI batteries:
Pinel, P. et al. (2007)
HCP batteries:
Barch, D. M. et al. (2013)
n = 13
Pinho, A.L. et al. Hum Brain Mapp(2021) 9/35

Effect of subject and task on brain activity
Per-voxel one-way ANOVA qFDR < 0.05
x=10
L R
z=10 -28
-14
0
14
28
L R
y=-50
Subject effect
x=10
L R
z=10 -37
-19
0
19
37
L R
y=-50
Condition effect
x=-6
L R
z=3 -12
-5.9
0
5.9
12
L R
y=45
Phase encoding effect
Pinho, A.L. et al. SciData(2018)
10/35

Effect of subject and task on brain activity
Per-voxel one-way ANOVA qFDR < 0.05
x=10
L R
z=10 -28
-14
0
14
28
L R
y=-50
Subject effect
x=10
L R
z=10 -37
-19
0
19
37
L R
y=-50
Condition effect
x=-6
L R
z=3 -12
-5.9
0
5.9
12
L R
y=45
Phase encoding effect
IBC data is suitable for cognitive mapping and
individual-brain modeling!
10/35

Activation similarity fits task similarity
Similarity between
activation maps
of elementary contrasts
a
r
c
h
i
e
m
o
t
i
o
n
a
l
a
r
c
h
i
s
o
c
i
a
l
a
r
c
h
i
s
p
a
t
i
a
l
a
r
c
h
i
s
t
a
n
d
a
r
d
h
c
p
e
m
o
t
i
o
n
h
c
p
g
a
m
b
l
i
n
g
h
c
p
l
a
n
g
u
a
g
e
h
c
p
m
o
t
o
r
h
c
p
r
e
l
a
t
i
o
n
a
l
h
c
p
s
o
c
i
a
l
h
c
p
w
m
r
s
v
p
l
a
n
g
u
a
g
e
archi emotional
archi social
archi spatial
archi standard
hcp emotion
hcp gambling
hcp language
hcp motor
hcp relational
hcp social
hcp wm
rsvp language 0
1
Similarity between
cognitive description
a
r
c
h
i
e
m
o
t
i
o
n
a
l
a
r
c
h
i
s
o
c
i
a
l
a
r
c
h
i
s
p
a
t
i
a
l
a
r
c
h
i
s
t
a
n
d
a
r
d
h
c
p
e
m
o
t
i
o
n
h
c
p
g
a
m
b
l
i
n
g
h
c
p
l
a
n
g
u
a
g
e
h
c
p
m
o
t
o
r
h
c
p
r
e
l
a
t
i
o
n
a
l
h
c
p
s
o
c
i
a
l
h
c
p
w
m
r
s
v
p
l
a
n
g
u
a
g
e
archi emotional
archi social
archi spatial
archi standard
hcp emotion
hcp gambling
hcp language
hcp motor
hcp relational
hcp social
hcp wm
rsvp language 0
1
11/35

Similarity between
activation maps of
elementary contrasts
a
r
c
h
i
e
m
o
t
io
n
a
l
a
r
c
h
i
s
o
c
ia
l
a
r
c
h
i
s
p
a
t
ia
l
a
r
c
h
i
s
t
a
n
d
a
r
d
h
c
p
e
m
o
t
io
n
h
c
p
g
a
m
b
li
n
g
h
c
p
la
n
g
u
a
g
e
h
c
p
m
o
t
o
r
h
c
p
r
e
la
t
io
n
a
l
h
c
p
s
o
c
ia
l
h
c
p
w
m
r
s
v
p
la
n
g
u
a
g
e
archi emotional
archi social
archi spatial
archi standard
hcp emotion
hcp gambling
hcp language
hcp motor
hcp relational
hcp social
hcp wm
rsvp language 0
1
Similarity between
a
r
c
h
i
e
m
o
t
io
n
a
l
a
r
c
h
i
s
o
c
ia
l
a
r
c
h
i
s
p
a
t
ia
l
a
r
c
h
i
s
t
a
n
d
a
r
d
h
c
p
e
m
o
t
io
n
h
c
p
g
a
m
b
li
n
g
h
c
p
la
n
g
u
a
g
e
h
c
p
m
o
t
o
r
h
c
p
r
e
la
t
io
n
a
l
h
c
p
s
o
c
ia
l
h
c
p
w
m
r
s
v
p
la
n
g
u
a
g
e
archi emotional
archi social
archi spatial
archi standard
hcp emotion
hcp gambling
hcp language
hcp motor
hcp relational
hcp social
hcp wm
rsvp language 0
1
Second release:
• Mental Time Travel battery
Gauthier, B., & van Wassenhove, V. (2016a,b)
• Preference battery
Lebreton, M. et al. (2015)
• ToM + Pain Matrices battery
Dodell-Feder, D. et al. (2010)
Jacoby, N. et al. (2015)
Richardson, H. et al. (2018)
• Visual Short-Term Memory + Enumeration
tasks
Knops, A. et al. (2014)
• Self-Reference Effect task
Genon, S. et al. (2014)
• “Bang!” task
Campbell, K. L. et al. (2015)
First + Second releases:
All contrasts: 279
12/35

Similarity between
activation maps of
elementary contrasts
a
r
c
h
i
e
m
o
t
io
n
a
l
a
r
c
h
i
s
o
c
ia
l
a
r
c
h
i
s
p
a
t
ia
l
a
r
c
h
i
s
t
a
n
d
a
r
d
h
c
p
e
m
o
t
io
n
h
c
p
g
a
m
b
li
n
g
h
c
p
la
n
g
u
a
g
e
h
c
p
m
o
t
o
r
h
c
p
r
e
la
t
io
n
a
l
h
c
p
s
o
c
ia
l
h
c
p
w
m
r
s
v
p
la
n
g
u
a
g
e
archi emotional
archi social
archi spatial
archi standard
hcp emotion
hcp gambling
hcp language
hcp motor
hcp relational
hcp social
hcp wm
rsvp language 0
1
Similarity between
a
r
c
h
i
e
m
o
t
io
n
a
l
a
r
c
h
i
s
o
c
ia
l
a
r
c
h
i
s
p
a
t
ia
l
a
r
c
h
i
s
t
a
n
d
a
r
d
h
c
p
e
m
o
t
io
n
h
c
p
g
a
m
b
li
n
g
h
c
p
la
n
g
u
a
g
e
h
c
p
m
o
t
o
r
h
c
p
r
e
la
t
io
n
a
l
h
c
p
s
o
c
ia
l
h
c
p
w
m
r
s
v
p
la
n
g
u
a
g
e
archi emotional
archi social
archi spatial
archi standard
hcp emotion
hcp gambling
hcp language
hcp motor
hcp relational
hcp social
hcp wm
rsvp language 0
1
Second release:
• Mental Time Travel battery
Gauthier, B., & van Wassenhove, V. (2016a,b)
• Preference battery
Lebreton, M. et al. (2015)
• ToM + Pain Matrices battery
Dodell-Feder, D. et al. (2010)
Jacoby, N. et al. (2015)
Richardson, H. et al. (2018)
• Visual Short-Term Memory + Enumeration
tasks
Knops, A. et al. (2014)
• Self-Reference Effect task
Genon, S. et al. (2014)
• “Bang!” task
Campbell, K. L. et al. (2015)
First + Second releases:
All contrasts: 279
Spearman correlation
First Release: 0.21 (p ≤ 10−17)
Second Release: 0.21 (p ≤ 10−13)
First+Second Releases: 0.23 (p ≤ 10−72)
12/35

Individual Functional Atlasing
13/35

Variability of Functional Signatures
Pinho, A.L. et al. Hum Brain Mapp(2021) n = 13
Individual z-maps
14/35

Variability of Functional Signatures
0.00 0.25 0.50
read sentence vs. listen to sentence
read sentence vs. checkerboard
left hand vs. right hand
horizontal checkerboard vs. vertical checkerboard
mental subtraction vs. sentence
saccade vs. fixation
guess which hand vs. hand palm or back
object grasping vs. mimic orientation
false-belief story vs. mechanistic story
false-belief tale vs. mechanistic tale
expression intention vs. expression gender
face trusty vs. face gender
face image vs. shape outline
punishment vs. reward
0.00 0.25 0.50
tongue vs. any motion
right foot vs. any motion
left foot vs. any motion
right hand vs. any motion
left hand vs. any motion
tale vs. mental addition
relational processing vs. visual matching
tool image vs. any image
place image vs. any image
face image vs. any image
body image vs. any image
2-back vs. 0-back
read pseudowords vs. consonant strings
read words vs. consonant strings
read words vs. read pseudowords
read sentence vs. read jabberwocky
read sentence vs. read words
inter-subject correlation
intra-subject correlation
Intra- and inter- subject correlation of brain maps
14/35

Study 1
Dictionary of cognitive components

Decomposition of 51 contrasts
with dictionary learning
Individual topographies of
20 components (n = 13)
Each component gets the name
of the active condition from the
contrast with the highest value in
the dictionary.
Multi-subject, sparse dictionary learning:
min(Us )s=1...n,V∈C
n
X
s=1

∥Xs
− Us
V∥2
+ λ∥Us
∥1

,
with Xs
p×c , Us
p×k and Vk×c
• Functional correspondence: dictionary
of functional profiles (V) common to
all subjects
• Sparsity: ℓ1−norm penalty and
Us ≥ 0 , ∀s ∈ [n]
16/ 35

Components are consistently mapped across subjects.
16/ 35

0.25 0.30 0.35 0.40 0.45 0.50 0.55
Intra-subject
correlation
Inter-subject
correlation
Correlations of the dictionary components on split-half data
Variability of topographies linked to individual differences.
16/ 35

Study 2
Reconstruction of functional contrasts

Leave-p-out CV (p=3 subjects)
experiment to learn the shared
representations from contrasts of
eleven tasks. (n = 13)
Predict all contrasts from the
remaining task
18/ 35

Leave-p-out CV (p=3 subjects)
experiment to learn the shared
representations from contrasts of
eleven tasks. (n = 13)
Predict all contrasts from the
remaining task
Train a Ridge-regression model with individual
contrast maps i of tasks −j to predict task j on
individual contrast-maps s ̸= i:
b
ws,λ,j
= argminw∈Rc−1
X
i̸=s
∥Xi
j − Xi
−j w∥2
+ λ∥w∥2
Prediction output for one contrast of task j in
subject s:
b
Xs
j = Xs
−j b
ws,λ,j
.
Cross-validated R-squared for task j at location i:
R2
i (j) = 1 − means∈[n]
∥b
Xs
i,j − Xs
i,j ∥2
∥Xs
i,j ∥2
18/ 35

Pinho, A.L. et al. Hum Brain Mapp(2021)
n = 13
max R2
Most of the brain regions
are covered by the
predicted functional
signatures.
18/ 35

n = 13
Pinho, A.L. et al. Hum Brain Mapp(2021)
Ridge-Regression model
for the scrambled case:
b
ws,λ,j
= argminw∈Rc−1
X
i,k ̸= s
∥Xi
j −Xk
−j w∥2
+λ∥w∥2
Cross-validated R-squared:
R2
i (j) = 1 − means∈[n]
∥b
Xs
i,j − Xs′
i,j ∥2
∥Xs′
i,j ∥2
Permutations of subjects
decrease the proportion of
well-predicted voxels in all
tasks, showing that
topographies are driven by
subject-specific variability.
18/ 35

Study 3
Example: Functional mapping of the language network

Ex: Functional mapping of the language network
Goal: Cognitive profile of ROIs based on IBC language-related contrasts
Select ROIs / Select IBC contrasts
Individualize ROIs using dual-regression
and the left-out contrasts
R(s) = R pinv X(s)

X(s)
Voxelwise z-scores average for each
ROI at every selected contrast Pinho, A.L. et al. Hum Brain Mapp(2021)
20/ 35

Ex: Functional mapping of the language network
Linear SVC (upper triangle)
Dummy Classifier (lower triangle)
LOGOCV scheme
Prediction within pairs of ROIs
13 groups = 13 participants
Pinho, A.L. et al. Hum Brain Mapp(2021) 20/ 35

Conclusions
Functional atlasing using a large dataset in the task dimension
• Investigation of common functional profiles between tasks
• Common
functional profiles
Shared
behavioral responses
Mental
functions
21/ 35

Conclusions
Functional atlasing using a large dataset in the task dimension
• Investigation of common functional profiles between tasks
• Common
functional profiles
Shared
behavioral responses
Mental
functions
Individual brain modeling using data with higher spatial resolution
• generalize across subjects
• elicit variability between subjects
21/ 35

Encoding models for naturalistic stimuli
using the IBC Third-Release
• Clips task:
4 fMRI Sessions / 21 Runs
Nishimoto, S. et al. (2011)
• Raiders task:
2 fMRI Sessions: 13 Runs
Haxby, J. V. et al. (2011)
22/ 35

Analyzing naturalistic-stimuli fMRI data with FastSRM
• Shared Response Model by Chen et al. (2015)
• Fast Shared Response Model (FastSRM) by Richard et al. (2019):
https://hugorichard.github.io/FastSRM/
23/ 35

Benefits:
• Standard GLM applied to naturalistic stimuli leads to high-dimensional
controlled-design models.
23/ 35

Benefits:
• Unsupervised data-driven approach on fMRI timeseries where the design matrix and the
spatial maps are learnt jointly is more wieldy.
23/ 35

Benefits:
• Unsupervised data-driven approach on fMRI timeseries where the design matrix and the
spatial maps are learnt jointly is more wieldy.
• High-dimensional data (many voxels) require a decomposition method with scalability.
23/ 35

Description of FastSRM
Richard et al. (2019)
For all subjects and time frames, (Fast)SRM
can be formally defined as follows:
X = SW + E
X ∈ RG×nv
→ concatenation of G brain
images with v vertices for n=12 subjects
S ∈ RG×k
→ shared response:
concatenation of the weights across
time frames
W ∈ Rk×nv
→ concatenation of the k spatial
components with v vertices for the n
subjects
E → the additive noise
24/ 35

Double K-Fold CV for FastSRM for each task
CV scheme applied for each task with
K = 3 for 12 subjects and
K = 2 for R runs
Co-Smoothing described in Wu, A. et al.(2018) NeurIPS
Image credit to Thomas Chapalain
25/ 35

Group correlation of original vs. reconstructed data
26/ 35

Group-level activation between Raiders and Clips
q ⩽ 0.05
Top 10 regions of Glasser atlas w/ areas displaying
≥ 5% of significant voxels in both hemispheres
1 Auditory Association Cortex
2 Temporo-Parieto-Occipital Junction
3 Posterior Cingulate Cortex
4 Superior Parietal Cortex
5 Inferior Parietal Cortex
6 Early Auditory Cortex
7 Dorsal Stream Visual Cortex
8 Lateral Temporal Cortex
9 MT+Complex and Neighboring Visual Areas
10 Primary Visual Cortex (V1)
27/ 35

Remarks about FastSRM
• FastSRM combined with Co-Smoothing CV successfully extracts brain networks
directly from task-fMRI time-series.
• Useful to analyse high-dimensional paradigms, such as naturalistic stimuli.
• Limitations: No prior information about what properties of the stimulus may
drive activations.
28/ 35

Analyzing naturalistic-stimuli fMRI data with Deep CNN
29/ 35

Encoding Pipeline
• Extract features:
• Resize frames
• Temporal downsampling of t samples matching the TR
• CNN outputs d reduced intermediate representations
29/ 35

Encoding Pipeline
• Resize frames
• Representations are convolved with the hrf
29/ 35

Encoding Pipeline
• Resize frames
• Build a design matrix of size Nt × Nd
29/ 35

Encoding Pipeline
• Resize frames
• Build a design matrix of size Nt × Nd
• Fit the resulting GLM built from a CNN block with fMRI data
29/ 35

Cross-Validation Procedure
Image credit to Thomas Chapalain 30/ 35

The hierarchy of the Visual Cortex
Results for the Raiders task using CORnet-Z CNN
Block 1 Block 2 Block 3 Block 4
Hierarchical Convolution Layers of the Model
0.04
0.06
0.08
0.10
0.12
Mean
correlation
across
subjects
(a.u)
CORNet-Z Predictions of the Early Visual Cortex
Areas :
V1
V2
V3
V4
0.04
0.06
0.08
0.10
0.12
Mean
correlation
across
subjects
(a.u)
CORNet-Z Predictions of the Visual Dorsal Pathway
Areas :
V3A
V3B
V7
IP0
0.04
0.06
0.08
0.10
0.12
Mean
correlation
across
subjects
(a.u)
CORNet-Z Predictions of the Visual Ventral Pathway
Areas :
LO1
LO2
LO3
MT
MST
FST
PIT
Credit to Thomas Chapalain 31/ 35

Remarks about deep CNN encoding model
• Ecological stimuli combined with CNN provides insights about the hierarchy of the
visual system.
• Limitations:
• Correlations of areas within the Primary Visual Cortex due to long temporal
durations are not explored.
• Not all CNN models are easily interpretable.
32/ 35

Resources
OpenNeuro: ds002685
NeuroVault: id = 6618
EBRAINS: Individual Brain Charting (IBC) project
GitHub Analyses Repository: hbp-brain-charting/analysis pipeline
GitHub Protocols Repository: hbp-brain-charting/public protocols
Website: https://project.inria.fr/IBC/
FastSRM on GitHub: https://hugorichard.github.io/software/software-2/
33/ 35

Thanks!
Bertrand Thirion
The IBC volunteers!
34/ 35

Thank you for your attention.
35/ 35

Deep behavioral phenotyping in functional MRI for cognitive mapping of the human brain

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Deep behavioral phenotyping in functional MRI for cognitive mapping of the human brain

Semelhante a Deep behavioral phenotyping in functional MRI for cognitive mapping of the human brain (20)

Mais de Ana Luísa Pinho

Mais de Ana Luísa Pinho (18)

Último

Último (20)

Deep behavioral phenotyping in functional MRI for cognitive mapping of the human brain