Practical 4: The canonical Gaussian connective field model#
This is the notebook for the connective field practical of the workshop. This practical covers the canonical Gaussian connective field (CF) model, its different computational parts, and how to fit it to empirical data from a visual functional magnetic resonance imaging (fMRI) experiment.
We recommend going through the first practical on the canonical Gaussian population receptive field (pRF) model first to get familiar with the prfmodel package before starting with this practical.
The first part of this practical goes through the canonical Gaussian CF model step-by-step. The model input and output as well as every submodel is motivated and explained with interactive visualizations. This part only looks at model prediction but not at fitting the model to data. This is what the second part is about: Fitting the model to an empirical dataset obtained with a 7T fMRI scanner. This part covers data inspection and visualization on an inflated surface of the brain. It goes through two consecutive fitting stages (grid search, least-squares) and concludes with evaluating the model fit and interpreting its parameters.
Installation and Setup#
First, we need to install prfmodel from PyPI. We install the package with the TensorFlow backend (for more information on the different backends, see the Backends section in the package documentation).
Please make sure to use a GPU instance when installing prfmodel and running this notebook in Colab.
!pip install "prfmodel[tensorflow]==0.2.0"
We set the backend to TensorFlow and a default for printing.
import os
import pandas as pd
# Set keras backend to 'tensorflow' (this is normally the default)
os.environ["KERAS_BACKEND"] = "tensorflow"
# Print parameter DataFrames with three decimals
pd.set_option("display.precision", 3)
The canonical Gaussian CF model#
A connective field (CF) model is a topographically informed model of connectivity between neuronal populations. It tries to find the patch within a source region (the CF) that best explains the signal in a target population. You can think of this as an adaptation of traditional PRF models, where this source ROI replaces the stimulus space as the space in which the PRF/CF is defined. In this example, we use the canonical CF model adapted from Haak et al. (2013) where the CF is a 2D Gaussian in V1. Hence, we want to find the 2D Gaussian patch in V1 whose average Gaussian-weighted activity predicts the activity in a given target population best by varying its center location within V1 and size.
In this example, we map the recorded blood oxygenation level-dependent (BOLD) response from an fMRI experiment of the source populations covered by the CF to all other neuron populations in the brain (i.e., one population per vertex on the surface). The BOLD response is recorded while the subject is exposed to an experimental stimulus. In this case, this is a series of bar apertures that move across the visual field on a computer screen.
Input: Source region response and distance matrix#
The canonical CF model requires two inputs: The recorded activity of all neuron populations in the source ROI and a matrix that contains the distance between the source populations on the surface. Whereas a pRF model is defined in the feature space of the experimental stimulus (e.g., visual space), a CF model is defined on the distance matrix of the source region neuron populations. In the canonical formulation by Haak et al. (2013), the center of a CF must always be located at the known position of a source neuron population on the surface. The distance matrix then indicates how far away other source populations are from the CF center. The distance is used to compute the strength of the CF for all other source populations. Thus, the distance matrix of a CF model has a similar function as the stimulus grid in a pRF model. Because the distance is measured in 2D space, canonical CF models always use a 2D Gaussian to model CFs.
The recorded neural activity in the source ROI acts as the stimulus design matrix of the CF model, which multiplies the CF with the source region activity and sums the product over all source region populations.
In this practical, we use the BOLD response of a single subject who watched a moving bar stimulus in a 7 Tesla MRI scanner. The response was recorded with a repetition time (TR) of 1.5 seconds and is given for every vertex of a standardized cortical surface from the Human Connectome Project (HCP; Van Essen et al., 2012). We load the dataset with the tag 7t-retbar-visual for both hemispheres together with the standardized surface with the tag hcp-999999-surface. The surface comes with an atlas that assigns every vertex to a brain region, which we need to select the source ROI.
from prfmodel.examples import load_dataset
# Downloads on first use and caches in a user data directory (see prfmodel.examples.get_data_dir)
surface = load_dataset("hcp-999999-surface", surface_type="inflated")
dataset = load_dataset("7t-retbar-visual", hemisphere="both")
dataset.response.shape # One timecourse for every vertex (left hemisphere first)
The raw BOLD response has an arbitrary scale that differs between vertices. We convert it to percent signal change (PSC) relative to the mean of each timecourse, so that every timecourse fluctuates around zero.
import numpy as np
response_raw = dataset.response
response_psc = (response_raw / response_raw.mean(axis=1, keepdims=True) - 1.0) * 100.0
Exercise 1
Visualize the experimental stimulus of the dataset. The dataset does not contain a ready-made stimulus object, only the raw stimulus design matrix (hint: load it with scipy.io.loadmat(dataset.files["design"])["stim"] and plot single time frames with matplotlib.pyplot.imshow). What do the dimensions of the design matrix refer to? How many bar passes in which directions does the stimulus design contain?
Solution
import matplotlib.pyplot as plt
from scipy.io import loadmat
design_raw = loadmat(dataset.files["design"])["stim"]
print(design_raw.shape, design_raw.min(), design_raw.max())
# Plot every fifth time frame
fig, axes = plt.subplots(4, 6, figsize=(14, 6), layout="constrained")
for ax, frame in zip(axes.flat, range(0, design_raw.shape[-1], 5)):
ax.imshow(design_raw[..., frame], cmap="gray", vmin=0, vmax=255)
ax.set(title=f"Time frame {frame}", xticks=[], yticks=[])
The design matrix has the shape (281, 500, 120): The first two dimensions are the rows (y) and columns (x) of pixels on the screen (with an aspect ratio of 16:9), and the last dimension contains the 120 time frames (one per TR). Note that this order differs from the design matrices in prfmodel, where time frames are the first dimension. The values are pixel intensities from 0 (background) to 255 (bar).
The stimulus contains four bar passes, separated by blank periods:
Time frames 16–28: horizontal bar from top to bottom
Time frames 31–50: vertical bar from right to left
Time frames 68–87: vertical bar from left to right
Time frames 90–102: horizontal bar from bottom to top
# Your code goes here
We select V1 as the source ROI. The atlas is a multi-modal parcellation of the cortex (Glasser et al., 2016) in which V1 has the label 1. The response of the vertices in V1 is the source region response.
# V1 has the label 1 in the atlas
label_v1 = 1
# Concatenate the labels of both hemispheres (left first)
is_v1 = np.concatenate([surface.atlas["left"] == label_v1, surface.atlas["right"] == label_v1])
source_response = response_psc[is_v1]
source_response.shape # One timecourse for every V1 vertex
The distance matrix contains the distance between every pair of V1 vertices along the cortical surface. We compute it on the white matter surface of each hemisphere: The surface is a mesh of triangles, which we turn into a graph that connects neighboring vertices with edges as long as the distance between them. Dijkstra’s algorithm then finds the shortest path between every pair of V1 vertices along the edges of the graph.
import nibabel as nib
from scipy.sparse import csr_matrix
from scipy.sparse.csgraph import dijkstra
def calculate_distance_matrix(surface_path: str | os.PathLike, source_indices: np.ndarray) -> np.ndarray:
"""Calculate the shortest distance along a surface mesh between every pair of source vertices."""
coords, triangles = nib.load(surface_path).agg_data()
# Every triangle has three edges between its corners
i, j, k = triangles.T
edges = np.concatenate([np.column_stack((i, j)), np.column_stack((j, k)), np.column_stack((k, i))])
edge_lengths = np.linalg.norm(coords[edges[:, 0]] - coords[edges[:, 1]], axis=1)
num_vertices = coords.shape[0]
graph = csr_matrix((edge_lengths, (edges[:, 0], edges[:, 1])), shape=(num_vertices, num_vertices))
# Only keep the edges between source vertices, so that the shortest paths stay inside the source region
source_graph = graph[source_indices][:, source_indices]
return dijkstra(source_graph, directed=False)
We compute the distance matrix for each hemisphere separately. A CF cannot extend across hemispheres, so we set the distance between vertices in different hemispheres to infinity.
# Indices of the V1 vertices in each hemisphere
v1_indices_left = np.where(surface.atlas["left"] == label_v1)[0]
v1_indices_right = np.where(surface.atlas["right"] == label_v1)[0]
distance_matrix_left = calculate_distance_matrix(surface.files["wm_lh"], v1_indices_left)
distance_matrix_right = calculate_distance_matrix(surface.files["wm_rh"], v1_indices_right)
# Vertices in different hemispheres are infinitely far apart
num_left, num_right = len(v1_indices_left), len(v1_indices_right)
distance_matrix = np.block([
[distance_matrix_left, np.full((num_left, num_right), np.inf)],
[np.full((num_right, num_left), np.inf), distance_matrix_right],
])
distance_matrix.shape # One row and one column for every V1 vertex
We combine the source region response and the distance matrix into a CFStimulus object, which is the input of the CF model.
from prfmodel.stimuli import CFStimulus
stimulus = CFStimulus(distance_matrix=distance_matrix, source_response=source_response)
print(stimulus)
Building the model step by step#
The canonical Gaussian CF model gets from input (i.e., source region response and distance matrix) to output (i.e., predicted BOLD response) in several steps. Each step is performed by a submodel that does a dedicated computation:
The tuning profile describes which part of the source ROI predicts activity in the target population.
The stimulus encoding computes the overlap between the tuning profile and the recorded neural activity in the source ROI, which gives the predicted target neuron population activity.
The scaling adjusts the baseline and amplitude of the predicted BOLD response.
Importantly, since both the source and target populations are BOLD signals, there is no need for an impulse model to account for the shape of the HRF
Exercise 2
Imagine the source response was recorded with a method that does not measure the BOLD response (e.g., electrocorticography). How would the CF model need to be adjusted to account for the difference in measurement?
Solution
It depends on how the target response is measured:
Target also measured with ECoG: Source and target have the same temporal dynamics, so the model needs no adjustment (apart from the scaling, which already adapts the units).
Target measured with BOLD fMRI: The source response reflects neural activity almost instantly, but the target response is the slow and delayed BOLD response to that activity. The model would need an impulse response submodel, like a pRF model for BOLD data: The CF-weighted source response (i.e., the predicted target neural activity) is convolved with the hemodynamic impulse response before scaling. The source and target recordings would also need to be aligned in time and sampled at the same rate.
The opposite case (source measured with BOLD, target with ECoG) is much harder because we would have to undo the slow hemodynamic response of the source (deconvolution), which amplifies the noise.
We build up the model one submodel at a time and then combine all submodels into the full model. To explore how the parameters of each submodel affect its predictions, we use interactive figures.
The first hidden cell defines helper functions for the interactive figures. Some of them are shared by all practicals and are imported from the module plotting_helpers.py, which lies next to this notebook.
The figures show V1 as a flat map. Because the cortical surface is curved, we cannot draw V1 on a plane without distorting it. Instead, we arrange the V1 vertices of each hemisphere on a plane so that the distances between them match the distance matrix as closely as possible (with classical multidimensional scaling). On this map, distances are approximately the same as along the cortical surface, so a CF with size sigma covers about a circle with radius sigma. The horizontal axis of the map runs from the back (posterior) to the front (anterior) of V1 and the vertical axis from its lower (ventral) to its upper (dorsal) part.
Tuning profiles#
In prfmodel, the core of every CF model is called tuning profile, just like in pRF models. In the canonical Gaussian CF model, the tuning profile is an isotropic Gaussian density on the cortical surface of the source ROI. It is centered on a source vertex and decreases with the distance from this center vertex along the surface. It describes how strongly the activity of every source neuron population contributes to the activity of the target neuron population.
To illustrate this, we create an instance of a Gaussian CF tuning profile and evaluate it on the distance matrix. The Gaussian CF tuning profile only has two parameters, namely, the index of the source vertex at the CF center (center_index; i.e., a row of the distance matrix), and the size defined as the standard deviation (sigma, in mm along the surface).
from prfmodel.models.cf import GaussianCFTuning
cf_tuning = GaussianCFTuning()
tuning_params = pd.DataFrame(
{
"center_index": [448],
"sigma": [2.0],
},
)
# Predict the tuning profile
simulated_tuning_profile = cf_tuning(stimulus, tuning_params)
simulated_tuning_profile.shape # One value for every V1 vertex
The predicted tuning profile assigns a value to every V1 vertex. The figure below shows it on the map of V1 (left) and against the distance of every vertex to the CF center (right). The darkest vertices lie at the CF center (center_index), and the spread of the profile is determined by its size (sigma). The center_index slider moves the CF center through V1: The first vertices lie in the left hemisphere, the remaining ones in the right hemisphere. You can also type a vertex index into the box next to the slider.
Stimulus encoding#
In CF models, the tuning profile is encoded with the source region response through a dot-product operation, just like the stimulus design matrix in pRF models. At every time frame, this operation multiplies the response of every source vertex with its value in the tuning profile and sums the products over all source vertices. The result is the predicted activity of the target neuron population.
We illustrate this by creating an instance of the CF stimulus encoder.
from prfmodel.models.cf import CFStimulusEncoder
cf_encoder = CFStimulusEncoder()
# We first predict the tuning profile
simulated_tuning_profile = cf_tuning(stimulus, tuning_params)
# Then we stimulus-encode it
simulated_encoded_profile = cf_encoder(stimulus, simulated_tuning_profile, tuning_params)
simulated_encoded_profile.shape # One value per time frame
The figure below shows the source region response at a single time frame on the map of V1 together with the CF (left; the circle has a radius of sigma) and the predicted target neuron population activity (right). The time_frame slider selects the time frame shown on the left, which is marked with a dot on the right.
Scaling models#
The encoded response has a certain amplitude (i.e., scale) and a baseline. However, the observed BOLD responses of target neuron populations often differ in their amplitude and baseline from the source region response. This is mainly because target neuron populations receive input from multiple source regions, perform different computations, and are often less sensitive to the experimental stimulus. To account for these differences, a scaling submodel can modify the amplitude and baseline of the predicted response, just like in pRF models.
We create an instance of a scaling model and modify the encoded response.
from prfmodel.scaling import BaselineAmplitude
scaling_model = BaselineAmplitude()
scaling_params = pd.DataFrame({"baseline": [0.0], "amplitude": [1.0]})
scaled_response = scaling_model(simulated_encoded_profile, scaling_params)
scaled_response.shape
The figure below shows the encoded response before (blue) and after (orange) scaling. The dashed gray line shows the scaled response for the starting parameters.
The full canonical model#
Now that we have seen every submodel, we can combine them into the full CF model. In prfmodel, we call this a canonical model because it performs all necessary steps to get from input (i.e., source region response and distance matrix) to output (i.e., predicted BOLD response): tuning profile, stimulus encoding, and scaling. We create an instance of the canonical Gaussian CF model, which uses the submodels from above by default, and make a prediction with all parameters at once.
from prfmodel.models.cf import GaussianCFModel
cf_model = GaussianCFModel()
start_params = pd.DataFrame(
{
"center_index": [448],
"sigma": [2.0],
"baseline": [0.0],
"amplitude": [1.0],
},
)
# Make prediction with CF model
simulated_response = cf_model(stimulus, start_params)
simulated_response.shape # One predicted timecourse with one value per time frame
The figure below shows the CF on the map of V1 (left) and the predicted BOLD response of the full model (right). The dashed gray line shows the prediction for the starting parameters above. Use the sliders to see how all parameters together shape the predicted BOLD response.
Fitting the model to data#
Inspecting the data#
Before we fit the canonical Gaussian CF model to empirical data, we inspect the BOLD response timecourses in percent signal change that we computed at the beginning (response_psc). The dataset contains one timecourse for every vertex of the standardized surface in both hemispheres. We start with the source region response, which is the input of the CF model.
Exercise 3
Plot the recorded BOLD response in V1 (hint: use prfmodel.plotting.plot_response_heatmap). How does V1 respond to the experimental stimulus?
Solution
from prfmodel.plotting import plot_response_heatmap
plot_response_heatmap(
source_response,
vmin=-3,
vmax=3,
cmap="RdBu_r", # Diverging colormap: positive responses are red, negative responses blue
xlabel="Time frame (in TR)",
ylabel="V1 vertex index",
colorbar_label="BOLD response (in PSC)",
figsize=(6, 6),
);
The first 1,509 rows are the V1 vertices of the left hemisphere, the remaining rows those of the right hemisphere. In each hemisphere, a group of vertices responds strongly to each of the four bar passes. The responses form slanted bands: As the bar moves through the visual field, it activates one V1 vertex after the other because neighboring vertices in V1 represent neighboring positions in the visual field (retinotopy). Each response peaks a few time frames after the bar passed because of the delay of the BOLD response. Many other V1 vertices hardly respond or show a weak negative response during the bar passes. They likely represent parts of the visual field that the bar does not cover, for example, the periphery beyond the edges of the screen.
# Your code goes here
To visualize the response on the inflated surface, we create a helper function with make_surface_plot from plotting_helpers.py like in the first practical. The helper shows three views that focus on the visual cortex in the occipital lobe: the inner (medial) side of each hemisphere, where the calcarine sulcus lies, and both hemispheres from behind (posterior). In contrast to the first practical, the response is already given for every vertex of the surface, so we do not need to project it from a volume onto the surface.
from plotting_helpers import make_surface_plot
plot_surf_stat_map_helper = make_surface_plot(surface.mesh)
We plot the standard deviation of each timecourse on the surface to get an overview of the variability in the BOLD timecourses across the surface. Vertices with missing values (NaN) are not colored.
# Calculate standard deviation of each timecourse
response_sd = response_psc.std(axis=1)
plot_surf_stat_map_helper(response_sd, vmax=5.0, title="BOLD response standard deviation (in PSC)");
To orient ourselves on the surface, we also plot the location of the early visual areas V1 to V4. In the atlas, V2, V3, and V4 have the labels 4, 5, and 6. We will use these areas later to interpret the estimated CF parameters.
# Atlas labels of the early visual areas
visual_area_labels = {"V1": 1, "V2": 4, "V3": 5, "V4": 6}
# Concatenate the labels of both hemispheres (left first)
atlas_labels = np.concatenate([surface.atlas["left"], surface.atlas["right"]])
# Name of the visual area of each vertex; empty for vertices outside of V1 to V4
visual_area = np.full(atlas_labels.shape, "", dtype=object)
for name, label in visual_area_labels.items():
visual_area[atlas_labels == label] = name
# Number the areas from 1 (V1) to 4 (V4) for plotting
visual_area_number = np.full(atlas_labels.shape, np.nan)
for number, name in enumerate(visual_area_labels, start=1):
visual_area_number[visual_area == name] = number
fig, _ = plot_surf_stat_map_helper(
visual_area_number,
vmin=0.5,
vmax=4.5,
cmap=mpl.colors.ListedColormap(["#2a78d6", "#eb6834", "#1baf7a", "#8c5bd6"]),
title="Visual areas V1 to V4",
)
# Label the colors of the colorbar (the last axes of the figure) with the names of the areas
fig.axes[-1].set_yticks([1, 2, 3, 4], labels=list(visual_area_labels));
Selecting valid vertices#
Some vertices do not have a valid timecourse, that is, their timecourse contains missing values (NaN). We filter out all vertices with missing values.
response_is_valid = np.all(np.isfinite(response_psc), axis=1)
response_valid = response_psc[response_is_valid]
response_is_valid.mean() # Fraction of valid vertices
Grid search#
We fit the canonical Gaussian CF model that we created above (cf_model) to the response of every valid vertex. In contrast to the pRF model in the first practical, we do not need to adjust the model to the TR of the recording: The CF model has no impulse response submodel, and its prediction has the same time frames as the source region response.
We fit the model in two stages. In the first stage, we perform a “brute-force” grid search to find good estimates for the parameters of the CF tuning profile (i.e., center_index and sigma). For center_index, we test every V1 vertex. Because center_index is a whole number that selects a row of the distance matrix, the grid search is the only way to estimate it in prfmodel (it cannot be optimized with stochastic gradient descent). The values for sigma are log-spaced between 0.5 and 20 mm. A CF with a sigma of 0.5 mm practically only includes the center vertex because neighboring vertices are about 1.5 mm apart.
For baseline and amplitude, we only provide a single value so that they will stay constant across the entire grid.
grid_param_ranges = {
"center_index": np.arange(len(distance_matrix)), # Every V1 vertex
"sigma": np.geomspace(0.5, 20.0, 10), # Log-spaced (in mm)
"baseline": [0.0],
"amplitude": [1.0],
}
To run the grid search, we construct the prfmodel.fitters.GridFitter. The grid has 29,390 parameter combinations (2,939 V1 vertices times 10 values for sigma). We set batch_size=20 to let the fitter evaluate 20 parameter combinations at the same time which saves us a lot of memory. By default, the loss (i.e., the metric the fitter minimizes between model predictions and data) is the negative correlation coefficient, which ignores differences in baseline and amplitude between model predictions and observed data.
from prfmodel.fitters import GridFitter
# Create grid fitter object
grid_fitter = GridFitter(
model=cf_model,
stimulus=stimulus,
compile_step=True, # Setting 'compile_step=True' speeds up the fitting
)
# Run grid search
grid_history, grid_params = grid_fitter.fit(
data=response_valid,
parameter_values=grid_param_ranges,
batch_size=20,
)
We can print the parameters estimated by the grid search.
grid_params
Because the grid search can only return values that are in the grid, it is worth checking how the estimates are distributed across the grid. The figure below shows how many vertices have their CF center at each V1 vertex on the map of V1 (left) and the distribution of the CF sizes with prfmodel.plotting.plot_grid_parameter_distribution (right). CF sizes at the smallest or largest value of the grid are highlighted in red.
Least-squares#
In the grid search, we ignored the baseline and amplitude parameters. In prfmodel, we estimate them with least-squares by regressing the observed responses on the predicted CF model responses for each vertex. The intercept and slope of the least-squares regression become the baseline and amplitude parameter, respectively.
from prfmodel.fitters import LeastSquaresFitter
# Create least-squares fitter
ls_fitter = LeastSquaresFitter(
model=cf_model,
stimulus=stimulus,
)
# Run least squares fit
ls_history, ls_params = ls_fitter.fit(
data=response_valid,
parameters=grid_params,
slope_name="amplitude", # Names of parameters to be optimized with least squares
intercept_name="baseline",
batch_size=200,
)
We can print the parameter estimates from the grid search together with the least-squares parameter estimates.
ls_params
Evaluating the model fit#
Now that the CF model parameters are estimated, we can compare the model predictions against the observed responses. Because we want to make predictions for all vertices at once, we wrap our cf_model in the prfmodel.utils.batched function. The modifier changes the behavior of the model to make predictions for batches of vertices sequentially. This saves us a lot of memory at the expense of minimal runtime overhead.
from prfmodel.utils import batched
predict_batched = batched(cf_model)
# Make predictions with the estimated parameters
pred_response = np.asarray(predict_batched(stimulus, ls_params, batch_size=200))
We can quantify how well the predictions align with the observed timecourses using the R-squared metric, which we compute with prfmodel.utils.calculate_r_squared. This metric indicates the proportion of variance in the observed data explained by our model predictions.
from prfmodel.utils import calculate_r_squared
r_squared = calculate_r_squared(response_valid, pred_response) # One score per vertex
r_squared.shape
Exercise 4
Plot the distribution of R-squared values (hint: use prfmodel.plotting.plot_r_squared_hist). Why is there a large spike of R-squared values close to 1?
Solution
from prfmodel.plotting import plot_r_squared_hist
# A log scale also shows the bins with few vertices
plot_r_squared_hist(r_squared, value_range=(-0.1, 1.0), log=True);
is_v1_valid = is_v1[response_is_valid]
print("Fraction of V1 vertices with R-squared > 0.99:", np.mean(r_squared[is_v1_valid] > 0.99))
print("Fraction of other vertices with R-squared > 0.99:", np.mean(r_squared[~is_v1_valid] > 0.99))
print("Fraction of V1 vertices with the smallest CF size:", np.mean(np.isclose(ls_params["sigma"][is_v1_valid], 0.5)))
The spike consists of the V1 vertices. We fitted the model to all vertices, including those in V1, which are also the source vertices. For a V1 vertex, the best CF is centered on the vertex itself and has the smallest size in the grid, so that it practically only includes the vertex itself. The prediction is then the response of the vertex, and the fit is (almost) perfect. These R-squared scores are trivial and tell us nothing about the relationship between V1 and other areas, which is why we exclude V1 vertices when interpreting the parameters below.
# Your code goes here
To plot the R-squared scores on the surface, we need one value for every vertex, including the invalid ones. We define a helper function that fills the values of the valid vertices into an array with one value per vertex and sets the invalid vertices to NaN.
def fill_valid_vertices(values: np.ndarray) -> np.ndarray:
"""Fill values for the valid vertices into an array with one value per vertex."""
values_full = np.full((response_psc.shape[0],), fill_value=np.nan)
values_full[response_is_valid] = values
return values_full
plot_surf_stat_map_helper(fill_valid_vertices(r_squared), vmin=0.0, vmax=1.0, title="Variance explained (R-squared)");
Vertices far outside the visual cortex reach R-squared scores of about 0.3, although they do not respond to the visual stimulus. The grid search picks the best of 29,390 predictions for every vertex, so some prediction always matches the noise in a timecourse to some degree. Such vertices often have the smallest CF size in the grid because a CF that only includes a single (noisy) V1 vertex can follow the noise more closely than a larger CF that averages over many V1 vertices.
We also compare the predicted against the observed timecourses for a few vertices with prfmodel.plotting.plot_observed_predicted. We pick the vertex with the median R-squared score in each of V2, V3, and V4.
from prfmodel.plotting import plot_observed_predicted
visual_area_valid = visual_area[response_is_valid]
fig, axes = plt.subplots(3, 1, figsize=(10, 7.5), sharex=True, layout="constrained")
for ax, area in zip(axes, ["V2", "V3", "V4"]):
# Index of the valid vertex with the median R-squared score in the area
area_indices = np.flatnonzero(visual_area_valid == area)
vertex_index = area_indices[np.argsort(r_squared[area_indices])[len(area_indices) // 2]]
plot_observed_predicted(response_valid[vertex_index], pred_response[vertex_index], ax=ax)
ax.set(xlabel="", ylabel="BOLD response (in PSC)", title=f"{area}: R-squared = {r_squared[vertex_index]:.2f}")
axes[-1].set_xlabel("Time frame (in TR)");
Exercise 5
Why does the fit for the vertex in V2 look so different from the fits in V3 and V4? Think about what the response of V1 to the experimental stimulus looks like and what a response in V2-V4 should look like.
Solution
V1 vertices that represent the stimulated part of the visual field respond with a clear peak to each bar pass (see Exercise 3). A vertex in V2–V4 that represents the same part of the visual field should respond in the same way, and the CF model can predict it by averaging the responses of the corresponding V1 vertices. This is what we see for the vertices in V3 and V4: The observed response has four clear peaks, and the prediction follows them.
The observed response of the vertex in V2 does not show any peaks for the bar passes. Instead, it slowly drifts upwards over the run and fluctuates from time frame to time frame. This vertex likely represents a part of the visual field that the bar does not cover, so its response is not driven by the stimulus. Nevertheless, the CF model reaches an R-squared of 0.65: The grid search finds a CF in V1 whose response contains a similar drift and similar fluctuations, for example, because of noise that is shared between brain regions (e.g., from breathing or head motion). The CF model can map any signal that source and target have in common, not only the response to the stimulus. A high R-squared therefore does not guarantee that the CF reflects how the stimulus-driven response is relayed from V1 to the target vertex.
Interpreting the model parameters#
To analyze and interpret the CF parameters, we zoom in on vertices for which we can trust the CF estimates. We select vertices that fulfill three criteria:
The R-squared is above 0.5. This threshold is somewhat arbitrary, but it excludes most vertices whose R-squared scores only reflect noise (see above).
The vertex lies outside V1. V1 vertices are perfectly predicted by their own response (see Exercise 4).
The CF size is below the largest
sigmavalue in the grid. An estimate at the ceiling of the grid suggests that the best fitting CF size lies outside the grid.
center_index = ls_params["center_index"].to_numpy().astype(int)
sigma = ls_params["sigma"].to_numpy()
r_squared_threshold = 0.5
selection_criteria = pd.DataFrame({
f"r_squared > {r_squared_threshold}": r_squared > r_squared_threshold,
"outside V1": ~is_v1[response_is_valid],
# Least squares does not change sigma, so it is always one of the grid values
"sigma below grid ceiling": ~np.isclose(sigma, np.max(grid_param_ranges["sigma"])),
})
is_selected = selection_criteria.all(axis=1).to_numpy()
def fill_selected(values: np.ndarray) -> np.ndarray:
"""Fill values of the selected vertices into an array with one value per vertex; all others are NaN."""
return fill_valid_vertices(np.where(is_selected, values, np.nan))
# Fraction of valid vertices that fulfill each criterion and all criteria together
selection_criteria.assign(all_criteria=is_selected).mean()
First, we look at the CF size indicated by sigma and plot it on the surface.
plot_surf_stat_map_helper(fill_selected(sigma), vmin=0.0, vmax=10.0, title="CF size (sigma, in mm)");
Second, to compare the CF sizes between the visual areas, we plot the average sigma of the selected vertices in V2, V3, and V4 with prfmodel.plotting.plot_parameter_by_roi. The error bars show the standard error of the mean.
from prfmodel.plotting import plot_parameter_by_roi
plot_parameter_by_roi(
ls_params[is_selected],
"sigma",
visual_area_valid[is_selected],
order=["V2", "V3", "V4"],
error="sem",
xlabel="Visual area",
ylabel="CF size (sigma, in mm)",
figsize=(4, 4),
);
Exercise 6
Interpret the CF size surface plot and target ROI comparison. What does it mean when higher-order visual areas have larger CFs in V1?
Solution
On the surface, the CF size increases with the distance from V1: Vertices in V2 next to V1 have small CFs, and the CFs become larger in V3, V4, and the areas beyond. The ROI comparison shows the same: The average CF size increases from about 2 mm in V2 over about 3 mm in V3 to about 6 mm in V4.
A larger CF means that the response of a target vertex is best predicted by averaging the responses of a larger patch of V1. Neighboring V1 vertices represent neighboring positions in the visual field, so a larger patch of V1 covers a larger part of the visual field. Thus, neuron populations in higher-order visual areas integrate information from a larger part of V1, which is in line with their larger pRFs. This convergence along the visual hierarchy is what Haak et al. (2013) also reported.
Exercise 7
CF models weight the response of all neuron populations in the source region. When these responses substantially differ in scale, this can distort the CF estimates. Instead of expressing the source response in PSC, we can also express it in z-scores. Does this change the CF estimates for this example dataset? Does z-scoring source responses change the interpretation of the CF model parameters in general?
response_z = (response_raw - np.mean(response_raw, axis=1, keepdims=True)) / np.std(response_raw, axis=1, keepdims=True)
response_z.shape
Solution
We create a stimulus with the z-scored source response and repeat the grid search and least-squares. To save time, we only fit the vertices in V2, V3, and V4. The grid search uses the correlation as loss, which does not depend on the scale of the target response, so we can keep the target response in PSC.
stimulus_z = CFStimulus(distance_matrix=distance_matrix, source_response=response_z[is_v1])
is_target = np.isin(visual_area_valid, ["V2", "V3", "V4"])
_, grid_params_z = GridFitter(model=cf_model, stimulus=stimulus_z, compile_step=True).fit(
data=response_valid[is_target],
parameter_values=grid_param_ranges,
batch_size=20,
)
_, ls_params_z = LeastSquaresFitter(model=cf_model, stimulus=stimulus_z).fit(
data=response_valid[is_target],
parameters=grid_params_z,
slope_name="amplitude",
intercept_name="baseline",
batch_size=200,
)
r_squared_z = calculate_r_squared(
response_valid[is_target], np.asarray(predict_batched(stimulus_z, ls_params_z, batch_size=200)),
)
We compare the estimates for the vertices that we selected above.
is_compared = is_selected[is_target]
estimates_psc = ls_params[is_target][is_compared].reset_index(drop=True)
estimates_z = ls_params_z[is_compared].reset_index(drop=True)
# Distance between the CF centers estimated with both source responses (in mm)
center_distance = distance_matrix[
estimates_psc["center_index"].astype(int), estimates_z["center_index"].astype(int)
]
print("Fraction with the same CF center:", np.mean(center_distance == 0.0))
print("Fraction with CF centers less than 3 mm apart:", np.mean(center_distance < 3.0))
print("Fraction with the same CF size:", np.mean(np.isclose(estimates_psc["sigma"], estimates_z["sigma"])))
print("Mean R-squared (PSC):", r_squared[is_target][is_compared].mean())
print("Mean R-squared (z-scores):", r_squared_z[is_compared].mean())
# Mean CF size per visual area
pd.DataFrame({
"PSC": estimates_psc["sigma"],
"z-scores": estimates_z["sigma"],
"area": visual_area_valid[is_target][is_compared],
}).groupby("area").mean()
For this dataset, z-scoring hardly changes the estimates: About two thirds of the vertices get the same CF center and more than 80% get a CF center less than 3 mm away, most vertices get the same CF size, and the mean CF size per area and the R-squared are almost identical.
In general, z-scoring changes how the source vertices are weighted. In PSC, vertices with large responses (e.g., close to large veins) contribute more to the weighted sum than vertices with small responses. In z-scores, every source vertex contributes with the same variance, which also gives more weight to vertices that mostly contain noise. The CF center and size keep their meaning as the location and extent of the CF on the surface of V1. However, the predicted response is no longer a weighted average of the actual responses, and amplitude and baseline change their units (from PSC per PSC to PSC per standard deviation).
# Your code goes here
Exercise 8
Are CF models symmetric? Would the CF for V1 \(\rightarrow\) V2 be the same as the CF for V2 \(\rightarrow\) V1?
Solution
No. The CF model predicts the target response as a weighted sum (i.e., a spatial average) of source responses. A V2 population has a larger pRF than a V1 population, and averaging the responses of neighboring V1 populations with small, overlapping pRFs mimics a larger pRF. Thus, V1 \(\rightarrow\) V2 works well, and the CF size tells us how much of V1 a V2 population integrates over.
In the other direction, averaging V2 responses can only make the effective pRF larger, never smaller. The best prediction of a V1 population is then a CF that is as small as possible (a single V2 vertex at the corresponding position), and the fit is worse because the V2 response is blurrier than the V1 response. The CF centers would still roughly correspond to each other because both areas are retinotopically organized, but the CF sizes would differ and mean something different. Note also that the CF is a statistical mapping between responses, which does not imply that information flows only in the modeled direction: There are both feedforward and feedback connections between V1 and V2.
Exercise 9
Fit a CF model with V2 as the source ROI and V3 and V4 as the target ROIs. Do you see a similar relationship between CF size and visual hierarchy?
Solution
We compute the distance matrix and the source response for the V2 vertices in the same way as for V1. All V2 vertices have valid responses.
label_v2 = visual_area_labels["V2"]
is_v2 = atlas_labels == label_v2
v2_indices_left = np.where(surface.atlas["left"] == label_v2)[0]
v2_indices_right = np.where(surface.atlas["right"] == label_v2)[0]
distance_matrix_v2_left = calculate_distance_matrix(surface.files["wm_lh"], v2_indices_left)
distance_matrix_v2_right = calculate_distance_matrix(surface.files["wm_rh"], v2_indices_right)
num_left_v2, num_right_v2 = len(v2_indices_left), len(v2_indices_right)
distance_matrix_v2 = np.block([
[distance_matrix_v2_left, np.full((num_left_v2, num_right_v2), np.inf)],
[np.full((num_right_v2, num_left_v2), np.inf), distance_matrix_v2_right],
])
stimulus_v2 = CFStimulus(distance_matrix=distance_matrix_v2, source_response=response_psc[is_v2])
We fit the model to the valid vertices in V3 and V4 with a grid that contains every V2 vertex as a CF center.
is_target_v2 = np.isin(visual_area_valid, ["V3", "V4"])
grid_param_ranges_v2 = grid_param_ranges | {"center_index": np.arange(len(distance_matrix_v2))}
_, grid_params_v2 = GridFitter(model=cf_model, stimulus=stimulus_v2, compile_step=True).fit(
data=response_valid[is_target_v2],
parameter_values=grid_param_ranges_v2,
batch_size=20,
)
_, ls_params_v2 = LeastSquaresFitter(model=cf_model, stimulus=stimulus_v2).fit(
data=response_valid[is_target_v2],
parameters=grid_params_v2,
slope_name="amplitude",
intercept_name="baseline",
batch_size=200,
)
r_squared_v2 = calculate_r_squared(
response_valid[is_target_v2], np.asarray(predict_batched(stimulus_v2, ls_params_v2, batch_size=200)),
)
# Same selection criteria as with V1 as the source ROI
is_selected_v2 = (r_squared_v2 > r_squared_threshold) & ~np.isclose(
ls_params_v2["sigma"], np.max(grid_param_ranges_v2["sigma"]),
)
plot_parameter_by_roi(
ls_params_v2[is_selected_v2],
"sigma",
visual_area_valid[is_target_v2][is_selected_v2],
order=["V3", "V4"],
error="sem",
xlabel="Visual area",
ylabel="CF size in V2 (sigma, in mm)",
figsize=(4, 4),
);
Yes, the CFs of V4 vertices in V2 (about 2.7 mm) are larger than those of V3 vertices (about 2.2 mm). However, the difference between V3 and V4 is much smaller than with V1 as the source ROI (about 3 mm vs. 6 mm). V2 populations already integrate over a larger part of the visual field than V1 populations, so fewer source vertices are needed to build the larger pRFs of V3 and V4. Note that CF sizes in V1 and V2 cannot be compared directly because the same distance on the surface of V1 and V2 corresponds to different distances in the visual field (cortical magnification differs between areas).
# Your code goes here
Exercise 10 (advanced)
In their study, Haak et al. (2013) correlate the pRF eccentricity and polar angle of source and target neuron populations in the CF model. Fit a canonical Gaussian 2D pRF model to V1-V4 and correlate the pRF eccentricity and polar angle associated with the CF center of every target population in V2-V4 with its actual pRF eccentricity and polar angle. Do you see similar correlations as reported by Haak et al. (2013)? Hint: The dataset does not contain a ready-made pRF stimulus, so you first need to create a prfmodel.stimuli.PRFStimulus from the raw design matrix (see Exercise 1).
Solution
First, we create a pRF stimulus from the raw design matrix. We binarize it, downsample it to a grid of 56 x 100 cells to make the fit faster, and move the time frames to the first dimension. The dataset does not tell us how large the screen was in degrees of visual angle, so we assume a height of 10 degrees. Scaling the visual field by a constant factor scales all pRF centers by the same factor, so the correlations below do not depend on this assumption.
from prfmodel.fitters import grid_values_2d_prf
from prfmodel.impulse import DerivativeTwoGammaImpulse
from prfmodel.models.prf import Gaussian2DPRFModel
from prfmodel.stimuli import PRFStimulus
from prfmodel.utils import calculate_eccentricity, calculate_polar_angle
from scipy.io import loadmat
design_raw = loadmat(dataset.files["design"])["stim"] # Shape (y, x, time frames), values from 0 to 255
# Binarize, crop to 280 x 500 pixels, and average blocks of 5 x 5 pixels
design_binary = (design_raw[:280] > 127).astype("float32")
design_small = design_binary.reshape(56, 5, 100, 5, -1).mean(axis=(1, 3))
# Move the time frames to the first dimension and flip the rows, so that the first row is the bottom of the screen
design_prf = np.ascontiguousarray(np.moveaxis(design_small, -1, 0)[:, ::-1, :])
# Assumed height of the screen in degrees of visual angle
screen_height = 10.0
cell_size = screen_height / design_prf.shape[1]
y_coords = (np.arange(design_prf.shape[1]) - (design_prf.shape[1] - 1) / 2) * cell_size
x_coords = (np.arange(design_prf.shape[2]) - (design_prf.shape[2] - 1) / 2) * cell_size
grid = np.stack(np.meshgrid(y_coords, x_coords, indexing="ij"), axis=-1) # Shape (y, x, 2)
tr = 1.5
prf_stimulus = PRFStimulus(design=design_prf, grid=grid, dimension_labels=["y", "x"], frame_duration=tr)
Second, we fit the canonical Gaussian 2D pRF model to the valid vertices in V1 to V4 with grid search and least-squares, like in the first practical.
is_v1_to_v4 = visual_area_valid != ""
prf_model = Gaussian2DPRFModel(impulse_model=DerivativeTwoGammaImpulse(resolution=tr))
prf_grid_values = grid_values_2d_prf(prf_stimulus, num_mu=21, mu_extent=1.0, num_sigma=10) | {
"weight_deriv": [-0.5],
"baseline": [0.0],
"amplitude": [1.0],
}
_, prf_grid_params = GridFitter(model=prf_model, stimulus=prf_stimulus, compile_step=True).fit(
data=response_valid[is_v1_to_v4],
parameter_values=prf_grid_values,
batch_size=20,
)
_, prf_params = LeastSquaresFitter(model=prf_model, stimulus=prf_stimulus).fit(
data=response_valid[is_v1_to_v4],
parameters=prf_grid_params,
slope_name="amplitude",
intercept_name="baseline",
batch_size=200,
)
prf_r_squared = calculate_r_squared(
response_valid[is_v1_to_v4], np.asarray(batched(prf_model)(prf_stimulus, prf_params, batch_size=200)),
)
Third, we look up the pRF of the V1 vertex at the CF center of every target vertex. Each hemisphere represents the opposite half of the visual field, so the polar angles of the right hemisphere lie around \(\pm \pi\), where they wrap around. To avoid this, we mirror the left half of the visual field onto the right half before computing the polar angle. It then runs from \(-\pi/2\) (lower vertical meridian) to \(\pi/2\) (upper vertical meridian). We only compare target vertices with a good CF fit for which the pRF model also fits well to the vertex and its CF center.
def fill_v1_to_v4(values: np.ndarray) -> np.ndarray:
"""Fill values for the vertices in V1 to V4 into an array with one value per valid vertex; NaN for all others."""
values_valid = np.full(len(response_valid), np.nan)
values_valid[is_v1_to_v4] = values
return values_valid
mu_x_prf = prf_params["mu_x"].to_numpy()
mu_y_prf = prf_params["mu_y"].to_numpy()
prf_eccentricity = fill_v1_to_v4(calculate_eccentricity(mu_x_prf, mu_y_prf))
prf_polar_angle = fill_v1_to_v4(calculate_polar_angle(np.abs(mu_x_prf), mu_y_prf)) # Mirrored visual field
prf_r_squared_valid = fill_v1_to_v4(prf_r_squared)
# Row of response_valid of the V1 vertex at the CF center of every valid vertex (all V1 vertices are valid)
v1_rows = np.flatnonzero(is_v1[response_is_valid])
cf_center_rows = v1_rows[center_index]
prf_r_squared_threshold = 0.3
is_compared = (
is_selected
& np.isin(visual_area_valid, ["V2", "V3", "V4"])
& (prf_r_squared_valid > prf_r_squared_threshold)
& (prf_r_squared_valid[cf_center_rows] > prf_r_squared_threshold)
)
fig, axes = plt.subplots(1, 2, figsize=(10, 4.5), layout="constrained")
for ax, values, label in zip(
axes, [prf_eccentricity, prf_polar_angle], ["Eccentricity (in degrees)", "Polar angle (in radians)"],
):
for area in ["V2", "V3", "V4"]:
is_area = is_compared & (visual_area_valid == area)
correlation = np.corrcoef(values[cf_center_rows][is_area], values[is_area])[0, 1]
ax.scatter(
values[cf_center_rows][is_area], values[is_area], s=3, alpha=0.3, label=f"{area} (r = {correlation:.2f})",
)
ax.legend(frameon=False, markerscale=4)
ax.set(xlabel=f"{label} of the CF center in V1", ylabel=f"{label} of the target vertex")
Yes. The pRF position associated with the CF center agrees well with the pRF position of the target vertex itself. The correlations are very high for eccentricity (about 0.98 in V2 and V3 and 0.91 in V4) and somewhat lower for polar angle (about 0.9 in V2, 0.8 in V3, and 0.7 in V4). Like Haak et al. (2013), we find that the CF model recovers the retinotopic organization of the target areas from the V1 responses alone, without any knowledge of the stimulus. The correlations decrease along the visual hierarchy because the larger CFs and pRFs of higher-order areas are less precisely localized.
# Your code goes here
References#
Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C. F., Jenkinson, M., Smith, S. M., & Van Essen, D. C. (2016). A multi-modal parcellation of human cerebral cortex. Nature, 536(7615), 171-178. https://doi.org/10.1038/nature18933
Haak, K. V., Winawer, J., Harvey, B. M., Renken, R., Dumoulin, S. O., Wandell, B. A., & Cornelissen, F. W. (2013). Connective field modeling. NeuroImage, 66, 376-384. https://doi.org/10.1016/j.neuroimage.2012.10.037
Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., Della Penna, S., Feinberg, D., Glasser, M. F., Harel, N., Heath, A. C., Larson-Prior, L., Marcus, D., Michalareas, G., Moeller, S., … WU-Minn HCP Consortium. (2012). The Human Connectome Project: A data acquisition perspective. NeuroImage, 62(4), 2222-2231. https://doi.org/10.1016/j.neuroimage.2012.02.018