Practical 3: Model comparison and selection#
This is the notebook for the third practical of the workshop. This practical covers how to compare and select between different population receptive field (pRF) models on empirical data from visual functional magnetic resonance imaging (fMRI) experiments.
We recommend first going through the first practical and second practical because this practical builds on package-specific basics and general concepts that are covered there.
The typical goal in computational modelling is to build a model that explains and predicts a target pattern in empirical data. That is, the model should not only predict the pattern in one set of observations but generalize to all observations that stem from the same data-generating process. The second practical introduced cross-validation to assess how well a pRF model can generalize its predictions from training data to new observations. Cross-validation is used to compare models that vary in complexity, for example, in the number of free parameters, and select the model that shows the best generalization. More complex models can fit the training data more precisely, meaning that they are in theory better able to capture the target pattern. That is, they are less likely to underfit the training data. However, this means that they are also more prone to fitting aspects of the data that are not part of the data-generating process for the target pattern. This is called overfitting the training data.
Cross-validation is a means to assess whether a pRF model is over- or underfitting. When underfitting, it will show relatively low prediction performance on the training and test data compared to other models (or a noise ceiling). This is also called bias. Conversely, when overfitting, the model will show high prediction performance on the training but relatively low prediction performance on the test data. Moreover, the prediction performance will be more variable on different test datasets the more a model overfits the training data. This test variance occurs because the model test prediction performance depends on variable characteristics of the data that do not belong to the data-generating process of the target pattern. The tension between bias (underfitting) and variance (overfitting) is called bias-variance trade-off. Given fixed training and test data, the complexity of a model typically navigates this trade-off and the goal is to find the model complexity that sits on the optimal point between under- and overfitting.
Different pRF models make different assumptions about how a neuron population responds to visual stimulation. The canonical Gaussian 2D pRF model from the first practical assumes that the response grows linearly with the part of the pRF that is covered by the stimulus. Other models add, for example, a suppressive surround around the pRF center or a compression of the response when a larger part of the pRF is stimulated. Each model is a hypothesis about the response, and comparing the models on empirical data tells us which hypothesis describes the data best.
In this practical, we compare four models: The canonical Gaussian 2D pRF model, the Compressive Spatial Summation (CSS) model, the Difference of Gaussians (DoG) model, and the Divisive Normalization (DN) model. The latter three models can mimic the Gaussian model and have more free parameters, so in principle they always fit the training data at least as well as the Gaussian model. Whether they also predict new data better is what we test with cross-validation. We also estimate the noise ceiling of each voxel, which tells us how well any model could possibly predict the data given the noise in it. Finally, the best model can differ between brain regions, so we select a model for each voxel separately.
Contents#
Installation and Setup#
First, we need to install prfmodel from PyPI. We install the package with the TensorFlow backend (for more information on the different backends, see the Backends section in the package documentation).
Please make sure to use a GPU instance when installing prfmodel and running this notebook in Colab.
!pip install "prfmodel[tensorflow]==0.1.0"
We set the backend to TensorFlow and a default for printing.
import os
import numpy as np
import pandas as pd
# Set keras backend to 'tensorflow' (this is normally the default)
os.environ["KERAS_BACKEND"] = "tensorflow"
# Print parameter DataFrames with three decimals
pd.set_option("display.precision", 3)
Loading the dataset#
In this practical, we use the Arrow-of-Time (AOT) dataset from the first practical. The BOLD response was recorded with a 7 Tesla MRI scanner while the subject watched a moving bar stimulus. The dataset contains a single timecourse per voxel, which is the median over several runs of the experiment. We load the dataset together with the inflated surface of the subject.
from prfmodel.examples import load_dataset
# Downloads on first use and caches in a user data directory (see prfmodel.examples.get_data_dir)
dataset = load_dataset("7t-aot-visual", surface_type="inflated")
stimulus = dataset.stimulus
Like in the first practical, we crop the design matrix and grid to the rectangular screen on which the stimulus was shown, that is, to the cells that the bar covers in at least one time frame.
from prfmodel.stimuli import PRFStimulus
# Cells that the bar covers in at least one time frame lie on the screen
is_on_screen = stimulus.design.max(axis=0) > 0.0
# The screen is a rectangle, so we keep the rows (y) and columns (x) of the design matrix between its edges
rows_on_screen = np.flatnonzero(is_on_screen.any(axis=1))
cols_on_screen = np.flatnonzero(is_on_screen.any(axis=0))
rows = slice(rows_on_screen[0], rows_on_screen[-1] + 1)
cols = slice(cols_on_screen[0], cols_on_screen[-1] + 1)
stimulus = PRFStimulus(
design=stimulus.design[:, rows, cols],
grid=stimulus.grid[rows, cols],
frame_duration=stimulus.frame_duration,
dimension_labels=stimulus.dimension_labels,
)
print(stimulus)
# The grid stores y in grid[..., 0] and x in grid[..., 1]
(y_min, x_min), (y_max, x_max) = stimulus.grid.min(axis=(0, 1)), stimulus.grid.max(axis=(0, 1))
print(f"Screen: x from {x_min:.2f} to {x_max:.2f} degrees, y from {y_min:.2f} to {y_max:.2f} degrees")
Like in the first practical, we subtract the baseline from the raw BOLD response and only keep voxels with a response that varies over time.
response_no_baseline = dataset.response - 100.0
# Voxels with a response that varies over time
response_is_valid = response_no_baseline.std(axis=1) > 0.0
response_valid = response_no_baseline[response_is_valid]
def fill_valid_voxels(values: np.ndarray) -> np.ndarray:
"""Fill values for the valid voxels into an array with one value per voxel in the mask."""
values_full = np.full((response_no_baseline.shape[0],), fill_value=np.nan)
values_full[response_is_valid] = values
return values_full
response_valid.shape
We reuse the helper functions from the first practical to project values for each voxel onto the surface (project_to_surface) and to plot them on the inflated surface (plot_surf_stat_map_helper). We create them with the functions from the module plotting_helpers.py, which lies next to this notebook.
from plotting_helpers import make_surface_plot, make_surface_projection
project_to_surface = make_surface_projection(dataset)
plot_surf_stat_map_helper = make_surface_plot(dataset.mesh)
Splitting the run into two halves#
For cross-validation, we need data to fit the models to (training data) and data to evaluate their predictions on (test data). The AOT dataset only contains a single (median) run, so we split the run into two halves. The stimulus contains eight bar sweeps that are separated by blank periods. The first and the second half of the run contain the same four bar sweeps, but in a different order:
Bar sweep |
First half (frames) |
Second half (frames) |
|---|---|---|
Horizontal bar, top to bottom |
30–49 |
287–307 |
Vertical bar, left to right |
53–79 |
258–284 |
Horizontal bar, bottom to top |
101–120 |
216–236 |
Vertical bar, right to left |
124–150 |
186–212 |
Between the two halves lies a long blank period from frame 151 to frame 185. We split the run in the middle of this blank period.
import matplotlib.pyplot as plt
num_frames = stimulus.design.shape[0]
# A frame shows the bar if any pixel of the design matrix is non-zero
is_bar_visible = stimulus.design.reshape(num_frames, -1).max(axis=1) > 0
split = 168 # Middle of the blank period between frames 151 and 185
fig, ax = plt.subplots(figsize=(10, 2), layout="constrained")
ax.step(np.arange(num_frames), is_bar_visible, where="mid")
ax.axvline(split, color="black", linestyle="--", label="Split")
ax.set(xlabel="Time frame", yticks=[0, 1], yticklabels=["Blank", "Bar"])
ax.legend(loc="upper right");
Exercise 1
Using what you learned in the first practical, create a video of the stimulus (hint: use prfmodel.plotting.animate_2d_prf_stimulus). Can you confirm the bar sweeps in the table above?
Solution
from IPython.display import HTML
from prfmodel.plotting import animate_2d_prf_stimulus
ani = animate_2d_prf_stimulus(stimulus, interval=25) # Pause 25 ms between time frames
HTML(ani.to_html5_video())
Yes. The video shows the four sweeps of the first half (top to bottom, left to right, bottom to top, right to left), then the long blank period, and then the same four sweeps in the reverse order (right to left, bottom to top, left to right, top to bottom). To check the time frames of a sweep, you can also plot single frames of the design matrix, for example, plt.imshow(stimulus.design[30], origin="lower").
# Your code goes here
We create a separate stimulus for each half that only contains the frames of that half. The stimulus grid stays the same.
first_half = slice(None, split) # Frames 0 to 167
second_half = slice(split, None) # Frames 168 to 339
stimulus_first = PRFStimulus(
design=stimulus.design[first_half],
grid=stimulus.grid,
frame_duration=stimulus.frame_duration,
dimension_labels=["y", "x"],
)
stimulus_second = PRFStimulus(
design=stimulus.design[second_half],
grid=stimulus.grid,
frame_duration=stimulus.frame_duration,
dimension_labels=["y", "x"],
)
stimulus_first.design.shape, stimulus_second.design.shape
Exercise 2
Why do we split the run in the middle of the blank period and not directly after the last bar sweep of the first half (i.e., at frame 151)? Think about how long the BOLD response to a stimulus lasts (hint: look at the impulse response in the first practical).
Solution
The BOLD response to a stimulus peaks about 5 seconds later and, including the undershoot, lasts for 20 seconds or longer. If we split the run at frame 151, the response to the last sweep of the first half would continue into the second half. The model would not be able to predict this part of the second half because the stimulus of the second half does not contain the sweep that caused it: The model assumes that there is no response before the first frame of a stimulus. At the same time, the end of the response would be missing from the first half. Splitting in the middle of the blank period gives the response 17 frames (about 15 seconds) to return close to baseline before the second half starts, and the second half starts with another 18 blank frames before its first sweep.
Both halves were recorded in the same session and the noise in both halves comes from the same runs. However, the two halves contain different time frames with different noise, and the bar sweeps appear in a different order and with blank periods of different lengths in between. Thus, the model must predict the response to the same bar sweeps in a new sequence, which is similar to the test with a new stimulus in the second practical. In the following, we use the first half for training and the second half for testing.
Selecting voxels with a high noise ceiling#
Not all voxels in the gray matter mask respond to the stimulus, and among those that do, the noise level varies a lot. Comparing models on voxels that are dominated by noise is not informative because no model can predict noise. Therefore, we focus on voxels with a reliable response to the stimulus.
How reliable a response is can be estimated without any model when the same stimulus is shown twice: The correlation between the two responses tells us how well the response can be reproduced. Suppose that each response is the sum of the same stimulus-driven signal and independent noise with the same variance. Then, this correlation is the fraction of the variance of one response that is caused by the stimulus. Even a perfect model cannot explain more variance than that, so the correlation estimates the noise ceiling: The highest R-squared that a model can reach on one of the responses.
It is important that we select voxels with a criterion that does not depend on any of the candidate models. If we selected voxels by the R-squared of the Gaussian model, we would favor voxels that the Gaussian model already describes well and could miss voxels where other models make a difference.
The AOT dataset does not contain repeated runs, but both halves of the run contain the same four bar sweeps. We cut out the frames of each sweep from both halves and correlate them. To account for the delay of the BOLD response, we shift each sweep by 5 frames (4.5 seconds). Some sweeps last one frame longer in the second half, so we use the shorter duration for both halves.
# Frame at which each bar sweep starts in the first and second half, and the number of frames we use for it
sweeps = pd.DataFrame(
{
"onset_first": [30, 53, 101, 124],
"onset_second": [287, 258, 216, 186],
"num_frames": [20, 27, 20, 27],
},
index=[
"Horizontal bar, top to bottom",
"Vertical bar, left to right",
"Horizontal bar, bottom to top",
"Vertical bar, right to left",
],
)
lag = 5 # Delay of the BOLD response in frames
def cut_sweeps(onsets: pd.Series) -> np.ndarray:
"""Concatenate the responses of all valid voxels to the bar sweeps, shifted by the delay of the BOLD response."""
return np.concatenate(
[response_valid[:, onset + lag:onset + lag + n] for onset, n in zip(onsets, sweeps["num_frames"])],
axis=1,
)
sweep_response_first = cut_sweeps(sweeps["onset_first"])
sweep_response_second = cut_sweeps(sweeps["onset_second"])
sweep_response_first.shape
For each voxel, we compute the correlation between the responses to the sweeps in both halves.
def correlate_rows(x: np.ndarray, y: np.ndarray) -> np.ndarray:
"""Pearson correlation between each row of x and the same row of y."""
x = x - x.mean(axis=1, keepdims=True)
y = y - y.mean(axis=1, keepdims=True)
return (x * y).sum(axis=1) / np.sqrt((x**2).sum(axis=1) * (y**2).sum(axis=1))
# Voxels with a constant response during the sweeps get a NaN correlation
with np.errstate(invalid="ignore"):
noise_ceiling = correlate_rows(sweep_response_first, sweep_response_second)
noise_ceiling.shape
We look at the distribution of the noise ceiling across voxels with prfmodel.plotting.plot_r_squared_hist and plot the noise ceiling on the surface.
from prfmodel.plotting import plot_r_squared_hist
plot_r_squared_hist(noise_ceiling[np.isfinite(noise_ceiling)], value_range=(-1.0, 1.0), log=True, xlabel="Noise ceiling");
plot_surf_stat_map_helper(
project_to_surface(fill_valid_voxels(noise_ceiling)), vmin=0.0, vmax=1.0, title="Noise ceiling",
);
Exercise 3
Where on the surface do the voxels with a high noise ceiling lie? Compare the map with the map of the response standard deviation from the first practical (hint: compute the standard deviation of response_valid and project it onto the surface). Are there voxels with a large standard deviation but a low noise ceiling? What could cause their variability?
Solution
response_sd = response_valid.std(axis=1)
plot_surf_stat_map_helper(
project_to_surface(fill_valid_voxels(response_sd)), vmin=0.0, vmax=5.0, title="BOLD response standard deviation",
)
fig, axes = plt.subplots(1, 2, figsize=(15, 4), gridspec_kw={"width_ratios": [1, 2]}, layout="constrained")
# Noise ceiling against standard deviation for all valid voxels
axes[0].hexbin(response_sd, noise_ceiling, xscale="log", bins="log", gridsize=50, cmap="inferno")
axes[0].set(xlabel="Standard deviation", ylabel="Noise ceiling")
# Example of a voxel with a large standard deviation (top 1%) but a low noise ceiling
is_variable_unreliable = (response_sd > np.quantile(response_sd, 0.99)) & (noise_ceiling < 0.2)
voxel_index = np.flatnonzero(is_variable_unreliable)[0]
axes[1].plot(response_valid[voxel_index])
axes[1].axvline(split, color="black", linestyle="--")
axes[1].set(
xlabel="Time frame",
ylabel="BOLD response",
title=f"Voxel {voxel_index}: noise ceiling = {noise_ceiling[voxel_index]:.2f}, standard deviation = {response_sd[voxel_index]:.1f}",
)
plt.show()
print(f"Voxels with a large standard deviation but a low noise ceiling: {is_variable_unreliable.sum()}")
# Where do these voxels lie?
plot_surf_stat_map_helper(
project_to_surface(fill_valid_voxels(is_variable_unreliable.astype(float))),
vmin=0.0,
vmax=0.3,
title="Large standard deviation, low noise ceiling",
);
Voxels with a high noise ceiling lie in the occipital lobe: around the occipital pole and the calcarine sulcus on the medial side, and on the posterior surface, that is, in the early and higher visual areas. This is roughly where the standard deviation is elevated, too. However, the two maps differ: The standard deviation is also large in regions far from the visual cortex, for example, next to the front end of the medial wall and along the lower edge of the surface, where the noise ceiling is close to zero. Of the 1% of voxels with the largest standard deviation, most (859 of 1,055) have a noise ceiling below 0.2.
The timecourses of these voxels fluctuate strongly, but the fluctuations are unrelated to the stimulus and do not repeat between the two halves. They lie in regions that are known for strong noise in fMRI: close to large veins and sinuses, close to air-filled cavities (which distort the magnetic field), or close to cerebrospinal fluid. Their variability comes from breathing, the heartbeat, head motion, and these artifacts, not from neuronal responses to the stimulus. This is why the standard deviation alone is not a good criterion for selecting voxels, and the noise ceiling, which measures how well the response repeats, is better.
# Your code goes here
Exercise 4
The two halves do not show exactly the same stimulus: The same sweep follows a different sweep or blank period in each half, and some sweeps last one frame longer in one of the halves. Why does this make our estimate of the noise ceiling lower than the noise ceiling we would get from two runs with the identical stimulus?
Solution
The noise ceiling assumes that the two responses we correlate contain the same stimulus-driven signal and only differ in noise. This is not quite true here:
The BOLD response is slow, so the response during a sweep still contains the tail of the response to the preceding sweep or blank period. This tail differs between the halves because the order of the sweeps differs.
When a sweep lasts one frame longer in one half, the bar moves at a slightly different speed or starts at a slightly different time. The responses of the two halves are then slightly shifted against each other.
We use the same fixed delay of 5 frames for all voxels and sweeps, which is only an approximation of the delay of the BOLD response.
These differences are stimulus-driven, but in the correlation they look like noise and reduce it. Thus, our noise ceiling is a conservative (too low) estimate of how well a model can predict the data. Two runs with the identical stimulus would only differ in the noise and would give a higher correlation.
We select voxels with a noise ceiling above 0.7. Note that this threshold is somewhat arbitrary. We split the responses of the selected voxels into the two halves.
noise_ceiling_threshold = 0.7
# NaN values are never above the threshold
is_selected = noise_ceiling > noise_ceiling_threshold
response_first = response_valid[is_selected][:, first_half]
response_second = response_valid[is_selected][:, second_half]
noise_ceiling_selected = noise_ceiling[is_selected]
def fill_selected_voxels(values: np.ndarray) -> np.ndarray:
"""Fill values for the selected voxels into an array with one value per voxel in the mask, NaN for all others."""
values_valid = np.full((response_valid.shape[0],), fill_value=np.nan)
values_valid[is_selected] = values
return fill_valid_voxels(values_valid)
response_first.shape, response_second.shape
Exercise 5
How many voxels would you select with a threshold of 0.5 or 0.9? What are the advantages and disadvantages of a lower and a higher threshold for comparing models?
Solution
for threshold in [0.5, 0.7, 0.9]:
print(f"Noise ceiling above {threshold}: {(noise_ceiling > threshold).sum()} voxels")
With a threshold of 0.5, we would select 5,961 voxels, with 0.7 (our choice) 3,103 voxels, and with 0.9 only 406 voxels.
Lower threshold: We compare the models on more voxels, which makes the comparison more representative of the visual cortex as a whole (e.g., it includes more voxels from higher visual areas, where the responses are weaker). However, the additional voxels are noisier: Noise makes the differences between models smaller and their ranking less reliable, and each fit takes longer.
Higher threshold: The remaining voxels have very reliable responses, so differences between the models are more likely to be real. However, they are few and mostly come from regions with the strongest responses (e.g., early visual cortex), so the result may not generalize to other regions. The threshold also depends on our noise ceiling estimate, which is conservative (Exercise 4).
# Your code goes here
Candidate models#
We compare four pRF models that make different assumptions about how the neuron population responds to the stimulus:
Model |
Class in |
Assumption |
Additional parameters |
Free parameters |
|---|---|---|---|---|
Gaussian (Dumoulin & Wandell, 2008) |
|
The response grows linearly with the stimulated part of the pRF |
– |
5 |
CSS (Kay et al., 2013) |
|
The response grows less than linearly (compressive) with the stimulated part of the pRF |
|
6 |
DoG (Zuiderbaan et al., 2012) |
|
Stimulation around the pRF center suppresses the response (suppressive surround) |
|
7 |
DN (Aqil et al., 2021) |
|
The response to the pRF center is divided by the response to a larger normalization pool, which produces both suppression and compression |
|
8 |
The free parameters are the parameters that we estimate for each voxel (we fix weight_deriv for all models). The models are ordered by their number of free parameters. They can (approximately) mimic each other: The CSS model with n equal to 1 and the DoG model without a surround (amplitude_surround equal to 0) are equivalent to the Gaussian model, and the DN model behaves like the DoG model when normalization is weak and produces compression like the CSS model when it is strong (Aqil et al., 2021).
Fitting models with many parameters is difficult because the loss function has many local minima. We therefore fit the models in sequence and use the estimates of simpler models as starting values for more complex ones. prfmodel provides helper functions for this, for example, prfmodel.models.prf.init_css_from_gaussian. After initializing a model, we estimate its amplitude and baseline parameters with least-squares and then fine-tune all free parameters with stochastic gradient descent (SGD).
For a fair comparison, we use the same SGD settings for all models, including the Gaussian model. Otherwise, a model could fit better only because it was optimized for longer. We use a larger learning rate than the default of the Adam optimizer (0.001) so that 500 steps are enough to reach a stable fit.
from keras import ops
from keras.optimizers import Adam
from prfmodel.fitters import GridFitter, LeastSquaresFitter, SGDFitter, grid_values_2d_prf
from prfmodel.fitters.adapter import Adapter, ParameterTransform
from prfmodel.impulse import DerivativeTwoGammaImpulse
# Same SGD settings for all models
num_steps = 500
learning_rate = 0.01
# Define repetition time (TR)
tr = 0.9
# Create custom impulse model that is shared by all models
impulse_model = DerivativeTwoGammaImpulse(resolution=tr)
To see how the additional parameters of the CSS, DoG, and DN models change their predictions, we use interactive figures like in the first practical. The code that creates the figures is hidden: Run a hidden cell to show its figure and move the sliders to change the parameters.
Each figure shows a pRF in the center of the visual field with a size of 1 degree. The left panel shows the predicted neuron population response \(r\) (before the convolution with the impulse response) while a vertical bar like the one in the experiment is at different horizontal positions in the visual field. The right panel shows the predicted BOLD response to the first half of the run. The dashed gray lines show the predictions of the Gaussian model with the same center and size. Because the additional parameters also change the scale of the predictions, we scale all predictions to a maximum of 1 so that we can compare their shapes.
The next hidden cell defines helper functions for the interactive figures. Like in the first practical, some of them are imported from the module plotting_helpers.py.
Fitting the models to the first half#
Gaussian model#
All four models build on the Gaussian model, so we first recall how it makes a prediction. Its tuning profile is the isotropic 2D Gaussian density centered at \((\mu_x, \mu_y)\) with standard deviation \(\sigma\):
The tuning profile is encoded with the stimulus design \(S\) by summing the overlap over all points \((x, y)\) of the stimulus grid at every time frame \(t\):
For the Gaussian model, this encoded response is the predicted neuron population response \(r(t) = P(t; \mu_x, \mu_y, \sigma)\). It is convolved with the impulse response \(h\) and scaled with the parameters amplitude (\(\beta\)) and baseline (\(\alpha\)) to give the predicted BOLD response:
The other models only differ in how they compute \(r(t)\) from one or two encoded Gaussians. All Gaussians of a model share the same center \((\mu_x, \mu_y)\).
We fit the Gaussian model in the same way as in the first practical: grid search, least-squares, and SGD. We only use the stimulus and the responses of the first half.
Unlike in the first practical, the grid only contains pRF centers within the visual field covered by the stimulus. The grid search compares predictions and responses with a correlation, which does not depend on the scale of the prediction. For a small pRF with a center far outside the stimulus, the prediction only comes from the outer tail of the Gaussian and is tiny. Its correlation with the response can still be high, but the prediction is so small that least-squares cannot estimate its amplitude, and the fit fails. Starting the pRF centers within the stimulus avoids this, and SGD can still move them outside.
from prfmodel.models.prf import Gaussian2DPRFModel
gaussian_model = Gaussian2DPRFModel(impulse_model=impulse_model)
grid_param_ranges = grid_values_2d_prf(
stimulus_first,
num_mu=11,
mu_extent=1.0, # Only span the visual field covered by the stimulus
num_sigma=10,
sigma_range=(0.05, 4.0), # Log-spaced by default
) | { # Add constant parameters
"weight_deriv": [-0.5],
"baseline": [0.0],
"amplitude": [1.0],
}
grid_fitter = GridFitter(
model=gaussian_model,
stimulus=stimulus_first,
compile_step=True, # Setting 'compile_step=True' speeds up the fitting
)
grid_history, grid_params = grid_fitter.fit(
data=response_first,
parameter_values=grid_param_ranges,
batch_size=20,
)
ls_fitter = LeastSquaresFitter(
model=gaussian_model,
stimulus=stimulus_first,
)
ls_history, ls_params = ls_fitter.fit(
data=response_first,
parameters=grid_params,
slope_name="amplitude",
intercept_name="baseline",
batch_size=200,
)
gaussian_sgd_fitter = SGDFitter(
model=gaussian_model,
stimulus=stimulus_first,
adapter=Adapter([ParameterTransform(["sigma"], transform_fun=ops.log, inverse_fun=ops.exp)]),
optimizer=Adam(learning_rate=learning_rate),
compile_step=True,
)
gaussian_history, gaussian_params = gaussian_sgd_fitter.fit(
data=response_first,
init_parameters=ls_params,
fixed_parameters=["weight_deriv"],
num_steps=num_steps,
)
gaussian_params
CSS model#
The CSS model raises the stimulus-encoded pRF response to the power of the compression exponent n before it is convolved with the impulse response:
With n below 1, the response grows less than linearly with the part of the pRF that is covered by the stimulus. The CSS model of Kay et al. (2013) also has a gain parameter that multiplies the compressed response. It has the same effect as amplitude, so the CSS model in prfmodel uses amplitude instead of a separate gain parameter. The predicted BOLD response \(\hat{y}(t)\) is computed in the same way as for the Gaussian model.
Move the n slider in the figure below to see how the exponent changes the response to the bar and the predicted BOLD response. With n equal to 1, the CSS model makes the same predictions as the Gaussian model.
We start from the Gaussian estimates and an exponent of 0.5. Because the exponent changes the scale of the prediction, we re-estimate amplitude and baseline with least-squares. In SGD, we optimize sigma and n in log space to keep them positive.
from prfmodel.models.prf import Gaussian2DCSSPRFModel, init_css_from_gaussian
css_model = Gaussian2DCSSPRFModel(impulse_model=impulse_model)
# Start from the Gaussian estimates with n = 0.5
css_init_params = init_css_from_gaussian(gaussian_params)
_, css_init_params = LeastSquaresFitter(model=css_model, stimulus=stimulus_first).fit(
data=response_first,
parameters=css_init_params,
slope_name="amplitude",
intercept_name="baseline",
batch_size=200,
)
css_sgd_fitter = SGDFitter(
model=css_model,
stimulus=stimulus_first,
adapter=Adapter([ParameterTransform(["sigma", "n"], transform_fun=ops.log, inverse_fun=ops.exp)]),
optimizer=Adam(learning_rate=learning_rate),
compile_step=True,
)
css_history, css_params = css_sgd_fitter.fit(
data=response_first,
init_parameters=css_init_params,
fixed_parameters=["weight_deriv"],
num_steps=num_steps,
)
css_params
DoG model#
The DoG model subtracts the response of a larger surround Gaussian from the response of a center Gaussian:
The parameters \(\beta_c\), \(\beta_s\), \(\sigma_c\), and \(\sigma_s\) are called amplitude_center, amplitude_surround, sigma_center, and sigma_surround. A positive amplitude_surround means that stimulation of the surround suppresses the response. Because the two amplitudes already scale the response, the DoG model has no separate amplitude parameter, and the predicted BOLD response is \(\hat{y}(t) = (h * r)(t) + \alpha\).
In the figure below, the center Gaussian has amplitude_center and sigma_center equal to 1. Move the sliders to see how the size and the strength of the surround change the response to the bar and the predicted BOLD response.
We start from the Gaussian estimates with a surround that is three times as large as the center and very weak (amplitude_surround equal to 0.001). Both amplitudes enter the prediction linearly, so we can estimate them together with the baseline with least-squares. For some voxels, least-squares returns a negative amplitude_surround, that is, a surround that increases the response. In SGD, we optimize amplitude_surround in log space to keep the surround suppressive, so we only use the least-squares estimates for voxels with a suppressive surround. The other voxels keep the starting values from the Gaussian estimates with the very weak surround.
from prfmodel.models.prf import DoG2DPRFModel, init_dog_from_gaussian
dog_model = DoG2DPRFModel(impulse_model=impulse_model)
# Start from the Gaussian estimates with a weak surround that is three times as large as the center
dog_init_params = init_dog_from_gaussian(gaussian_params, sigma_ratio=3.0)
_, dog_ls_params = LeastSquaresFitter(model=dog_model, stimulus=stimulus_first).fit(
data=response_first,
parameters=dog_init_params,
slope_name=["amplitude_center", "amplitude_surround"],
intercept_name="baseline",
batch_size=200,
)
# Only use the least-squares estimates for voxels with a suppressive surround
is_suppressive = dog_ls_params["amplitude_surround"] > 0.0
dog_init_params.loc[is_suppressive] = dog_ls_params.loc[is_suppressive]
is_suppressive.mean()
dog_sgd_fitter = SGDFitter(
model=dog_model,
stimulus=stimulus_first,
adapter=Adapter([
ParameterTransform(
["sigma_center", "sigma_surround", "amplitude_surround"], transform_fun=ops.log, inverse_fun=ops.exp,
),
]),
optimizer=Adam(learning_rate=learning_rate),
compile_step=True,
)
dog_history, dog_params = dog_sgd_fitter.fit(
data=response_first,
init_parameters=dog_init_params,
fixed_parameters=["weight_deriv"],
num_steps=num_steps,
)
dog_params
DN model#
The DN model combines an activation Gaussian and a larger normalization Gaussian. The encoded response of the activation Gaussian \(A(t) = P(t; \mu_x, \mu_y, \sigma_A)\) is divided by the encoded response of the normalization Gaussian \(N(t) = P(t; \mu_x, \mu_y, \sigma_N)\):
The standard deviations \(\sigma_A\) and \(\sigma_N\) are called sigma_activation and sigma_normalization.
The parameters \(a\), \(b\), \(c\), and \(d\) are called amplitude_activation, baseline_activation, amplitude_normalization, and baseline_normalization. Multiplying all four parameters by the same factor does not change the predicted response, meaning that different combinations of the four parameters can lead to the same predicted response and model fit. The consequence is that the DN model with four free parameters is non-identifiable which makes estimating the parameters difficult. To make parameter estimation easier and the model identifiable, we only estimate three of the four parameters and fix \(c\) to 1.
The subtraction of \(b/d\) makes sure that the response is zero when there is no stimulus. Like for the DoG model, \(a\) already scales the response, so the predicted BOLD response is \(\hat{y}(t) = (h * r)(t) + \alpha\).
In the figure below, \(a\), \(c\), and sigma_activation are equal to 1. Move the sliders to see how the size of the normalization Gaussian and the two baselines change the response to the bar and the predicted BOLD response. Because the baselines can be very small or very large, their sliders change them on a logarithmic scale.
Exercise 6
Check that the DN model is non-identifiable: Create a DivNormGaussian2DPRFModel and parameters for a single voxel (e.g., with the values from the interactive figure above) and predict the response to the first half of the run. Then multiply amplitude_activation, baseline_activation, amplitude_normalization, and baseline_normalization by the same constant (e.g., 2 or 10) and predict the response again. How does the prediction change? What happens when you only multiply some of the four parameters by the constant?
Solution
from prfmodel.models.prf import DivNormGaussian2DPRFModel
dn_check_model = DivNormGaussian2DPRFModel(impulse_model=impulse_model)
# Parameters for a single voxel with the starting values of the interactive figure
dn_check_params = pd.DataFrame({
"mu_x": [0.0],
"mu_y": [0.0],
"sigma_activation": [1.0],
"sigma_normalization": [2.0],
"amplitude_activation": [1.0],
"baseline_activation": [1.0],
"amplitude_normalization": [1.0],
"baseline_normalization": [1.0],
"weight_deriv": [-0.5],
"baseline": [0.0],
})
constant = 10.0
def predict_scaled(names: list[str]) -> np.ndarray:
"""Predict the response to the first half after multiplying the parameters `names` by the constant."""
params = dn_check_params.copy()
params[names] *= constant
return np.asarray(dn_check_model(stimulus_first, params))[0]
prediction = predict_scaled([])
scaled_predictions = {
"a, b, c, and d": predict_scaled(
["amplitude_activation", "baseline_activation", "amplitude_normalization", "baseline_normalization"],
),
"a and b": predict_scaled(["amplitude_activation", "baseline_activation"]),
"c and d": predict_scaled(["amplitude_normalization", "baseline_normalization"]),
"only a": predict_scaled(["amplitude_activation"]),
}
for name, scaled_prediction in scaled_predictions.items():
print(
f"Multiplying {name}: identical = {np.allclose(scaled_prediction, prediction, atol=1e-6)}, "
f"ratio of the maxima = {scaled_prediction.max() / prediction.max():.2f}, "
f"correlation = {np.corrcoef(scaled_prediction, prediction)[0, 1]:.2f}"
)
Multiplying all four parameters by the same constant does not change the prediction at all:
Thus, the data cannot distinguish between these parameter combinations. When we only multiply some of the parameters, the prediction changes:
Multiplying only \(a\) and \(b\) (the numerator) multiplies the response by the constant, and multiplying only \(c\) and \(d\) (the denominator) divides it by the constant. The shape of the response stays the same (correlation of 1), only its scale changes.
Multiplying a single parameter (e.g., only \(a\)) changes the balance between activation and normalization, which changes the shape of the response (the correlation drops below 1).
Because the response only depends on the ratios between the four parameters, fixing one of them (here \(c = 1\)) makes the model identifiable without losing any flexibility.
# Your code goes here
Good starting values for the DN model are hard to find. The helper function prfmodel.models.prf.init_div_norm_from_dog_css approximates them from the DoG estimates (suppression) and the CSS estimates (compression). It expects a CSS exponent below 1, so we limit the exponent to 0.9 for the few voxels where the CSS model estimated no compression. It also warns if the DoG surround of a voxel ended up smaller than its center. This can happen for a few voxels and is not a problem because SGD can still adjust the sizes of both Gaussians. The response is linear in \(a\) and \(b\), so we estimate them with least-squares. Like for the DoG model, least-squares returns negative values for some voxels, which we replace with very small positive values because we optimize \(a\) and \(b\) in log space.
from prfmodel.models.prf import DivNormGaussian2DPRFModel, init_div_norm_from_dog_css
dn_model = DivNormGaussian2DPRFModel(impulse_model=impulse_model)
# Approximate suppression from the DoG estimates and compression from the CSS estimates (n must be below 1)
dn_init_params = init_div_norm_from_dog_css(dog_params, css_n=css_params["n"].clip(upper=0.9), stimulus=stimulus_first)
_, dn_init_params = LeastSquaresFitter(model=dn_model, stimulus=stimulus_first).fit(
data=response_first,
parameters=dn_init_params,
slope_name=["amplitude_activation", "baseline_activation"],
intercept_name="baseline",
batch_size=200,
)
# Keep a and b positive
dn_init_params[["amplitude_activation", "baseline_activation"]] = (
dn_init_params[["amplitude_activation", "baseline_activation"]].clip(lower=1e-3)
)
dn_sgd_fitter = SGDFitter(
model=dn_model,
stimulus=stimulus_first,
adapter=Adapter([
ParameterTransform(
[
"sigma_activation",
"sigma_normalization",
"amplitude_activation",
"baseline_activation",
"baseline_normalization",
],
transform_fun=ops.log,
inverse_fun=ops.exp,
),
]),
optimizer=Adam(learning_rate=learning_rate),
compile_step=True,
)
dn_history, dn_params = dn_sgd_fitter.fit(
data=response_first,
init_parameters=dn_init_params,
fixed_parameters=["weight_deriv", "amplitude_normalization"],
num_steps=num_steps,
)
dn_params
Exercise 7
Print the parameter names of each model (e.g., css_model.parameter_names) and compare them with the table above. Then, look at the distributions of the CSS exponent n and the ratio between the DoG surround and center sizes (sigma_surround / sigma_center). What do they tell you about compression and suppression in the selected voxels?
Solution
for name, model in zip(["Gaussian", "CSS", "DoG", "DN"], [gaussian_model, css_model, dog_model, dn_model]):
print(f"{name}: {model.parameter_names}")
dog_sigma_ratio = dog_params["sigma_surround"] / dog_params["sigma_center"]
fig, axes = plt.subplots(1, 2, figsize=(10, 4), layout="constrained")
axes[0].hist(css_params["n"], bins=np.linspace(0.0, 2.0, 41))
axes[0].axvline(1.0, color="black", linestyle="--") # No compression
axes[0].set(xlabel="n", ylabel="Number of voxels", title="CSS exponent")
axes[1].hist(dog_sigma_ratio, bins=np.geomspace(0.5, 100.0, 41))
axes[1].set(xscale="log", xlabel="sigma_surround / sigma_center", ylabel="Number of voxels", title="DoG size ratio")
plt.show()
print(css_params["n"].describe())
print(dog_sigma_ratio.describe())
# Total response of the surround relative to the center (the Gaussians are normalized)
print((dog_params["amplitude_surround"] / dog_params["amplitude_center"]).describe())
The parameter names contain more parameters than the table: the parameters of the impulse response model (delay, dispersion, undershoot, u_dispersion, ratio, and weight_deriv). The first five are not part of our parameter DataFrames, so the model uses their default values, and we fix weight_deriv. Without them, the names match the free parameters in the table. For the DN model, the names also contain amplitude_normalization (\(c\)), which we fix to 1.
Compression: The CSS exponent is below 1 for 96% of the selected voxels and below 0.5 for 69% (median of 0.30). Thus, the responses of most voxels are strongly compressive: Covering a larger part of the pRF increases the response much less than linearly. A few voxels have an exponent above 1, which is probably not well constrained by the data.
Suppression: The DoG surround is about 2 to 3.5 times as large as the center for most voxels (median of 2.6). Many ratios lie close to the starting value of 3, which suggests that the data do not constrain the size of the surround well (see Exercise 18). The strength of the surround varies a lot: For a fifth of the voxels, the total response of the surround is below 1% of the center, so these voxels show practically no suppression, whereas the surround of a quarter of the voxels has more than 65% of the total response of the center.
# Your code goes here
Comparing the in-sample fit#
We first compare how well the models fit the training data. We collect the models and their parameter estimates in dictionaries and compute the in-sample R-squared of each model.
from prfmodel.utils import batched, calculate_r_squared
model_names = ["Gaussian", "CSS", "DoG", "DN"] # From the simplest to the most complex model
models = dict(zip(model_names, [gaussian_model, css_model, dog_model, dn_model]))
model_params = dict(zip(model_names, [gaussian_params, css_params, dog_params, dn_params]))
# Make predictions for batches of voxels to save memory
predict_batched = {name: batched(model) for name, model in models.items()}
pred_response_first = {
name: np.asarray(predict_batched[name](stimulus_first, model_params[name], batch_size=200)) for name in model_names
}
# One column per model and one row per selected voxel
r_squared_train = pd.DataFrame(
{name: calculate_r_squared(response_first, pred_response_first[name]) for name in model_names},
)
r_squared_train.agg(["mean", "median"])
We compare the in-sample R-squared of each model against the Gaussian model for each voxel.
from prfmodel.plotting import plot_r_squared_comparison
fig, axes = plt.subplots(1, 3, figsize=(15, 4.5), layout="constrained")
for ax, name in zip(axes, model_names[1:]):
plot_r_squared_comparison(
r_squared_train["Gaussian"],
r_squared_train[name],
value_range=(0.0, 1.0),
xlabel="R-squared Gaussian (training set)",
ylabel=f"R-squared {name} (training set)",
ax=ax,
)
Exercise 8
Which model fits the training data best? Why do the CSS, DoG, and DN models almost never fit worse than the Gaussian model? What would it mean if a more complex model fit a voxel clearly worse than the Gaussian model? Does the in-sample fit tell us which model is the best description of the data?
Solution
# Improvement of the in-sample R-squared over the Gaussian model
improvement_train = r_squared_train[model_names[1:]].sub(r_squared_train["Gaussian"], axis=0)
pd.DataFrame({
"mean improvement": improvement_train.mean(),
"fraction better": (improvement_train > 0.0).mean(),
"fraction clearly worse (< -0.01)": (improvement_train < -0.01).mean(),
})
The DN model fits the training data best (mean R-squared of 0.690), followed by the CSS (0.679), DoG (0.667), and Gaussian model (0.654).
The CSS, DoG, and DN models can mimic the Gaussian model (e.g., the CSS model with n equal to 1), and we start their fits from the Gaussian estimates. Thus, they can always reach at least the fit of the Gaussian model, and with their additional parameters, they can fit the data more closely. The CSS model fits better than the Gaussian model for 91% of the voxels, the DoG model for 77%, and the DN model for 84%. Almost none of the voxels (at most 1% for the DN model) fit clearly worse.
If a more complex model fits a voxel clearly worse than the Gaussian model, its fit failed: For example, SGD got stuck in a poor local minimum, or the starting values or the constraints of the parameters (e.g., keeping amplitude_surround positive) prevented it from reaching the solution of the Gaussian model. We should check these voxels and, for example, refit them with different starting values.
The in-sample fit does not tell us which model describes the data best. A model with more parameters can also fit the noise in the training data better (overfitting), so a better in-sample fit is expected even if the additional parameters do not describe anything real. To compare the models, we need their out-of-sample fit.
# Your code goes here
Comparing the out-of-sample fit#
Now, we compare the predictions of the models for the second half, which none of the models were fitted to. We make predictions with the stimulus of the second half and the parameters estimated on the first half.
pred_response_second = {
name: np.asarray(predict_batched[name](stimulus_second, model_params[name], batch_size=200)) for name in model_names
}
r_squared_test = pd.DataFrame(
{name: calculate_r_squared(response_second, pred_response_second[name]) for name in model_names},
)
pd.DataFrame({"training set": r_squared_train.mean(), "test set": r_squared_test.mean()})
fig, axes = plt.subplots(1, 3, figsize=(15, 4.5), layout="constrained")
for ax, name in zip(axes, model_names[1:]):
plot_r_squared_comparison(
r_squared_test["Gaussian"],
r_squared_test[name],
value_range=(0.0, 1.0),
xlabel="R-squared Gaussian (test set)",
ylabel=f"R-squared {name} (test set)",
ax=ax,
)
Exercise 9
Compare the out-of-sample R-squared of the models. Is the ranking of the models the same as for the in-sample fit? Which model generalizes best on average? How large is the advantage of the more complex models compared to the in-sample fit?
Solution
improvement_test = r_squared_test[model_names[1:]].sub(r_squared_test["Gaussian"], axis=0)
pd.DataFrame({
"mean improvement (training set)": improvement_train.mean(),
"mean improvement (test set)": improvement_test.mean(),
"fraction better (training set)": (improvement_train > 0.0).mean(),
"fraction better (test set)": (improvement_test > 0.0).mean(),
})
The ranking is the same as for the in-sample fit: The DN model predicts the test data best on average (mean R-squared of 0.656), closely followed by the CSS model (0.652), the DoG model (0.635), and the Gaussian model (0.625). All models predict the test data somewhat worse than they fit the training data.
The advantage of the more complex models shrinks slightly: The mean improvement over the Gaussian model drops from 0.036 to 0.030 for the DN model and from 0.013 to 0.010 for the DoG model, whereas it stays at about 0.026 for the CSS model. The CSS model also improves the prediction for the most voxels (84%, compared to 78% for the DN model and 54% for the DoG model). Thus, the DN model generalizes best on average, but only by a small margin over the CSS model, which has two parameters less and improves the prediction more consistently.
# Your code goes here
We compare the out-of-sample R-squared of each model with the noise ceiling of each voxel. Voxels on the identity line are predicted as well as the noise allows.
fig, axes = plt.subplots(1, 4, figsize=(20, 4.5), layout="constrained")
for ax, name in zip(axes, model_names):
plot_r_squared_comparison(
noise_ceiling_selected,
r_squared_test[name],
value_range=(0.0, 1.0),
xlabel="Noise ceiling",
ylabel=f"R-squared {name} (test set)",
ax=ax,
)
Exercise 10
How close do the models get to the noise ceiling? Are there voxels for which a model predicts the test data better than the noise ceiling? Considering how we estimated the noise ceiling, how is this possible? There are also voxels with a high noise ceiling that none of the models predicts well. Plot the responses of some of these voxels: What do they have in common, and why can’t our models describe them?
Solution
from prfmodel.plotting import plot_observed_predicted
# Ratio between out-of-sample R-squared and noise ceiling, and fraction of voxels above the noise ceiling
print(pd.DataFrame({
"median ratio to noise ceiling": r_squared_test.div(noise_ceiling_selected, axis=0).median(),
"fraction above noise ceiling": r_squared_test.gt(noise_ceiling_selected, axis=0).mean(),
}))
# Voxels with a high noise ceiling that no model predicts well
is_poorly_predicted = (noise_ceiling_selected > 0.8) & (r_squared_test.max(axis=1) < 0.3)
print(f"Voxels with a noise ceiling above 0.8 and an out-of-sample R-squared below 0.3: {is_poorly_predicted.sum()}")
fig, axes = plt.subplots(3, 1, figsize=(12, 10.5), sharex=True, layout="constrained")
for ax, voxel_index in zip(axes, np.flatnonzero(is_poorly_predicted)):
plot_observed_predicted(
response_second[voxel_index],
{name: pred_response_second[name][voxel_index] for name in model_names},
ylabel="BOLD response",
ax=ax,
)
ax.set_title(f"Voxel {voxel_index}: noise ceiling = {noise_ceiling_selected[voxel_index]:.2f}", loc="left")
For a typical voxel, the models reach about 80% (Gaussian and DoG) to 85% (CSS and DN) of the noise ceiling. The out-of-sample R-squared lies above the noise ceiling for 3% of the voxels with the Gaussian and DoG model and for about 10% with the CSS and DN model. This is possible because the noise ceiling is only an estimate: It is conservative (Exercise 4), so differences between the halves that a model can predict (e.g., the different order of the sweeps) count as noise. It also contains noise itself because we compute it from only 94 time frames per half.
There are 22 voxels with a noise ceiling above 0.8 for which no model reaches an R-squared of 0.3. Most of them have a clear response with large peaks (some above 10%), but the peaks occur 3 to 5 time frames (about 3 to 4.5 seconds) later than predicted, and they are often broader. When we shift the predictions by this delay, most of them correlate highly with the response (about 0.8 to 0.9). Such large and delayed responses are typical for voxels close to large veins, which drain blood from a larger region and react more slowly. Many of these voxels lie next to each other (e.g., voxels 2624 to 2631). Our models use the same impulse response with a fixed shape for all voxels, so they cannot shift their predictions in time. A few other voxels have strong negative responses (e.g., voxel 345) that none of the models describe. To describe the delayed voxels, we would also need to estimate the parameters of the impulse response (e.g., delay) for each voxel.
# Your code goes here
Exercise 11
Find the voxels for which the out-of-sample R-squared of the CSS, DoG, or DN model is much higher than for the Gaussian model. Plot the observed test timecourses together with the predictions of all four models for some of these voxels (hint: use prfmodel.plotting.plot_observed_predicted with a dictionary of predictions). Which features of the response does the Gaussian model miss?
Solution
# Only consider voxels for which the Gaussian fit did not fail (in-sample R-squared above 0.3)
has_gaussian_fit = r_squared_train["Gaussian"] > 0.3
fig, axes = plt.subplots(3, 1, figsize=(12, 10.5), sharex=True, layout="constrained")
# For each complex model, the voxel with the largest improvement over the Gaussian model on the test set
plotted_voxels = []
for ax, name in zip(axes, model_names[1:]):
voxel_index = improvement_test.loc[has_gaussian_fit, name].drop(plotted_voxels).idxmax()
plotted_voxels.append(voxel_index)
plot_observed_predicted(
response_second[voxel_index],
{model_name: pred_response_second[model_name][voxel_index] for model_name in model_names},
ylabel="BOLD response",
ax=ax,
)
ax.set_title(
f"Voxel {voxel_index}: R-squared Gaussian = {r_squared_test['Gaussian'][voxel_index]:.2f}, "
f"{name} = {r_squared_test[name][voxel_index]:.2f}",
loc="left",
)
CSS: The voxel responds to a sweep with a broad, plateau-like response: The response is almost as large when the bar only covers the edge of the pRF as when it covers the center. The Gaussian model predicts narrower peaks with a dip in between. Compression flattens the response, so the CSS model describes the plateau better.
DoG and DN: The response drops clearly below baseline when the bar is close to but not on the pRF center, and it lies above this level during the blank periods. The Gaussian model can only predict an increase of the response (followed by the short undershoot of the impulse response) and misses these negative responses. The suppressive surround of the DoG model and the normalization of the DN model predict them.
We only consider voxels for which the Gaussian model fits the training data with an R-squared above 0.3. Without this restriction, the largest improvements of the CSS model come from voxels for which the Gaussian fit failed: Its prediction is flat (R-squared close to 0), although the voxel responds clearly. There, the improvement reflects a failed fit rather than compression. Because the DoG and DN models start from the Gaussian estimates, their fits fail for these voxels, too.
# Your code goes here
So far, we compared the models across all selected voxels. However, neuron populations in different parts of the visual cortex can respond differently to stimulation: For example, compression is stronger in higher visual areas than in V1 (Kay et al., 2013). A model that describes one region best may not be the best model for another region. Therefore, we now select a model for each voxel separately.
Selecting a model per voxel#
For each voxel, we select the model with the highest out-of-sample R-squared. However, the out-of-sample R-squared is itself an estimate that contains noise. When two models predict the test data almost equally well, which of them comes out on top is partly chance. In such cases, we prefer the simpler model because it explains the data with fewer assumptions. We therefore only select a more complex model if its out-of-sample R-squared is higher than that of every simpler model by more than a minimum gain. Note that the value of the minimum gain is somewhat arbitrary.
def select_models(r_squared: pd.DataFrame, min_gain: float) -> np.ndarray:
"""Select the most complex model that beats all simpler models by more than min_gain for each voxel.
The columns of r_squared must be ordered from the simplest to the most complex model.
"""
scores = r_squared.to_numpy()
selected = np.zeros(scores.shape[0], dtype=int) # Start with the simplest model for all voxels
for i in range(1, scores.shape[1]):
beats_simpler = np.all(scores[:, [i]] > scores[:, :i] + min_gain, axis=1)
selected[beats_simpler] = i
return r_squared.columns.to_numpy()[selected]
min_gain = 0.01
best_model = select_models(r_squared_test[model_names], min_gain=min_gain)
# Fraction of selected voxels for which each model is the best model
pd.Series(best_model).value_counts(normalize=True).reindex(model_names, fill_value=0.0)
Selecting the model with the highest score among several noisy scores has a catch: The score of the selected model overestimates how well it will predict new data. Some models win only because the noise in the test data happens to favor them. This is the same problem as with the in-sample fit in the second practical, only one level higher: We used the test data to select the model, so we cannot use the same test data to fairly evaluate the selection. For a fair evaluation, we need data that was not used for the selection, for example, the other fold of the cross-validation (see the optional exercise below).
To see where on the surface each model is selected, we cannot simply project the model names (or numbers) onto the surface because the projection averages neighboring voxels. Instead, we project for each model whether it was selected (1) or not (0). After the projection, each vertex shows the fraction of voxels around it for which the model was selected.
for name in model_names:
is_best = (best_model == name).astype(float)
plot_surf_stat_map_helper(
project_to_surface(fill_selected_voxels(is_best)),
vmin=0.0,
vmax=1.0,
title=f"Fraction of voxels for which {name} is selected",
)
Exercise 12
Where on the surface is each model selected? Compare the maps with the map of the noise ceiling. Do you see systematic differences between regions?
Solution
plot_surf_stat_map_helper(
project_to_surface(fill_selected_voxels(noise_ceiling_selected)),
vmin=0.7,
vmax=1.0,
title="Noise ceiling of the selected voxels",
)
plt.show()
# Fraction of the selected models within bins of the noise ceiling
pd.crosstab(pd.cut(noise_ceiling_selected, [0.7, 0.8, 0.9, 1.0]), best_model, normalize="index")
The Gaussian model is mostly selected around the occipital pole on the posterior surface and along the calcarine sulcus on the medial side, where the center of the visual field is represented in early visual cortex. The CSS model is mostly selected in the outer parts of the selected region, on the lateral, dorsal, and ventral side of the posterior surface, where the higher visual areas and the more peripheral parts of the visual field are represented. This agrees with Kay et al. (2013), who found stronger compression in higher visual areas. The DN model is selected in scattered patches, mostly in the outer parts as well, and the DoG model is rarely selected.
The noise ceiling is highest along the calcarine sulcus and around the occipital pole and lower towards the edges of the selected region. However, the fractions of the selected models hardly change with the noise ceiling: For example, the CSS model is selected for 45% to 47% of the voxels in each bin. Thus, the regional differences are not just a consequence of differences in noise.
# Your code goes here
Exercise 13
Select the models with a minimum gain of 0 and of 0.05. How do the fractions of selected models change and why?
Solution
pd.DataFrame({
f"min_gain = {gain}": pd.Series(select_models(r_squared_test[model_names], min_gain=gain))
.value_counts(normalize=True)
.reindex(model_names, fill_value=0.0)
for gain in [0.0, 0.01, 0.05]
})
Model |
|
|
|
|---|---|---|---|
Gaussian |
6% |
37% |
77% |
CSS |
47% |
46% |
20% |
DoG |
5% |
4% |
3% |
DN |
41% |
13% |
1% |
With a minimum gain of 0, any improvement is enough, however small. The complex models fit at least as well as the Gaussian model and slightly better for most voxels, often only by chance, so the Gaussian model is only selected for 6% of the voxels, and the most flexible DN model for 41%. With a minimum gain of 0.05, a model must clearly beat all simpler models, so the Gaussian model is selected for most voxels. The DN model almost disappears because it must also beat the CSS model by 0.05, which already captures most of the compression. The CSS model is still selected for a fifth of the voxels, which shows that compression clearly improves the prediction for these voxels. The minimum gain controls the trade-off between missing real improvements (high gain) and selecting complex models because of noise (low gain).
# Your code goes here
Exercise 14
Compute how much the out-of-sample R-squared of each model improves over the Gaussian model for each voxel. Plot the improvement against the eccentricity and the size (sigma) of the pRF estimated with the Gaussian model (hint: use prfmodel.utils.calculate_eccentricity; averaging the improvement within bins of eccentricity or size makes the pattern easier to see). You can also plot the improvement on the surface with a diverging colormap (e.g., cmap="RdBu_r" with a symmetric color range). For which pRFs do the more complex models make the largest difference? Keep in mind that the stimulus only covers the visual field up to about 3.9 degrees from the center horizontally and 2.8 degrees vertically: How well can the models describe pRFs with centers outside of this region?
Solution
from prfmodel.utils import calculate_eccentricity
gaussian_eccentricity = calculate_eccentricity(gaussian_params["mu_x"], gaussian_params["mu_y"])
eccentricity_bins = pd.cut(gaussian_eccentricity, [0, 1, 2, 3, 4, 6, 10, np.inf])
sigma_bins = pd.cut(gaussian_params["sigma"], [0, 0.5, 1, 2, 3, 4, 6, np.inf])
fig, axes = plt.subplots(1, 2, figsize=(14, 4.5), layout="constrained")
# Average improvement within bins of eccentricity and size
improvement_test.groupby(eccentricity_bins, observed=True).mean().plot(
ax=axes[0], marker="o", xlabel="Eccentricity (degrees)", ylabel="Improvement over Gaussian (test set)",
)
improvement_test.groupby(sigma_bins, observed=True).mean().plot(
ax=axes[1], marker="o", xlabel="sigma (degrees)", ylabel="Improvement over Gaussian (test set)",
)
for ax in axes:
ax.axhline(0.0, color="black", linestyle="--")
ax.tick_params(axis="x", labelrotation=30)
plt.show()
# Number of voxels per bin
print(pd.Series(eccentricity_bins).value_counts(sort=False))
print(pd.Series(sigma_bins).value_counts(sort=False))
for name in model_names[1:]:
plot_surf_stat_map_helper(
project_to_surface(fill_selected_voxels(improvement_test[name].to_numpy())),
vmin=-0.1,
vmax=0.1,
cmap="RdBu_r",
title=f"Improvement of {name} over Gaussian (test set)",
)
The improvement grows with eccentricity: For pRFs within 1 to 2 degrees of the center of the visual field, the more complex models hardly improve the prediction (the DoG model not at all up to 3 degrees). For pRFs with centers between 6 and 10 degrees, the CSS and DN models improve the R-squared by about 0.06. The CSS and DN models also improve most for large pRFs (sigma above 2 degrees), whereas the DoG model improves most for small pRFs (below 1 degree). On the surface, the improvement is close to zero around the occipital pole and largest in the outer parts of the selected region.
Note that more than a third of the selected voxels (36%) have a Gaussian pRF center outside the stimulus. For these voxels, the mean improvement of the CSS and DN models is about twice as large as for the voxels with a center inside the stimulus, and the Gaussian model is only selected for 15% of them (compared to 50% inside). For these pRFs, the stimulus only covers their edge, so the response only tells us how the edge of the pRF responds. Different combinations of center, size, and the additional parameters predict almost the same response, for example, a center further outside the stimulus together with a larger size or amplitude. Their parameters are therefore poorly constrained, and the additional flexibility of the complex models helps them describe the response to the edge of the pRF. Thus, a better prediction for these voxels does not necessarily mean that their neuron populations show more compression or suppression, and we should not interpret their parameters. To describe these pRFs, we would need a stimulus that covers a larger part of the visual field.
# Your code goes here
Exercise 15 (optional)
Repeat the fitting with the halves swapped: Fit all four models to the second half and evaluate them on the first half. Select the models for this fold with select_models. For which fraction of the voxels is the same model selected in both folds? What does this tell you about how reliable the selection for individual voxels is? Also average the out-of-sample R-squared of each model across both folds and compare the models again.
Solution
We put the fitting steps from above into a function so that we can reuse it (also in the next exercise). Note that fitting all four models takes about as long as the fits above.
def fit_models(stimulus: PRFStimulus, response: np.ndarray) -> dict[str, pd.DataFrame]:
"""Fit all four models to the response with the same steps as above and return their parameter estimates."""
log_transform = {"transform_fun": ops.log, "inverse_fun": ops.exp}
def fit_least_squares(model, init_params, slope_name):
_, params = LeastSquaresFitter(model=model, stimulus=stimulus).fit(
data=response, parameters=init_params, slope_name=slope_name, intercept_name="baseline", batch_size=200,
)
return params
def fit_sgd(model, init_params, log_names, fixed_names):
sgd_fitter = SGDFitter(
model=model,
stimulus=stimulus,
adapter=Adapter([ParameterTransform(log_names, **log_transform)]),
optimizer=Adam(learning_rate=learning_rate),
compile_step=True,
)
_, params = sgd_fitter.fit(
data=response, init_parameters=init_params, fixed_parameters=fixed_names, num_steps=num_steps,
)
return params
# Gaussian model: grid search, least-squares, and SGD
grid_values = grid_values_2d_prf(stimulus, num_mu=11, mu_extent=1.0, num_sigma=10, sigma_range=(0.05, 4.0)) | {
"weight_deriv": [-0.5],
"baseline": [0.0],
"amplitude": [1.0],
}
_, params = GridFitter(model=gaussian_model, stimulus=stimulus, compile_step=True).fit(
data=response, parameter_values=grid_values, batch_size=20,
)
params = fit_least_squares(gaussian_model, params, "amplitude")
gaussian = fit_sgd(gaussian_model, params, ["sigma"], ["weight_deriv"])
# CSS model: start from the Gaussian estimates
params = fit_least_squares(css_model, init_css_from_gaussian(gaussian), "amplitude")
css = fit_sgd(css_model, params, ["sigma", "n"], ["weight_deriv"])
# DoG model: start from the Gaussian estimates and only keep suppressive least-squares estimates
params = init_dog_from_gaussian(gaussian, sigma_ratio=3.0)
dog_ls_params = fit_least_squares(dog_model, params, ["amplitude_center", "amplitude_surround"])
is_suppressive = dog_ls_params["amplitude_surround"] > 0.0
params.loc[is_suppressive] = dog_ls_params.loc[is_suppressive]
dog = fit_sgd(dog_model, params, ["sigma_center", "sigma_surround", "amplitude_surround"], ["weight_deriv"])
# DN model: start from the DoG and CSS estimates
params = init_div_norm_from_dog_css(dog, css_n=css["n"].clip(upper=0.9), stimulus=stimulus)
params = fit_least_squares(dn_model, params, ["amplitude_activation", "baseline_activation"])
params[["amplitude_activation", "baseline_activation"]] = (
params[["amplitude_activation", "baseline_activation"]].clip(lower=1e-3)
)
dn = fit_sgd(
dn_model,
params,
["sigma_activation", "sigma_normalization", "amplitude_activation", "baseline_activation", "baseline_normalization"],
["weight_deriv", "amplitude_normalization"],
)
return {"Gaussian": gaussian, "CSS": css, "DoG": dog, "DN": dn}
# Fit the models to the second half
params_second = fit_models(stimulus_second, response_second)
# Predict the first half with the parameters estimated on the second half
r_squared_test_swapped = pd.DataFrame({
name: calculate_r_squared(
response_first, np.asarray(predict_batched[name](stimulus_first, params_second[name], batch_size=200)),
)
for name in model_names
})
best_model_swapped = select_models(r_squared_test_swapped[model_names], min_gain=min_gain)
print(f"Same model selected in both folds: {(best_model == best_model_swapped).mean():.2f}")
print(pd.crosstab(pd.Series(best_model, name="First fold"), pd.Series(best_model_swapped, name="Second fold")))
# Out-of-sample R-squared averaged across both folds
pd.DataFrame({
"first fold": r_squared_test.mean(),
"second fold": r_squared_test_swapped.mean(),
"average": ((r_squared_test + r_squared_test_swapped) / 2).mean(),
})
The same model is selected in both folds for 69% of the voxels. This is clearly more than expected by chance (36% for selections with the same fractions of models but no relation between the folds), but for almost a third of the voxels, the selection changes when we swap the halves:
The selection of the Gaussian model is the most reliable: 85% of the voxels for which it is selected in the first fold also select it in the second fold.
The CSS model is selected again for 66% of its voxels. Most of the other voxels switch to the Gaussian model.
The DoG and DN models are selected again for fewer than half of their voxels (41% and 44%).
Thus, the selection for an individual voxel is not very reliable, especially for the DoG and DN models, whose advantage over the simpler models is small and noisy. The selection becomes more reliable when we look at many voxels together, for example, at the fraction of voxels for which a model is selected within a region.
Averaged across both folds, the out-of-sample R-squared is 0.623 for the Gaussian, 0.645 for the CSS, 0.631 for the DoG, and 0.646 for the DN model. In the second fold, the CSS model even predicts better than the DN model (0.639 compared to 0.635). Thus, the CSS and DN models are equally good on average, and the CSS model achieves this with two parameters less. The first fold alone made the DN model look slightly better than it is.
# Your code goes here
Fitting and interpreting the selected models#
Once we have selected a model for each voxel, we want to estimate its parameters as precisely as possible so that we can interpret them. Like in the second practical, we now use all data, that is, the full run with both halves. The in-sample R-squared of these fits is optimistic, but we accept this because cross-validation has already told us how well the models generalize.
When we interpret the parameters of the selected models, we need to keep in mind that the parameters of different models do not always mean the same thing. All four models have a pRF center (mu_x and mu_y), so we can compare the polar angle and eccentricity of the pRF across voxels with different selected models. In contrast, the size parameters differ between models: For example, the sigma of the CSS model describes the Gaussian before the compression, which makes the pRF appear larger than its sigma suggests (Kay et al., 2013). The size of the center Gaussian of the DoG model (sigma_center) is also not the same as the size of a Gaussian pRF without a surround. A map of pRF sizes that mixes the size parameters of different models would therefore mix different definitions of size.
Exercise 16
Fit all four models to the full run (stimulus and response_valid[is_selected]) using the same steps as above. For each voxel, take the pRF center of its selected model (hint: use np.select with one condition per model) and plot the polar angle and eccentricity on the surface (hint: use prfmodel.utils.calculate_polar_angle and prfmodel.utils.calculate_eccentricity, and project mu_x and mu_y onto the surface separately like in the first practical). Do the maps differ from the maps you get from the Gaussian model alone?
Solution
We reuse the function fit_models from the solution of Exercise 15. Fitting the full run takes about twice as long as fitting one half.
from prfmodel.utils import calculate_eccentricity, calculate_polar_angle
# Fit all four models to the full run with the function from the solution of Exercise 15
params_full = fit_models(stimulus, response_valid[is_selected])
# One condition per model: the voxels for which the model is selected
conditions = [best_model == name for name in model_names]
mu_x_selected = np.select(conditions, [params_full[name]["mu_x"] for name in model_names])
mu_y_selected = np.select(conditions, [params_full[name]["mu_y"] for name in model_names])
centers = {
"selected models": (mu_x_selected, mu_y_selected),
"Gaussian model": (params_full["Gaussian"]["mu_x"].to_numpy(), params_full["Gaussian"]["mu_y"].to_numpy()),
}
for label, (mu_x, mu_y) in centers.items():
# Project mu_x and mu_y onto the surface separately and compute polar angle and eccentricity afterwards
mu_x_surf = project_to_surface(fill_selected_voxels(mu_x))
mu_y_surf = project_to_surface(fill_selected_voxels(mu_y))
plot_surf_stat_map_helper(calculate_polar_angle(mu_x_surf, mu_y_surf), cyclic=True, title=f"Polar angle ({label})")
plot_surf_stat_map_helper(
calculate_eccentricity(mu_x_surf, mu_y_surf), vmin=0.0, vmax=5.0, title=f"Eccentricity ({label})",
)
# Distance between the pRF centers of the selected models and the Gaussian model
center_distance = np.hypot(mu_x_selected - centers["Gaussian model"][0], mu_y_selected - centers["Gaussian model"][1])
pd.Series(center_distance).groupby(best_model).describe()
The maps of the selected models and the Gaussian model look very similar: Both show the same retinotopic organization, with the contralateral visual field in each hemisphere and the polar angle and eccentricity changing in the same way across the surface (see the first practical). The pRF centers of the selected models differ only little from the Gaussian centers: The polar angle differs by a median of only 1 to 3 degrees. The center of the CSS model lies a median of 0.65 degrees away from the Gaussian center, mostly closer to the center of the visual field (its eccentricity is 0.5 degrees smaller), and the center of the DN model lies 0.92 degrees away. Where the CSS model is selected, the eccentricity map therefore shows somewhat smaller values. For the voxels for which the Gaussian model is selected, the maps are identical.
Thus, the choice of the model hardly changes the large-scale maps of the pRF centers, but it can shift the centers of individual voxels by up to a few degrees. The shifts are largest for pRFs near or outside the edge of the stimulus: For voxels with a Gaussian eccentricity above 4 degrees, the selected model moves the center by a median of 1 to 1.4 degrees, compared to 0.16 degrees within 2 degrees of the center of the visual field.
# Your code goes here
Exercise 17
Interpret the additional parameters of the more complex models, but only for the voxels for which the respective model is selected. For example, plot the CSS exponent n and the ratio between the DoG surround and center sizes against the eccentricity of the pRF. How do compression and suppression change with eccentricity?
Solution
We use the pRF centers of the selected models from Exercise 16 and plot the additional parameters against eccentricity, together with their median within bins of eccentricity (black). For the DN model, we use the ratio between the sizes of the normalization and activation Gaussians.
eccentricity_selected = calculate_eccentricity(mu_x_selected, mu_y_selected)
eccentricity_bins = pd.cut(eccentricity_selected, [0, 1, 2, 3, 4, 6, 12])
# Additional parameters of each complex model for the voxels for which it is selected
additional_parameters = {
"CSS": ("n", params_full["CSS"]["n"]),
"DoG": ("sigma_surround / sigma_center", params_full["DoG"]["sigma_surround"] / params_full["DoG"]["sigma_center"]),
"DN": (
"sigma_normalization / sigma_activation",
params_full["DN"]["sigma_normalization"] / params_full["DN"]["sigma_activation"],
),
}
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5), layout="constrained")
for ax, (name, (label, values)) in zip(axes, additional_parameters.items()):
is_model = best_model == name
ax.scatter(eccentricity_selected[is_model], values[is_model], s=5, alpha=0.5)
# Median within bins of eccentricity
binned_median = values[is_model].groupby(eccentricity_bins[is_model], observed=True).median()
ax.plot([interval.mid for interval in binned_median.index], binned_median, color="black", marker="o")
ax.set(
yscale="log",
xlabel="Eccentricity (degrees)",
ylabel=label,
title=f"{name} ({is_model.sum()} voxels)",
)
Compression: For the voxels for which the CSS model is selected, compression is strong: The median exponent
nis about 0.15 and does not change with eccentricity.Suppression (DoG): The DoG model is only selected for about a hundred voxels. The surround is mostly 2 to 3 times as large as the center, without a clear relation to eccentricity. Some ratios are extremely large (above 10), which means that the surround is hardly constrained for these voxels (see Exercise 18).
Normalization (DN): The normalization Gaussian is relatively larger for pRFs close to the center of the visual field (median ratio of about 2.3 within 2 degrees) than for pRFs further out (about 1.5 beyond 6 degrees).
These results should be interpreted with caution: pRFs with an eccentricity above about 3 to 4 degrees have their center outside the stimulus (which only reaches 2.8 degrees vertically and 3.9 degrees horizontally), so their parameters are poorly constrained (Exercise 14). Moreover, the selection of the DoG and DN models is not reliable for individual voxels (Exercise 15), and the eccentricity bins mix voxels from different visual areas, which can differ in compression and suppression (e.g., compression is stronger in higher visual areas; Kay et al., 2013).
# Your code goes here
Exercise 18
Plot the surround and normalization pRFs of the DoG and DN model on the stimulus grid. How much of these pRFs is covered by the stimulus? Does the stimulus design provide enough information to estimate these pRFs? How does this relate to whether the DoG and DN model fit a voxel better than the Gaussian and CSS model?
Solution
We use the estimates from the first half because we selected the models with them. For each Gaussian of the DoG and DN model, we compute the fraction of its volume that lies inside the visual field covered by the stimulus, and we plot the Gaussians of one voxel for which the DoG model is selected with prfmodel.models.prf.Gaussian2DPRFTuning (see the first practical).
from prfmodel.models.prf import Gaussian2DPRFTuning
from scipy.stats import norm
# Edges of the visual field covered by the stimulus (the grid stores y in grid[..., 0] and x in grid[..., 1])
(y_min, x_min), (y_max, x_max) = stimulus.grid.min(axis=(0, 1)), stimulus.grid.max(axis=(0, 1))
def fraction_inside_stimulus(mu_x: pd.Series, mu_y: pd.Series, sigma: pd.Series) -> np.ndarray:
"""Fraction of the volume of a 2D Gaussian that lies inside the visual field covered by the stimulus."""
fraction_x = norm.cdf(x_max, mu_x, sigma) - norm.cdf(x_min, mu_x, sigma)
fraction_y = norm.cdf(y_max, mu_y, sigma) - norm.cdf(y_min, mu_y, sigma)
return np.asarray(fraction_x * fraction_y)
# The Gaussians of the DoG and DN model and the names of their size parameters
gaussians = {
"DoG center": (dog_params, "sigma_center"),
"DoG surround": (dog_params, "sigma_surround"),
"DN activation": (dn_params, "sigma_activation"),
"DN normalization": (dn_params, "sigma_normalization"),
}
coverage = pd.DataFrame({
name: fraction_inside_stimulus(params["mu_x"], params["mu_y"], params[sigma_name])
for name, (params, sigma_name) in gaussians.items()
})
# Median fraction inside the stimulus, separately for the voxels for which each model is selected
print(coverage.groupby(best_model).median().reindex(model_names))
# The voxel with the highest out-of-sample R-squared among the voxels for which the DoG model is selected
voxel_index = r_squared_test["DoG"].where(best_model == "DoG").idxmax()
tuning = Gaussian2DPRFTuning()
fig, axes = plt.subplots(1, 4, figsize=(18, 4), layout="constrained")
for ax, (name, (params, sigma_name)) in zip(axes, gaussians.items()):
profile = np.asarray(tuning(stimulus, params.iloc[[voxel_index]].rename(columns={sigma_name: "sigma"})))[0]
image = ax.imshow(profile, origin="lower", extent=(x_min, x_max, y_min, y_max), cmap="inferno")
fig.colorbar(image, ax=ax, shrink=0.8)
ax.set(xlabel="x (degrees)", ylabel="y (degrees)", title=f"{name}: {coverage[name][voxel_index]:.0%} inside")
The surround and normalization Gaussians are much larger than the stimulus: The median sigma_surround is 5.6 degrees, and some estimates of sigma_surround and sigma_normalization reach tens of degrees, although the stimulus only spans about 7.9 × 5.6 degrees. For the median voxel, only 21% of the DoG surround and 35% of the DN normalization Gaussian lie inside the stimulus. Even for voxels with a pRF center inside the stimulus, only about a third of the DoG surround lies inside. The plotted voxel is one of the more favorable cases: Its pRF lies well inside the stimulus, and still about a third of its surround lies outside.
Thus, the stimulus design does not provide enough information to estimate these Gaussians well. The data only constrain the part of a Gaussian that lies inside the stimulus. A larger, stronger surround can predict almost the same response as a smaller, weaker one, so the size and strength of the surround trade off against each other.
This also determines when the DoG and DN models can fit better than the Gaussian and CSS models. Suppression only becomes visible when the bar stimulates the surround but not the center, which produces a negative response (Exercise 11). For a pRF in the middle of the stimulus, the bar always covers parts of the center and the surround at the same time, and the effect of the surround is hard to distinguish from a smaller or compressed center. The voxels for which the DoG and DN models are selected have pRFs that lie closer to the edge of the stimulus: Their center or activation Gaussian lies only 56% (DoG) or 37% (DN) inside the stimulus, compared to more than 95% for the voxels for which the Gaussian model is selected.
Summary#
Models with more free parameters that can mimic simpler models always fit the training data at least as well. Therefore, we cannot use the in-sample fit to compare models.
We compare models by their out-of-sample fit using cross-validation. When a dataset only contains a single run, we can split the run into two halves, as long as both halves sample the stimulus space well and the split leaves enough time for the BOLD response to return to baseline.
The noise ceiling estimates how well any model could possibly predict a voxel given the noise in its response. Selecting voxels by a criterion that does not depend on any of the candidate models avoids favoring one of them.
The best model can differ between voxels. When we select a model for each voxel, we should prefer simpler models when the difference is small, and keep in mind that the score of the selected model is optimistic. A fair evaluation of the selection requires data that was not used for it.
Parameters of different models are only directly comparable when they mean the same thing, such as the pRF center.
References#
Aqil, M., Knapen, T., & Dumoulin, S. O. (2021). Divisive normalization unifies disparate response signatures throughout the human visual hierarchy. Proceedings of the National Academy of Sciences, 118(46), e2108713118. https://doi.org/10.1073/pnas.2108713118
Dumoulin, S. O., & Wandell, B. A. (2008). Population receptive field estimates in human visual cortex. NeuroImage, 39(2), 647-660. https://doi.org/10.1016/j.neuroimage.2007.09.034
Kay, K. N., Winawer, J., Mezer, A., & Wandell, B. A. (2013). Compressive spatial summation in human visual cortex. Journal of Neurophysiology, 110(2), 481-494. https://doi.org/10.1152/jn.00105.2013
Zuiderbaan, W., Harvey, B. M., & Dumoulin, S. O. (2012). Modeling center-surround configurations in population receptive fields using fMRI. Journal of Vision, 12(3), 10. https://doi.org/10.1167/12.3.10