Practical 2: Cross-validation#

This is the notebook for the second practical of the workshop. This practical covers how to evaluate the out-of-sample model fit using cross-validation on empirical data from visual functional magnetic resonance imaging (fMRI) experiments. It uses the canonical Gaussian 2D population receptive field (pRF) model as an example and illustrates cross-validation across different runs and sessions from the same experiment.

We recommend first going through the first practical because this practical builds on package-specific basics that are covered there.

We also recommend going through this notebook before starting with the notebooks of the other practicals which build on this one.

In computational modelling, cross-validation is typically used to evaluate and compare the ability of different models to generalize their predictions to new observations. That is, from a set of candidate models, we want to select the model which can generalize best to unseen data. Here, we focus only on the cross-validation part and leave the model comparison and selection part to the next practical.

As all computational and machine learning models, pRF models can overfit the data that they are “trained” on (i.e., the data they are fit to in order to estimate their parameters). Overfitting means that the pRF model fits noise in the training data that is not present or different in unseen (i.e., test) data. It leads to a degradation in prediction performance from the training to test data. The more free parameters a model has, the more flexibly it can fit to noise in the training data and the more likely it is to overfit. While some degree of overfitting is unavoidable, we want to at least know when a pRF model overfits and whether we can trust that it is modelling a robust pattern in the data and not just noise. When comparing different models, we also want to be sure that a model leads to better predictions because it captures the underlying pattern in the data better and not just because it fits better to noise in the data (e.g., because it has more parameters).

Contents#

Installation and Setup#

First, we need to install prfmodel from PyPI. We install the package with the TensorFlow backend (for more information on the different backends, see the Backends section in the package documentation).

Please make sure to use a GPU instance when installing prfmodel and running this notebook in Colab.

!pip install "prfmodel[tensorflow]==0.1.0"

We set the backend to TensorFlow and a default for printing.

import os

import numpy as np
import pandas as pd

# Set keras backend to 'tensorflow' (this is normally the default)
os.environ["KERAS_BACKEND"] = "tensorflow"
# Print parameter DataFrames with three decimals
pd.set_option("display.precision", 3)

Loading the dataset#

In this example, we use the Center for Human Neuroscience (CHN) dataset (Chang et al., 2024, 2025) that includes BOLD responses recorded with a 3 Tesla MRI scanner while subjects were exposed to two different bar stimuli. We only use the standard (fixed) bar stimulus here.

We load the dataset which is included in the prfmodel package. The load_dataset function has a split argument that we can use to load a specific run of the experiment. This allows us to use different runs for fitting (i.e., training) and evaluating the out-of-sample fit (i.e., testing) the model. The dataset contains three runs from a single subject:

Split

Session

Stimulus

Role in this practical

ses-01_run-01

1

Sequence of bar positions A

Training

ses-02_run-01

2

Sequence of bar positions A (identical to training)

Test: same stimulus, new session

ses-01_run-02

1

Sequence of bar positions B (different from training)

Test: new stimulus, same session

The two test runs allow us to ask two different questions about how well the model generalizes: Can it predict a new measurement of the same experiment, recorded in a different session? And can it predict the response to a new stimulus that it was not fitted to?

We start with loading the training data which is the first run of the first session. We select the standard bar stimulus with the task argument. By default, load_dataset only loads the left hemisphere. On first use, every run downloads about 360 MB, so loading can take a moment.

from prfmodel.examples import load_dataset

# Downloads on first use and caches in a user data directory (see prfmodel.examples.get_data_dir)
dataset_train = load_dataset("3t-chn-visual", split="ses-01_run-01", task="fixedbar")

# Select the stimulus used for training
stimulus_train = dataset_train.stimulus

Exercise 1

Using what you learned in the first practical, inspect the stimulus design matrix and grid. How many bar passes are in the design matrix and in which directions do they run? What is the size of the stimulus grid? (hint: use prfmodel.plotting.animate_2d_prf_stimulus to create a video of the stimulus).

Solution
from IPython.display import HTML
from prfmodel.plotting import animate_2d_prf_stimulus

print(stimulus_train)
print("Grid from", stimulus_train.grid.min(axis=(0, 1)), "to", stimulus_train.grid.max(axis=(0, 1)), "(y, x)")

# In this dataset, the first row of the grid is the upper edge of the visual field
ani = animate_2d_prf_stimulus(stimulus_train, origin="upper", interval=25)

HTML(ani.to_html5_video())

The design matrix has 305 time frames and a grid of 108 x 108 cells that spans -8 to 8 degrees of visual angle in both directions. The bar moves within a circular aperture with a radius of 8 degrees. Unlike in the first practical, the first row of the grid is the upper edge of the visual field, so we set origin="upper" to show the video the right way up.

The design contains eight bar passes of 37–38 time frames each, without blank periods in between (the bar is visible from time frame 2 to 301). Some of the passes are slightly tilted against the horizontal and vertical axes.

# Your code goes here

Before fitting a pRF model to the data, we inspect the BOLD response timecourses. Note that this dataset contains more timecourses than the 7t-aot-visual dataset from the first practical and each timecourse corresponds to a vertex on the surface (not to a voxel in the volume). This means that we don’t need to project timecourse statistics and model parameters onto the surface for visualization.

response_train = dataset_train.response
response_train.shape

Exercise 2

Inspect the BOLD timecourses. What is the unit of the BOLD response? Can you see the response to the bar sweeps when visualizing them?

Solution
print("Largest absolute mean of a timecourse:", np.abs(response_train.mean(axis=1)).max())
print("Median standard deviation of the timecourses:", np.median(response_train.std(axis=1)))

Every timecourse has a mean of zero, and the timecourses vary by several hundred units. Thus, the response is not in percent signal change like in the first practical, but in the arbitrary units of the scanner signal with the mean of each timecourse subtracted.

We plot the timecourses of all vertices as a heatmap, sorted by their standard deviation (SD). The heatmap has many more vertices than the figure has pixel rows, so it can only show a fraction of them. Therefore, we also zoom in on the 500 most variable vertices at the bottom of the heatmap.

import matplotlib.pyplot as plt
from prfmodel.plotting import plot_response_heatmap

# All vertices, sorted by the standard deviation of their response (the most variable at the bottom)
sorted_vertices = np.argsort(response_train.std(axis=1))

fig, axes = plt.subplots(2, 1, figsize=(10, 8), sharex=True, layout="constrained")

for ax, vertices, title in zip(
    axes, [sorted_vertices, sorted_vertices[-500:]], ["All vertices", "500 most variable vertices"],
):
    plot_response_heatmap(
        response_train[vertices],
        vmin=-300,
        vmax=300,
        cmap="RdBu_r",
        xlabel="",
        ylabel="Vertex (sorted by SD)",
        colorbar_label="BOLD response",
        ax=ax,
    )
    ax.set_title(title)

axes[-1].set_xlabel("Time frame (in TR)");

Only for a small fraction of the vertices. In the heatmap of all vertices, the rows at the top have a constant response, and the other rows mostly show noise without a clear pattern: Most vertices of the hemisphere do not respond to the visual stimulus. Among the 500 most variable vertices, we can see vertical stripes where many vertices respond at the same time frames, roughly once for each of the eight bar passes. However, some of the most variable vertices (e.g., the rows at the very bottom) fluctuate quickly from one time frame to the next without following the bar passes. Their variability is caused by noise, so a large SD alone does not tell us that a vertex responds to the stimulus. Below, we will select the vertices in the visual cortex.

# Your code goes here

Exercise 3

Using the helper function below, plot the standard deviation (SD) of each vertex on an inflated surface. Which areas of the brain have the largest variability in BOLD response? Can you find V1 on the surface?

from plotting_helpers import make_surface_plot

# The surface is the same for all splits of the dataset, so we take it from the training data
plot_surf_stat_map_helper = make_surface_plot(dataset_train.mesh)
Solution
plot_surf_stat_map_helper(
    response_train.std(axis=1), vmin=0.0, vmax=150.0, title="BOLD response standard deviation",
);

The largest variability lies in the occipital lobe at the back of the brain, especially on the medial side around the calcarine sulcus and at the occipital pole. This is where V1 lies. There are also patches with a large SD in other places, for example, around the medial wall (the black area without cortex) and on the lateral surface. Here, the variability is likely not caused by the visual stimulus but by noise.

# Your code goes here

Selecting vertices in visual cortex#

The response contains a timecourse for every vertex of the left hemisphere, but the pRF model only makes sense for vertices that respond to visual stimulation. Fitting the model to all vertices would also take a long time. The dataset includes a region of interest (ROI) label for each vertex, which tells us whether the vertex belongs to one of the visual areas V1, V2, or V3 (or to none of them).

roi_index = dataset_train.roi_index  # One ROI label per vertex

dataset_train.roi_mapping  # Which ROI each label refers to
# Only color vertices that belong to one of the ROIs; at the borders between ROIs, merged vertices get in-between colors
plot_surf_stat_map_helper(
    np.where(roi_index > 0, roi_index, np.nan),
    vmin=1,
    vmax=3,
    cmap="viridis",
    title="ROIs (1 = V1, 2 = V2, 3 = V3)",
);

Exercise 4

Compare the ROIs to the map of the response SD that you plotted above. Do the vertices with the largest variability lie inside the ROIs?

Solution

Partly. The ROIs cover the occipital pole and the area around the calcarine sulcus, where many vertices have a large SD because they respond to the bar. However, not all vertices inside the ROIs are highly variable (e.g., the parts of V1–V3 further to the front, which represent the periphery of the visual field beyond the stimulus), and many highly variable vertices lie outside the ROIs. The SD does not distinguish between variability caused by the stimulus and variability caused by noise, so it is not a good criterion for selecting vertices on its own.

We select the vertices that belong to one of the ROIs. We also make sure that their responses vary over time: A vertex with a constant response contains no information about the stimulus.

# Vertices in V1, V2, or V3 with a response that varies over time
response_is_valid = (roi_index > 0) & (response_train.std(axis=1) > 0.0)

response_train_valid = response_train[response_is_valid]


def fill_valid_vertices(values: np.ndarray) -> np.ndarray:
    """Fill values for the valid vertices into an array with one value per vertex, using NaN for all other vertices."""
    values_full = np.full((response_is_valid.shape[0],), fill_value=np.nan)
    values_full[response_is_valid] = values
    return values_full


response_train_valid.shape

Exercise 5

How many vertices of the whole hemisphere have a constant response (hint: response_train.std(axis=1) == 0.0) and where do they lie on the surface? Are any of them inside the ROIs?

Solution
is_constant = response_train.std(axis=1) == 0.0

print("Vertices with a constant response:", is_constant.sum(), "of", len(is_constant))
print("Of those inside the ROIs:", np.sum(is_constant & (roi_index > 0)))

# Only color the vertices with a constant response
plot_surf_stat_map_helper(
    np.where(is_constant, 1.0, np.nan), vmin=0.0, vmax=1.0, title="Vertices with a constant response",
);

About 15,600 of the 221,709 vertices (7%) have a constant response. Most of them lie in the medial wall, where the surface closes the hemisphere but there is no cortex. Others lie along the top edge of the hemisphere, which was probably outside the part of the brain that was scanned, and a few are scattered over the lateral surface. None of them lie inside the ROIs.

# Your code goes here

Fitting the model#

We create a canonical Gaussian 2D pRF model with a custom impulse response submodel to fit to the training data. The repetition time (TR) for this dataset is 1.2 seconds.

from prfmodel.impulse import DerivativeTwoGammaImpulse
from prfmodel.models.prf import Gaussian2DPRFModel

# Define repetition time (TR)
tr = 1.2

# Create custom impulse model
impulse_model = DerivativeTwoGammaImpulse(resolution=tr)

# Define pRF model with custom impulse response submodel
prf_model = Gaussian2DPRFModel(
    impulse_model=impulse_model,
)

Exercise 6 (optional)

Why do we need to create a custom impulse response model with the TR of the dataset? Read the relevant Important Details section.

Solution

The impulse response describes how the BOLD signal unfolds over time in seconds: It peaks about 5 seconds after neural activity and returns to baseline after about 20–30 seconds. The model convolves the impulse response with the predicted neuron population activity, which has one value per time frame. For the convolution to be correct, the impulse response must be sampled with the same time step as the stimulus and the data, that is, once per TR. If we sampled it every second instead, every sample of the impulse response would be interpreted as one time frame of 1.2 seconds. The predicted BOLD response would then be stretched in time: For example, the peak 5 seconds after the activity would end up 5 time frames (6 seconds) later.

To keep things simple, we only use grid search and least-squares in this example to fit the pRF model to the training data. We start by defining ranges of the grid search.

from prfmodel.fitters import grid_values_2d_prf

grid_param_ranges = grid_values_2d_prf(
    stimulus_train,  # We use the stimulus from the training data
    num_mu=21,
    mu_extent=2.0,  # Span twice the visual field covered by the stimulus
    num_sigma=10,
    sigma_range=(0.05, 4.0),  # Log-spaced by default
) | {  # Add constant parameters
    "weight_deriv": [-0.5],
    "baseline": [0.0],
    "amplitude": [1.0],
}

grid_param_ranges

Then, we create the GridFitter and run the grid search on the timecourses of the selected vertices.

from prfmodel.fitters import GridFitter

# Create grid fitter object
grid_fitter = GridFitter(
    model=prf_model,
    stimulus=stimulus_train,
    compile_step=True,  # Setting 'compile_step=True' speeds up the fitting
)

# Run grid search
grid_history, grid_params = grid_fitter.fit(
    data=response_train_valid,  # Only the selected vertices
    parameter_values=grid_param_ranges,
    batch_size=20,
)

Exercise 7

Inspect the results from the grid search. Are there any vertices for which the center of the pRF lies exactly on or outside the stimulus grid (hint: use prfmodel.plotting.plot_2d_prf_centers)? Are there any vertices for which the pRF size (sigma) lies on the floor or ceiling of the sigma range (hint: use prfmodel.plotting.plot_grid_parameter_distribution)? How should the parameters of these vertices be handled or interpreted?

Solution
import matplotlib.pyplot as plt
from prfmodel.plotting import plot_2d_prf_centers, plot_grid_parameter_distribution

fig, axes = plt.subplots(1, 2, figsize=(10, 4.5), layout="constrained")

plot_2d_prf_centers(grid_params, grid_param_ranges, stimulus=stimulus_train, ax=axes[0])
axes[0].set(xlabel="mu_x (in degrees)", ylabel="mu_y (in degrees)", title="pRF centers")

plot_grid_parameter_distribution(grid_params, grid_param_ranges, "sigma", ax=axes[1])
axes[1].set(xlabel="sigma (in degrees)", title="pRF sizes");
grid_min_xy, grid_max_xy = stimulus_train.grid.min(axis=(0, 1)), stimulus_train.grid.max(axis=(0, 1))
edge_tolerance = 1e-3  # in degrees

is_on_or_outside_edge = (
    (grid_params["mu_x"] < grid_min_xy[1] + edge_tolerance)
    | (grid_params["mu_x"] > grid_max_xy[1] - edge_tolerance)
    | (grid_params["mu_y"] < grid_min_xy[0] + edge_tolerance)
    | (grid_params["mu_y"] > grid_max_xy[0] - edge_tolerance)
)

print("Fraction with a center on or outside the edge of the stimulus:", is_on_or_outside_edge.mean())
print("Fraction with sigma at the floor:", np.isclose(grid_params["sigma"], np.min(grid_param_ranges["sigma"])).mean())
print("Fraction with sigma at the ceiling:", np.isclose(grid_params["sigma"], np.max(grid_param_ranges["sigma"])).mean())

Yes. Most pRF centers lie inside the stimulus and in the right half of the visual field, which the left hemisphere represents. However, about a third of the vertices have a center on the edge of or outside the stimulus. About 11% of the vertices have a sigma at the floor (0.05 degrees) and about 7% at the ceiling (4 degrees) of the grid.

We should not interpret the parameters of these vertices at face value:

  • Center on or outside the stimulus edge: The stimulus never covers the pRF center, so the data only constrain the center weakly. Many of these vertices also have a response that is dominated by noise. We usually exclude them from the interpretation.

  • sigma at the ceiling: The best fitting size may lie beyond the grid, so the estimate is only a lower bound. We can extend the grid or exclude these vertices.

  • sigma at the floor: The size is smaller than what the stimulus can resolve (e.g., the width of the bar), so the estimate is only an upper bound. Very small pRFs are also often a sign of a noisy fit.

# Your code goes here

After the grid search, we fit the baseline and amplitude parameters of the model with least-squares.

from prfmodel.fitters import LeastSquaresFitter

# Create least-squares fitter
ls_fitter = LeastSquaresFitter(
    model=prf_model,
    stimulus=stimulus_train,
)

# Run least squares fit
ls_history, ls_params = ls_fitter.fit(
    data=response_train_valid,
    parameters=grid_params,
    slope_name="amplitude",  # Names of parameters to be optimized with least squares
    intercept_name="baseline",
    batch_size=200,
)

Evaluating the model fit in-sample#

Before we look at the out-of-sample fit of the pRF model, we look at its fit on the training data. For that, we make predictions with the training stimulus and the training parameters.

from prfmodel.utils import batched

predict_batched = batched(prf_model)

# Make predictions with the estimated parameters
pred_response_train = np.asarray(predict_batched(stimulus_train, ls_params, batch_size=200))

We compare the training data predictions against the observed BOLD timecourses using R-squared.

from prfmodel.utils import calculate_r_squared

r_squared_train = calculate_r_squared(response_train_valid, pred_response_train)  # One score per selected vertex
r_squared_train.shape

Exercise 8

Visualize the distribution of the training R-squared values (hint: use prfmodel.plotting.plot_r_squared_hist). What fraction of the selected vertices has an R-squared above 0.3? Plot the R-squared on the inflated surface (hint: use fill_valid_vertices). Where do the well-fitting vertices lie?

Solution
from prfmodel.plotting import plot_r_squared_hist

plot_r_squared_hist(r_squared_train);

print("Fraction of vertices with R-squared > 0.3:", np.mean(r_squared_train > 0.3))
plot_surf_stat_map_helper(
    fill_valid_vertices(r_squared_train), vmin=0.0, vmax=1.0, title="Variance explained (R-squared, training data)",
);

About 38% of the selected vertices have an R-squared above 0.3. The well-fitting vertices lie in the posterior parts of V1–V3 around the occipital pole, which represent the central part of the visual field covered by the stimulus. Towards the front of the ROIs on the medial surface, which represents the periphery of the visual field beyond the stimulus, the R-squared drops close to zero.

# Your code goes here

Same stimulus, new session#

To assess the out-of-sample model fit, we first evaluate its predictions on the first run of the second session. The subject saw exactly the same stimulus as in the training run. What differs is the noise in the measurement and everything that changes between sessions, for example, the position of the head in the scanner, the alignment of the data to the anatomy, and the state of the subject. This tells us whether the model captures a pattern in the responses that we can reproduce in a new measurement, and not just the noise of the training run.

We load the dataset for this run using the split argument.

dataset_test_ses = load_dataset("3t-chn-visual", split="ses-02_run-01", task="fixedbar")

stimulus_test_ses = dataset_test_ses.stimulus

response_test_ses = dataset_test_ses.response

# Select the same vertices as for the training data
response_test_ses_valid = response_test_ses[response_is_valid]

Exercise 9

Check that the stimulus of this test dataset is identical to the training stimulus (hint: use np.array_equal on the design matrices). What does this imply for the model predictions for training and test data? Where can differences in prediction performance between training and test data then come from?

Solution
print("Identical design matrices:", np.array_equal(stimulus_train.design, stimulus_test_ses.design))
print("Identical grids:", np.array_equal(stimulus_train.grid, stimulus_test_ses.grid))

Both are identical. Thus, the model with the training parameters makes exactly the same predictions for the training and the test data. Any difference in prediction performance must come from differences in the observed responses:

  • Noise: Thermal and physiological noise (e.g., breathing and heartbeat) differ between the runs. The model is partly fitted to the noise of the training run, which does not repeat in the test run.

  • Session: The position of the head in the scanner, the alignment of the data to the anatomy, and the scanner itself (e.g., drifts and calibration) differ between sessions.

  • Subject: The state of the subject, for example, attention, fatigue, and how well they fixated, can differ between sessions.

# Your code goes here

Exercise 10

Visualize the BOLD response from this test dataset. Do you see any fundamental differences compared to the training data?

Solution

We plot the responses of the selected vertices in both runs next to each other with the same color range.

import matplotlib.pyplot as plt
from prfmodel.plotting import plot_response_heatmap

fig, axes = plt.subplots(1, 2, figsize=(12, 5), sharey=True, layout="constrained")

for ax, response, title in zip(
    axes, [response_train_valid, response_test_ses_valid], ["Training (ses-01_run-01)", "Test (ses-02_run-01)"],
):
    plot_response_heatmap(
        response,
        vmin=-300,
        vmax=300,
        cmap="RdBu_r",
        xlabel="Time frame (in TR)",
        ylabel="Vertex index",
        colorbar_label="BOLD response",
        ax=ax,
    )
    ax.set_title(title)

print("Median SD (training):", np.median(response_train_valid.std(axis=1)))
print("Median SD (test):", np.median(response_test_ses_valid.std(axis=1)))

No. Both runs show the same pattern of responses to the bar passes at the same time frames, and the variability of the responses is similar (a median SD of about 77 vs. 74). The runs only differ in the details, that is, in the noise.

# Your code goes here

We make model predictions using the stimulus of the test data and the parameters of the training data. We can reuse the batched model object because nothing in it is specific to the training or test data. We compare the test predictions against the observed test timecourses.

pred_response_test_ses = np.asarray(predict_batched(stimulus_test_ses, ls_params, batch_size=200))

r_squared_test_ses = calculate_r_squared(response_test_ses_valid, pred_response_test_ses)
r_squared_test_ses.shape

Exercise 11

Compare the training against the test R-squared (hint: use prfmodel.plotting.plot_r_squared_comparison). Also compare the predicted vs. observed timecourses for the best fitting vertices (hint: use prfmodel.plotting.plot_observed_predicted). Can you explain the differences in prediction performance between training and test data? Can you explain why some vertices have negative out-of-sample R-squared scores?

Solution
import matplotlib.pyplot as plt
from prfmodel.plotting import plot_observed_predicted, plot_r_squared_comparison

plot_r_squared_comparison(
    r_squared_train,
    r_squared_test_ses,
    xlabel="R-squared (training: ses-01_run-01)",
    ylabel="R-squared (test: ses-02_run-01)",
)

print("Mean R-squared (training):", r_squared_train.mean())
print("Mean R-squared (test):", r_squared_test_ses.mean())
print("Fraction of vertices with a negative test R-squared:", np.mean(r_squared_test_ses < 0.0))

Neighboring vertices on the surface have almost the same response, so we do not plot the three best vertices but the best vertex, the vertex at the top 1%, and the vertex at the top 10% of the training R-squared. We also show the observed training response in gray. The prediction is the same for both runs.

# Vertices sorted from the highest to the lowest training R-squared
order = np.argsort(r_squared_train)[::-1]
example_vertices = order[[0, len(order) // 100, len(order) // 10]]

fig, axes = plt.subplots(3, 1, figsize=(10, 8), sharex=True, layout="constrained")

for ax, vertex in zip(axes, example_vertices):
    plot_observed_predicted(
        response_test_ses_valid[vertex], pred_response_test_ses[vertex], observed_label="Observed (test)", ax=ax,
    )
    ax.plot(response_train_valid[vertex], color="gray", linewidth=1, alpha=0.6, label="Observed (training)")
    ax.legend(frameon=False, loc="upper right")
    ax.set(
        xlabel="",
        ylabel="BOLD response",
        title=f"R-squared training = {r_squared_train[vertex]:.2f}, test = {r_squared_test_ses[vertex]:.2f}",
    )

axes[-1].set_xlabel("Time frame (in TR)");

For most vertices, the test R-squared is somewhat lower than the training R-squared (the points lie below the identity line), and the mean R-squared drops from about 0.22 to 0.19. The model was fitted to the training run, including part of its noise. The test run contains different noise, so this part of the fit does not carry over. For the best vertices, the predicted peaks match the peaks in both runs, but the noise between the peaks differs.

The R-squared compares the squared errors of the prediction with the variance of the observed response around its mean. When the prediction is worse than simply predicting the mean of the test response, the R-squared becomes negative. This cannot happen for the training data because least-squares can always set the amplitude to zero and the baseline to the mean. It can happen for the test data, where the parameters are fixed. About a third of the vertices have a slightly negative test R-squared. They almost all have a training R-squared close to zero: The model fitted noise in the training run, and this noise prediction only adds errors in the test run.

# Your code goes here

For full leave-one-out cross-validation, we would repeat the out-of-sample prediction but swap the training and the test data: The split ses-02_run-01 would become the training and ses-01_run-01 the test data. We would then average the out-of-sample R-squared scores across both directions (also called folds) for each vertex. In this way, our generalization assessment does not depend on the order of the two runs which we assume to be independent.

Exercise 12 (optional)

Compute the out-of-sample R-squared in reverse order (using ses-02_run-01 as training and ses-01_run-01 as test data). How does out-of-sample R-squared differ between the two folds?

Solution

We fit the model to the first run of the second session in the same way as to the training data and evaluate it on the first run of the first session.

from prfmodel.plotting import plot_r_squared_comparison

_, grid_params_rev = GridFitter(model=prf_model, stimulus=stimulus_test_ses, compile_step=True).fit(
    data=response_test_ses_valid,
    parameter_values=grid_param_ranges,
    batch_size=20,
)

_, ls_params_rev = LeastSquaresFitter(model=prf_model, stimulus=stimulus_test_ses).fit(
    data=response_test_ses_valid,
    parameters=grid_params_rev,
    slope_name="amplitude",
    intercept_name="baseline",
    batch_size=200,
)

pred_response_rev = np.asarray(predict_batched(stimulus_train, ls_params_rev, batch_size=200))
r_squared_test_rev = calculate_r_squared(response_train_valid, pred_response_rev)

plot_r_squared_comparison(
    r_squared_test_ses,
    r_squared_test_rev,
    xlabel="Out-of-sample R-squared (fold 1)",
    ylabel="Out-of-sample R-squared (fold 2)",
)

# Average across both folds for every vertex
r_squared_cv = (r_squared_test_ses + r_squared_test_rev) / 2.0

print("Mean out-of-sample R-squared (fold 1):", r_squared_test_ses.mean())
print("Mean out-of-sample R-squared (fold 2):", r_squared_test_rev.mean())
print("Mean cross-validated R-squared:", r_squared_cv.mean())

The two folds give very similar results: The mean out-of-sample R-squared is about 0.19 in both directions, and the scores of individual vertices are highly correlated between the folds. Small differences remain because each fold is fitted to and evaluated on different noise. For this dataset, the order of the runs hardly matters, but averaging both folds still gives a more reliable estimate than a single fold.

# Your code goes here

New stimulus, same session#

Next, we evaluate the model predictions on the second run of the first session. This run was recorded in the same session as the training run, but the bar moves through the visual field in a different sequence. This is a different test because the model must predict the response to a stimulus that it was not fitted to. A pRF model that captures how a vertex responds to stimulation in different parts of the visual field should be able to do that. In contrast, a model that only fits the timecourse of the training run (including its noise) would fail.

dataset_test_run = load_dataset("3t-chn-visual", split="ses-01_run-02", task="fixedbar")

stimulus_test_run = dataset_test_run.stimulus

response_test_run = dataset_test_run.response

# Select the same vertices as for the training data
response_test_run_valid = response_test_run[response_is_valid]

Exercise 13

Compare the stimulus of this test dataset against the training stimulus (hint: create a video of both stimuli). How does the sequence of bar positions differ? Compute and evaluate the out-of-sample prediction performance of the pRF model for this dataset.

Solution
from IPython.display import HTML, display
from prfmodel.plotting import animate_2d_prf_stimulus

for stimulus, title in [(stimulus_train, "Training (ses-01_run-01)"), (stimulus_test_run, "Test (ses-01_run-02)")]:
    ani = animate_2d_prf_stimulus(stimulus, title=title, origin="upper", interval=25)
    display(HTML(ani.to_html5_video()))

Both stimuli have the same timing: eight bar passes of 37–38 time frames each without blank periods. However, the passes run in different directions and in a different order. For example, the first pass of the test run moves from the lower left to the upper right instead of from the upper left to the lower right, and the second pass moves from left to right instead of from right to left. Hence, the bar is at a different position at almost every time frame.

from prfmodel.plotting import plot_r_squared_comparison

pred_response_test_run = np.asarray(predict_batched(stimulus_test_run, ls_params, batch_size=200))
r_squared_test_run = calculate_r_squared(response_test_run_valid, pred_response_test_run)

plot_r_squared_comparison(
    r_squared_train,
    r_squared_test_run,
    xlabel="R-squared (training: ses-01_run-01)",
    ylabel="R-squared (test: ses-01_run-02)",
)

print("Mean R-squared (training):", r_squared_train.mean())
print("Mean R-squared (test, new stimulus):", r_squared_test_run.mean())

The model generalizes to the new stimulus: Vertices with a high training R-squared also have a high test R-squared. Thus, the model captures how the vertices respond to stimulation at different positions in the visual field and not only the timecourse of the training run. However, the test R-squared is lower than the training R-squared (a mean of about 0.17 vs. 0.22).

# Your code goes here

Exercise 14

Compare the out-of-sample prediction performance for this test dataset to the previous test dataset. For which test dataset is it better? Think about what differs between the training data and each of the test datasets (noise, session, stimulus) and how this can explain the difference.

Solution
from prfmodel.plotting import plot_r_squared_comparison

plot_r_squared_comparison(
    r_squared_test_ses,
    r_squared_test_run,
    xlabel="R-squared (same stimulus, new session)",
    ylabel="R-squared (new stimulus, same session)",
)

pd.DataFrame({
    "training": r_squared_train,
    "same stimulus, new session": r_squared_test_ses,
    "new stimulus, same session": r_squared_test_run,
}).agg(["mean", "median"])

The prediction performance is better for the test run with the same stimulus in a new session (a mean R-squared of about 0.19) than for the test run with a new stimulus in the same session (about 0.17).

  • Same stimulus, new session: Only the noise and the session differ. The prediction is identical to the training prediction, so the model only has to reproduce the same stimulus-driven response in a new measurement. Any error in the pRF estimates that does not matter for the training stimulus does not matter for the test run either.

  • New stimulus, same session: The noise and the stimulus differ, but the session is the same. The model has to predict the response to new bar positions. Errors in the pRF estimates and in the assumptions of the model (e.g., the Gaussian shape of the pRF or the impulse response) that were hidden by the specific training stimulus now show up.

For this dataset, generalizing to a new stimulus is harder than generalizing to a new session, even though the new session adds more differences in the measurement.

# Your code goes here

Fitting and interpreting the final model#

Cross-validation is primarily used for comparing the out-of-sample performance between different candidate models and selecting the model that best generalizes to new observations. Model comparison and selection will be part of the next practical.

Once we have selected a model, our goal changes: We no longer want to evaluate the model but to estimate its parameters as precisely as possible so that we can interpret them. For that, we want to use as much data with the least amount of noise that we can get, including the data that we held out for testing before. This means that there is no held-out data left to evaluate the final fit, and that its in-sample R-squared is optimistic for the same reasons as above. We accept this because cross-validation has already told us how well this model generalizes to new data.

For the CHN dataset, we can for example average the BOLD responses of the two runs that show the identical stimulus (the first runs of both sessions) and fit the pRF model on the averaged responses.

# Average the first runs of both sessions, which show the identical stimulus
response_avg = np.stack([
    response_train_valid,
    response_test_ses_valid,
], axis=-1).mean(axis=-1)

response_avg.shape

Exercise 15

Compare the averaged responses against the responses from only the first session. Do you observe differences in the noise level?

Solution

We plot both responses for the vertex with the highest training R-squared. To quantify the noise, we use the difference between the two sessions: It only contains noise because the stimulus-driven response is the same in both sessions.

import matplotlib.pyplot as plt

vertex = np.argmax(r_squared_train)

fig, ax = plt.subplots(figsize=(10, 3), layout="constrained")
ax.plot(response_train_valid[vertex], color="gray", linewidth=1, label="First session (ses-01_run-01)")
ax.plot(response_avg[vertex], color="black", linewidth=1.5, label="Average of both sessions")
ax.legend(frameon=False, loc="upper right")
ax.set(xlabel="Time frame (in TR)", ylabel="BOLD response")

# For independent noise with the same SD in both sessions, the difference between the sessions has sqrt(2) times
# the noise SD of a single session and 2 times the noise SD of the average
session_difference_sd = (response_train_valid - response_test_ses_valid).std(axis=1)

print("Median noise SD (single session):", np.median(session_difference_sd / np.sqrt(2.0)))
print("Median noise SD (average):", np.median(session_difference_sd / 2.0))

Yes. The averaged response is smoother: The peaks for the bar passes stay, but the fluctuations between them become smaller. Averaging two sessions with independent noise reduces the noise SD by a factor of \(\sqrt{2}\) (from about 62 to 44 units for the median vertex).

# Your code goes here

Exercise 16

Why is averaging the responses across sessions a valid choice for this dataset?

Solution

Both runs show the identical stimulus with the same timing, and the responses of both sessions are given for the same vertices of the same surface. Thus, the stimulus-driven part of the response is the same in both runs at every time frame, whereas the noise is independent between the sessions. Averaging keeps the stimulus-driven response and reduces the noise (for independent noise with equal variance, the noise variance is halved). This assumes that the pRFs of the subject do not change between the sessions and that the data of both sessions are well aligned to the same anatomy.

Exercise 17

Why can we average the first run of the first session with the first run of the second session, but not with the second run of the first session?

Solution

The second run of the first session shows a different sequence of bar positions. At the same time frame, the bar is at a different position than in the first run, so the stimulus-driven responses of the two runs differ. Averaging them would mix the responses to two different stimuli into a timecourse that does not belong to any stimulus, and the pRF model could not predict it with either design matrix. To use both runs, we can instead fit the model to both runs at the same time, for example, by concatenating their design matrices and responses along the time dimension.

Exercise 18

Fit the Gaussian 2D pRF model to the averaged responses (response_avg) using grid search and least-squares. Evaluate the model predictions on the averaged responses. How does the in-sample R-squared compare to the in-sample R-squared of the model fitted to only the first run of the first session?

Solution

Both runs have the same stimulus, so we use the training stimulus.

from prfmodel.plotting import plot_r_squared_comparison

_, grid_params_avg = GridFitter(model=prf_model, stimulus=stimulus_train, compile_step=True).fit(
    data=response_avg,
    parameter_values=grid_param_ranges,
    batch_size=20,
)

_, ls_params_avg = LeastSquaresFitter(model=prf_model, stimulus=stimulus_train).fit(
    data=response_avg,
    parameters=grid_params_avg,
    slope_name="amplitude",
    intercept_name="baseline",
    batch_size=200,
)

pred_response_avg = np.asarray(predict_batched(stimulus_train, ls_params_avg, batch_size=200))
r_squared_avg = calculate_r_squared(response_avg, pred_response_avg)

plot_r_squared_comparison(
    r_squared_train,
    r_squared_avg,
    value_range=(0.0, 1.0),
    xlabel="In-sample R-squared (first session)",
    ylabel="In-sample R-squared (average)",
)

print("Mean in-sample R-squared (first session):", r_squared_train.mean())
print("Mean in-sample R-squared (average):", r_squared_avg.mean())

The in-sample R-squared is higher for the averaged responses (a mean of about 0.29 vs. 0.22), and it increases for most vertices. The stimulus-driven response is the same in both runs, but the noise is reduced, so the model can explain a larger fraction of the variance. Note that this in-sample R-squared is still optimistic because the model is partly fitted to the remaining noise.

# Your code goes here

Exercise 19

Subset the model parameter estimates using an R-squared threshold. Plot the parameter subset on the inflated surface (hint: set the estimates of vertices below the threshold to np.nan and use fill_valid_vertices to get one value for every vertex on the surface). Do the results align with your findings in the first practical?

Solution

We define a function that plots the polar angle, eccentricity, and size of the pRFs of a subset of the vertices, so that we can reuse it in the next exercise.

from prfmodel.utils import calculate_eccentricity, calculate_polar_angle


def plot_prf_maps(params: pd.DataFrame, is_shown: np.ndarray, label: str) -> None:
    """Plot the polar angle, eccentricity, and size of the pRFs of the shown vertices on the surface."""
    mu_x = np.where(is_shown, params["mu_x"], np.nan)
    mu_y = np.where(is_shown, params["mu_y"], np.nan)
    sigma = np.where(is_shown, params["sigma"], np.nan)

    plot_surf_stat_map_helper(
        fill_valid_vertices(calculate_polar_angle(mu_x, mu_y)), cyclic=True, title=f"Polar angle ({label})",
    )
    plot_surf_stat_map_helper(
        fill_valid_vertices(calculate_eccentricity(mu_x, mu_y)), vmin=0.0, vmax=8.0, title=f"Eccentricity ({label})",
    )
    plot_surf_stat_map_helper(fill_valid_vertices(sigma), vmin=0.0, vmax=4.0, title=f"pRF size ({label})")


r_squared_threshold = 0.3
is_above_threshold = r_squared_avg > r_squared_threshold

print("Fraction of vertices above the threshold:", is_above_threshold.mean())

plot_prf_maps(ls_params_avg, is_above_threshold, f"in-sample R-squared > {r_squared_threshold}")

Yes, the maps show the same organization as in the first practical:

  • Polar angle: The left hemisphere represents the right visual field, so the polar angles lie between \(-\pi/2\) and \(\pi/2\). The upper visual field (positive angles) is represented below the calcarine sulcus and the lower visual field (negative angles) above it.

  • Eccentricity: The eccentricity increases from the occipital pole (center of the visual field) towards the front of the medial surface (periphery). Many vertices at the front of the ROIs have an eccentricity at or beyond the edge of the stimulus (8 degrees).

  • pRF size: The pRF size increases with eccentricity.

# Your code goes here

For thresholding vertices for which we can interpret the parameters, we face a similar problem as for model selection: We want to select those vertices for which the pRF model robustly estimates the parameters without overfitting to noise in the data. Thus, we should not rely on R-squared based on the training data to select vertices for interpretation. Instead, we can for example use out-of-sample R-squared from cross-validation.

Exercise 20

Use a threshold based on test R-squared on the ses-01_run-02 split to select vertices to plot on the inflated surface. Do you see any differences from the results of the previous exercise?

Solution

We use the test R-squared of the model fitted to the training run (r_squared_test_run, see Exercise 13) to select the vertices, but still plot the parameters estimated from the averaged responses.

is_above_test_threshold = r_squared_test_run > r_squared_threshold

print("Fraction of vertices above the threshold:", is_above_test_threshold.mean())
print("Selected by both thresholds:", np.mean(is_above_threshold & is_above_test_threshold))
print("Selected only by the in-sample threshold:", np.mean(is_above_threshold & ~is_above_test_threshold))
print("Selected only by the out-of-sample threshold:", np.mean(~is_above_threshold & is_above_test_threshold))

plot_prf_maps(ls_params_avg, is_above_test_threshold, f"test R-squared > {r_squared_threshold}")

The maps show the same organization, but fewer vertices pass the threshold (about 31% instead of 48%). Almost all vertices selected by the out-of-sample threshold are also selected by the in-sample threshold, but a third of the vertices selected by the in-sample threshold do not generalize well enough to the new stimulus. Many of them lie near the occipital pole and have small pRFs close to the center of the visual field, so there are more gaps in this region of the maps. Their in-sample R-squared is partly due to fitted noise, or their small pRFs are sensitive to small differences between runs, for example, in fixation. With the out-of-sample threshold, we only interpret the parameters of vertices for which the model makes reliable predictions for new data.

# Your code goes here

Summary#

  • The in-sample (training) R-squared is an optimistic estimate of how well a pRF model describes the data because the model is partly fitted to noise in the training data.

  • Cross-validation evaluates the model on data that it was not fitted to. What the test data has in common with the training data determines which kind of generalization we test: A new session with the same stimulus tests whether the model captures a pattern that we can reproduce in a new measurement; a new stimulus tests whether the model captures how a vertex responds to stimulation in different parts of the visual field.

  • The out-of-sample R-squared also depends on the noise in the test data. It can be negative when the model predicts the test data worse than the mean of the test data.

  • We use cross-validation to compare models and to select vertices for which we can trust the parameter estimates. To interpret the parameters, we fit the selected model to all comparable data.

References#

Chang, K., Fine, I., & Boynton, G. M. (2024). CHN Retinotopic Mapping Dataset. OpenNeuro. https://doi.org/10.18112/openneuro.ds004698.v1.0.1

Chang, K., Fine, I., & Boynton, G. M. (2025). Improving the reliability and accuracy of population receptive field measures using a logarithmically warped stimulus. Journal of Vision, 25(1), 5. https://doi.org/10.1167/jov.25.1.5