Video C3D features learned by deep network correlate with functional MRI signal variation associated with the video

To gain further insights into the mechanisms of deep network learning from the perspective of brain imaging, we compared spatio-temporal features of video segments extracted via a 3-dimensional convolutional network (3D ConvNets) with video representations in human brain characterized by functional MRI signal variation when viewing video segments. We found correlations between C3D features and fMRI signal variation in brain regions selectively activated by video segments after the optimization of time lag due to the hemodynamic response function (HRF). Distinct activation patterns were also revealed by functional MRI for video segments classified as different classes of activity.