Fast multimodal image fusion with deep 3D convolutional networks for neurosurgical guidance – A preliminary study

Multimodality fusion in neurosurgical guidance aids neurosurgeons in making critical clinical decisions regarding safe maximal resection of tumors. It is challenging to have registration methods that automatically update pre-surgical MRI on intra-operative ultrasound, adjusting for the brain-shift for surgical guidance. A 3D deep learning-based convolutional network was developed for fast, multimodal alignment of pre-surgical MRI and intra-operative ultrasound volumes. The neural network is a combination of some well-known deep-learning architectures like FlowNet, Spatial Transformer Networks and UNet to achieve fast alignment of multimodal images. The CuRIOUS 2018 challenge training data was used to evaluate the accuracy of the developed method.