UMPM-Net: A Multimodal Deep Learning Framework for Colorectal Cancer Prognosis

Colorectal cancer (CRC) is one of the most common cancers
worldwide, for which early prediction of patient outcomes is important for
personalised treatment planning and to improve survival. However, current
prediction systems are limited in the fact that they are based on singlemodality data and cannot capture the complex interactions between clinical,
imaging, and genomic factors. To overcome these shortcomings, this study
presents a Unified Multimodal Predictive Modeling Network (UMPM-Net)
which incorporates Electronic Health Records (EHRs), medical imaging
biomarkers and genomic signatures in a deep learning model. The proposed
approach uses U-Net for accurate tumor segmentation, EfficientNet for
imaging feature extraction and DenseNet for classification and survival
prediction. The multimodal features are then fused at the feature level as a
way of obtaining complementary information across the modalities.
Experiments were performed on a multi-institutional dataset of a CRC with
1200 distinct patients with paired imaging, EHR and genomic profiles.
Quantitative results show better performance than five state-of-the-art
models with Dice score 0.92 for segmentation, accuracy score 0.90, ROCAUC score 0.95 for outcome classification and concordance index (C-index)
score 0.80 for predicting survival. The ablation analysis further confirms that
the fusion of imaging, clinical and genomic modalities has a significant
impact on the predictive accuracy and clinical interpretability. In conclusion,
the proposed UMPM-Net is a comprehensive, explainable and robust
multimodal learning framework that is capable of accurate, interpretable, and
clinically relevant prognostication for colorectal cancer