Explainable multimodal deep learning for cancer segmentation and prognostic decision support
Qifeng Wang
Sichuan University Jinjiang College
Zijie Liu
Sichuan University Jinjiang College
Chunfeng Wei
Sichuan University Jinjiang College
Kailin He
Sichuan University Jinjiang College
DOI: https://doi.org/10.59429/esta.v13i3.15561
Keywords: multimodal deep learning; cancer segmentation; prognosis; explainable artificial intelligence; medical imaging; survival analysis
Abstract
This study proposes an explainable multimodal deep-learning framework for cancer segmentation and prognostic decision support, aligned with the research plan of CT/MRI/PET fusion, small-lesion and fuzzy-boundary segmentation, interpretability, generalization, and clinical use. The model combines modality-specific encoders, masked reliability-aware fusion, a boundary-sensitive segmentation branch, and a censored survival branch. Missing examinations are represented by a modality mask. Explanations are organized at spatial, modality, and prognostic-feature levels, with uncertainty-based manual review. Public PET/CT and multimodal MRI cohorts can support development; true CT-MRI-PET fusion requires patient-matched scans and outcomes. The paper defines a reproducible method and validation protocol without claiming unmeasured performance.
References
[1] Tang Y, Yang D, Li W, Roth HR, Landman BA, Xu D, et al. Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis. Proc IEEE/CVF Conf Comput Vis Pattern Recognit. 2022:20730-20740.
[2] Andrearczyk V, Oreiller V, Boughdad S, et al. Automatic Head and Neck Tumor segmentation and outcome prediction relying on FDG-PET/CT images: Findings from the second edition of the HECKTOR challenge. Med Image Anal. 2023;90:102972.
[3] Pahud de Mortanges A, Luo H, Shu SZ, et al. Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging. npj Digit Med. 2024;7:195.
[4] Wiegrebe S, Kopper P, Sonabend R, Bischl B, Bender A. Deep learning for survival analysis: a review. Artif Intell Rev. 2024;57:65.
[5] Zhang G, Ma C, Yan C, et al. Multimodal Deep Learning for Cancer Survival Prediction: A Review. Current Bioinformatics. 2025;20(4):299-322.
[6] Mohamed YA, Khoo BE, Mohd Asaari MS, Aziz ME, Ghazali FR. Decoding the black box: Explainable AI for cancer diagnosis, prognosis, and treatment planning-A state-of-the-art systematic review. Int J Med Inform. 2025;193:105689.