MULTI-MODAL DEEP LEARNING FOR ENHANCED MELANOMA METASTASIS DIAGNOSIS

Abstract

This study presents a novel multi-modal deep learning framework for enhancing the prediction of melanoma metastasis by integrating primary melanoma pathology images with patient demographic and clinical information. Our approach leverages a biomarker mining network, a percolation depth prediction module, and a patient information integration mechanism, culminating in a fully connected layer classifier for comprehensive metastasis risk assessment. The biomarker mining network, enhanced by a spatial attention mechanism, identifies critical biomarkers with high sensitivity (92%) and specificity (88%). The percolation depth prediction module achieves a mean absolute error (MAE) of 0.15 mm, significantly improving depth assessment accuracy. By integrating patient information through a unique hot encoding method, our model captures inter-case variations, enabling personalized predictions. The fully connected layer classifier achieves an overall accuracy of 87%, outperforming traditional methods such as Breslow and Clark grading, as well as unimodal deep learning models. Visualization techniques, including Gradient-weighted Class Activation Mapping (Grad-CAM), provide interpretable insights into the model’s decision-making process. Our results demonstrate the efficacy of multi-modal deep learning in improving melanoma metastasis diagnosis, offering a robust tool for clinical decision-making and personalized treatment planning.