Explainable Multimodal Artificial Intelligence for Early Disease Detection Using Clinical, Laboratory, and Medical Imaging Data

Abstract

Early disease detection is vital for better outcomes and timely intervention. We developed an explainable multimodal AI framework combining clinical and imaging data. We evaluated clinical-only, imaging-only, and multimodal models, comparing them on accuracy, precision, recall, specificity, F1-score, and ROC-AUC. Our attention-based multimodal model performed best, with an accuracy of 94%, precision of 93%, recall of 93%, specificity of 95%, F1-score of 93%, and ROC-AUC of 97%, surpassing both clinical-only and imaging-only models. The model consisted of a clinical data encoder, an imaging encoder based on a DenseNet121, and an attention-based fusion mechanism to describe the complementary information across modalities. SHAP analysis emphasized the contributions of clinical and laboratory factors, and Grad-CAM revealed image regions associated with model predictions. These findings highlight the potential of explainable multimodal AI in enhancing disease-detection performance and providing transparent evidence for clinical decision support. We provide a roadmap for deployment in U.S. healthcare: external validation, human supervision, data protection, interoperability, and proper regulatory assessment under clinical supervision.