Cross-Modal Attention Fusion for Clinical Risk Prediction: Integrating EHR Data with Dual-Attribution Explain ability
Abstract
Clinical prediction models have historically relied on structured electronic health record data, leaving the rich prognostic signal embedded in free-text clinical documentation systematically underutilized. The proposed Clinical Multimodal Adaptive Clinical Prediction architecture — a dual-pathway prediction system integrating structured EHR features through gradient-boosted tree processing with unstructured clinical note embeddings through Clinical BERT encoding, unified by a cross-modal attention mechanism that weighs each modality's contribution by predictive confidence. The architecture is designed for geriatric acute care settings where clinical notes capture disease severity, symptom trajectory, and functional context unavailable from structured data alone. A federated fine-tuning protocol enables privacy- preserving multi-institutional model development across health systems with different EHR platforms and patient populations. An attribution framework combining SHAP-based structured feature importance with Integrated Gradients note concept attribution provides role-differentiated clinical explanations, with a concordance quality badge signalling modality alignment. A governance framework specifies deployment standards, post-market performance monitoring, and equity evaluation protocols calibrated to multi-site clinical NLP systems. The CMACP framework addresses the technical and governance requirements for safe deployment of multimodal clinical AI in settings where older adult patients are most vulnerable to prediction errors. External validation across multiple health system types, equity stratification by race and documentation quality environment, and prospective evaluation of attribution quality are identified as priority research requirements. Index Terms — multimodal clinical AI; cross-modal attention; ClinicalBERT; structured EHR; natural language processing; dual attribution; federated learning; explainability
References
Abdallah, A., Maarof, M.A. and Zainal, A. (2016). Fraud detection system: A survey. Journal of Network and Computer Applications, 68, 90 to 113. Abernethy, A. P., Ahmad, A., Zafar, S. Y., Wheeler, J. L., Reese, J. B., and Lyerly, H. K. (2010). Electronic patient-reported data capture as a foundation of rapid learning cancer care. Medical Care, 48(6 Suppl), S32–S38. Ackerman, P. (2017). Industrial Cybersecurity: Efficiently Secure Critical Infrastructure Systems. Packt Publishing, Birmingham. Adebayo, A. (2020). Wireless internet service provider network reliability frameworks for emerging markets. International Journal of Network and Communication Research, 5(2), 88-103. Tizeti Inc., Nigeria. Adepu, S. and Mathur, A. (2016). An investigation into the response of a water treatment system to cyber attacks. HASE 2016, 141 to 148. Adepu, S., Brasser, F., Garcia, L., Rodler, M., Davi, L., Sadeghi, A.R. and Zonouz, S. (2020). Control behavior integrity for distributed cyber physical systems. ACM/IEEE ICCPS 2020. Adeyoyin, O., Awanye, E. N., Morah, O. O., and Ekpedo, L. (2020). A Conceptual Framework Linking Financial Strategy and Operational Excellence in Manufacturing Firms. Adler-Milstein, J., and Jha, A. K. (2017). HITECH act drove large gains in hospital electronic health record adoption. Health Affairs, 36(8), 1416–1422. Agbabiaka, J., Okonkwo, C.S., Ogunwole, O., Mayo, W. and Okeke, O.T. (2019). Supply Chain Risk Management Model for EPC and Gas Processing Projects. IRE Journals, 3(2), 968 to 980. DOI: 10.64388/IREV3I2-1713124. Aggarwal, C.C. (2017). Outlier Analysis (2nd ed.). Springer, Cham. Ahmed, M., Mahmood, A.N. and Hu, J. (2016). A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60, 19 to 31. Ahmed, K.S. and Odejobi, O.D. (2018). Conceptual Framework for Scalable and Secure Cloud Architectures for Enterprise Messaging. IRE Journals, 2(1), 1-15. Ahmed, K. S., Odejobi, O. D., and Oshoba, T. O. (2019). Algorithmic model for constraint satisfaction in cloud network resource allocation. IRE Journals, 2(12). ISSN: 2456-8880. Ahmed, K.S., Odejobi, O.D. and Oshoba, T.O. (2020). Predictive Model for Cloud Resource Scaling Using Machine Learning Techniques. Journal of Frontiers in Multidisciplinary Research, 1(1), 173-183. DOI: 10.54660/.Ijfmr.2020.1.1.173-183. Aifuwa, S. E., Oshoba, T. O., Ogbuefi, E., Ike, P. N., Nnabueze, S. B., and Olatunde-Thorpe, J. (2020). Predictive analytics models enhancing supply chain demand forecasting accuracy and reducing inventory management inefficiencies. International Journal of Multidisciplinary Research and Growth Evaluation, 1(3), 171-181. DOI: 10.54660/.IJMRGE.2020.1.3.171-181. Ajala, C., Okonkwo, V., Okezie, C., and Anioke, S. (2022). Effectiveness of HIV prevention strategies (ART, pre-exposure prophylaxis, and awareness campaigns) in reducing HIV incidence in Nigeria. Open Access Research Journal of Science and Technology, 6(2), 86– 112. https://oarjst.com/ Akeju, B., Edivri, J., Ogbole, J. I., Okoruwa, P. O., Fadayomi, O., and Abolaji, T. O. (2018). Conceptual model for insider threat classification and risk modeling in complex digital systems. IRE Journals, 1(9). https://doi.org/10.64388/IREV1I9-1713778 Akhtar, N. and Mian, A. (2018). Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6, 14410 to 14430. Akinlolu, V. S., Omaghomi, T. T., and Fapohunda, M. (2024). A Data-Driven Framework for Improving Health Facility Preparedness and Disaster Risk Mitigation in Urban Hospitals. International Journal of Advanced Multidisciplinary Research and Studies, 4(6), 2757– 2769. Akinlolu, V. S., Fapohunda, M., Omaghomi, T. T., Atima, M. E., and Igweonu, C. (2023). A Proposed Care-Coordination Framework for Reducing Readmissions Among Chronic Disease Patients. International Journal of Multidisciplinary Research and Growth Evaluation, 4(5), 1187–1195. https://doi.org/10.54660/.IJMRGE.2023.4.5.1187-1195 Akinola, A. S., Okafor, C. M., Dako, O. F., and Adesanya, O. S. (2020b). Evidence-informed Advisory for Ultra-High-Net-Worth Clients: Portfolio Governance and Fiduciary Risk Controls. Journal of Frontiers in Multidisciplinary Research, 1(2), 112-120. Akinola, A. S., Adesanya, O. S., Okafor, C. M., and Farounbi, B. O. (2018). Automated Payroll Compliance Assurance: Linking Withholding Algorithms to Financial Statement Reliability. IRE Journals, 1(7). Akinola, A. S., Farounbi, B. O., Onyelucheya, O. P., and Okafor, C. M. (2020a). Translating finance bills into strategy: Sectoral impact mapping and regulatory scenario analysis. Journal of Frontiers in Multidisciplinary Research, 1(1), 102-111. Aldaraani, N. and Begum, Z. (2018). Understanding the impact of ransomware: A survey on its evolution, mitigation and prevention techniques. NCC 2018, 1 to 5. Alexander, O., Belisle, M. and Steele, J. (2020). MITRE ATT&CK for Industrial Control Systems: Design and Philosophy. MITRE Corporation, McLean, VA. Aliliele, C., Mbonu, I.S. and Iwuanyanwu, U., 2023. A Review of API Governance and Risk Prioritization Frameworks in Modern Financial Institutions. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 9(10), pp.395-433. Aliliele, C., Mbonu, I.S. and Iwuanyanwu, U., 2023. A Conceptual Framework for Continuous Cloud Misconfiguration Monitoring and Enterprise Risk Mitigation Strategies. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 9(10), pp.373-394. Aliliele, C., Mbonu, I.S. and Iwuanyanwu, U., 2023. Advances in Predictive Analytics Models for Student Retention and Institutional Risk Management Systems. International Journal of Advanced Multidisciplinary Research and Studies, 3(6), pp.2692-2711. Alladi, T., Chamola, V. and Zeadally, S. (2020). Industrial control systems: Cyberattack trends and countermeasures. Computer Communications, 155, 1 to 8. Allodi, L. and Massacci, F. (2014). Comparing vulnerability severity and exploits using case control studies. ACM TISSEC, 17(1), 1 to 20. Almorsy, M., Grundy, J. and Muller, I. (2016). An analysis of the cloud computing security problem. arXiv preprint arXiv:1609.01107. Alsentzer, E., Murphy, J. R., Boag, W., Weng, W. H., Jin, D., Naumann, T., and McDermott, M. (2019). Publicly available clinical BERT embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop, 72–78. Amin, M. and Wollenberg, B.F. (2005). Toward a smart grid: Power delivery for the 21st century. IEEE Power and Energy Magazine, 3(5), 34 to 41. Aminu-Ibrahim, A. Y., Ogbete, J. C., and Ambali, K. B. (2018). Developing sustainable diagnostic laboratory infrastructure models for emerging and resource constrained health systems. Iconic Research and Engineering Journals, 1(8), 118 to 132.https://doi.org/10.64388/IREV1I8 to 1713586 Aminu-Ibrahim, A.Y., Ogbete, J.C. and Ambali, K.B. (2019). Capital Project Delivery Models for High Risk Healthcare Infrastructure in Developing National Health Systems. Iconic Research and Engineering Journals, 2(10), 626 to 649. DOI: 10.64388/IREV2I10 to 1713588. Aminu-Ibrahim, A.Y., Ogbete, J.C. and Ambali, K.B. (2020). Infrastructure Driven Expansion of Diagnostic Access Across Underserved and Rural Healthcare Regions. International Journal of Multidisciplinary Research and Growth Evaluation, 1(5), 691 to 706. DOI: 10.54660/IJMRGE.2020.1.5.691 to 706. Aminu-Ibrahim, A.Y. and Ogbete, J.C., 2023. Healthcare infrastructure as a public health intervention using evidence from large laboratory networks. Shodhshauryam, International Scientific Refereed Research Journal, 6(1), pp.256-286. Anderson, R. (2008). Security Engineering: A Guide to Building Dependable Distributed Systems (2nd ed.). Wiley, Indianapolis, IN. Ani, U.P.D., He, H. and Tiwari, A. (2017). Review of cybersecurity issues in industrial critical infrastructure: Manufacturing in perspective. Journal of Cyber Security Technology, 1(1), 32 to 74. Anichukwueze, C. C., Osuji, V. C., and Oguntegbe, E. E. (2019). Global marketing law and consumer protection challenges: a strategic framework for multinational compliance. IRE Journals, 3(6), 325-333. Anioke, S., and Atima, M. (2018). Regulatory analytics approaches for improving occupational health safety outcomes across public and private workplaces. Iconic Research and Engineering Journals, 3(6), 119–218. https://www.irejournals.com/ Anioke, S., and Atima, M. (2019). Digital employer risk rating frameworks supporting public health-oriented social insurance compliance systems. Iconic Research and Engineering Journals, 3(5), 411–433. https://www.irejournals.com/ Anioke, S., and Atima, M. (2020a). Community-based public health compliance models supporting vulnerable workers and informal sector populations. International Journal of Multidisciplinary Research and Growth Evaluation, 1(5), 509–522. https://doi.org/10.54660/IJMRGE.2020.1.5.509-522 Anioke, S., and Atima, M. (2020b). Data-driven strategies for preventing workplace injuries and improving employee health protection outcomes. International Journal of Multidisciplinary Research and Growth Evaluation, 1(5), 523–536. https://doi.org/10.54660/IJMRGE.2020.1.5.523-536 Anioke, S., and Atima, M. (2023a). Business intelligence applications for mental health resource allocation and public health program accountability. International Journal of Multidisciplinary Research and Studies, 3(6), 2549–2563. https://doi.org/10.62225/2583049X.2023.3.6.5492 Anioke, S., and Atima, M. (2023b). Public health governance models using process optimization and performance metrics for regulatory oversight. International Journal of Advanced Multidisciplinary Research and Studies, 3(6), 2534–2548. https://doi.org/10.62225/2583049X.2023.3.6.5491 Anioke, S., and Atima, M. (2023c). Public health informatics frameworks for protecting vulnerable populations through data-driven policy enforcement. International Journal of Multidisciplinary Research and Studies, 3(6), 2564–2579. https://doi.org/10.62225/2583049X.2023.3.6.5493 Anioke, S., and Atima, M. (2024). Predictive analytics systems strengthening public health surveillance and epidemic preparedness decision making. Iconic Research and Engineering Journals, 7(2), 157–203. https://doi.org/10.32628/SHISRRJ Anthony, P., Ezeh, F. E., Oparah, O. S., Gado, P., Adeleke, A. S., Gbaraba, S. V., Okoli, A. O., and Omotayo, O. (2023). Adaptive multi-modal AI systems for continuous disease monitoring via IoT and wearable devices. International Journal of Advanced Multidisciplinary Research and Studies, 3(1), 1557–1570. https://doi.org/10.62225/2583049X.2023.3.1.5158 Anwar, S. and Soltesz, B. (2016). Securing the smart grid. International Journal of Advanced Research in Computer Science, 7(1). Apruzzese, G., Colajanni, M., Ferretti, L., Guido, A. and Marchetti, M. (2018). On the effectiveness of machine and deep learning for cyber security. CyCon 2018, 371 to 390. Apruzzese, G., Colajanni, M., Ferretti, L. and Marchetti, M. (2019). Addressing adversarial attacks against security systems based on machine learning. International Conference on Cyber
More Articles from INTERNATIONAL JOURNAL OF MEDICAL EVALUATION AND PHYSICAL REPORT
COVID-19 and Its Impact on Government Workers Performance
in Nigeria: A Study of Oredo Local Government Area, Edo
Author: Gabriel Favour Eke, PhD, Mack Ebohol Oriakhi PhD
Effect of Acha ( Digitaria exilis ) Pudding Consumption on Fasting and Postprandial Blood Glucose Levels Among Patients with Type 2 Diabetes Mellitus at the Bingham University Teaching Hospital, Jos- Nigeria
Author: Dabe Grace, Oluchukwu Ogechukwu Anunobi, and Daikwo Moses Alilu
The Role of Medicine Quality Failures in the Emergence and Spread of Antimicrobial Resistance: A Narrative Review
Author: Oluchi Beatrice Aneke, Abimbola Caleb Adesemoye, Esther Sydney
Regulatory Harmonization and Convergence in Global Medicine Quality Assurance: A Review of Frameworks, Pathways, and Implementation Barriers
Author: Oluchi Beatrice Aneke, Abimbola Caleb Adesemoye, Esther Sydney
Psychosocial Impact of Infertility Among Women of Reproductive Age Attending Irrua Specialist Teaching Hospital, Edo State, Nigeria
Author: Mary Saidu, Phoebe Nwamaka Kanikwu, and Juliana Afemikhe