Explainable artificial intelligence (XAI) is routinely proposed as the mechanism that will make clinical machine learning trustworthy, yet the explanations themselves are almost never validated: in the applied healthcare literature a SHAP or LIME plot is presented as evidence rather than tested as a claim. This study asks whether post-hoc explanations of clinical risk models are faithful, stable and reproducible, and whether these properties can be measured routinely. We propose CLARA (Clinical Explanation Reliability Assessment), a four-axis evaluation protocol that scores an explainer on faithfulness (area over the perturbation curve for progressive feature deletion and insertion, benchmarked against a random-attribution control), stability (top-k set overlap under input perturbation), consensus (rank agreement between independent explainers) and computational cost, and aggregates these into an Explanation Quality Index. CLARA was applied to three public clinical cohorts (Cleveland heart disease, n = 297; Pima diabetes, n = 768; Wisconsin diagnostic breast cancer, n = 569), four model families (logistic regression, random forest, gradient boosting, multilayer perceptron) and two local explainers (SHAP and a local surrogate following the LIME formulation), producing 14,400 instance-level explanations across 24 configurations. Both explainers were far more faithful than random feature orderings, and SHAP was significantly more faithful in six of twelve model-dataset cells (Wilcoxon signed-rank, p < 0.01) while the surrogate prevailed in two. The main finding is that agreement between explainers is governed by the model rather than by the explainer: mean rank correlation fell from 0.91 for logistic regression to 0.54 for gradient boosting, whereas hold-out AUROC was uncorrelated with explanation quality (ρ = 0.08, p = 0.70). Faithfulness and consensus should therefore be reported alongside discrimination whenever an explained model is proposed for clinical use.

ARULMURUGAN R.

PhD, associate professor, Department of information technology university, Mattu University, Mattu, Ethiopia

E-mail: arulmr@gmail.com, https://orcid.org/0000-0003-1030-9565

TESHOME D.A.

M. Sc, head of department IT, Mattu University, College of engineering and technology department of information technology, Mattu, Ethiopia

E-mail: teshome.debushe@meu.edu.et, https://orcid.org/0000-0002-4316-8474

1. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) [Electronic resource] // Official Journal of the European Union. – OJ L, 2024/1689, 12.07.2024. – Access mode: https://eur-lex.europa.eu/eli/reg/2024/1689/oj (date of access: 01.09.2026).
2. Lundberg S.M., Lee S.-I. A unified approach to interpreting model predictions // Advances in Neural Information Processing Systems 30 (NeurIPS 2017). – Long Beach, 2017. – P. 4765-4774.
3. Ribeiro M.T., Singh S., Guestrin C. «Why should I trust you?»: Explaining the predictions of any classifier // Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. – San Francisco, 2016. – P. 1135-1144. https://doi.org/10.1145/2939672.2939778
4. Sundararajan M., Taly A., Yan Q. Axiomatic attribution for deep networks // Proceedings of the 34th International Conference on Machine Learning, PMLR. – Sydney, 2017. – Vol. 70. – P. 3319-3328. URL: https://proceedings.mlr.press/v70/sundararajan17a.html
5. Selvaraju R.R., Cogswell M., Das A., Vedantam R., Parikh D., Batra D. Grad-CAM: Visual explanations from deep networks via gradient-based localization // Proceedings of the IEEE International Conference on Computer Vision (ICCV). – Venice, 2017. – P. 618-626. https://doi.org/10.1109/ICCV.2017.74
6. Loh H.W., Ooi C.P., Seoni S., Barua P.D., Molinari F., Acharya U.R. Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011-2022). Computer Methods and Programs in Biomedicine, 2022. Vol. 226. 107161. https://doi.org/10.1016/j.cmpb.2022.107161
7. Antoniadi A.M., Du Y., Guendouz Y., Wei L., Mazo C., Becker B.A., Mooney C. Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: A systematic review. Applied Sciences, 2021. Vol. 11. Issue 11. 5088. https://doi.org/10.3390/app11115088
8. Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 2019. Vol. 1. Issue 5. 206-215. https://doi.org/10.1038/s42256-019-0048-x
9. Ghassemi M., Oakden-Rayner L., Beam A.L. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 2021. Vol. 3. Issue 11. e745-e750. https://doi.org/10.1016/S2589-7500(21)00208-9
10. Slack D., Hilgard S., Jia E., Singh S., Lakkaraju H. Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods // Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. – New York, 2020. – P. 180-186. https://doi.org/10.1145/3375627.3375830
11. Alvarez-Melis D., Jaakkola T.S. On the robustness of interpretability methods [Electronic resource]. – arXiv:1806.08049, 2018. – Access mode: https://arxiv.org/abs/1806.08049 (date of access: 01.09.2026).
12. DeGrave A.J., Janizek J.D., Lee S.-I. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 2021. Vol. 3. 610-619. https://doi.org/10.1038/s42256-021-00338-7
13. Abbas Q., Jeong W., Lee S.W. Explainable AI in clinical decision support systems: A meta-analysis of methods, applications, and usability challenges. Healthcare, 2025. Vol. 13. Issue 17. 2154. https://doi.org/10.3390/healthcare13172154
14. Zhang K., Wang D., Lin F., Xie J., Zhou W. A comprehensive review of explainable artificial intelligence in healthcare: methods, evaluation, and clinical integration. iScience, 2026. Vol. 29. Issue 3. 115026. https://doi.org/10.1016/j.isci.2026.115026
15. Lundberg S.M., Erion G., Chen H., DeGrave A., Prutkin J.M., Nair B., et al. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2020. Vol. 2. Issue 1. 56-67. https://doi.org/10.1038/s42256-019-0138-9
16. van der Velden B.H.M., Kuijf H.J., Gilhuijs K.G.A., Viergever M.A. Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Medical Image Analysis, 2022. Volume 79. 102470. https://doi.org/10.1016/j.media.2022.102470
17. Payrovnaziri S.N., Chen Z., Rengifo-Moreno P., Miller T., Bian J., Chen J.H., Liu X., He Z. Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review. Journal of the American Medical Informatics Association, 2020. Vol. 27. Issue 7. 1173-1185. https://doi.org/10.1093/jamia/ocaa053
18. Markus A.F., Kors J.A., Rijnbeek P.R. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. Journal of Biomedical Informatics, 2021. Vol. 113. 103655. https://doi.org/10.1016/j.jbi.2020.103655
19. Van Dort B.A., Engelsma T., Sneekes P., Peute L., Medlock S. Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives. PLOS Digital Health, 2026. Vol. 5. Issue 5. e0001417. https://doi.org/10.1371/journal.pdig.0001417
20. Caruana R., Lou Y., Gehrke J., Koch P., Sturm M., Elhadad N. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission // Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. – Sydney, 2015. – P. 1721-1730. https://doi.org/10.1145/2783258.2788613
21. Amann J., Vetter D., Blomberg S.N., Christensen H.C., Coffee M., Gerke S., et al. To explain or not to explain? Artificial intelligence explainability in clinical decision support systems. PLOS Digital Health, 2022. Vol. 1. Issue 2. e0000016. https://doi.org/10.1371/journal.pdig.0000016
22. Doshi-Velez F., Kim B. Towards a rigorous science of interpretable machine learning [Electronic resource]. – arXiv:1702.08608, 2017. – Access mode: https://arxiv.org/abs/1702.08608 (date of access: 01.09.2026).
23. Petsiuk V., Das A., Saenko K. RISE: Randomized input sampling for explanation of black-box models // Proceedings of the British Machine Vision Conference (BMVC). – Newcastle, 2018. – arXiv: URL: https://arxiv.org/abs/1806.07421.
24. Hooker S., Erhan D., Kindermans P.-J., Kim B. A benchmark for interpretability methods in deep neural networks // Advances in Neural Information Processing Systems 32 (NeurIPS 2019). – Vancouver, 2019. – arXiv: URL: https://arxiv.org/abs/1806.10758.
25. Detrano R., Janosi A., Steinbrunn W., Pfisterer M., Schmid J.-J., Sandhu S., Guppy K.H., Lee S., Froelicher V. International application of a new probability algorithm for the diagnosis of coronary artery disease. American Journal of Cardiology, 1989. Vol. 64. Issue 5. 304-310. https://doi.org/10.1016/0002-9149(89)90524-9
26. Smith J.W., Everhart J.E., Dickson W.C., Knowler W.C., Johannes R.S. Using the ADAP learning algorithm to forecast the onset of diabetes mellitus // Proceedings of the Annual Symposium on Computer Application in Medical Care. – Washington, 1988. – P. 261-265. URL: https://pmc.ncbi.nlm.nih.gov/articles/PMC2245318/
27. Street W.N., Wolberg W.H., Mangasarian O.L. Nuclear feature extraction for breast tumor diagnosis // IS&T/SPIE International Symposium on Electronic Imaging: Science and Technology, Biomedical Image Processing and Biomedical Visualization. – San Jose, 1993. – Vol. 1905. – P. 861-870. https://doi.org/10.1117/12.148698
28. Breiman L. Random forests. Machine Learning, 2001. Vol. 45. Issue 1. 5-32. https://doi.org/10.1023/A:1010933404324
29. Chen T., Guestrin C. XGBoost: A scalable tree boosting system // Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. – San Francisco, 2016. – P. 785-794. https://doi.org/10.1145/2939672.2939785
30. Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011. Vol. 12. P. 2825-2830. URL: https://www.jmlr.org/papers/v12/pedregosa11a.html
31. -2830.

explainable artificial intelligence, clinical decision support, SHAP, LIME, explanation faithfulness, explanation stability, machine learning in medicine, model evaluation