Publication:
A Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extraction

Simple item page
dc.citedby3
dc.contributor.authorAlamoodi A.H.en_US
dc.contributor.authorZughoul O.en_US
dc.contributor.authorDavid D.en_US
dc.contributor.authorGarfan S.en_US
dc.contributor.authorPamucar D.en_US
dc.contributor.authorAlbahri O.S.en_US
dc.contributor.authorAlbahri A.S.en_US
dc.contributor.authorYussof S.en_US
dc.contributor.authorSharaf I.M.en_US
dc.contributor.authorid57205435311en_US
dc.contributor.authorid57204659906en_US
dc.contributor.authorid58043918300en_US
dc.contributor.authorid57213826607en_US
dc.contributor.authorid54080216100en_US
dc.contributor.authorid57201013684en_US
dc.contributor.authorid57201009814en_US
dc.contributor.authorid16023225600en_US
dc.contributor.authorid17435789800en_US
dc.date.accessioned2025-03-03T07:41:33Z
dc.date.available2025-03-03T07:41:33Z
dc.date.issued2024
dc.description.abstractArtificial intelligence (AI) has become a crucial element of modern technology, especially in the healthcare sector, which is apparent given the continuous development of large language models (LLMs), which are utilized in various domains, including medical beings. However, when it comes to using these LLMs for the medical domain, there?s a need for an evaluation platform to determine their suitability and drive future development efforts. Towards that end, this study aims to address this concern by developing a comprehensive Multi-Criteria Decision Making (MCDM) approach that is specifically designed to evaluate medical LLMs. The success of AI, particularly LLMs, in the healthcare domain, depends on their efficacy, safety, and ethical compliance. Therefore, it is essential to have a robust evaluation framework for their integration into medical contexts. This study proposes using the Fuzzy-Weighted Zero-InConsistency (FWZIC) method extended to p, q-quasirung orthopair fuzzy set (p, q-QROFS) for weighing evaluation criteria. This extension enables the handling of uncertainties inherent in medical decision-making processes. The approach accommodates the imprecise and multifaceted nature of real-world medical data and criteria by incorporating fuzzy logic principles. The MultiAtributive Ideal-Real Comparative Analysis (MAIRCA) method is employed for the assessment of medical LLMs utilized in the case study of this research. The results of this research revealed that ?Medical Relation Extraction? criteria with its sub-levels had more importance with (0.504) than ?Clinical Concept Extraction? with (0.495). For the LLMs evaluated, out of 6 alternatives, (A4) ?GatorTron S 10B? had the 1st rank as compared to (A1) ?GatorTron 90B? had the 6th rank. The implications of this study extend beyond academic discourse, directly impacting healthcare practices and patient outcomes. The proposed framework can help healthcare professionals make more informed decisions regarding the adoption and utilization of LLMs in medical settings. ? The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.en_US
dc.description.natureFinalen_US
dc.identifier.ArtNo81
dc.identifier.doi10.1007/s10916-024-02090-y
dc.identifier.issue1
dc.identifier.scopus2-s2.0-85202703844
dc.identifier.urihttps://www.scopus.com/inward/record.uri?eid=2-s2.0-85202703844&doi=10.1007%2fs10916-024-02090-y&partnerID=40&md5=4502a35b5cacea143aa18aae89abdd53
dc.identifier.urihttps://irepository.uniten.edu.my/handle/123456789/36196
dc.identifier.volume48
dc.publisherSpringeren_US
dc.sourceScopus
dc.sourcetitleJournal of Medical Systems
dc.subjectArtificial Intelligence
dc.subjectDecision Making
dc.subjectFuzzy Logic
dc.subjectHumans
dc.subjectArticle
dc.subjectcase study
dc.subjectdata analysis
dc.subjectdata integration
dc.subjectdiscourse analysis
dc.subjectevaluation study
dc.subjectfeature extraction
dc.subjectfuzzy logic
dc.subjectfuzzy weighted zero inconsistency method
dc.subjecthealth care practice
dc.subjectlarge language model
dc.subjectmedical decision making
dc.subjectmedical education
dc.subjectmedical research
dc.subjectmultiatributive ideal real comparative analysis
dc.subjectmulticriteria decision analysis
dc.subjectsensitivity analysis
dc.subjecttreatment outcome
dc.subjectuncertainty
dc.subjectartificial intelligence
dc.subjectdecision making
dc.subjecthuman
dc.titleA Novel Evaluation Framework for Medical LLMs: Combining Fuzzy Logic and MCDM for Medical Relation and Clinical Concept Extractionen_US
dc.typeArticleen_US
dspace.entity.typePublication
Files
Collections
SCOPUS