Objective: With the increasing prevalence of large language models (LLMs) in the medical field, patients are increasingly turning to advanced online resources for information related to liver cirrhosis due to its long-term management processes. Therefore, a comprehensive evaluation of real-world performance of LLMs in these specialized medical areas is necessary. Methods: This study evaluates the performance of four mainstream LLMs (ChatGPT-4o, Claude-3.5 Sonnet, Gemini-1.5 Pro, and Llama-3.1) in answering 39 questions related to liver cirrhosis. The information quality, readability and accuracy were assessed using Ensuring Quality Information for Patients tool, Flesch-Kincaid metrics and consensus scoring. The simplification and their self-correction ability of LLMs were also assessed. Results: Significant performance differences were observed among the models. Gemini scored highest in providing high-quality information. While the readability of all four LLMs was generally low, requiring a college-level reading comprehension ability, they exhibited strong capabilities in simplifying complex information. ChatGPT performed best in terms of accuracy, with a "Good" rating of 80%, higher than Claude (72%), Gemini (49%), and Llama (64%). All models received high scores for comprehensiveness. Each of the four LLMs demonstrated some degree of self-correction ability, improving the accuracy of initial answers with simple prompts. ChatGPT's and Llama's accuracy improved by 100%, Claude's by 50% and Gemini's by 67%. Conclusion: LLMs demonstrate excellent performance in generating health information related to liver cirrhosis, yet they exhibit differences in answer quality, readability and accuracy. Future research should enhance their value in healthcare, ultimately achieving reliable, accessible and patient-centered medical information dissemination.
基金:
This research was funded by the high-level Chinese Medicine Key Discipline Construction Project (No. zyyzdxk-2023005, to Xianbo Wang); Capital’s Funds for Health improvement and Research (No. 2024-1-2173, to Xianbo Wang); National Natural Science Foundation of China (No. 82474419, to Xianbo Wang and 82474426, to Ying Feng); Beijing Municipal Natural Science Foundation (No. 7232272, to Ying Feng); and Beijing Traditional Chinese Medicine Technology Development Fund Project (No. BJZYZD-2023-12, to Xianbo Wang).
语种:
外文
WOS:
PubmedID:
中科院(CAS)分区:
出版当年[2025]版:
大类|2 区医学
小类|2 区卫生保健与服务3 区计算机:信息系统3 区医学:信息
最新[2025]版:
大类|2 区医学
小类|2 区卫生保健与服务3 区计算机:信息系统3 区医学:信息
JCR分区:
出版当年[2023]版:
Q1HEALTH CARE SCIENCES & SERVICESQ2COMPUTER SCIENCE, INFORMATION SYSTEMSQ2MEDICAL INFORMATICS
最新[2023]版:
Q1HEALTH CARE SCIENCES & SERVICESQ2COMPUTER SCIENCE, INFORMATION SYSTEMSQ2MEDICAL INFORMATICS
第一作者机构:[1]Capital Med Univ, Beijing Ditan Hosp, Ctr Integrat Med, Beijing, Peoples R China
共同第一作者:
通讯作者:
推荐引用方式(GB/T 7714):
Li Yanqiu,Li Zhuojun,Li Jinze,et al.The actual performance of large language models in providing liver cirrhosis-related information: A comparative study[J].INTERNATIONAL JOURNAL OF MEDICAL INFORMATICS.2025,201:doi:10.1016/j.ijmedinf.2025.105961.
APA:
Li, Yanqiu,Li, Zhuojun,Li, Jinze,Liu, Long,Liu, Yao...&Wang, Xianbo.(2025).The actual performance of large language models in providing liver cirrhosis-related information: A comparative study.INTERNATIONAL JOURNAL OF MEDICAL INFORMATICS,201,
MLA:
Li, Yanqiu,et al."The actual performance of large language models in providing liver cirrhosis-related information: A comparative study".INTERNATIONAL JOURNAL OF MEDICAL INFORMATICS 201.(2025)