114027
论文已发表
注册即可获取德孚的最新动态
IF 收录期刊
- 3.4 Breast Cancer (Dove Med Press)
- 3.2 Clin Epidemiol
- 2.6 Cancer Manag Res
- 2.9 Infect Drug Resist
- 3.7 Clin Interv Aging
- 5.1 Drug Des Dev Ther
- 3.1 Int J Chronic Obstr
- 6.6 Int J Nanomed
- 2.6 Int J Women's Health
- 2.9 Neuropsych Dis Treat
- 2.8 OncoTargets Ther
- 2.0 Patient Prefer Adher
- 2.2 Ther Clin Risk Manag
- 2.5 J Pain Res
- 3.0 Diabet Metab Synd Ob
- 3.2 Psychol Res Behav Ma
- 3.4 Nat Sci Sleep
- 1.8 Pharmgenomics Pers Med
- 2.0 Risk Manag Healthc Policy
- 4.1 J Inflamm Res
- 2.0 Int J Gen Med
- 3.4 J Hepatocell Carcinoma
- 3.0 J Asthma Allergy
- 2.2 Clin Cosmet Investig Dermatol
- 2.4 J Multidiscip Healthc
已发表论文
关于大型语言模型在幽门螺杆菌相关中文医疗咨询中的表现
Authors Zhang M, Zhou S, Zhang S, Yi T, Jiang B, Jiang X
Received 6 August 2025
Accepted for publication 15 December 2025
Published 20 December 2025 Volume 2025:18 Pages 6769—6777
DOI https://doi.org/10.2147/IDR.S553523
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Dr Hazrat Bilal
Mingjun Zhang,* Shiming Zhou,* Shulin Zhang, Ting Yi, Bo Jiang, Xuan Jiang
Department of Gastroenterology, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University, Beijing, People’s Republic of China
*These authors contributed equally to this work
Correspondence: Xuan Jiang, Email jxa01998@btch.edu.cn Bo Jiang, Email drjiang@163.com
Background: H. pylori infection is a worldwide health issue, fueling rising demand for medical counseling. LLMs have the potential to serve in medical counseling. However, their performance remains unclear.
Objective: This study aimed to evaluate the effectiveness of LLMs in providing H. pylori related medical counseling in a Chinese context.
Methods: 20 H. pylori-related questions were collected, covering four domains: definition and symptoms, diagnosis, treatment, and prevention. Each question was asked thrice in Chinese to each LLM. We assessed the responses across five dimensions (accuracy, relevance, completeness, clarity, and reliability).
Results: 1. In the first batch of tests, the overall performance distribution was 33.3% good, 66.1% medium, and 0.6% poor, respectively. No significant differences were observed among the three LLMs (p=0.158). Good performance was observed with 47.8% in accuracy, 53.9% in relevance, 68.3% in completeness, 36.7% in clarity, and 36.1% in reliability. No significant differences were observed in accuracy, relevance, completeness, or clarity. Reliability differed significantly (p< 0.001), with Ernie Bot achieving the best performance. 2. The second test batch yielded performance rates of 70.6% good, 29.4% medium, and 0% poor, with a significant difference among the three LLMs (p=0.018). Doubao attained the best performance, surpassing other models in relevance and clarity. 3. The newly assessed AI batch showed markedly superior overall performance to the counterpart evaluated more than a year prior.
Conclusion: This study is the first to evaluate the effectiveness of various LLMs in H. pylori-related medical counseling in a real-world setting. The study showed that while LLMs generally performed acceptably in terms of accuracy, relevance, and completeness, their clarity and reliability were less satisfactory. Ernie Bot, developed by Chinese company, outperformed ChatGPT in certain aspects of medical counseling in Chinese. With the guidance of professionals, LLMs can serve as potential aids for medical counseling.
Keywords: artificial intelligence, large language model, medical counseling, Helicobacter pylori
Department of Gastroenterology, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University, Beijing, People’s Republic of China
*These authors contributed equally to this work
Correspondence: Xuan Jiang, Email jxa01998@btch.edu.cn Bo Jiang, Email drjiang@163.com
Background: H. pylori infection is a worldwide health issue, fueling rising demand for medical counseling. LLMs have the potential to serve in medical counseling. However, their performance remains unclear.
Objective: This study aimed to evaluate the effectiveness of LLMs in providing H. pylori related medical counseling in a Chinese context.
Methods: 20 H. pylori-related questions were collected, covering four domains: definition and symptoms, diagnosis, treatment, and prevention. Each question was asked thrice in Chinese to each LLM. We assessed the responses across five dimensions (accuracy, relevance, completeness, clarity, and reliability).
Results: 1. In the first batch of tests, the overall performance distribution was 33.3% good, 66.1% medium, and 0.6% poor, respectively. No significant differences were observed among the three LLMs (p=0.158). Good performance was observed with 47.8% in accuracy, 53.9% in relevance, 68.3% in completeness, 36.7% in clarity, and 36.1% in reliability. No significant differences were observed in accuracy, relevance, completeness, or clarity. Reliability differed significantly (p< 0.001), with Ernie Bot achieving the best performance. 2. The second test batch yielded performance rates of 70.6% good, 29.4% medium, and 0% poor, with a significant difference among the three LLMs (p=0.018). Doubao attained the best performance, surpassing other models in relevance and clarity. 3. The newly assessed AI batch showed markedly superior overall performance to the counterpart evaluated more than a year prior.
Conclusion: This study is the first to evaluate the effectiveness of various LLMs in H. pylori-related medical counseling in a real-world setting. The study showed that while LLMs generally performed acceptably in terms of accuracy, relevance, and completeness, their clarity and reliability were less satisfactory. Ernie Bot, developed by Chinese company, outperformed ChatGPT in certain aspects of medical counseling in Chinese. With the guidance of professionals, LLMs can serve as potential aids for medical counseling.
Keywords: artificial intelligence, large language model, medical counseling, Helicobacter pylori