Health AI Central
Peer-Reviewed Publication
J Am Med Inform Assoc2025;32(4):626-637.April 1, 2025Journal Article

Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments.

Chuan Hong1, Anand Chowdhury2, Anthony D Sorrentino2, Haoyuan Wang1, Monica Agrawal1, Armando Bedoya1,2, Sophia Bessias3, Nicoleta J Economou-Zavlanos1, Ian Wong1,2, Christian Pean2, Fan Li1,4, Kathryn I Pollak5,6, Eric G Poon1,2, Michael J Pencina1
1Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC 27710, United States.
2Department of Medicine, Duke University School of Medicine, Durham, NC 27710, United States.
3Duke Clinical and Translational Science Institute, Duke University School of Medicine, Durham, NC 27710, United States.
4Department of Statistical Science, Duke University, Durham, NC 27710, United States.
5Cancer Prevention and Control Research Program, Duke Cancer Institute, Durham, NC 27710, United States.
6Department of Population Health Sciences, Duke University School of Medicine, Durham, NC 27710, United States.
PubMed PMC DOI PDF

Abstract

OBJECTIVES: Large language models (LLMs) are increasingly utilized in healthcare, transforming medical practice through advanced language processing capabilities. However, the evaluation of LLMs predominantly relies on human qualitative assessment, which is time-consuming, resource-intensive, and may be subject to variability and bias. There is a pressing need for quantitative metrics to enable sc…

Create a free account to keep reading

Free members get 10 full research views every month across publications, clinical trials, FDA clearances, adverse events, and NIH grants. No credit card required.

Create Free AccountSign In

Want unlimited research access? See Pro plans

Data Accuracy Notice: Research intelligence on Health AI Central is aggregated from public sources (PubMed, ClinicalTrials.gov, FDA, NIH, CMS, and others) and refreshed nightly. Classifications and derived metrics are produced by automated methods described in our Methodology. We recommend verifying critical data points against the primary sources before making decisions.