DOE OSTI · 3029007
Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools
Abstract
Large language models (LLMs) show promise in supporting differential diagnosis, but their performance is challenging to evaluate due to the unstructured nature of their responses, and their accuracy compared to existing diagnostic tools is not well characterized. To assess the current capabilities of LLMs to diagnose genetic diseases, we benchmarked these models on 5213 previously published case reports using the Phenopacket Schema, the Human Phenotype Ontology and Mondo disease ontology. Prompts generated from each phenopacket were sent to seven LLMs, including four generalist models and three LLMs specialized for medical applications. The same phenopackets were used as input to a widely used diagnostic tool, Exomiser, in phenotype-only mode. The best LLM ranked the correct diagnosis first in 23.6% of cases, whereas Exomiser did so in 35.5% of cases. While the performance of LLMs for supporting differential diagnosis has been improving, it has not reached the level of commonly used traditional bioinformatics tools. Future research is needed to determine the best approach to incorporate LLMs into diagnostic pipelines.
Keep this discovery
Explore connections, maps & timelines
Reese, Justin T. [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)] (ORCID:0000000221702250), Chimirri, Leonardo [Charité Universitätsmedizin Berlin (Germany)] (ORCID:0000000269128518), Bridges, Yasemin [Queen Mary University of London (United Kingdom)], Danis, Daniel [Charité Universitätsmedizin Berlin (Germany)], Caufield, J. Harry [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)] (ORCID:0000000157057831), Gargano, Michael A. [Jackson Laboratory for Genomic Medicine, Farmington, CT (United States)], Kroll, Carlo [Queen Mary University of London (United Kingdom)], Schmeder, Andrew [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)], Liu, Fengchen [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)], Wissink, Kyran [Charité Universitätsmedizin Berlin (Germany)], McMurry, Julie A. [University of North Carolina, Chapel Hill, NC (United States)] (ORCID:0000000293535498), Graefe, Adam S. L. [Charité Universitätsmedizin Berlin (Germany)], Niyonkuru, Enock [Jackson Laboratory for Genomic Medicine, Farmington, CT (United States)], Korn, Daniel R. [University of North Carolina, Chapel Hill, NC (United States)], Casiraghi, Elena [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Università degli Studi di Milano (Italy); European Laboratory for Learning and Intelligent Systems (ELLIS), Milan (Italy)] (ORCID:0000000320247572), Valentini, Giorgio [Università degli Studi di Milano (Italy); European Laboratory for Learning and Intelligent Systems (ELLIS), Milan (Italy)] (ORCID:0000000256943919), Jacobsen, Julius O. B. [Queen Mary University of London (United Kingdom)], Haendel, Melissa [University of North Carolina, Chapel Hill, NC (United States)] (ORCID:0000000191148737), Smedley, Damian [Queen Mary University of London (United Kingdom)] (ORCID:0000000258369850), Mungall, Christopher J. [Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)], Robinson, Peter N. [Charité Universitätsmedizin Berlin (Germany); Jackson Laboratory for Genomic Medicine, Farmington, CT (United States); European Laboratory for Learning and Intelligent Systems (ELLIS), Milan (Italy)] (ORCID:0000000207369199). 2026-02-24. Systematic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools. https://doi.org/10.1038/s41431-026-02054-5
Cite the original work for its findings. Save a collection to share your selection of sources.