MedScribe

Partners in Evaluation

Background

Every clinical visit requires documentation. Clinicians typically record encounters using SOAP notes, a structured format that divides information into Subjective, Objective, Assessment, and Plan sections. Unfortunately, this lengthy process often costs twice the amount of time clinicians spend in direct patient care¹ and remains a leading cause of burnout². In response, many healthcare organizations have integrated AI scribes into their workflows, with about 30% of physician practitioners using some form of AI technology³. However, healthcare organizations lack methods to compare the real-world performance⁴ of these documentation systems.

Through our MedScribe benchmark, we aim to fulfill this evaluation gap by assessing whether or not current AI systems are able to reduce the documentation burden without sacrificing accuracy or compliance. In collaboration with Protege, we created a dataset with 100 rubrics that score SOAP notes on documentation quality, providing an objective framework for AI scribing assessment.

Results

Claude Opus 5.5 leads the field, followed by Claude Fable 5.1, Claude Sonnet 5.5, Claude Opus 5, and Muse Spark 1.2.

Accuracy tends to improve as model outputs get longer—GPT 5.1 scores significantly better while also producing significantly longer responses.

Performance varies only slightly across SOAP categories, with most models performing marginally worse on the P (Plan) section. This section includes concrete actions such as orders, prescriptions, referrals, and follow-ups. Even small errors here can affect care continuity, patient safety, and billing accuracy.

Looking more closely into pass rates across specific subcategories, some tasks appear universally harder. For example, “General Survey” and “HPI” sections were especially difficult for models.

Methodology

Evaluating the quality of SOAP notes from real clinical conversations is challenging due to patient privacy constraints. Thus, we adapted the NoteChat framework⁵ to generate synthetic transcripts from real, de-identified SOAP notes from our partner Protege. We conducted A/B testing for realism with our experts (medical scribes and a physician assistant) and found that our synthetic transcripts were indistinguishable from our 10 real transcript samples. Across 80 transcripts (40 real, 40 simulated), the experts identified the true transcript type with 39% accuracy (chance = 50%, p > 0.05).

Our experts developed gold-standard rubrics for each of our transcripts. They first created a standardized template designating the required content and section structure of the SOAP note, then independently annotated each transcript. Any conflicting rubrics were reviewed collaboratively to ensure reliability, quality, and consistency for our documentation standards.

Each model was given the transcripts and prompted to output corresponding SOAP notes given the same template our experts outlined. All models are evaluated with temperature 1, and produce at most 30k tokens.

We observed significant variability depending on which evaluation model was used. Unlike human reviewers, models penalized minor formatting issues, such as placing details in the wrong SOAP section, more harshly than humans. To quantify alignment with human judgment, we ran an alignment study on 3 samples, covering a total of 180 checks. By modifying the judge system prompt, we were able to achieve an 80.5% agreement rate with the majority human opinion, and model pass rates closely matched those assigned by trained medical scribes.

Citations

[1] Sinsky, C., Colligan, L., Li, L., Prgomet, F., Reynolds, L., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016). Allocation of physician time in ambulatory practice: A time and motion study in four specialties. Annals of Internal Medicine, 165(11), 753–760. https://doi.org/10.7326/M16-0961

[2] American Medical Association. (2025, February 14). Physicians’ greatest use for AI? Cutting administrative burdens. AMA Digital Health. https://www.ama-assn.org/practice-management/digital-health/physicians-greatest-use-ai-cutting-administrative-burdens

[3] Vahidy, F. S., Rajkomar, A., & Sounderajah, V. (2025). Beyond human ears: Navigating the uncharted risks of AI scribes in clinical practice. npj Digital Medicine, 8(1), Article 95. https://doi.org/10.1038/s41746-025-01895-6

[4] Kanaparthy, A., Barot, A., Mehta, N., Bhandari, M., & De Simone, A. (2025). Real-world evidence synthesis of digital scribes using ambient listening and generative AI: A systematic review. npj Digital Medicine, 8(1), Article 124. https://doi.org/10.2196/76743

[5] Agarwal, S., Zhang, T., Lee, J., & Rumshisky, A. (2024). NoteChat: Synthesizing doctor–patient conversations from clinical notes for training and evaluating medical dialogue systems. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 14532–14548). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.901