🔍 Read the full analysis: What ‘Beats Doctors’ Means In Microsoft’s AI Study on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft’s MAI-DxO scored 83.2% on 304 difficult New England Journal of Medicine case records, compared with 36.1% for generalist physicians using reference materials. The peer-reviewed study tested text-based historical cases in a simulator, not diagnosis or treatment of real patients; a prospective study alongside usual care is planned.
Microsoft’s diagnostic AI system, MAI-DxO, scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, according to a Science paper published October 22. The result describes performance on historical, text-based cases in a simulator—not a trial showing that the system diagnoses patients better in hospitals.
The paper, “Sequential diagnosis with language models,” evaluates a benchmark designed to imitate parts of the diagnostic process. Each case begins with a short summary. A doctor or AI system must request additional information, such as a test result, from a simulated gatekeeper that holds the rest of the case. The benchmark scores both the final diagnosis and the estimated cost of obtaining information.
On the 304 complex cases drawn from the New England Journal of Medicine, Microsoft reports 83.2% diagnostic accuracy for MAI-DxO. Generalist physicians scored 36.1% when they could use textbooks and the internet, and 22.3% without those resources. On a separate set of 325 emergency-department cases, the reported scores were 80.3% for the AI and 46.8% for physicians.
Microsoft describes MAI-DxO as a system that uses a language model with a panel of virtual agents. They propose diagnoses, request questions or tests, and critique one another over multiple rounds. The company’s results also show that GPT-5 alone scored 78.9% on the complex cases; the orchestration system added roughly four percentage points, according to the supplied account of the results.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why the Doctor Comparison Matters
The results suggest that language models may be able to work through difficult diagnostic cases by gathering information in stages, rather than only selecting an answer from a complete case file. That is relevant because real diagnosis often involves deciding what to ask, which test to order, and when to stop. The benchmark also attempts to account for test costs, rather than scoring diagnosis alone.
But the phrase “beats doctors” can imply a broader clinical comparison than the study supports. The physician group consisted of primary-care doctors and hospital generalists working individually, while MAI-DxO operates as a coordinated set of virtual agents. The cases were selected for difficulty and, as the source account notes, such cases may be referred to specialists or handled by teams in practice. The study did not compare the AI with specialists or multidisciplinary clinical teams.
For hospitals and patients, the practical question is not only whether an AI can identify a published diagnosis. It is whether using it with real patients improves decisions, reduces avoidable delays or costs, and does so without causing harm. The reported benchmark does not answer those questions. Its scores are a research result, not evidence of better patient outcomes.
Products Worth Considering
Canon PIXMA TR160 Wireless Portable Color Inkjet Printer
Dell XPS Tower 8960 Desktop, Intel Core i7-13700, NVIDIA GeForce RTX 4060, 16GB DDR5, 512GB SSD
Apple iPad Air 13-inch (M4), 128GB, Wi-Fi 7, Blue
As an affiliate, we earn on qualifying purchases.
How the Diagnostic Benchmark Works
Medical systems are often assessed using exam-style questions in which all case facts are supplied at once. Microsoft’s Sequential Diagnosis Benchmark instead starts with a limited presentation and makes further information available only when requested. This tests information gathering as well as the final answer, while assigning estimated costs to tests.
The source account says some methodological details come from a 2025 preprint, because the full Science article is behind a paywall and details may differ in the final version. In that preprint, a separate model judged answers against the published diagnosis. The account also reports that the gatekeeper could generate synthetic results for tests missing from a case record, and that test prices were estimates based on billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit charge. These are benchmark assumptions, not measured costs from live care.
The comparison has also changed since the earlier preprint. That version reported an 80% AI score against 20% for doctors on its hard-case comparison and used OpenAI’s o3 model. The Science paper’s figures cited here include an open-book physician group and report the system’s later results. The paper lists Microsoft-affiliated authors, including Microsoft AI chief executive Mustafa Suleyman. Peer review means the paper underwent journal review; it does not establish independent replication.
““Building Medical Superintelligence””
— Microsoft, in its blog post announcing the system
What the Study Cannot Establish
The study did not involve real patients receiving care, physical examinations, or live imaging. Its cases were historical records presented through a text-based simulator. It therefore remains unclear how the system would perform when information is incomplete, patient accounts are ambiguous, or clinicians must weigh findings that the benchmark does not capture.
The supplied account says the complex-case set contains no healthy patients or harmless explanations, limiting the ability to measure false alarms. It also says the physician and AI evaluations did not use identical sets in the earlier preprint: doctors were assessed on 56 held-out cases, while the AI was assessed on all 304. The final paper’s reported figures are the basis for the comparison above, but the complete final methods could not be checked in the source account. The degree to which the scores generalize beyond these selected cases, and whether the system’s estimated cost advantage would hold in actual hospitals, remain open questions.
Microsoft’s results have not, in the supplied material, been independently reproduced. The study also does not establish that MAI-DxO can replace doctors, specialists, or clinical teams. Those claims would require evidence from prospective testing and patient outcomes.
Prospective Testing Alongside Care
The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients will interact with the system alongside their normal care. Unlike a retrospective benchmark, this would test the system in a live clinical setting, though the supplied material does not specify the study’s start date, size, evaluation measures, or publication timeline.
Results from that work will be needed to assess how MAI-DxO performs with real patients and how its recommendations fit into clinicians’ decisions. Until those findings are available, the 83.2% figure should be read as a score on a defined set of difficult cases—not as a measure of accuracy in routine practice or proof of improved care.
Key Questions
What does “beats doctors” mean in this study?
It means MAI-DxO scored higher than the participating generalist physicians on specified sets of historical cases in a simulator. It does not mean the system was shown to outperform doctors treating real patients.
What scores did Microsoft report?
Microsoft reported 83.2% accuracy for MAI-DxO on 304 complex cases, versus 36.1% for physicians using textbooks and the internet and 22.3% without those resources. On 325 emergency-department cases, the reported scores were 80.3% and 46.8%, respectively.
Were specialists included in the comparison?
The supplied account describes the comparison physicians as primary-care doctors and hospital generalists working alone. It says specialists and physician teams were not included.
Did the study test the AI on real patients?
No. It used historical, text-based cases in a simulator. A prospective study at Beth Israel Deaconess Medical Center is planned to involve consenting patients alongside their usual care.
Does the result show that AI can replace doctors?
No. The study reports benchmark performance and does not establish that MAI-DxO can replace clinicians or improve patient outcomes. Its performance in live care remains to be tested.
Source: ThorstenMeyerAI.com
