Microsoft says its AI diagnosis system beat doctors on test cases in a Science study
Microsoft researchers say an AI system that works through a diagnosis step by step was more than twice as accurate as general physicians on hundreds of complex medical cases, according to a study published Thursday in the journal Science, Reuters reported. The results come from benchmark tests, not real patients; Microsoft says a hospital trial in Boston is next.
What the system does: The tool, called the Microsoft AI Diagnostic Orchestrator (MAI-DxO), decides what to ask about a patient’s history, which exams and tests to order, and updates its working diagnosis as results come in, Reuters reported. Microsoft said it operates like a panel of virtual physicians that propose diagnoses and challenge one another over several rounds.
The results: On 304 complex cases from The New England Journal of Medicine, MAI-DxO reached 83% accuracy, compared with 36% for physicians using textbooks and the internet and 22% for physicians without those resources, Microsoft said. On more typical emergency-department cases, it reached 80% accuracy versus 47% for physicians, according to Microsoft. Researchers also found the AI’s workups cost less than those designed by doctors, Reuters reported.
The limits: These are Microsoft’s figures from a test set of written cases, not results from treating patients. “That’s not real-world,” Dr. Dominic King of Microsoft AI said of older multiple-choice benchmarks, Reuters reported. The company said it is open-sourcing part of the benchmark so other researchers can check the findings.
What comes next: Microsoft said it will run a prospective study with Beth Israel Deaconess Medical Center in Boston in which patients interact with the system directly and its performance is compared with physicians’ assessments. Reuters reported that the first test will look at triaging patients in emergency department waiting rooms.