AI Chatbot Defeats Healthcare Professionals in Medical Tests

Researchers developed an AI chatbot that reportedly defeated medical practitioners in a series of medical tests.

Last week, doctors from the Royal College of Physicians in London were bested by an AI chatbot developed by researchers from the medicare startup Babylon Health. The demonstration marked the first time an artificial intelligence robot performed a clinical test.

The event was reportedly organized to show how a chatbot would perform at Royal College of General Practitioners final test for trainee doctors known as the MRCGP. For the past five years, Babylon Health Director Dr. Mobasher Butt said that general practitioners have only averaged a score of 72 percent at the exam. However, the company’s AI chatbot was able to get an 81 percent score.

“Tonight’s results clearly illustrate how AI-augmented health services can reduce the burden on healthcare systems around the world. Our mission is to put accessible and affordable health services into the hands of every person on Earth,” Dr. Ali Parsa, Babylon Health’s founder and CEO, said in a statement.

Read More: Inbenta Chatbot Uses Natural Language Processing for AI-minded Support

Babylon also collaborated with RCP, Dr. Megan Mahoney of Stanford University, and Dr. Arnold DoRosario of Yale-New Haven Health to further test the ability of the AI system against seven medical professionals. For the second trial, the company used about 100 independently-devised symptom sets known as vignettes.

The chatbot got a 98% percent accuracy percentage while the doctors achieved 64 to 94 percent accuracy.

“The accuracy of the AI was 98% when assessed against conditions seen most frequently in
primary care medicine. In comparison, when Babylon’s research team assessed experienced clinicians using the same measure, their accuracy ranged from 52-99%,” the company said in a press release statement.

“Babylon’s latest artificial intelligence capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on-par with top-rated practicing clinicians,” Dr. Parsa went on to say.

Babylon’s research paper on its AI chatbot entitled A Comparative Study of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis is available for download here.

https://youtu.be/4yiZF9WHBWk