Updates from the Frontier: Large Language Models, Diagnosis, and Clinical Reasoning

In an exciting demonstration of what could be the future of diagnosis and clinical reasoning, Daniel Restrepo, MD, SFHM, went toe-to-toe with an artificial intelligence (AI) tool known as Dr. CaBot. Prior to Dr. Restrepo and Dr. CaBot squaring off, Adam Rodman, MD, MPH, FACP, introduced the session by asking the audience for a definition of AI. Notably, he called on Robert Wachter, MD, who gave a concise and fitting definition: computers performing tasks typically requiring human intelligence. Dr. Rodman then walked the audience through a brief history of the use of computers for diagnosis. He highlighted physicians from decades ago who envisioned a future where offloading clinical work to computers would be beneficial. In doing so, Dr. Rodman attempted to make a historical case for tools like Dr. CaBot to improve the quality of diagnosis. He concluded his introductory remarks by detailing the history of clinicopathological conferences at his institution, in which master clinicians detail their approaches to challenging clinical cases.

Next, a case in the style of a clinicopathological conference was presented by Dr. Rodman, and the two discussants took turns explaining their approach and conclusions to the audience. The case was that of a man in his 70s with a history of bladder cancer, spinal surgery, and cardiac disease who presented with fever and back pain. Dr. Restrepo expertly navigated the case by weighing the importance of different signs and symptoms, which he used to organize his differential diagnosis and determine the most likely diagnosis. This was followed by an audio file of the AI tool, Dr. CaBot, presenting a summary of the case, including an explanation of what diagnosis they chose as the most likely and what the next best step in management would be. Dr. CaBot’s description included human speech patterns, including the use of “umm,” and human language, such as describing a key clinical data point as “spicy.” Ultimately, both Dr. Restrepo and Dr. CaBot were successful in identifying the correct diagnosis.

While the audience was impressed by Dr. Restrepo’s discussion, many members of the audience were struck by the potential impact of tools like Dr. CaBot, such that the line for questions from the audience proved longer than the duration of the session. In fact, Dr. Rodman was answering questions from the audience outside the conference room for more than 30 minutes after the session concluded.

After the discussants concluded their remarks, Dr. Rodman provided some additional commentary. Citing research, including his own, he encouraged the audience to harness the power of AI as a second opinion. Using no uncertain terms, he stated that it may become a moral imperative for clinicians to operationalize tools like Dr. CaBot for the sake of our patients. Emphasizing that medical errors continue to occur, the use of tools like Dr. CaBot as a second opinion could protect our patients from the harms of misdiagnosis. Dr. Rodman expects a future in which most diagnoses are double-checked by an AI tool. Further, he encouraged those in leadership roles to pursue scalable oversight of diagnoses by incorporating systems-level AI tools.

Facing questions from skeptics and enthusiasts in the audience, Dr. Rodman referred to a randomized controlled trial he is conducting that he believes will demonstrate that what Dr. CaBot does is more than a parlor trick and is a real skill. It is possible that publications like this will demonstrate the benefits of incorporating AI tools into a hospitalist’s workflow and lead to greater adoption. When asked about the impact on medical education, he shared concern about the risk of deskilling and stated that his practice has come to expect learners to take an initial attempt at clinical problem-solving without the use of an AI tool. 

Dr. James

Dr. James is an assistant professor of medicine at Emory University in Atlanta.