Artificial intelligence program ChatGPT has demonstrated promise in responding to medical questions, offering both accuracy and completeness in its answers, but it’s not yet a perfect solution, according to a recent study published in JAMA Network Open. ChatGPT responded to over 280 medical queries from various specialties with mostly to almost completely correct answers. This technology could potentially evolve into a reliable medical information resource, but certain challenges need to be addressed before it becomes a go-to source for medical inquiries.
1. ChatGPT’s Accuracy and Completeness
The study found that ChatGPT provided relatively accurate and comprehensive information with an average accuracy rating of 4.8 on a 6-point scale. While it was not perfect, ChatGPT’s responses scored well for easy, medium, and hard questions. In particular, it excelled in answering questions about common conditions, melanoma, and immunotherapy, with scores reaching 5.7 and 5.2.
The AI program also delivered fairly complete responses, scoring 2.5 on a 3-point scale. However, there were variances in its ability to respond to different question types, performing better in “yes/no” questions compared to open-ended queries.
2. Challenges and Room for Improvement
Some questions revealed limitations in ChatGPT’s capabilities. For example, it struggled with questions that contained phrases like “may be used,” which might have caused misunderstandings. Additionally, certain questions received incomplete or incorrect responses, reflecting potential language comprehension issues.
3. Need for Expert Oversight and Collaborative Use
The study also highlighted the potential benefits of having a second AI program review ChatGPT’s initial responses. This process improved the accuracy of answers and demonstrated the potential for the collaborative use of AI in addressing medical questions.
4. The Role of AI in Healthcare
Both the study researchers and experts emphasized that while AI like ChatGPT holds promise in providing medical information, it should not replace physicians. Instead, it can serve as a valuable tool for doctors and patients alike. Physicians could use AI to access additional information regarding complex diagnoses, while patients might utilize AI as a “health coach” for common illnesses. It’s unlikely that AI will fully replace doctors, but it may shift their roles to prioritize more complex tasks.
The Future of AI in Healthcare
ChatGPT has shown significant potential as a medical information resource, but there are challenges to overcome before it becomes a reliable, standalone solution. With further refinement and a focus on addressing language comprehension issues, AI chatbots could become valuable partners in healthcare, providing additional information and guidance.
