Chatbots in Healthcare

Not all medical chatbots are created equal, as a recent JAMA Network Open study reveals. The study found that some chatbots are better at tailoring health information to patient health literacy than others. Chatbots in Healthcare may not be ready for prime time.

Thank you for reading this post, don't forget to subscribe!

The report compared the free and paid versions of ChatGPT, showing that while the paid version initially provided more readable health information, the difference was minimal once researchers asked the chatbots to explain things at a sixth-grade reading level. The findings suggest that both versions of ChatGPT could potentially widen health disparities in terms of information access and literacy.

Chatbots like ChatGPT are becoming increasingly prominent in healthcare, showing potential in improving patient access to health information. However, their quality can vary. The study evaluated the free and paid versions of ChatGPT using the Flesch Reading Ease score for readability and the DISCERN instrument for consumer health information quality.

Researchers tested both versions using the five most popular cancer-related queries from 2021 to 2023. They found that while the paid version had slightly higher readability scores (52.6) compared to the free version (62.48) on a 100-point scale, both scores were deemed suboptimal.

The study revealed that prompting the free version of ChatGPT to explain concepts at a sixth-grade reading level improved its readability score to 71.55, outperforming the paid version under similar conditions. Even so, when both versions were asked to simplify answers to a sixth-grade reading level, the paid version scored slightly higher at 75.64.

Despite these improvements, the overall readability of responses was still problematic. Without the simplification prompt, responses were roughly at a 12th-grade reading level. Even with the prompt, they remained closer to an eighth- or tenth-grade level, possibly due to chatbot confusion about the request.

The study raises concerns about health equity. If the paid version of ChatGPT provides more accessible information, individuals with the means to purchase it might have a clear advantage. This disparity could exacerbate existing health inequities, especially for those using the free version.

The researchers concluded that until chatbots consistently provide information at a lower reading level, clinicians should guide patients on how to effectively use these tools and encourage them to request information at simpler reading levels.