High Capability of ChatGPT to Answer Patient Questions on Male Infertility Reviewed by Reproductive Urologists
Sharath S. Reddy, MD1, Brooke A. Harnisch, MD2, Stanton Honig, MD1, Katherine Rotker, MD1.
1Department of Urology, Yale School of Medicine, New Haven, CT, USA, 2Department of Urology, University of Connecticut, Farmington, CT, USA.
BackgroundNovel technologies utilizing artificial intelligence (AI) have shown their potential to reduce provider workload and improve task efficiency in certain clinical settings. ChatGPT is an AI-powered chatbot developed by OpenAI which utilizes natural language processing (NLP) technology to provide human-like responses to open-ended questions posed by users. Recent studies have shown ChatGPT's ability to effectively provide diabetes counseling in lieu of trained educators, and develop a differential diagnosis when posed with standardized clinical vignettes. Within urology, male infertility is one subspecialty with a significant lack of high-quality online sources of information. The objective of this preliminary study was to evaluate the ability of ChatGPT to respond to common patient questions encountered by male infertility specialists.
MethodsWe composed a list of 15 questions commonly asked by patients pertaining to topics in male infertility. The questions were submitted to ChatGPT Plus with the prompt: "Answer the following questions in 50 words or less". 3 fellowship-trained andrologists then rated the accuracy of ChatGPT's responses on a scale from 1-5 (1 representing a completely inaccurate response and 5 representing a response in line with what they would recommend to patients). Scores were tabulated and summarized (Table 1). Descriptive statistics were calculated.
ResultsMedian score across all 15 responses from 3 raters was 4.5 (IQR 1.0). The most common rating was 5, indicating a high level of agreement between ChatGPT's responses and those of andrologists. The lowest score assigned was 3 (indicating partial accuracy), and was only received by 4 responses. Overall, ChatGPT was able to provide relatively accurate and concise responses to these common questions.
ConclusionsOur results demonstrate the capability of using ChatGPT as a potentially reliable source of information for patients on male infertility. Given the limited access to male infertility specialists, ChatGPT could possibly improve patient satisfaction and reduce provider burden. Similar to the revolution brought upon by Google and other search engines, ChatGPT and similar technologies may be the next stage for delivery of patient-focused medical advice. Further collection of real-world data is needed to better assess the accuracy and reliability of ChatGPT's responses. Urologists should be aware of the strengths and limitations of emerging NLP technologies as a medium for providing accurate health information to patients. 
Back to 2023 Abstracts