BACKGROUND: Generative Artificial Intelligence (AI) has become prominent as an oncology aid in clinical documentation, which has implications for treatment decisions. National guidelines favor synoptic reporting of prostate biopsy specimens to aid in risk stratification of prostate cancer (PCa). AI is a potential tool to reduce variation in algorithmic risk stratification. We questioned whether a trained AI model could review pathology reports, along with clinical vignettes, for PCa to provide an accurate risk assessment.METHODS: Retrospective review of localized PCa patients (pts) from January 2022 - October 2023 was performed. Following initial prostate biopsy, all pts were seen in consult and risk stratified based on NCCN guidelines by a urologic oncologist. We provided a generative AI model, ChatGPT, the PCa risk stratification algorithm from NCCN Version 1.2023, then queried the model to ensure accurate recall of the algorithm rules. PSA, PSA density, cT stage, and synoptic path report from each case were provided. The model was prompted to “risk stratify the patient.”RESULTS: 186 pts were reviewed. The model correctly risk stratified 117/186 (62.9%) pts. 62/186 (33.3%) pts were incorrectly stratified into alternative groups (Figure 1). Notably, 68 pts were correctly assigned as intermediate risk, but the model required further prompting for categorization into favorable or unfavorable risk. Once prompted, only 56/91 (61.5%) pts were correctly risk stratified (Figure 1). The model also recommended treatment modalities appropriate to the stated risk group, however with the caveat recommendation of consulting a urologic oncologist. CONCLUSIONS:
Generative AI demonstrated poor performance in algorithmic risk stratification. The implications of inaccurate assessments need to be considered as more healthcare systems look to incorporate generative AI into clinical tools. Close review is needed to vet any clinical tool. Ultimately when it comes to the nuance of the PCa, a trained provider is needed to interpret the information and recommendations in line with the patient’s goals and preferences.
