Peer-reviewed article
How do large language models answer ADHD-related questions? A comparative study of ChatGPT, Gemini, and DeepSeek.
Bilgiç B, Turan S, Avcil S, Pekcanlar AA ·
Summary
Written by AI, not yet AICS's view · claude-haiku-4-5-20251001This study evaluates how three large language models (ChatGPT, Gemini, and DeepSeek) answer questions about ADHD. Researchers presented 22 common ADHD questions to each model and had psychiatrists rate the accuracy and quality of responses. All three models performed well overall, with ChatGPT scoring highest at 91% accuracy, followed by Gemini at 89% and DeepSeek at 87%.
Key points
- ChatGPT (GPT-4o) achieved the highest accuracy (91%) and reproducibility (89%) scores
- All three models showed high overall accuracy (87-91%) on ADHD-related questions
- ChatGPT performed better on treatment and long-term outcome questions, while Gemini and DeepSeek performed better on basic knowledge and diagnosis
- The study found significant differences in quality, usefulness, and reliability across the three models
- The authors conclude that LLMs may serve as supportive information sources but should not replace professional clinical evaluation
Limitations
This study evaluates information quality rather than child health outcomes or well-being. It does not assess whether access to these LLM responses actually improves health decision-making or child outcomes for ADHD patients and families.
Abstract
Objective Large language models (LLMs) are increasingly used by patients and caregivers as sources of health information. However, their performance in addressing attention-deficit/hyperactivity disorder (ADHD)-related questions has not been systematically compared. This study aimed to evaluate and compare the accuracy, reproducibility, quality, usefulness, and reliability of responses generated by ChatGPT (GPT-4o), Gemini, and DeepSeek R1. Methods In this cross-sectional comparative study, 22 commonly asked ADHD-related questions identified from publicly available digital sources were categorized into four domains: basic knowledge, diagnosis and assessment, treatment and medication, and long-term outcomes. Each question was presented to all three models using the same standardized prompts in separate chat sessions. The generated responses were independently evaluated by two specialists in child and adolescent psychiatry. Reproducibility was examined by repeating the same queries on different days. Descriptive statistics and non-parametric repeated-measures analyses were used to compare model performance. Results All models showed high overall accuracy, with mean scores of 91% for ChatGPT (GPT-4o), 89% for Gemini, and 87% for DeepSeek R1. Reproducibility followed a similar pattern (89%, 86%, and 84%, respectively). Gemini and DeepSeek performed relatively better in basic knowledge and diagnostic domains, whereas ChatGPT (GPT-4o) showed stronger performance in treatment and long-term outcome-related questions. Significant differences were observed in quality, usefulness, and reliability across models, with ChatGPT (GPT-4o) achieving the highest overall expert-rated scores. Conclusion Although large language models generally provided accurate responses to ADHD-related questions, notable differences were observed in the depth, clarity, and clinical usefulness of the information across models. These systems may serve as supportive sources of information for patients and caregivers; however, their responses should be interpreted with caution and should not replace professional clinical evaluation or medical advice.