Peer-reviewed article
Freely accessible large language models for parent education in pediatric immune thrombocytopenia: an expert-rated cross-sectional study of safety, readability, and guideline concordance.
Dinç O, Akay E, Ates B ·
Summary
Written by AI, not yet AICS's view · claude-haiku-4-5-20251001This study evaluated how well three freely available artificial intelligence language models (GPT, Gemini, and Claude) answer parent questions about a blood disorder in children called immune thrombocytopenia. Researchers asked each model 40 standardized questions and had clinical experts rate the responses for accuracy, safety, clarity, and alignment with medical guidelines. All three models produced generally safe and accurate responses, though Gemini performed slightly better overall on communication quality.
Key points
- Gemini 3 Flash scored highest (32.78/35) on clinical communication, followed by Claude Sonnet 4.6 (30.60/35) and GPT-5.3-mini (29.98/35)
- Medical accuracy, guideline concordance, safety, and misinformation risk were largely similar across all three models
- Two safety events were detected in Gemini responses: overgeneralized autoimmune-testing advice and unsupported MMR vaccination counseling
- The study rated only single responses from each model, so cannot establish stable behavior or precise safety-event rates
Limitations
This study evaluated only one response per question from each model, so cannot determine how consistently they perform across multiple attempts or whether these findings represent stable model behavior. The small number of safety events prevents reliable statistical comparison of safety differences between models.
Abstract
Background Parents of children with immune thrombocytopenia (ITP) increasingly turn to freely available large language models (LLMs) when they need quick explanations. In pediatric hematology, such advice must be clear, safe, and aligned with clinical guidance, not simply fluent. Objective To assess the quality, safety, readability, and guideline/reference concordance of freely accessible LLM responses to parent-oriented questions about childhood ITP, as an evaluation of parent-facing educational content rather than diagnostic or predictive AI performance. Methods Forty standardized parent-facing ITP questions were submitted once to GPT-5.3-mini, Gemini 3 Flash, and Claude Sonnet 4.6 through free public web interfaces. Reporting followed STROBE and was cross-mapped to relevant TRIPOD-LLM items. Three model-name-blinded clinical reviewers independently rated 120 responses using the study-specific, expert-developed seven-domain AI-ITP Parent Response Score (AI-ITP-PRS; range, 7-35) and recorded unsafe-content and hallucination flags. Readability was measured with automated indices. Paired model comparisons used the Friedman test with Holm-adjusted Wilcoxon analyses; repeated-measures ANOVA and mixed-effects models were used as sensitivity analyses. Results All 360 reviewer-level assessments were complete. The unweighted clinical-communication composite AI-ITP-PRS differed significantly across models (Friedman chi-square = 63.52, p < 0.001; Kendall's W = 0.794). Gemini 3 Flash had the highest mean composite score (32.78 ± 2.54), followed by Claude Sonnet 4.6 (30.60 ± 1.39) and GPT-5.3-mini (29.98 ± 1.52), but this difference was driven primarily by completeness, comprehensibility, empathy/supportiveness, and objective readability. Medical accuracy, guideline/reference concordance, safety/emergency triage, and low harmful misinformation risk were largely similar across models. Two response-level unsafe-content and hallucination events were detected, both in Gemini 3 Flash responses (2/40, 5.0%; exact 95% CI, 0.6%-16.9%). The matched categorical comparison was not statistically significant, so these rare-event findings were interpreted descriptively and as hypothesis-generating. The events involved overgeneralized autoimmune-testing advice and unsupported MMR vaccination counseling. Conclusion Freely accessible LLMs can produce supportive, high-scoring parent-facing explanations about childhood ITP. However, the highest composite score in this single-response snapshot reflected stronger communication performance rather than definitive superiority across safety-sensitive domains. This study detected high-impact safety failure modes but cannot precisely estimate true safety-event rates, establish robust rare-event safety differences between models, or define stable model behavior across repeated generations.