Large Language Models for the National Radiological Technologist Licensure Examination in Japan: Cross-Sectional Comparative Benchmarking and Evaluation of Model-Generated Items Study

Mock examinations are widely used in health professional education to assess learning and prepare candidates for national licensure. However, instructor-written multiple-choice items can vary in difficulty, coverage, and clarity. Recently, large language models (LLMs) have achieved high accuracy in...

Full description

Saved in:
Bibliographic Details
Published in:JMIR medical education Vol. 11; p. e81807
Main Authors: Ito, Toshimune, Ishibashi, Toru, Hayashi, Tatsuya, Kojima, Shinya, Sogabe, Kazumi
Format: Journal Article
Language:English
Published: Canada JMIR Publications 13.11.2025
Subjects:
Accuracy
Artificial intelligence
Benchmarking
Cross-Sectional Studies
Educational Measurement - methods
Educational Measurement - standards
Humans
Japan
Language
Large Language Models
Licenses
Licensure - standards
Multiple choice
Patient safety
Radiation
Safety management
Statistical analysis
Statistical significance
Technology, Radiologic - education
Technology, Radiologic - standards
Japan
large language models
item generation
medical education
licensing exam
radiology, educational evaluation
ISSN:2369-3762, 2369-3762
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Mock examinations are widely used in health professional education to assess learning and prepare candidates for national licensure. However, instructor-written multiple-choice items can vary in difficulty, coverage, and clarity. Recently, large language models (LLMs) have achieved high accuracy in medical examinations, highlighting their potential for assisting item-bank development; however, their educational quality remains insufficiently characterized. This study aimed to (1) identify the most accurate LLM for the Japanese National Examination for Radiological Technologists and (2) use the top model to generate blueprint-aligned multiple-choice questions and evaluate their educational quality. Four LLMs-OpenAI o3, o4-mini, o4-mini-high (OpenAI), and Gemini 2.5 Flash (Google)-were evaluated on all 200 items of the 77th Japanese National Examination for Radiological Technologists in 2025. Accuracy was analyzed for overall items and for 173 nonimage items. The best-performing model (o3) then generated 192 original items across 14 subjects by matching the official blueprint (image-based items were excluded). Subject-matter experts (≥5 y as coordinators and routine mock examination authors) independently rated each generated item on five criteria using a 5-point scale (1=unacceptable, 5=adoptable): item difficulty, factual accuracy, accuracy of content coverage, appropriateness of wording, and instructional usefulness. Cochran Q with Bonferroni-adjusted McNemar tests compared model accuracies, and one-sided Wilcoxon signed-rank tests assessed whether the median ratings exceeded 4. OpenAI o3 achieved the highest accuracy overall (90.0%; 95% CI 85.1%-93.4%) and on nonimage items (92.5%; 95% CI 87.6%-95.6%), significantly outperforming o4-mini on the full set (P=.02). Across models, accuracy differences on the non-image subset were not significant (Cochran Q, P=.10). Using o3, the 192 generated items received high expert ratings for item difficulty (mean, 4.29; 95% CI 4.11-4.46), factual accuracy (4.18; 95% CI 3.98-4.38), and content coverage (4.73; 95% CI 4.60-4.86). Ratings were comparatively lower for appropriateness of wording (3.92; 95% CI 3.73-4.11) and instructional usefulness (3.60; 95% CI 3.41-3.80). For these two criteria, the tests did not support a median rating >4 (one-sided Wilcoxon, P=.45 and P≥.99, respectively). Representative low-rated examples (ratings 1-2) and the rationale for those scores-such as ambiguous phrasing or generic explanations without linkage to stem cues-are provided in the supplementary materials. OpenAI o3 can generate radiological licensure items that align with national standards in terms of difficulty, factual correctness, and blueprint coverage. However, wording clarity and the pedagogical specificity of explanations were weaker and did not meet an adoptable threshold without further editorial refinement. These findings support a practical workflow in which LLMs draft syllabus-aligned items at scale, while faculty perform targeted edits to ensure clarity and formative feedback. Future studies should evaluate image-inclusive generation, use Application Programming Interface (API)-pinned model snapshots to increase reproducibility, and develop guidance to improve explanation quality for learner remediation.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:2369-3762
2369-3762
DOI:10.2196/81807