Optimizing RAG in Programming Education: A Comparative Study of Models and Strategies

This study examines how Retrieval-Augmented Generation (RAG) enhances the effectiveness of generative artificial intelligence in programming education. This study compares commercial and open-source large language models within a RAG system, examining how retrieval design and prompt engineering affe...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE access Jg. 13; S. 190890 - 190903
Hauptverfasser: Yang, Christopher C. Y., Liu, Guan-Jie, Yu, Cheng-Jui
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Piscataway IEEE 2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:
Accuracy
Adaptation models
AI in education
Artificial intelligence
Biological system modeling
Cognition
Comparative studies
Costs
Education
Effectiveness
generative AI
Generative artificial intelligence
Large language models
LLMs
Machine learning
Open source software
Performance enhancement
Programming
programming education
Programming profession
Prompt engineering
Recall
Retrieval
Retrieval augmented generation
Retrieval-augmented generation (RAG)
retrieval-augmented generation assessment (RAGAS)
ISSN:2169-3536, 2169-3536
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:This study examines how Retrieval-Augmented Generation (RAG) enhances the effectiveness of generative artificial intelligence in programming education. This study compares commercial and open-source large language models within a RAG system, examining how retrieval design and prompt engineering affect response quality. By creating a database specific to machine learning courses and using the RAGAS (Retrieval-Augmented Generation Assessment) evaluation framework, a comparative analysis of five prominent models (GPT-4o, Claude-3.7-Sonnet, Gemini-2.0-Flash, Llama3.3-70b, and Ministral-8b) is conducted based on five quality metrics, namely Context Precision, Context Recall, Context Entities Recall, Faithfulness, and Response Relevance. The findings reveal considerable performance differences between different model types, with structured reasoning prompts (Chain-of-Thought and Take a Step Back) significantly improved the overall model performance. Re-ranking was identified as the most effective retrieval approach, especially in enhancing the performance of lightweight open-source models. This research provides empirical evidence for the effectiveness of economically feasible RAG systems in programming education, thus helping bridge the gap in the performance of commercial and open-source models and providing real-world solutions for resource-constrained educational settings.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2025.3629843