Optimizing RAG in Programming Education: A Comparative Study of Models and Strategies

This study examines how Retrieval-Augmented Generation (RAG) enhances the effectiveness of generative artificial intelligence in programming education. This study compares commercial and open-source large language models within a RAG system, examining how retrieval design and prompt engineering affe...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE access Ročník 13; s. 190890 - 190903
Hlavní autoři: Yang, Christopher C. Y., Liu, Guan-Jie, Yu, Cheng-Jui
Médium: Journal Article
Jazyk:angličtina
Vydáno: Piscataway IEEE 2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
Accuracy
Adaptation models
AI in education
Artificial intelligence
Biological system modeling
Cognition
Comparative studies
Costs
Education
Effectiveness
generative AI
Generative artificial intelligence
Large language models
LLMs
Machine learning
Open source software
Performance enhancement
Programming
programming education
Programming profession
Prompt engineering
Recall
Retrieval
Retrieval augmented generation
Retrieval-augmented generation (RAG)
retrieval-augmented generation assessment (RAGAS)
ISSN:2169-3536, 2169-3536
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:This study examines how Retrieval-Augmented Generation (RAG) enhances the effectiveness of generative artificial intelligence in programming education. This study compares commercial and open-source large language models within a RAG system, examining how retrieval design and prompt engineering affect response quality. By creating a database specific to machine learning courses and using the RAGAS (Retrieval-Augmented Generation Assessment) evaluation framework, a comparative analysis of five prominent models (GPT-4o, Claude-3.7-Sonnet, Gemini-2.0-Flash, Llama3.3-70b, and Ministral-8b) is conducted based on five quality metrics, namely Context Precision, Context Recall, Context Entities Recall, Faithfulness, and Response Relevance. The findings reveal considerable performance differences between different model types, with structured reasoning prompts (Chain-of-Thought and Take a Step Back) significantly improved the overall model performance. Re-ranking was identified as the most effective retrieval approach, especially in enhancing the performance of lightweight open-source models. This research provides empirical evidence for the effectiveness of economically feasible RAG systems in programming education, thus helping bridge the gap in the performance of commercial and open-source models and providing real-world solutions for resource-constrained educational settings.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2025.3629843