JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models

Code generation benchmarks such as HumanEval are widely adopted to evaluate LLMs' capabilities. However, after consolidating the latest 24 benchmarks, we noticed three significant imbalances. First, imbalanced programming language. 95.8% of benchmarks involve Python, while only 5 benchmarks inv...

Full description

Bibliographic Details

Published in:	IEEE/ACM International Conference on Automated Software Engineering : [proceedings] pp. 870 - 882
Main Authors:	Cao, Jialun, Chen, Zhiyong, Wu, Jiarong, Cheung, Shing-Chi, Xu, Chang
Format:	Conference Proceeding
Language:	English
Published:	ACM 27.10.2024
Subjects:	Benchmark testing Codes Java Large Language Model Measurement Object oriented modeling Object-Oriented Programming Program Synthesis Python Skeleton Software Software engineering Systematics
ISSN:	2643-1572
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!