Best practices for comparing optimization algorithms

Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. W...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org
Main Authors: Beiranvand, Vahid, Hare, Warren, Lucet, Yves
Format: Paper
Language:English
Published: Ithaca Cornell University Library, arXiv.org 24.09.2017
Subjects:
Algorithms
Benchmarks
Optimization
Optimization algorithms
ISSN:2331-8422
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. We provide suggestions for each step of the comparison process and highlight the pitfalls to avoid when evaluating the performance of optimization algorithms. We also discuss various methods of reporting the benchmarking results. Finally, some suggestions for future research are presented to improve the current benchmarking process.
AbstractList Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation. In this paper, we systematically review the benchmarking process of optimization algorithms, and discuss the challenges of fair comparison. We provide suggestions for each step of the comparison process and highlight the pitfalls to avoid when evaluating the performance of optimization algorithms. We also discuss various methods of reporting the benchmarking results. Finally, some suggestions for future research are presented to improve the current benchmarking process.
Author Lucet, Yves
Beiranvand, Vahid
Hare, Warren
Author_xml – sequence: 1
  givenname: Vahid
  surname: Beiranvand
  fullname: Beiranvand, Vahid
– sequence: 2
  givenname: Warren
  surname: Hare
  fullname: Hare, Warren
– sequence: 3
  givenname: Yves
  surname: Lucet
  fullname: Lucet, Yves
BookMark eNotjs1Kw0AURgexYG37AO4GXCfe-cvcLLVoFQpuui_TyU2d0mTiTCri0xvQ1dl8nO_csus-9sTYnYBSozHw4NJ3-CqFhboElFpesblUShSopbxhq5xPACArK41Rc6afKI98SM6PwVPmbUzcx25wKfRHHocxdOHHjSH23J2PMYXxo8tLNmvdOdPqnwu2e3nerV-L7fvmbf24LZyRWJAVpHUzJbSVB9NihaYxQqM6oPO1VAcC7y3RNBQOPAmBgLVARbohqtWC3f9phxQ_L1Pn_hQvqZ8e9xJsZbUxNapfhAdItw
ContentType Paper
Copyright 2017. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml – notice: 2017. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID 8FE
8FG
ABJCF
ABUWG
AFKRA
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
HCIFZ
L6V
M7S
PHGZM
PHGZT
PIMPY
PKEHL
PQEST
PQGLB
PQQKQ
PQUKI
PRINS
PTHSS
DOI 10.48550/arxiv.1709.08242
DatabaseName ProQuest SciTech Collection
ProQuest Technology Collection
Materials Science & Engineering Collection
ProQuest Central (Alumni)
ProQuest Central UK/Ireland
ProQuest Central Essentials - QC
ProQuest Central
ProQuest Technology Collection
ProQuest One
ProQuest Central Korea
SciTech Collection (ProQuest)
ProQuest Engineering Collection
Engineering Database
Proquest Central Premium
ProQuest One Academic (New)
ProQuest Publicly Available Content Database
ProQuest One Academic Middle East (New)
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Applied & Life Sciences
ProQuest One Academic (retired)
ProQuest One Academic UKI Edition
ProQuest Central China
Engineering Collection
DatabaseTitle Publicly Available Content Database
Engineering Database
Technology Collection
ProQuest One Academic Middle East (New)
ProQuest Central Essentials
ProQuest One Academic Eastern Edition
ProQuest Central (Alumni Edition)
SciTech Premium Collection
ProQuest One Community College
ProQuest Technology Collection
ProQuest SciTech Collection
ProQuest Central China
ProQuest Central
ProQuest One Applied & Life Sciences
ProQuest Engineering Collection
ProQuest One Academic UKI Edition
ProQuest Central Korea
Materials Science & Engineering Collection
ProQuest Central (New)
ProQuest One Academic
ProQuest One Academic (New)
Engineering Collection
DatabaseTitleList Publicly Available Content Database
Database_xml – sequence: 1
  dbid: PIMPY
  name: ProQuest Publicly Available Content Database
  url: http://search.proquest.com/publiccontent
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Physics
EISSN 2331-8422
Genre Working Paper/Pre-Print
GroupedDBID 8FE
8FG
ABJCF
ABUWG
AFKRA
ALMA_UNASSIGNED_HOLDINGS
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
FRJ
HCIFZ
L6V
M7S
M~E
PHGZM
PHGZT
PIMPY
PKEHL
PQEST
PQGLB
PQQKQ
PQUKI
PRINS
PTHSS
ID FETCH-LOGICAL-a528-e71e44d242f6c05f8685d51483b8ac923be0cc7eee711a0ce118089183e4dee93
IEDL.DBID M7S
IngestDate Mon Jun 30 09:41:32 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a528-e71e44d242f6c05f8685d51483b8ac923be0cc7eee711a0ce118089183e4dee93
Notes SourceType-Working Papers-1
ObjectType-Working Paper/Pre-Print-1
content type line 50
OpenAccessLink https://www.proquest.com/docview/2076745598?pq-origsite=%requestingapplication%
PQID 2076745598
PQPubID 2050157
ParticipantIDs proquest_journals_2076745598
PublicationCentury 2000
PublicationDate 20170924
PublicationDateYYYYMMDD 2017-09-24
PublicationDate_xml – month: 09
  year: 2017
  text: 20170924
  day: 24
PublicationDecade 2010
PublicationPlace Ithaca
PublicationPlace_xml – name: Ithaca
PublicationTitle arXiv.org
PublicationYear 2017
Publisher Cornell University Library, arXiv.org
Publisher_xml – name: Cornell University Library, arXiv.org
SSID ssj0002672553
Score 1.6344167
SecondaryResourceType preprint
Snippet Comparing, or benchmarking, of optimization algorithms is a complicated task that involves many subtle considerations to yield a fair and unbiased evaluation....
SourceID proquest
SourceType Aggregation Database
SubjectTerms Algorithms
Benchmarks
Optimization
Optimization algorithms
Title Best practices for comparing optimization algorithms
URI https://www.proquest.com/docview/2076745598
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV07T8MwED5BCxITb_EoVQZWQx5u7ExIRa1goIqgQ5kq53yBSjQtSan4-dhuChIDC6Nly_Lz7nx3_j6Ay0RSos3Th-Ui0YyjIpYZPcJyjEMldOSHuXZkE2IwkKNRktYOt6pOq1zLRCeo9Qytj9x6QmLBLZz4zfydWdYoG12tKTQ2oWlREgKXuvf07WMJY2Es5mgVzHTQXdeq_JwsrwJhYSplaAnaf4lgp1f6u_8d0R40UzWnch82qDiAbZfPidUh8K7px1t_gao8Y5p6q3Rzo6q8mZET0_oDpqfeXkyvi9dpdQTDfm94e8dqggSmOqFkJALiXJtZ5DH6nVzGsqONASSjTCo0lltGPqIgMg0D5SNZuDeZmEtMXBMl0TE0illBJ-ApgRFipAQJybMcE02ZUIJLDLXKgugUWus1GNeHvBr_LMDZ39XnsBNabeiCOS1oLMoPuoAtXC4mVdmGZrc3SB_bbu9MKb1_SJ-_AKASpjI
linkProvider ProQuest
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V07T8MwELaqFgQTb_EokAFGQ-KksTMgJB5Vq5aqEh26RY59gUr0QVIK_Cj-I2e3AYmBjYHZHpzz5b7zvT5CTiIBkcanD015pGmgJNAEcYSmKmSSa99lqbZkE7zTEf1-1C2Rj6IXxpRVFjbRGmo9ViZGbiIhIQ_MOPHLyTM1rFEmu1pQaMzVogXvr_hkyy-aN3i_p4zVb3vXDbpgFaCyxgQF7kEQaESmNFRuLRWhqGn0GoSfCKnQ3UnAVYoD4EZPugrMjDQRoeZDoAHM7CW0-BX0IlhkKwXvv0I6LOTooPvz3KmdFHYus7fB7MzjZiqmYIYP_ofFtzBWX_tnAlgnla6cQLZBSjDaJMu2WlXlWyS4wmM7RYNX7qDj7cyL6RGInTFaweGivdSRTw_4EdPHYb5Nen9xzh1SHo1HsEscyZWvlC85cBEkqYo0JFzyQCimZeL5e6RaiDxe_MJ5_C3v_d-Xj8lKo3fXjtvNTuuArDKD-zZtVSXlafYCh2RJzaaDPDuy6uKQ-I9v5xNGSf5C
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Best+practices+for+comparing+optimization+algorithms&rft.jtitle=arXiv.org&rft.au=Beiranvand%2C+Vahid&rft.au=Hare%2C+Warren&rft.au=Lucet%2C+Yves&rft.date=2017-09-24&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422&rft_id=info:doi/10.48550%2Farxiv.1709.08242