Comparison between Adam, AdaMax and Adam W optimizers to implement a Weather Forecast based on Neural Networks for the Andean city of Quito
The main function of an optimizer is to determine in what measure to change the weights and the learning rate of the neural network to reduce losses. One of the best known optimizers is Adam, which main advantage is the invariance of the magnitudes of the parameter updates with respect to the change...
Saved in:
| Published in: | 2021 IEEE Fifth Ecuador Technical Chapters Meeting (ETCM) pp. 1 - 6 |
|---|---|
| Main Authors: | , , , |
| Format: | Conference Proceeding |
| Language: | English |
| Published: |
IEEE
12.10.2021
|
| Subjects: | |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | The main function of an optimizer is to determine in what measure to change the weights and the learning rate of the neural network to reduce losses. One of the best known optimizers is Adam, which main advantage is the invariance of the magnitudes of the parameter updates with respect to the change of scale of the gradient. However, other optimizers are often chosen because they generalize in a better manner. AdamW is a variant of Adam where the weight decay is performed only after controlling the parameter-wise step size. In order to present a comparative scenario for optimizers in the present work, a Temperature Forecast for the Andean city of Quito using a neural network structure with uncertainty reduction was implemented and three optimizers (Adam, AdaMax and AdamW) were analyzed. In order to do the comparison three error metrics were obtained per hour in order to determine the effectiveness of the prediction. From the analysis it can be seen that Adam and AdaMax behave similarly reaching a maximum MSE per hour of 2.5°C nevertheless AdamW allows to reduce this error around 1.3°C. |
|---|---|
| AbstractList | The main function of an optimizer is to determine in what measure to change the weights and the learning rate of the neural network to reduce losses. One of the best known optimizers is Adam, which main advantage is the invariance of the magnitudes of the parameter updates with respect to the change of scale of the gradient. However, other optimizers are often chosen because they generalize in a better manner. AdamW is a variant of Adam where the weight decay is performed only after controlling the parameter-wise step size. In order to present a comparative scenario for optimizers in the present work, a Temperature Forecast for the Andean city of Quito using a neural network structure with uncertainty reduction was implemented and three optimizers (Adam, AdaMax and AdamW) were analyzed. In order to do the comparison three error metrics were obtained per hour in order to determine the effectiveness of the prediction. From the analysis it can be seen that Adam and AdaMax behave similarly reaching a maximum MSE per hour of 2.5°C nevertheless AdamW allows to reduce this error around 1.3°C. |
| Author | Fontaine, Allyx Lupera, Pablo Yacoubi, Samira El Llugsi, Ricardo |
| Author_xml | – sequence: 1 givenname: Ricardo surname: Llugsi fullname: Llugsi, Ricardo email: ricardo.llugsi@epn.edu.ec organization: Telecommunications and Information Networks (DETRI),National Polytechnic School Department of Electronics,Quito,Ecuador – sequence: 2 givenname: Samira El surname: Yacoubi fullname: Yacoubi, Samira El email: yacoubi@univ-perp.fr organization: Université de Perpignan Via Domitia,UMR Espace-Dev,Perpignan,France – sequence: 3 givenname: Allyx surname: Fontaine fullname: Fontaine, Allyx email: allyx.fontaine@univ-guyane.fr organization: Université de Guyane,UMR Espace-Dev,Cayenne,French Guiana – sequence: 4 givenname: Pablo surname: Lupera fullname: Lupera, Pablo email: pablo.lupera@epn.edu.ec organization: National Polytechnic School,Faculty of Electrical and Electronics Engineering,Quito,Ecuador |
| BookMark | eNotkNFKAzEQRSPog9Z-gSDzAbZms8nu5rEUq4JVhEIfy2R3QoPdZMmm1PoL_rSrloG5l-Fy4M4VO_fBE2O3GZ9mGdf3D6v5UuWFzKeCi2yqleZFlZ2xsS6rrCiUlNkwl-x7HtoOo-uDB0PpQORh1mB797uX-Anom78DrCF0ybXui2IPKYBrux215BMgrAnTliIsQqQa-wQGe2pgYL7SPuJukHQI8aMHGyIMUZj5htBD7dIRgoX3vUvhml1Y3PU0PumIrRZDj6fJy9vj83z2MtlWgk9saVBpLauSUCoqBc-lriquy8Ji3ehClFQPXgppjVFWoG6UMLUgjaZUlI_YzT_WEdGmi67FeNycPpT_AFlIYdc |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/ETCM53643.2021.9590681 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9781665441414 1665441410 |
| EndPage | 6 |
| ExternalDocumentID | 9590681 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IL CBEJK RIE RIL |
| ID | FETCH-LOGICAL-h820-f7ba599487ea45e720349880976facd9627ec76f424fbb5f2a9d52bc2e9ab75e3 |
| IEDL.DBID | RIE |
| IngestDate | Thu Jun 29 18:38:10 EDT 2023 |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-h820-f7ba599487ea45e720349880976facd9627ec76f424fbb5f2a9d52bc2e9ab75e3 |
| PageCount | 6 |
| ParticipantIDs | ieee_primary_9590681 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-Oct.-12 |
| PublicationDateYYYYMMDD | 2021-10-12 |
| PublicationDate_xml | – month: 10 year: 2021 text: 2021-Oct.-12 day: 12 |
| PublicationDecade | 2020 |
| PublicationTitle | 2021 IEEE Fifth Ecuador Technical Chapters Meeting (ETCM) |
| PublicationTitleAbbrev | ETCM |
| PublicationYear | 2021 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| Score | 1.7729363 |
| Snippet | The main function of an optimizer is to determine in what measure to change the weights and the learning rate of the neural network to reduce losses. One of... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 1 |
| SubjectTerms | Adam Adam W AdaMax Bayesian Convolutional Dropout Encoder-Decoder Neural Network Neural networks Optimizer Temperature distribution Time Series Time series analysis Uncertainty Urban areas Walk-Forward validation Weather forecasting Weight measurement |
| Title | Comparison between Adam, AdaMax and Adam W optimizers to implement a Weather Forecast based on Neural Networks for the Andean city of Quito |
| URI | https://ieeexplore.ieee.org/document/9590681 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LawIxEA5WeuipLVr6Zg49urqPrDFHkUoPVSxI9SZ5zFKh7hZdS-lf6J_uJC6WQi-97IYQsjBhZ74k38zH2J3VGTdG8CDUiQo45yKQCqOAHGaopY4S4eV8nh_FeNybz-Wkxlr7XBhE9OQzbLumv8u3hdm6o7KOTGXYdXnWB0J0d7laVdJvFMrO_XQwShOKsLTri6N2NfiXaooPGsPj_33uhDV_su9gso8rp6yGeYN9DfaCgVBxq6Bv1arlniP1ASq3vgNmUJAbWC0_CdhBWcByVVHEQcFsB_jACXIatSnBBTELNKcr0qFe6eVZ4RsgLAs0FPq5RZWDIbAORQZPW_IATTYdkhkegkpIIXihAB9kQqtUStqaoOIpuotXLum_JSSSKWOd_A4aavOYZ1qnWaykTWNtYpRKixSTM1bPixzPGSQSLSahyXhXcUxEzzqBq0wbV0dOYHzBGs6Oi7ddqYxFZcLLv7uv2JFbqsATRK5ZvVxv8YYdmvdyuVnf-vX9BgA1qaM |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Na8JAEF3EFtpTW7T0u3Po0WhMNq57FKlYqmJBqjfZjwkValI0ltK_0D_d2RgshV56SZZl2cAsmXm7-2YeY3dWx9wYwT1fh8rjnAtPKmx65DB9LXUzFLmcz_NAjEbt2UyOS6y2y4VBxJx8hnXXzO_ybWo27qisISPpt1ye9V7EeeBvs7WKtN-mLxv3k-4wCinG0r4vaNaL4b90U_Kw0Tv63wePWfUn_w7Gu8hywkqYVNhXdycZCAW7CjpWLWvuOVQfoBKbd8AUUnIEy8UnQTvIUlgsC5I4KJhuIR84SU6j1hm4MGaB5nRlOtQrvXJe-BoIzQINhU5iUSVgCK5DGsPThnxAlU16ZIa-V0gpeC8U4r1YaBVJSZsTVDxCd_XKJf25hEViZawT4EFDbR7wWOsoDpS0UaBNgFJpEWF4yspJmuAZg1CixdA3MW8pjqFoWydxFWvjKskJDM5Zxdlx_rYtljEvTHjxd_ctO-hPhoP54GH0eMkO3bJ5OV3kipWz1Qav2b55zxbr1U2-1t_8qazq |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+IEEE+Fifth+Ecuador+Technical+Chapters+Meeting+%28ETCM%29&rft.atitle=Comparison+between+Adam%2C+AdaMax+and+Adam+W+optimizers+to+implement+a+Weather+Forecast+based+on+Neural+Networks+for+the+Andean+city+of+Quito&rft.au=Llugsi%2C+Ricardo&rft.au=Yacoubi%2C+Samira+El&rft.au=Fontaine%2C+Allyx&rft.au=Lupera%2C+Pablo&rft.date=2021-10-12&rft.pub=IEEE&rft.spage=1&rft.epage=6&rft_id=info:doi/10.1109%2FETCM53643.2021.9590681&rft.externalDocID=9590681 |