SNAS: Fast Hardware-Aware Neural Architecture Search Methodology
Recently, automated neural architecture search (NAS) emerges as the default technique to find a state-of-the-art (SOTA) convolutional neural network (CNN) architecture with higher accuracy than manually designed architectures for image classification. In this article, we present a fast hardware-awar...
Saved in:
| Published in: | IEEE transactions on computer-aided design of integrated circuits and systems Vol. 41; no. 11; pp. 4826 - 4836 |
|---|---|
| Main Authors: | , , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York
IEEE
01.11.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 0278-0070, 1937-4151 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Recently, automated neural architecture search (NAS) emerges as the default technique to find a state-of-the-art (SOTA) convolutional neural network (CNN) architecture with higher accuracy than manually designed architectures for image classification. In this article, we present a fast hardware-aware NAS methodology, called S3NAS, reflecting the latest research results. It consists of three steps: 1) supernet design; 2) Single-Path NAS for fast architecture exploration; and 3) scaling and post-processing. In the first step, we design a supernet, superset of candidate networks with two features: one is to allow stages to have a different number of blocks, and the other is to enable blocks to have parallel layers of different kernel sizes (MixConv). Next, we perform a differential search by extending the Single-Path NAS technique to support the MixConv layer and to add a latency-aware loss term to reduce the hyperparameter search overhead. Finally, we use compound scaling to scale up the network maximally within the latency constraint. In addition, we add squeeze-and-excitation (SE) blocks and h-swish activation functions if beneficial in the post-processing step. Experiments with the proposed methodology on four different hardware platforms demonstrate the effectiveness of the proposed methodology. It is capable of finding networks with better latency-accuracy tradeoff than SOTA networks, and the network search can be done within 4 h using TPUv3. |
|---|---|
| AbstractList | Recently, automated neural architecture search (NAS) emerges as the default technique to find a state-of-the-art (SOTA) convolutional neural network (CNN) architecture with higher accuracy than manually designed architectures for image classification. In this article, we present a fast hardware-aware NAS methodology, called S3NAS, reflecting the latest research results. It consists of three steps: 1) supernet design; 2) Single-Path NAS for fast architecture exploration; and 3) scaling and post-processing. In the first step, we design a supernet, superset of candidate networks with two features: one is to allow stages to have a different number of blocks, and the other is to enable blocks to have parallel layers of different kernel sizes (MixConv). Next, we perform a differential search by extending the Single-Path NAS technique to support the MixConv layer and to add a latency-aware loss term to reduce the hyperparameter search overhead. Finally, we use compound scaling to scale up the network maximally within the latency constraint. In addition, we add squeeze-and-excitation (SE) blocks and h-swish activation functions if beneficial in the post-processing step. Experiments with the proposed methodology on four different hardware platforms demonstrate the effectiveness of the proposed methodology. It is capable of finding networks with better latency-accuracy tradeoff than SOTA networks, and the network search can be done within 4 h using TPUv3. |
| Author | Lee, Jaeseong Kang, Duseok Ha, Soonhoi Rhim, Jungsub |
| Author_xml | – sequence: 1 givenname: Jaeseong orcidid: 0000-0002-3311-3891 surname: Lee fullname: Lee, Jaeseong organization: Computer Science and Engineering Department, Seoul National University, Seoul, South Korea – sequence: 2 givenname: Jungsub surname: Rhim fullname: Rhim, Jungsub organization: Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea – sequence: 3 givenname: Duseok orcidid: 0000-0003-4985-0789 surname: Kang fullname: Kang, Duseok organization: Computer Science and Engineering Department, Seoul National University, Seoul, South Korea – sequence: 4 givenname: Soonhoi orcidid: 0000-0001-7472-9142 surname: Ha fullname: Ha, Soonhoi email: sha@snu.ac.kr organization: Computer Science and Engineering Department, Seoul National University, Seoul, South Korea |
| BookMark | eNp9kMFOwzAMhiM0JLbBAyAulTh3xEnaJJyoBmNIYxw2zlWauqxTWUeaCu3tabWJAwcutmz58y99IzLY1Tsk5BroBIDqu_U0eZwwymDCgQsl-BkZguYyFBDBgAwpkyqkVNILMmqaLaUgIqaH5GG1TFb3wcw0Ppgbl38bh2HS12CJrTNVkDi7KT1a33a7FZpuDF7Rb-q8ruqPwyU5L0zV4NWpj8n77Gk9nYeLt-eXabIILaPKhxw0EzqnIlOxtZnKkRspmIqNAQMac82VEAXmmZaFKESMjBlLM4uZgNhKPia3x797V3-12Ph0W7du10WmTDIVSaARdFdwvLKubhqHRbp35adxhxRo2otKe1FpLyo9ieoY-YexpTe-rHfembL6l7w5kiUi_ibpWMRaR_wHJXx2aQ |
| CODEN | ITCSDI |
| CitedBy_id | crossref_primary_10_1145_3665868 crossref_primary_10_1145_3687310 crossref_primary_10_1109_ACCESS_2023_3286030 crossref_primary_10_1109_TCAD_2022_3223852 crossref_primary_10_1016_j_rser_2025_116231 crossref_primary_10_1109_TCSII_2023_3241487 crossref_primary_10_1109_TNNLS_2023_3346169 crossref_primary_10_1109_TNNLS_2025_3552693 crossref_primary_10_1109_TAI_2024_3377147 crossref_primary_10_1016_j_engappai_2025_110775 |
| Cites_doi | 10.1109/ACCESS.2020.3022327 10.1007/978-3-030-46147-8_29 10.1109/CVPR.2019.00293 10.1109/CVPR.2018.00474 10.1109/CVPR.2018.00907 10.1109/CVPR.2019.01099 10.1109/CVPR.2017.761 10.1145/3400302.3415731 10.1109/ETFA46521.2020.9212130 10.1109/JSTSP.2020.2971421 10.1109/CVPR.2017.668 10.1007/978-3-030-58539-6_26 10.1109/CVPR.2017.195 10.1109/TPAMI.2019.2913372 10.1609/aaai.v34i04.5959 10.1007/978-3-319-46493-0_39 10.1109/CVPR42600.2020.01044 10.1109/ICCD46524.2019.00102 10.1109/ICCV.2019.00140 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TCAD.2021.3134843 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Xplore CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Xplore url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1937-4151 |
| EndPage | 4836 |
| ExternalDocumentID | 10_1109_TCAD_2021_3134843 9646995 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Research Foundation of Korea (NRF) funderid: 10.13039/501100003725 – fundername: Korea Government (MSIT) grantid: NRF-2019R1A2B5B02069406 funderid: 10.13039/501100003621 – fundername: TensorFlow Research Cloud (TFRC) Program |
| GroupedDBID | --Z -~X 0R~ 29I 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFS ACIWK ACNCT AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD HZ~ H~9 IBMZZ ICLAB IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P PZZ RIA RIE RNS TN5 VH1 VJK AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c208t-319249d04b86ccb8de3a74286aa1a19ed93844fedb97f4f46e22ac0bceb416c73 |
| IEDL.DBID | RIE |
| ISSN | 0278-0070 |
| IngestDate | Sun Nov 09 07:08:29 EST 2025 Sat Nov 29 03:31:49 EST 2025 Tue Nov 18 22:16:38 EST 2025 Wed Aug 27 02:14:20 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 11 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c208t-319249d04b86ccb8de3a74286aa1a19ed93844fedb97f4f46e22ac0bceb416c73 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0003-4985-0789 0000-0001-7472-9142 0000-0002-3311-3891 |
| PQID | 2728571051 |
| PQPubID | 85470 |
| PageCount | 11 |
| ParticipantIDs | ieee_primary_9646995 crossref_primary_10_1109_TCAD_2021_3134843 crossref_citationtrail_10_1109_TCAD_2021_3134843 proquest_journals_2728571051 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-11-01 |
| PublicationDateYYYYMMDD | 2022-11-01 |
| PublicationDate_xml | – month: 11 year: 2022 text: 2022-11-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on computer-aided design of integrated circuits and systems |
| PublicationTitleAbbrev | TCAD |
| PublicationYear | 2022 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 Liu (ref16) ref15 ref14 ref31 ref30 ref11 ref10 ref2 Cai (ref22) Li (ref1) ref18 Xu (ref19) ref24 ref23 ref26 ref20 Kim (ref3) 2015 Cai (ref17) ref21 ref27 ref29 Gupta (ref6) ref8 ref4 Mei (ref25) ref5 Ramachandran (ref28) Tan (ref7) Zoph (ref12) Tan (ref9) |
| References_xml | – volume-title: Proc. ICLR ident: ref17 article-title: ProxylessNAS: Direct neural architecture search on target task and hardware – volume-title: Proc. ICLR ident: ref1 article-title: Pruning filters for efficient ConvNets – volume-title: Proc. MLSys Workshop ident: ref6 article-title: Accelerator-aware neural network design using AutoML – ident: ref21 doi: 10.1109/ACCESS.2020.3022327 – ident: ref8 doi: 10.1007/978-3-030-46147-8_29 – volume-title: Proc. ICLR ident: ref22 article-title: Once-for-all: Train one network and specialize it for efficient deployment – ident: ref14 doi: 10.1109/CVPR.2019.00293 – ident: ref5 doi: 10.1109/CVPR.2018.00474 – start-page: 74 volume-title: Proc. BMVC ident: ref7 article-title: MixConv: Mixed depthwise convolutional kernels – ident: ref13 doi: 10.1109/CVPR.2018.00907 – ident: ref18 doi: 10.1109/CVPR.2019.01099 – ident: ref2 doi: 10.1109/CVPR.2017.761 – ident: ref15 doi: 10.1145/3400302.3415731 – ident: ref30 doi: 10.1109/ETFA46521.2020.9212130 – ident: ref20 doi: 10.1109/JSTSP.2020.2971421 – ident: ref26 doi: 10.1109/CVPR.2017.668 – ident: ref29 doi: 10.1007/978-3-030-58539-6_26 – ident: ref4 doi: 10.1109/CVPR.2017.195 – ident: ref10 doi: 10.1109/TPAMI.2019.2913372 – year: 2015 ident: ref3 article-title: Compression of deep convolutional neural networks for fast and low power mobile applications publication-title: arXiv:1511.06530 – volume-title: Proc. ICLR Workshop ident: ref28 article-title: Searching for activation functions – start-page: 6105 volume-title: Proc. ICML ident: ref9 article-title: EfficientNet: Rethinking model scaling for convolutional neural networks – ident: ref23 doi: 10.1609/aaai.v34i04.5959 – volume-title: Proc. Int. Conf. Learn. Represent. ident: ref19 article-title: PC-DARTS: Partial channel connections for memory-efficient architecture search – ident: ref31 doi: 10.1007/978-3-319-46493-0_39 – ident: ref24 doi: 10.1109/CVPR42600.2020.01044 – volume-title: Proc. ICLR ident: ref12 article-title: Neural architecture search with reinforcement learning – volume-title: Proc. ICLR ident: ref16 article-title: DARTS: Differentiable architecture search – ident: ref11 doi: 10.1109/ICCD46524.2019.00102 – volume-title: Proc. ICLR ident: ref25 article-title: AtomNAS: Fine-grained end-to-end neural architecture search – ident: ref27 doi: 10.1109/ICCV.2019.00140 |
| SSID | ssj0014529 |
| Score | 2.469639 |
| Snippet | Recently, automated neural architecture search (NAS) emerges as the default technique to find a state-of-the-art (SOTA) convolutional neural network (CNN)... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 4826 |
| SubjectTerms | Artificial neural networks Compounds Computer architecture Constraint-aware AutoML Convolution Convolutional neural networks convolutional neural networks (CNNs) Hardware Image classification Kernel Methodology Network latency neural architecture search (NAS) neural network design Neural networks Searching Space exploration |
| Title | SNAS: Fast Hardware-Aware Neural Architecture Search Methodology |
| URI | https://ieeexplore.ieee.org/document/9646995 https://www.proquest.com/docview/2728571051 |
| Volume | 41 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Xplore customDbUrl: eissn: 1937-4151 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014529 issn: 0278-0070 databaseCode: RIE dateStart: 19820101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwED-24YM--DXF6ZQ--CTWpWnaJD45xOGLQ9iEvZU0uYIgm2yd_vsmaTcVRfClpJCD9vJ1v_vl7gDOuY5zadIkVAZJyCzACBVaQ64oXJxPWrgkYr7YBB8OxWQiHxtwuY6FQUR_-QyvXNNz-Waml85V1pOpBXMyaUKT87SK1VozBo5A9P4UlzHWzuOawYyI7I3tT1kkSCMLUGMmWPztDPJFVX7sxP54Gez878N2Ybs2I4N-Ne570MDpPmx9SS7YhpvRsD-6DgZqUQaOn39Xcwz77hm4jBxe-pNECKp7x8GDryjtfe0H8DS4G9_eh3W9hFBTIkq7nTowZQjLRap1LgzGyiJfkSoVqUiikbFgrECTS16wgqVIqdIk15hbs0zz-BBa09kUjyCwq1IYTllOVMFitC-0oCh0SqQ2Oo46QFYazHSdTNzVtHjJPKggMnNKz5zSs1rpHbhYi7xWmTT-6tx2Wl53rBXcge5qmLJ6rS0yyqlIrKGURMe_S53AJnVBCz6CsAutcr7EU9jQb-XzYn7mp9EH0MPDQQ |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwED_mFNQHv6Y4ndoHn8S6NE3bxCeHOCZuQ9iEvZU0uYIgm-xD_32TrJuKIvhSUshBe_m63_1ydwDniQozoePIlxqJzwzA8CUaQy7PbZxPnNskYq7YRNLt8sFAPJbgchkLg4ju8hle2abj8vVIzayrrC5iA-ZEtAKrtnJWEa215Awsheg8KjZnrJnJBYcZEFHvm98yWJAGBqKGjLPw2ynkyqr82IvdAdPc_t-n7cBWYUh6jfnI70IJh3uw-SW9YAVuet1G79prysnUswz9uxyj37BPz-bkcNKfNII3v3nsdVxNaedt34en5l3_tuUXFRN8RQmfmg3VwilNWMZjpTKuMZQG-_JYykAGArUIOWM56kwkOctZjJRKRTKFmTHMVBIeQHk4GuIheGZdcp1QlhGZsxDNC80pchUTobQKgyqQhQZTVaQTt1UtXlIHK4hIrdJTq_S0UHoVLpYir_NcGn91rlgtLzsWCq5CbTFMabHaJilNKI-MqRQFR79LncF6q99pp-377sMxbFAbwuDiCWtQno5neAJr6m36PBmfuin1ASskxoo |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SNAS%3A+Fast+Hardware-Aware+Neural+Architecture+Search+Methodology&rft.jtitle=IEEE+transactions+on+computer-aided+design+of+integrated+circuits+and+systems&rft.au=Lee%2C+Jaeseong&rft.au=Rhim%2C+Jungsub&rft.au=Kang%2C+Duseok&rft.au=Ha%2C+Soonhoi&rft.date=2022-11-01&rft.issn=0278-0070&rft.eissn=1937-4151&rft.volume=41&rft.issue=11&rft.spage=4826&rft.epage=4836&rft_id=info:doi/10.1109%2FTCAD.2021.3134843&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TCAD_2021_3134843 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0278-0070&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0278-0070&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0278-0070&client=summon |