First three years of the international verification of neural networks competition (VNN-COMP)
This paper presents a summary and meta-analysis of the first three iterations of the annual International Verification of Neural Networks Competition (VNN-COMP), held in 2020, 2021, and 2022. In the VNN-COMP, participants submit software tools that analyze whether given neural networks satisfy speci...
Gespeichert in:
| Veröffentlicht in: | International journal on software tools for technology transfer Jg. 25; H. 3; S. 329 - 339 |
|---|---|
| Hauptverfasser: | , , , , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
Berlin/Heidelberg
Springer Berlin Heidelberg
01.06.2023
Springer Nature B.V |
| Schlagworte: | |
| ISSN: | 1433-2779, 1433-2787 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | This paper presents a summary and meta-analysis of the first three iterations of the annual International Verification of Neural Networks Competition (VNN-COMP), held in 2020, 2021, and 2022. In the VNN-COMP, participants submit software tools that analyze whether given neural networks satisfy specifications describing their input-output behavior. These neural networks and specifications cover a variety of problem classes and tasks, corresponding to safety and robustness properties in image classification, neural control, reinforcement learning, and autonomous systems. We summarize the key processes, rules, and results, present trends observed over the last three years, and provide an outlook into possible future developments. |
|---|---|
| AbstractList | This paper presents a summary and meta-analysis of the first three iterations of the annual International Verification of Neural Networks Competition (VNN-COMP), held in 2020, 2021, and 2022. In the VNN-COMP, participants submit software tools that analyze whether given neural networks satisfy specifications describing their input-output behavior. These neural networks and specifications cover a variety of problem classes and tasks, corresponding to safety and robustness properties in image classification, neural control, reinforcement learning, and autonomous systems. We summarize the key processes, rules, and results, present trends observed over the last three years, and provide an outlook into possible future developments. |
| Author | Johnson, Taylor T. Müller, Mark Niklas Bak, Stanley Liu, Changliu Brix, Christopher |
| Author_xml | – sequence: 1 givenname: Christopher surname: Brix fullname: Brix, Christopher email: brix@cs.rwth-aachen.de organization: RWTH Aachen University – sequence: 2 givenname: Mark Niklas surname: Müller fullname: Müller, Mark Niklas organization: ETH Zurich – sequence: 3 givenname: Stanley surname: Bak fullname: Bak, Stanley organization: Stony Brook University – sequence: 4 givenname: Taylor T. surname: Johnson fullname: Johnson, Taylor T. organization: Vanderbilt University – sequence: 5 givenname: Changliu surname: Liu fullname: Liu, Changliu organization: Carnegie Mellon University |
| BookMark | eNp9kEFPAyEQhYnRxLb6Bzxt4kUPKCy7y-7RNFZNautBvRlC6aDU7VKBavrvpbtGEw-9MLyZ95Hh9dF-YxtA6ISSC0oIv_TxJBUmKcNREoazPdSjGWM45SXf_73z6hD1vV8QQnnBqx56GRnnQxLeHECyAel8YnWUkJgmgGtkMLaRdfIJzmijWrl1NLB2sd1A-LLu3SfKLlcQTDs-e55M8HB6_3B-hA60rD0c_9QBehpdPw5v8Xh6cze8GmPFChYwlwUvoZrNeJlnpU5VqWVG5oqC1kWVz3OWMqlyrossjyPFpE5znkkKlFLCNBug0-7dlbMfa_BBLOw6Ll97kZZFQauSsyq6ys6lnPXegRbKhPZHwUlTC0rENkzRhSlimKINU2QRTf-hK2eW0m12Q6yDfDQ3r-D-ttpBfQOz-Yke |
| CitedBy_id | crossref_primary_10_1016_j_scico_2025_103280 crossref_primary_10_1016_j_solener_2024_113219 crossref_primary_10_3390_math12121884 crossref_primary_10_1016_j_ijar_2024_109206 crossref_primary_10_3389_fcomp_2025_1557977 crossref_primary_10_1515_auto_2024_0132 crossref_primary_10_1017_S0956792525000099 crossref_primary_10_1145_3720417 crossref_primary_10_1007_s10009_023_00715_0 crossref_primary_10_1109_TSE_2024_3385378 crossref_primary_10_1016_j_neucom_2024_127936 |
| Cites_doi | 10.1007/978-3-030-25540-4_26 10.1145/3290354 10.1109/TNNLS.2018.2808470 10.29007/wfgr 10.1007/978-3-319-63387-9_5 10.1007/978-3-319-68167-2_19 10.48550/arXiv.2104.06718 10.1007/978-3-319-63387-9_1 10.1145/3498704 10.1109/CVPR.2016.90 10.1109/SP.2018.00058 10.1109/DASC.2016.7778091 10.1109/ICSE43902.2021.00036 10.48550/ARXIV.2109.00498 10.24963/ijcai.2021/351 10.48550/ARXIV.1604.07316 10.24963/ijcai.2021/364 10.48550/arXiv.2206.14772 10.1109/FormaliSE.2019.00012 10.29007/rgv8 10.1109/ICNN.1988.23914 10.1007/978-3-030-81685-8_6 10.48550/arXiv.2208.05740 10.48550/arXiv.2101.05844 10.48550/arXiv.2210.04871 10.1613/jair.2490 10.48550/arXiv.2212.10376 10.1561/2400000035 10.1145/3365365.3382213 10.1007/978-3-031-13185-1_7 10.1109/ICRA48506.2021.9561956 |
| ContentType | Journal Article |
| Copyright | The Author(s) 2023 The Author(s) 2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| Copyright_xml | – notice: The Author(s) 2023 – notice: The Author(s) 2023. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| DBID | C6C AAYXX CITATION 3V. 7SC 7XB 8AL 8AO 8FD 8FE 8FG 8FK 8G5 ABJCF ABUWG AFKRA ARAPS AZQEC BENPR BGLVJ CCPQU DWQXO GNUQQ GUQSH HCIFZ JQ2 K7- L6V L7M L~C L~D M0N M2O M7S MBDVC P5Z P62 PADUT PHGZM PHGZT PKEHL PQEST PQGLB PQQKQ PQUKI PTHSS Q9U |
| DOI | 10.1007/s10009-023-00703-4 |
| DatabaseName | Springer Nature OA Free Journals CrossRef ProQuest Central (Corporate) Computer and Information Systems Abstracts ProQuest Central (purchase pre-March 2016) Computing Database (Alumni Edition) ProQuest Pharma Collection Technology Research Database ProQuest SciTech Collection ProQuest Technology Collection ProQuest Central (Alumni) (purchase pre-March 2016) Research Library Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland Advanced Technologies & Computer Science Collection ProQuest Central Essentials Local Electronic Collection Information ProQuest Central Technology Collection ProQuest One Community College ProQuest Central ProQuest Central Student Research Library Prep SciTech Premium Collection ProQuest Computer Science Collection Computer Science Database ProQuest Engineering Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional Computing Database Research Library Engineering Database Research Library (Corporate) ProQuest advanced technologies & aerospace journals ProQuest Advanced Technologies & Aerospace Collection Research Library China ProQuest Central Premium ProQuest One Academic (New) ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition Engineering Collection ProQuest Central Basic |
| DatabaseTitle | CrossRef Research Library Prep Computer Science Database ProQuest Central Student Technology Collection Technology Research Database Computer and Information Systems Abstracts – Academic ProQuest One Academic Middle East (New) ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Essentials ProQuest Computer Science Collection Computer and Information Systems Abstracts ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College Research Library (Alumni Edition) ProQuest Pharma Collection ProQuest Central ProQuest One Applied & Life Sciences ProQuest Engineering Collection ProQuest Central Korea ProQuest Research Library Research Library China ProQuest Central (New) Advanced Technologies Database with Aerospace Engineering Collection Advanced Technologies & Aerospace Collection ProQuest Computing Engineering Database ProQuest Central Basic ProQuest Computing (Alumni Edition) ProQuest One Academic Eastern Edition ProQuest Technology Collection ProQuest SciTech Collection Computer and Information Systems Abstracts Professional Advanced Technologies & Aerospace Database ProQuest One Academic UKI Edition Materials Science & Engineering Collection ProQuest One Academic ProQuest One Academic (New) ProQuest Central (Alumni) |
| DatabaseTitleList | CrossRef Research Library Prep |
| Database_xml | – sequence: 1 dbid: BENPR name: ProQuest Central (subscription) url: https://www.proquest.com/central sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 1433-2787 |
| EndPage | 339 |
| ExternalDocumentID | 10_1007_s10009_023_00703_4 |
| GrantInformation_xml | – fundername: RWTH Aachen University (3131) |
| GroupedDBID | -59 -5G -BR -EM -Y2 -~C .4S .86 .DC .VR 06D 0R~ 0VY 1N0 1SB 203 29J 2J2 2JN 2JY 2KG 2KM 2LR 2P1 2VQ 2~H 30V 3V. 4.4 406 408 409 40D 40E 5GY 5VS 67Z 6NX 8AO 8FE 8FG 8FW 8G5 8TC 95- 95. 95~ 96X AAAVM AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AARHV AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYQN AAYTO AAYZH ABAKF ABBBX ABBXA ABDZT ABECU ABFTD ABFTV ABHLI ABHQN ABJCF ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABQSL ABSXP ABTEG ABTHY ABTKH ABTMW ABULA ABUWG ABWNU ABXPI ACAOD ACBXY ACDTI ACGFS ACHSB ACHXU ACIWK ACKNC ACMDZ ACMLO ACOKC ACOMO ACPIV ACSNA ACZOJ ADHHG ADHIR ADINQ ADKNI ADKPE ADRFC ADTPH ADURQ ADYFF ADZKW AEBTG AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AEOHA AEPYU AESKC AETLH AEVLU AEXYK AFBBN AFGCZ AFKRA AFLOW AFQWF AFWTZ AFZKB AGAYW AGDGC AGGDS AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMXSW AMYLF AMYQR AOCGG ARAPS ARCSS ARMRJ ASPBG AVWKF AXYYD AYJHY AZFZN AZQEC B-. BA0 BDATZ BENPR BGLVJ BGNMA BPHCQ BSONS C6C CAG CCPQU COF CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 DWQXO EBLON EBS EDO EIOEI EJD ESBYG FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRRFC FSGXE FWDCC GGCAI GGRSB GJIRD GNUQQ GNWQR GQ6 GQ7 GQ8 GUQSH GXS H13 HCIFZ HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ I-F I09 IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X I~Z J-C J0Z JBSCW JCJTX JZLTJ K6V K7- KDC KOV L6V LAS LLZTM M0N M2O M4Y M7S MA- MK~ ML~ N2Q NB0 NPVJJ NQJWS NU0 O9- O93 O9J OAM P62 P9O PADUT PF0 PQQKQ PROAC PT4 PT5 PTHSS Q2X QOS R89 R9I RIG RNI ROL RPX RSV RZK S16 S1Z S27 S3B SAP SCO SDH SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 TSG TSK TSV TUC TUS U2A UG4 UOJIU UTJUX UZXMN VC2 VFIZW W23 W48 WK8 YLTOR Z45 Z7R Z7X Z7Z Z83 Z88 ZMTXR AAPKM AAYXX ABBRH ABDBE ABFSG ABRTQ ACSTC ADHKG AEZWR AFDZB AFFHD AFHIU AFOHR AGQPQ AHPBZ AHWEU AIXLP ATHPR AYFIA CITATION PHGZM PHGZT PQGLB 7SC 7XB 8AL 8FD 8FK JQ2 L7M L~C L~D MBDVC PKEHL PQEST PQUKI Q9U |
| ID | FETCH-LOGICAL-c363t-7a678e9bb78548f2c8fa40dc1eff695d5323ac57f6458fac3af2574a1e11103f3 |
| IEDL.DBID | K7- |
| ISICitedReferencesCount | 32 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000998184400001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1433-2779 |
| IngestDate | Wed Nov 05 01:53:49 EST 2025 Tue Nov 18 22:39:57 EST 2025 Sat Nov 29 03:07:49 EST 2025 Fri Feb 21 02:42:08 EST 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Keywords | Deep learning Adversarial robustness Neural networks Certified robustness Machine learning Formal methods Formal verification |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c363t-7a678e9bb78548f2c8fa40dc1eff695d5323ac57f6458fac3af2574a1e11103f3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| OpenAccessLink | https://link.springer.com/10.1007/s10009-023-00703-4 |
| PQID | 2866198739 |
| PQPubID | 46652 |
| PageCount | 11 |
| ParticipantIDs | proquest_journals_2866198739 crossref_citationtrail_10_1007_s10009_023_00703_4 crossref_primary_10_1007_s10009_023_00703_4 springer_journals_10_1007_s10009_023_00703_4 |
| PublicationCentury | 2000 |
| PublicationDate | 20230600 2023-06-00 20230601 |
| PublicationDateYYYYMMDD | 2023-06-01 |
| PublicationDate_xml | – month: 6 year: 2023 text: 20230600 |
| PublicationDecade | 2020 |
| PublicationPlace | Berlin/Heidelberg |
| PublicationPlace_xml | – name: Berlin/Heidelberg – name: Heidelberg |
| PublicationTitle | International journal on software tools for technology transfer |
| PublicationTitleAbbrev | Int J Softw Tools Technol Transfer |
| PublicationYear | 2023 |
| Publisher | Springer Berlin Heidelberg Springer Nature B.V |
| Publisher_xml | – name: Springer Berlin Heidelberg – name: Springer Nature B.V |
| References | ZhangH.ChenH.XiaoC.GowalS.StanforthR.LiB.BoningD.S.HsiehC.Towards stable and efficient training of verifiably robust neural networks8th International Conference on Learning Representations, ICLR 20202020https://openreview.net/forum?id=Skxuk1rFwB De Palma, A., Bunel, R., Desmaison, Alban., Dvijotham, K., Kohli, P., Torr, P.H.S., Kumar, M.P.: Improved branch and bound for neural network verification via lagrangian decomposition. CoRR (2021). https://doi.org/10.48550/arXiv.2104.06718 Khedr, H., Ferlez, J., Shoukry, Y.: Effective formal verification of neural networks using the geometry of linear regions. arXiv preprint (2020). arXiv:2006.10864 Brix, C., Noll, T.: Debona: decoupled boundary network analysis for tighter bounds and faster adversarial robustness proofs. CoRR (2020). arXiv:2006.09040 [abs] SinghG.GehrT.MirmanM.PüschelM.VechevM.BengioS.WallachH.LarochelleH.GraumanK.Cesa-BianchiN.GarnettR.Fast and effective robustness certificationAdvances in Neural Information Processing Systems2018Red HookCurran Associates1080210813http://papers.nips.cc/paper/8278-fast-and-effective-robustness-certification.pdf FerrariC.MüllerM.N.JovanovicN.VechevM.T.Complete verification via multi-neuron relaxation guided branch-and-bound10th International Conference on Learning Representations, ICLR 2022, Virtual Event2022https://openreview.net/forum?id=l_amHf1oaK GehrT.MirmanM.Drachsler-CohenD.TsankovP.ChaudhuriS.VechevM.T.AI2: safety and robustness certification of neural networks with abstract interpretation2018 IEEE Symposium on Security and Privacy, SP 2018, Proceedings2018Los AlamitosIEEE Comput. Soc.31810.1109/SP.2018.00058 De Palma, A., Behl, H.S., Bunel, R., Torr, P.H.S., Kumar, M.P.: Scaling the convex barrier with sparse dual algorithms. CoRR (2021). https://doi.org/10.48550/arXiv.2101.05844 LuJ.KumarM.P.Neural network branching for neural network verificationInternational Conference on Learning Representations2020 TranH.D.BakS.XiangW.JohnsonT.T.Verification of deep convolutional neural networks using imagestars32nd International Conference on Computer-Aided Verification (CAV)2020BerlinSpringer ZhangH.WengT.W.ChenP.Y.HsiehC.J.DanielL.Efficient neural network robustness certification with general activation functionsAdv. Neural Inf. Process. Syst.20183149394948https://arxiv.org/pdf/1811.00866.pdf BunelR.De PalmaA.DesmaisonA.DvijothamK.KohliP.TorrP.H.KumarM.P.Lagrangian decomposition for neural network verificationConference on Uncertainty in Artificial Intelligence2020 HeK.ZhangX.RenS.SunJ.Deep residual learning for image recognition2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)201677077810.1109/CVPR.2016.90 Bak, S., Liu, C., Johnson, T.: The second international verification of neural networks competition (VNN-COMP 2021): summary and results (2021). https://doi.org/10.48550/ARXIV.2109.00498 MirmanM.GehrT.VechevM.T.DyJ.G.KrauseA.Differentiable abstract interpretation for provably robust neural networksProceedings of the 35th International Conference on Machine Learning, ICML 2018201835753583http://proceedings.mlr.press/v80/mirman18b.html ShriverD.ElbaumS.DwyerM.B.SilvaA.LeinoK.R.M.DNNV: a framework for deep neural network verificationComputer Aided Verification2021ChamSpringer13715010.1007/978-3-030-81685-8_6 Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., Zieba, K.: End to end learning for self-driving cars (2016). https://doi.org/10.48550/ARXIV.1604.07316 Vincent, J.A., Schwager, M.: Reachable polyhedral marching (RPM): a safety verification algorithm for robotic systems with deep neural network components (2021) KatzG.BarrettC.DillD.L.JulianK.KochenderferM.J.MajumdarR.KunčakV.Reluplex: An efficient SMT solver for verifying deep neural networksComputer Aided Verification2017ChamSpringer9711710.1007/978-3-319-63387-9_5 SzegedyC.ZarembaW.SutskeverI.BrunaJ.ErhanD.GoodfellowI.J.FergusR.BengioY.LeCunY.Intriguing properties of neural networks2nd International Conference on Learning Representations, ICLR 2014, Conference Track Proceedings2014http://arxiv.org/abs/1312.6199 De Palma, A., Bunel, R., Dvijotham, K., Kumar, M.P., Stanforth, R.: IBP regularization for verified adversarial robustness via branch-and-bound. (2022). https://doi.org/10.48550/arXiv.2206.14772 Müller, M.N., Makarchuk, G., Singh, G., Püschel, M., Vechev, M.: Prima: precise and general neural network certification via multi-neuron convex relaxations. arXiv preprint (2021). arXiv:2103.03638 HenriksenP.LomuscioA.Deepsplit: an efficient splitting method for neural network verification via indirect effect analysisProceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI21)202110.24963/ijcai.2021/351 Wang, S., Zhang, H., Xu, K., Lin, X., Jana, S., Hsieh, C.J., Kolter, Z.: Beta-CROWN: efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. arXiv preprint (2021). arXiv:2103.06624 ZhouC.Computation of optical flow using a neural networkIEEE 1988 International Conference on Neural Networks1988717810.1109/ICNN.1988.23914 Müller, M.N., Brix, C., Bak, S., Liu, C., Johnson, T.T.: The third international verification of neural networks competition (VNN-COMP 2022): summary and results (2022). https://doi.org/10.48550/arXiv.2212.10376 HuangX.KwiatkowskaM.WangS.WuM.MajumdarR.KunčakV.Safety verification of deep neural networksComputer Aided Verification2017ChamSpringer32910.1007/978-3-319-63387-9_1 KatzG.HuangD.A.IbelingD.JulianK.LazarusC.LimR.ShahP.ThakoorS.WuH.ZeljićA.The Marabou framework for verification and analysis of deep neural networksInternational Conference on Computer Aided Verification2019BerlinSpringer44345210.1007/978-3-030-25540-4_26 SinghG.GehrT.PüschelM.VechevM.Boosting robustness certification of neural networksProc. International Conference on Learning Representations (ICLR)2019 MadryA.MakelovA.SchmidtL.TsiprasD.VladuA.Towards deep learning models resistant to adversarial attacks6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings2018https://openreview.net/forum?id=rJzIBfZAb De PalmaA.BehlH.S.BunelR.TorrP.H.S.KumarM.P.Scaling the convex barrier with active sets9th International Conference on Learning Representations, ICLR 2021, Conference Track Proceedings2021https://openreview.net/forum?id=uQfOy7LrlTR BunelR.LuJ.TurkaslanI.KohliP.TorrP.KumarM.P.Branch and bound for piecewise linear neural network verificationJ. Mach. Learn. Res.2020211574161240737751498.68153 FischerM.SprecherC.DimitrovD.I.SinghG.VechevM.T.ShohamS.VizelY.Shared certificates for neural network verificationComputer Aided Verification – 34th International Conference, CAV 2022, Proceedings, Part I2022BerlinSpringer12714810.1007/978-3-031-13185-1_7 ShiZ.WangY.ZhangH.YiJ.HsiehC.RanzatoM.BeygelzimerA.DauphinY.N.LiangP.VaughanJ.W.Fast certified robust training with short warmupAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Virtual20211833518349https://proceedings.neurips.cc/paper/2021/hash/988f9153ac4fd966ea302dd9ab9bae15-Abstract.html VNN-COMP2020 report. https://www.overleaf.com/project/5f0c85e8d15dc10001749fa9. Accessed: 2022-08-28 SinghG.GanvirR.PüschelM.VechevM.Beyond the single neuron convex barrier for neural network certificationAdvances in Neural Information Processing Systems2019Red HookCurran Associates1509815109 IsacO.BarrettC.ZhangM.KatzG.Neural network verification with proof production22nd International Conference on Formal Methods in Computer-Aided Design (FMCAD)2022 Bai, J., Lu, F., Zhang, K., et al.: Onnx: open neural network exchange (2019). https://github.com/onnx/onnx BunelR.TurkaslanI.TorrP.H.S.KohliP.MudigondaP.K.BengioS.WallachH.LarochelleH.GraumanK.Cesa-BianchiN.GarnettR.A unified view of piecewise linear neural network verificationAdvances in Neural Information Processing Systems2018Red HookCurran Associates47954804https://proceedings.neurips.cc/paper/2018/hash/be53d253d6bc3258a8160556dda3e9b2-Abstract.html TjengV.XiaoK.Y.TedrakeR.Evaluating robustness of neural networks with mixed integer programmingICLR2019 ZhangH.WengT.ChenP.HsiehC.DanielL.BengioS.WallachH.M.LarochelleH.GraumanK.Cesa-BianchiN.GarnettR.Efficient neural network robustness certification with general activation functionsAdvances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018201849444953https://proceedings.neurips.cc/paper/2018/hash/d04863f100d59b3eb688a11f95b0ae60-Abstract.html TranH.D.MusauP.LopezD.M.YangX.NguyenL.V.XiangW.JohnsonT.T.Parallelizable reachability analysis algorithms for feed-forward neural networksProceedings of the 7th International Workshop on Formal Methods in Software Engineering (FormaliSE’19), FormaliSE ’192019PiscatawayIEEE Press314010.1109/FormaliSE.2019.00012 BotoevaE.KouvarosP.KronqvistJ.LomuscioA.MisenerR.Efficient verification of neural networks via dependency analysisProceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI20)2020Menlo ParkAAAI Press LopezD.M.MusauP.TranH.D.DuttaS.CarpenterT.J.IvanovR.JohnsonT.T.FrehseG.AlthoffM.Arch-comp19 category report: artificial intelligence and neural network control systems (ainncs) for continuous and hybrid systems plantsARCH19. 6th International Workshop on Applied Verification of Continuous and Hybrid Systems201910311910.29007/rgv8 LopezD.M.AlthoffM.BenetL.ChenX.FanJ.ForetsM.HuangC.JohnsonT.T.LadnerT.LiW.SchillingC.ZhuQ.FrehseG.AlthoffM.SchoitschE.GuiochetJ.Arch-comp22 category report: artificial intelligence and neural network control systems (AINNCS) for continuous and hybrid systems plantsProceedings of 9th International Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH22)202214218410.29007/wfgr TranH.D.MusauP.LopezD.M.YangX.NguyenL.V.XiangW.JohnsonT.T.Star-based reachability analysis for deep neural networks23rd Inter H.D. Tran (703_CR55) 2020 M. Fischer (703_CR16) 2022 C. Szegedy (703_CR52) 2014 703_CR2 703_CR3 703_CR1 G. Katz (703_CR25) 2017 C. Liu (703_CR30) 2021; 4 E. Botoeva (703_CR8) 2020 R. Bunel (703_CR12) 2018 D. Shriver (703_CR46) 2021 703_CR19 X. Huang (703_CR23) 2017 703_CR59 A. De Palma (703_CR40) 2021 P. Henriksen (703_CR22) 2021 703_CR53 R. Bunel (703_CR10) 2020 A. Madry (703_CR34) 2018 G. Singh (703_CR48) 2019 F. Serre (703_CR44) 2021 K. Xu (703_CR62) 2020 O. Isac (703_CR39) 2022 P. Henriksen (703_CR21) 2020 H.D. Tran (703_CR58) 2020 H. Zhang (703_CR68) 2018; 31 703_CR7 703_CR4 703_CR5 J. Ferlez (703_CR14) 2020 P. Kouvaros (703_CR28) 2021 703_CR41 703_CR43 703_CR9 G. Katz (703_CR26) 2019 703_CR42 J. Lu (703_CR33) 2020 W. Xiang (703_CR61) 2018; 29 M. Balunovic (703_CR6) 2020 H.D. Tran (703_CR56) 2019 I.J. Goodfellow (703_CR18) 2015 H. Zhang (703_CR65) 2020 K.D. Julian (703_CR24) 2016 D.M. Lopez (703_CR32) 2019 K. Xu (703_CR63) 2021 M. Mirman (703_CR35) 2018 L. Xu (703_CR64) 2008; 32 C. Zhou (703_CR69) 1988 703_CR38 703_CR37 D. Shriver (703_CR47) 2021 V. Tjeng (703_CR54) 2019 703_CR36 R. Bunel (703_CR11) 2020; 21 G. Singh (703_CR51) 2019 H.D. Tran (703_CR57) 2019 G. Singh (703_CR50) 2019; 3 703_CR60 R. Ehlers (703_CR13) 2017 C. Ferrari (703_CR15) 2022 703_CR27 C. Liu (703_CR29) 2019 K. He (703_CR20) 2016 G. Singh (703_CR49) 2018 703_CR66 Z. Shi (703_CR45) 2021 H. Zhang (703_CR67) 2018 T. Gehr (703_CR17) 2018 D.M. Lopez (703_CR31) 2022 |
| References_xml | – reference: BunelR.LuJ.TurkaslanI.KohliP.TorrP.KumarM.P.Branch and bound for piecewise linear neural network verificationJ. Mach. Learn. Res.2020211574161240737751498.68153 – reference: KouvarosP.LomuscioA.Towards scalable complete verification of ReLU neural networks via dependency-based branchingProceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI21)202110.24963/ijcai.2021/364 – reference: BunelR.De PalmaA.DesmaisonA.DvijothamK.KohliP.TorrP.H.KumarM.P.Lagrangian decomposition for neural network verificationConference on Uncertainty in Artificial Intelligence2020 – reference: Tacchella, A., Pulina, L., Guidotti, D., Demarchi, S.: The verification of neural networks library (VNN-LIB) (2019). https://www.vnnlib.org – reference: VNN-COMP2020 report. https://www.overleaf.com/project/5f0c85e8d15dc10001749fa9. Accessed: 2022-08-28 – reference: TranH.D.MusauP.LopezD.M.YangX.NguyenL.V.XiangW.JohnsonT.T.Parallelizable reachability analysis algorithms for feed-forward neural networksProceedings of the 7th International Workshop on Formal Methods in Software Engineering (FormaliSE’19), FormaliSE ’192019PiscatawayIEEE Press314010.1109/FormaliSE.2019.00012 – reference: Brix, C., Noll, T.: Debona: decoupled boundary network analysis for tighter bounds and faster adversarial robustness proofs. CoRR (2020). arXiv:2006.09040 [abs] – reference: EhlersR.Formal verification of piece-wise linear feed-forward neural networksInternational Symposium on Automated Technology for Verification and Analysis201726928610.1007/978-3-319-68167-2_19 – reference: HenriksenP.LomuscioA.Efficient neural network verification via adaptive refinement and adversarial searchProceedings of the 24th European Conference on Artificial Intelligence (ECAI20)2020 – reference: HeK.ZhangX.RenS.SunJ.Deep residual learning for image recognition2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)201677077810.1109/CVPR.2016.90 – reference: ZhangH.WengT.ChenP.HsiehC.DanielL.BengioS.WallachH.M.LarochelleH.GraumanK.Cesa-BianchiN.GarnettR.Efficient neural network robustness certification with general activation functionsAdvances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018201849444953https://proceedings.neurips.cc/paper/2018/hash/d04863f100d59b3eb688a11f95b0ae60-Abstract.html – reference: BalunovicM.VechevM.T.Adversarial training and provable defenses: bridging the gap8th International Conference on Learning Representations, ICLR 20202020https://openreview.net/forum?id=SJxSDxrKDr – reference: GehrT.MirmanM.Drachsler-CohenD.TsankovP.ChaudhuriS.VechevM.T.AI2: safety and robustness certification of neural networks with abstract interpretation2018 IEEE Symposium on Security and Privacy, SP 2018, Proceedings2018Los AlamitosIEEE Comput. Soc.31810.1109/SP.2018.00058 – reference: Bak, S., Liu, C., Johnson, T.: The second international verification of neural networks competition (VNN-COMP 2021): summary and results (2021). https://doi.org/10.48550/ARXIV.2109.00498 – reference: GoodfellowI.J.ShlensJ.SzegedyC.BengioY.LeCunY.Explaining and harnessing adversarial examples3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings2015http://arxiv.org/abs/1412.6572 – reference: Müller, M.N., Brix, C., Bak, S., Liu, C., Johnson, T.T.: The third international verification of neural networks competition (VNN-COMP 2022): summary and results (2022). https://doi.org/10.48550/arXiv.2212.10376 – reference: TranH.D.BakS.XiangW.JohnsonT.T.Verification of deep convolutional neural networks using imagestars32nd International Conference on Computer-Aided Verification (CAV)2020BerlinSpringer – reference: Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T.A., Kohli, P.: On the effectiveness of interval bound propagation for training verifiably robust models. CoRR (2018). arXiv:1810.12715 [abs] – reference: XuK.ShiZ.ZhangH.WangY.ChangK.W.HuangM.KailkhuraB.LinX.HsiehC.J.Automatic perturbation analysis for scalable certified robustness and beyondAdvances in Neural Information Processing Systems2020 – reference: SerreF.MüllerC.SinghG.PüschelM.VechevM.Scaling polyhedral neural network verification on GPUsProc. Machine Learning and Systems (MLSys)2021 – reference: BunelR.TurkaslanI.TorrP.H.S.KohliP.MudigondaP.K.BengioS.WallachH.LarochelleH.GraumanK.Cesa-BianchiN.GarnettR.A unified view of piecewise linear neural network verificationAdvances in Neural Information Processing Systems2018Red HookCurran Associates47954804https://proceedings.neurips.cc/paper/2018/hash/be53d253d6bc3258a8160556dda3e9b2-Abstract.html – reference: De Palma, A., Bunel, R., Desmaison, Alban., Dvijotham, K., Kohli, P., Torr, P.H.S., Kumar, M.P.: Improved branch and bound for neural network verification via lagrangian decomposition. CoRR (2021). https://doi.org/10.48550/arXiv.2104.06718 – reference: FerrariC.MüllerM.N.JovanovicN.VechevM.T.Complete verification via multi-neuron relaxation guided branch-and-bound10th International Conference on Learning Representations, ICLR 2022, Virtual Event2022https://openreview.net/forum?id=l_amHf1oaK – reference: Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., Zieba, K.: End to end learning for self-driving cars (2016). https://doi.org/10.48550/ARXIV.1604.07316 – reference: IsacO.BarrettC.ZhangM.KatzG.Neural network verification with proof production22nd International Conference on Formal Methods in Computer-Aided Design (FMCAD)2022 – reference: Simple Adversarial Generator. https://github.com/stanleybak/simple_adversarial_generator. Accessed: 2022-09-13 – reference: De Palma, A., Behl, H.S., Bunel, R., Torr, P.H.S., Kumar, M.P.: Scaling the convex barrier with sparse dual algorithms. CoRR (2021). https://doi.org/10.48550/arXiv.2101.05844 – reference: KatzG.BarrettC.DillD.L.JulianK.KochenderferM.J.MajumdarR.KunčakV.Reluplex: An efficient SMT solver for verifying deep neural networksComputer Aided Verification2017ChamSpringer9711710.1007/978-3-319-63387-9_5 – reference: SinghG.GehrT.MirmanM.PüschelM.VechevM.BengioS.WallachH.LarochelleH.GraumanK.Cesa-BianchiN.GarnettR.Fast and effective robustness certificationAdvances in Neural Information Processing Systems2018Red HookCurran Associates1080210813http://papers.nips.cc/paper/8278-fast-and-effective-robustness-certification.pdf – reference: Wang, S., Zhang, H., Xu, K., Lin, X., Jana, S., Hsieh, C.J., Kolter, Z.: Beta-CROWN: efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. arXiv preprint (2021). arXiv:2103.06624 – reference: FerlezJ.ShoukryY.AReN: assured ReLU NN architecture for model predictive control of LTI systemsProceedings of the 23rd International Conference on Hybrid Systems: Computation and Control, HSCC ’202020New YorkACM10.1145/3365365.338221307300847 – reference: MirmanM.GehrT.VechevM.T.DyJ.G.KrauseA.Differentiable abstract interpretation for provably robust neural networksProceedings of the 35th International Conference on Machine Learning, ICML 2018201835753583http://proceedings.mlr.press/v80/mirman18b.html – reference: SinghG.GehrT.PüschelM.VechevM.Boosting robustness certification of neural networksProc. International Conference on Learning Representations (ICLR)2019 – reference: HenriksenP.LomuscioA.Deepsplit: an efficient splitting method for neural network verification via indirect effect analysisProceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI21)202110.24963/ijcai.2021/351 – reference: MadryA.MakelovA.SchmidtL.TsiprasD.VladuA.Towards deep learning models resistant to adversarial attacks6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings2018https://openreview.net/forum?id=rJzIBfZAb – reference: FischerM.SprecherC.DimitrovD.I.SinghG.VechevM.T.ShohamS.VizelY.Shared certificates for neural network verificationComputer Aided Verification – 34th International Conference, CAV 2022, Proceedings, Part I2022BerlinSpringer12714810.1007/978-3-031-13185-1_7 – reference: ZhangH.ChenH.XiaoC.GowalS.StanforthR.LiB.BoningD.S.HsiehC.Towards stable and efficient training of verifiably robust neural networks8th International Conference on Learning Representations, ICLR 20202020https://openreview.net/forum?id=Skxuk1rFwB – reference: Bak, S.: Execution-guided overapproximation (ego) for improving scalability of neural network verification (2020) – reference: LopezD.M.MusauP.TranH.D.DuttaS.CarpenterT.J.IvanovR.JohnsonT.T.FrehseG.AlthoffM.Arch-comp19 category report: artificial intelligence and neural network control systems (ainncs) for continuous and hybrid systems plantsARCH19. 6th International Workshop on Applied Verification of Continuous and Hybrid Systems201910311910.29007/rgv8 – reference: Zhang, H., Wang, S., Xu, K., Li, L., Li, B., Jana, S., Hsieh, C., Kolter, J.Z.: General cutting planes for bound-propagation-based neural network verification. CoRR (2022). https://doi.org/10.48550/arXiv.2208.05740 – reference: TjengV.XiaoK.Y.TedrakeR.Evaluating robustness of neural networks with mixed integer programmingICLR2019 – reference: ShiZ.WangY.ZhangH.YiJ.HsiehC.RanzatoM.BeygelzimerA.DauphinY.N.LiangP.VaughanJ.W.Fast certified robust training with short warmupAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Virtual20211833518349https://proceedings.neurips.cc/paper/2021/hash/988f9153ac4fd966ea302dd9ab9bae15-Abstract.html – reference: TranH.D.YangX.LopezD.M.MusauP.NguyenL.V.XiangW.BakS.JohnsonT.T.NNV: the neural network verification tool for deep neural networks and learning-enabled cyber-physical systems32nd International Conference on Computer-Aided Verification (CAV)2020 – reference: SinghG.GehrT.PüschelM.VechevM.An abstract domain for certifying neural networksProc. ACM Program. Lang.20193POPL41:141:3010.1145/3290354 – reference: LopezD.M.AlthoffM.BenetL.ChenX.FanJ.ForetsM.HuangC.JohnsonT.T.LadnerT.LiW.SchillingC.ZhuQ.FrehseG.AlthoffM.SchoitschE.GuiochetJ.Arch-comp22 category report: artificial intelligence and neural network control systems (AINNCS) for continuous and hybrid systems plantsProceedings of 9th International Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH22)202214218410.29007/wfgr – reference: ZhangH.WengT.W.ChenP.Y.HsiehC.J.DanielL.Efficient neural network robustness certification with general activation functionsAdv. Neural Inf. Process. Syst.20183149394948https://arxiv.org/pdf/1811.00866.pdf – reference: XuL.HutterF.HoosH.H.Leyton-BrownK.SATzilla: portfolio-based algorithm selection for SATJ. Artif. Intell. Res.200832156560610.1613/jair.24901182.68272 – reference: SinghG.GanvirR.PüschelM.VechevM.Beyond the single neuron convex barrier for neural network certificationAdvances in Neural Information Processing Systems2019Red HookCurran Associates1509815109 – reference: HuangX.KwiatkowskaM.WangS.WuM.MajumdarR.KunčakV.Safety verification of deep neural networksComputer Aided Verification2017ChamSpringer32910.1007/978-3-319-63387-9_1 – reference: KatzG.HuangD.A.IbelingD.JulianK.LazarusC.LimR.ShahP.ThakoorS.WuH.ZeljićA.The Marabou framework for verification and analysis of deep neural networksInternational Conference on Computer Aided Verification2019BerlinSpringer44345210.1007/978-3-030-25540-4_26 – reference: TranH.D.MusauP.LopezD.M.YangX.NguyenL.V.XiangW.JohnsonT.T.Star-based reachability analysis for deep neural networks23rd International Symposium on Formal Methods (FM’19)2019BerlinSpringer – reference: XuK.ZhangH.WangS.WangY.JanaS.LinX.HsiehC.J.Fast and complete: enabling complete neural network verification with rapid and massively parallel incomplete verifiersInternational Conference on Learning Representations2021https://openreview.net/forum?id=nVZtXBI6LNn – reference: Bai, J., Lu, F., Zhang, K., et al.: Onnx: open neural network exchange (2019). https://github.com/onnx/onnx – reference: JulianK.D.LopezJ.BrushJ.S.OwenM.P.KochenderferM.J.Policy compression for aircraft collision avoidance systems2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC)201611010.1109/DASC.2016.7778091 – reference: ShriverD.ElbaumS.G.DwyerM.B.Reducing DNN properties to enable falsification with adversarial attacks43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021202127528710.1109/ICSE43902.2021.00036IEEE – reference: LiuC.ArnonT.LazarusC.StrongC.BarrettC.KochenderferM.J.Algorithms for verifying deep neural networksFound. Trends Optim.202143–424440410.1561/2400000035 – reference: LuJ.KumarM.P.Neural network branching for neural network verificationInternational Conference on Learning Representations2020 – reference: SzegedyC.ZarembaW.SutskeverI.BrunaJ.ErhanD.GoodfellowI.J.FergusR.BengioY.LeCunY.Intriguing properties of neural networks2nd International Conference on Learning Representations, ICLR 2014, Conference Track Proceedings2014http://arxiv.org/abs/1312.6199 – reference: XiangW.TranH.JohnsonT.T.Output reachable set estimation and verification for multilayer neural networksIEEE Trans. Neural Netw. Learn. Syst.2018291157775783386788510.1109/TNNLS.2018.2808470 – reference: ZhouC.Computation of optical flow using a neural networkIEEE 1988 International Conference on Neural Networks1988717810.1109/ICNN.1988.23914 – reference: De PalmaA.BehlH.S.BunelR.TorrP.H.S.KumarM.P.Scaling the convex barrier with active sets9th International Conference on Learning Representations, ICLR 2021, Conference Track Proceedings2021https://openreview.net/forum?id=uQfOy7LrlTR – reference: Vincent, J.A., Schwager, M.: Reachable polyhedral marching (RPM): a safety verification algorithm for robotic systems with deep neural network components (2021) – reference: LiuC.ArnonT.LazarusC.KochenderferM.J.Neuralverification.jl: algorithms for verifying deep neural networksICLR 2019 Debugging Machine Learning Models Workshop2019https://debug-ml-iclr2019.github.io/cameraready/DebugML-19_paper_22.pdf – reference: BotoevaE.KouvarosP.KronqvistJ.LomuscioA.MisenerR.Efficient verification of neural networks via dependency analysisProceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI20)2020Menlo ParkAAAI Press – reference: De Palma, A., Bunel, R., Dvijotham, K., Kumar, M.P., Stanforth, R.: IBP regularization for verified adversarial robustness via branch-and-bound. (2022). https://doi.org/10.48550/arXiv.2206.14772 – reference: Müller, M.N., Eckert, F., Fischer, M., Vechev, M.T.: Certified training: small boxes are all you need. CoRR (2022). https://doi.org/10.48550/arXiv.2210.04871 – reference: Müller, M.N., Makarchuk, G., Singh, G., Püschel, M., Vechev, M.: Prima: precise and general neural network certification via multi-neuron convex relaxations. arXiv preprint (2021). arXiv:2103.03638 – reference: Khedr, H., Ferlez, J., Shoukry, Y.: Effective formal verification of neural networks using the geometry of linear regions. arXiv preprint (2020). arXiv:2006.10864 – reference: ShriverD.ElbaumS.DwyerM.B.SilvaA.LeinoK.R.M.DNNV: a framework for deep neural network verificationComputer Aided Verification2021ChamSpringer13715010.1007/978-3-030-81685-8_6 – start-page: 443 volume-title: International Conference on Computer Aided Verification year: 2019 ident: 703_CR26 doi: 10.1007/978-3-030-25540-4_26 – volume: 3 start-page: 41:1 issue: POPL year: 2019 ident: 703_CR50 publication-title: Proc. ACM Program. Lang. doi: 10.1145/3290354 – volume: 29 start-page: 5777 issue: 11 year: 2018 ident: 703_CR61 publication-title: IEEE Trans. Neural Netw. Learn. Syst. doi: 10.1109/TNNLS.2018.2808470 – start-page: 18335 volume-title: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Virtual year: 2021 ident: 703_CR45 – volume-title: 2nd International Conference on Learning Representations, ICLR 2014, Conference Track Proceedings year: 2014 ident: 703_CR52 – start-page: 142 volume-title: Proceedings of 9th International Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH22) year: 2022 ident: 703_CR31 doi: 10.29007/wfgr – volume-title: 8th International Conference on Learning Representations, ICLR 2020 year: 2020 ident: 703_CR6 – start-page: 97 volume-title: Computer Aided Verification year: 2017 ident: 703_CR25 doi: 10.1007/978-3-319-63387-9_5 – ident: 703_CR3 – volume-title: 32nd International Conference on Computer-Aided Verification (CAV) year: 2020 ident: 703_CR58 – volume-title: 32nd International Conference on Computer-Aided Verification (CAV) year: 2020 ident: 703_CR55 – start-page: 10802 volume-title: Advances in Neural Information Processing Systems year: 2018 ident: 703_CR49 – start-page: 4944 volume-title: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018 year: 2018 ident: 703_CR67 – start-page: 269 volume-title: International Symposium on Automated Technology for Verification and Analysis year: 2017 ident: 703_CR13 doi: 10.1007/978-3-319-68167-2_19 – start-page: 4795 volume-title: Advances in Neural Information Processing Systems year: 2018 ident: 703_CR12 – volume-title: International Conference on Learning Representations year: 2020 ident: 703_CR33 – ident: 703_CR42 doi: 10.48550/arXiv.2104.06718 – volume-title: 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings year: 2015 ident: 703_CR18 – ident: 703_CR27 – volume-title: ICLR year: 2019 ident: 703_CR54 – volume: 31 start-page: 4939 year: 2018 ident: 703_CR68 publication-title: Adv. Neural Inf. Process. Syst. – start-page: 3 volume-title: Computer Aided Verification year: 2017 ident: 703_CR23 doi: 10.1007/978-3-319-63387-9_1 – start-page: 3575 volume-title: Proceedings of the 35th International Conference on Machine Learning, ICML 2018 year: 2018 ident: 703_CR35 – volume-title: Proceedings of the 24th European Conference on Artificial Intelligence (ECAI20) year: 2020 ident: 703_CR21 – ident: 703_CR38 doi: 10.1145/3498704 – ident: 703_CR4 – start-page: 770 volume-title: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) year: 2016 ident: 703_CR20 doi: 10.1109/CVPR.2016.90 – start-page: 3 volume-title: 2018 IEEE Symposium on Security and Privacy, SP 2018, Proceedings year: 2018 ident: 703_CR17 doi: 10.1109/SP.2018.00058 – ident: 703_CR9 – start-page: 1 volume-title: 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC) year: 2016 ident: 703_CR24 doi: 10.1109/DASC.2016.7778091 – start-page: 275 volume-title: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021 year: 2021 ident: 703_CR47 doi: 10.1109/ICSE43902.2021.00036 – ident: 703_CR5 doi: 10.48550/ARXIV.2109.00498 – volume-title: Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI21) year: 2021 ident: 703_CR22 doi: 10.24963/ijcai.2021/351 – volume-title: 23rd International Symposium on Formal Methods (FM’19) year: 2019 ident: 703_CR57 – volume-title: 9th International Conference on Learning Representations, ICLR 2021, Conference Track Proceedings year: 2021 ident: 703_CR40 – volume-title: Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI20) year: 2020 ident: 703_CR8 – ident: 703_CR7 doi: 10.48550/ARXIV.1604.07316 – ident: 703_CR1 – volume-title: ICLR 2019 Debugging Machine Learning Models Workshop year: 2019 ident: 703_CR29 – volume-title: International Conference on Learning Representations year: 2021 ident: 703_CR63 – volume-title: Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI21) year: 2021 ident: 703_CR28 doi: 10.24963/ijcai.2021/364 – ident: 703_CR43 doi: 10.48550/arXiv.2206.14772 – start-page: 31 volume-title: Proceedings of the 7th International Workshop on Formal Methods in Software Engineering (FormaliSE’19), FormaliSE ’19 year: 2019 ident: 703_CR56 doi: 10.1109/FormaliSE.2019.00012 – start-page: 103 volume-title: ARCH19. 6th International Workshop on Applied Verification of Continuous and Hybrid Systems year: 2019 ident: 703_CR32 doi: 10.29007/rgv8 – start-page: 71 volume-title: IEEE 1988 International Conference on Neural Networks year: 1988 ident: 703_CR69 doi: 10.1109/ICNN.1988.23914 – volume-title: Proc. International Conference on Learning Representations (ICLR) year: 2019 ident: 703_CR51 – volume-title: Proc. Machine Learning and Systems (MLSys) year: 2021 ident: 703_CR44 – volume-title: Conference on Uncertainty in Artificial Intelligence year: 2020 ident: 703_CR10 – start-page: 137 volume-title: Computer Aided Verification year: 2021 ident: 703_CR46 doi: 10.1007/978-3-030-81685-8_6 – ident: 703_CR66 doi: 10.48550/arXiv.2208.05740 – ident: 703_CR41 doi: 10.48550/arXiv.2101.05844 – ident: 703_CR37 doi: 10.48550/arXiv.2210.04871 – volume-title: 10th International Conference on Learning Representations, ICLR 2022, Virtual Event year: 2022 ident: 703_CR15 – volume-title: 6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings year: 2018 ident: 703_CR34 – ident: 703_CR2 – start-page: 15098 volume-title: Advances in Neural Information Processing Systems year: 2019 ident: 703_CR48 – ident: 703_CR19 – ident: 703_CR53 – volume: 32 start-page: 565 issue: 1 year: 2008 ident: 703_CR64 publication-title: J. Artif. Intell. Res. doi: 10.1613/jair.2490 – ident: 703_CR36 doi: 10.48550/arXiv.2212.10376 – volume: 21 start-page: 1574 year: 2020 ident: 703_CR11 publication-title: J. Mach. Learn. Res. – volume: 4 start-page: 244 issue: 3–4 year: 2021 ident: 703_CR30 publication-title: Found. Trends Optim. doi: 10.1561/2400000035 – volume-title: 8th International Conference on Learning Representations, ICLR 2020 year: 2020 ident: 703_CR65 – volume-title: 22nd International Conference on Formal Methods in Computer-Aided Design (FMCAD) year: 2022 ident: 703_CR39 – volume-title: Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control, HSCC ’20 year: 2020 ident: 703_CR14 doi: 10.1145/3365365.3382213 – start-page: 127 volume-title: Computer Aided Verification – 34th International Conference, CAV 2022, Proceedings, Part I year: 2022 ident: 703_CR16 doi: 10.1007/978-3-031-13185-1_7 – ident: 703_CR59 doi: 10.1109/ICRA48506.2021.9561956 – volume-title: Advances in Neural Information Processing Systems year: 2020 ident: 703_CR62 – ident: 703_CR60 |
| SSID | ssj0017679 |
| Score | 2.5708885 |
| Snippet | This paper presents a summary and meta-analysis of the first three iterations of the annual International Verification of Neural Networks Competition... |
| SourceID | proquest crossref springer |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 329 |
| SubjectTerms | Aircraft Benchmarks Classification Competition Computer Science Deep learning Explanation Paradigms Leveraging Analytic Intuition Image classification Machine learning Neural networks Software Software Engineering Software Engineering/Programming and Operating Systems Software utilities Specifications Theory of Computation Trends Verification |
| SummonAdditionalLinks | – databaseName: SpringerLINK Contemporary 1997-Present dbid: RSV link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV3LS8MwGA86PXhxPnE6JQcPigbWJl2SowyHB63Dx9hFSpomMJBO1in43_sla7cpKugxzYPyPX_kewShYyVNlimwfjzNJHGAl4gg0ESmIjPKAEBlvlD4msexGAxkrywKK6ps9yok6S31QrGbv8gPKXE9aihhy2gF3J1w6nh335_FDnjbd9gDIEBJyLksS2W-P-OzO5pjzC9hUe9tuvX__ecGWi_RJb6YisMmWjL5FqpXLzfgUpG30VN3CKgPT4CRBr-DsBd4ZGFo8HDxhhCDnLtUIj90K1z3S_icT3PHC6w96vZZX_ikH8ekc3vTO91Bj93Lh84VKR9aIJq26YRwBS7LyDTlwBphQy2sYq1MB8batoyyiIZU6YjbNotgSlNlQdOZCgxYyha1dBfV8lFu9hC2WgtAAIADmWYiZGkAByseBWBHBdNRAwUVvRNddiF3j2E8J_P-yY5-CdAv8fRLWAOdzfa8THtw_Lq6WbExKfWxSEIBOEQKTmUDnVdsm0__fNr-35YfoLXQc95d0zRRbTJ-NYdoVb9NhsX4yMvpBw7z4KI priority: 102 providerName: Springer Nature |
| Title | First three years of the international verification of neural networks competition (VNN-COMP) |
| URI | https://link.springer.com/article/10.1007/s10009-023-00703-4 https://www.proquest.com/docview/2866198739 |
| Volume | 25 |
| WOSCitedRecordID | wos000998184400001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVPQU databaseName: Computer Science Database customDbUrl: eissn: 1433-2787 dateEnd: 20241207 omitProxy: false ssIdentifier: ssj0017679 issn: 1433-2779 databaseCode: K7- dateStart: 20190101 isFulltext: true titleUrlDefault: http://search.proquest.com/compscijour providerName: ProQuest – providerCode: PRVPQU databaseName: Engineering Database customDbUrl: eissn: 1433-2787 dateEnd: 20241207 omitProxy: false ssIdentifier: ssj0017679 issn: 1433-2779 databaseCode: M7S dateStart: 20190101 isFulltext: true titleUrlDefault: http://search.proquest.com providerName: ProQuest – providerCode: PRVPQU databaseName: ProQuest advanced technologies & aerospace journals customDbUrl: eissn: 1433-2787 dateEnd: 20241207 omitProxy: false ssIdentifier: ssj0017679 issn: 1433-2779 databaseCode: P5Z dateStart: 20190101 isFulltext: true titleUrlDefault: https://search.proquest.com/hightechjournals providerName: ProQuest – providerCode: PRVPQU databaseName: ProQuest Central (subscription) customDbUrl: eissn: 1433-2787 dateEnd: 20241207 omitProxy: false ssIdentifier: ssj0017679 issn: 1433-2779 databaseCode: BENPR dateStart: 20190101 isFulltext: true titleUrlDefault: https://www.proquest.com/central providerName: ProQuest – providerCode: PRVPQU databaseName: Research Library customDbUrl: eissn: 1433-2787 dateEnd: 20241207 omitProxy: false ssIdentifier: ssj0017679 issn: 1433-2779 databaseCode: M2O dateStart: 20190101 isFulltext: true titleUrlDefault: https://search.proquest.com/pqrl providerName: ProQuest – providerCode: PRVAVX databaseName: SpringerLINK Contemporary 1997-Present customDbUrl: eissn: 1433-2787 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0017679 issn: 1433-2779 databaseCode: RSV dateStart: 19971201 isFulltext: true titleUrlDefault: https://link.springer.com/search?facet-content-type=%22Journal%22 providerName: Springer Nature |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV3dSxwxEB_qx4MvaluLZ1Xy0IeWGmo-9pJ9KlU8BHU9_EIEWbLZBA7kTt2z0P--k1zWU0FffBnIZjcsOx_57cxkBuCbyV1dG7R-qqpzGgAv1YxZmle6dsYhQJXxoPChKgp9eZn3k8OtSWmVrU2Mhroe2eAj_8U17iT4gyzy37d3NHSNCtHV1EJjBuYY5yzI-YGij1EE1Y219hASCMqVytOhmXR0LoYFuKCh4o2g8vnGNEWbLwKkcd_pLb33jZdhMSFO8mciIh_hgxt-gqW2mwNJyv0ZrnsDRIJkjMx15B8qQENGHoeODJ56DQnKfkgvisNwR6iIiZeHk3zyhtiIxGMmGPl-URR09_io_2MFznt7Z7v7NDVfoFZ0xZgqg9uYy6tKIbu051Z7I7dry5z33TyrM8GFsZnyXZnhlBXGo_ZLwxxaz23hxReYHY6GbhWIt1YjKkBsKK3UXFYMFzYqY2hbtbRZB1j75UubKpOHBhk35bSmcuBWidwqI7dK2YGfj8_cTupyvHn3esuiMuloU07504GtlsnT6ddXW3t7ta-wwKNcBVfNOsyO7x_cBszbv-NBc78Jczt7Rf9kM0oq0iN-HKg6RdrPrpCenF78B_2_8Fc |
| linkProvider | ProQuest |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1Pb9MwFH8aGxJc1sFAdCvDB5BAYLHYTu0cEELbqk7twg4b6gVljmNLlaZ2awpTvxSfkWc3abdJ9LYDR8fOkxP__N6z3z-AtzqxRaGR-8m8SKhXeKmKIkOTXBVWW1RQRQgU7ss0VYNBcroGf-pYGO9WWfPEwKiLsfF35J-ZQkmCB2SefL26pr5qlLeu1iU05rDo2dkNHtnKL8eHuL7vGOscnR10aVVVgBre5lMqNfJnm-S5xHkox4xyWuwXJrLOtZO4iDnj2sTStUWMXYZrh7AWOrLIFva540j3EWwIrqTP1d-TdGG1kO2Q2w9VEE6ZlEkVpFOF6gUzBOPUZ9jhVNwVhEvt9p5BNsi5TuN_-0NbsFlp1OTbfAs8gzU7eg6NuloFqZjXNvzsDFHTJVMEryUznGdJxg6blgxv34oS3NvefSo0_Qif8RMfj-b-8iUx4aQRPN3I-x9pSg--n5x-eAHnD_KRL2F9NB7ZV0CcMQq1HtR9hRGKiTxCwlrGEcoOJUzchKhe6cxUmdd9AZDLbJkz2qMjQ3RkAR2ZaMLHxTtX87wjK0e3akhkFQ8qsyUemvCpBtWy-9_UdlZTewNPumcn_ax_nPZ24SkLmPbXUi1Yn05-2dfw2PyeDsvJXtgdBC4eGmx_AY_PSCc |
| linkToPdf | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V3NT9swFH8ChtAuMD6mlcHwASQQWBDbqZ0DQghWgQqhB0BcUOY4tlQJtUAKE__a_ro9uwkdk-DGgaNjx3Li3_uw3xfAqk5sUWjkfjIvEuoVXqqiyNAkV4XVFhVUEQKFT2SaqqurpDMGf-pYGO9WWfPEwKiLvvF35NtMoSTBAzJPtl3lFtE5bO3d3lFfQcpbWutyGkOItO3Tbzy-lbvHh7jXa4y1fp4fHNGqwgA1vMkHVGrk1TbJc4lrUo4Z5bTYKUxknWsmcRFzxrWJpWuKGLsM1w4hLnRkkUXscMdx3nH4hFI49jTWlvTZgiGbIc8fqiOcMimTKmCnCtsLJgnGqc-2w6l4KRRHmu5_xtkg81ozH_lvfYHpStMm-0PSmIUx25uDmbqKBamY2jxct7qoAZMBgtqSJ1xnSfoOm5Z0_70tJUjz3q0qNP0InwkUH_eGfvQlMeEEEjzgyPplmtKDs9POxgJcvMtHfoWJXr9nvwFxxijUhlAnFkYoJvIIJ9YyjlCmKGHiBkT1rmemysjuC4PcZKNc0h4pGSIlC0jJRAM2n9-5HeYjeXP0Ug2PrOJNZTbCRgO2aoCNul-fbfHt2VZgCjGWnRyn7e_wmQV4-9uqJZgY3D_YZZg0j4Nuef8jEAqBX--Ntb_QjVDh |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=First+three+years+of+the+international+verification+of+neural+networks+competition+%28VNN-COMP%29&rft.jtitle=International+journal+on+software+tools+for+technology+transfer&rft.au=Brix%2C+Christopher&rft.au=M%C3%BCller%2C+Mark+Niklas&rft.au=Bak%2C+Stanley&rft.au=Johnson%2C+Taylor+T.&rft.date=2023-06-01&rft.issn=1433-2779&rft.eissn=1433-2787&rft.volume=25&rft.issue=3&rft.spage=329&rft.epage=339&rft_id=info:doi/10.1007%2Fs10009-023-00703-4&rft.externalDBID=n%2Fa&rft.externalDocID=10_1007_s10009_023_00703_4 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1433-2779&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1433-2779&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1433-2779&client=summon |