Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts
Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation of 9 state-of-the-art automated analysis tool...
Gespeichert in:
| Veröffentlicht in: | 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) S. 530 - 541 |
|---|---|
| Hauptverfasser: | , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
ACM
01.10.2020
|
| Schlagworte: | |
| ISSN: | 1558-1225 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation of 9 state-of-the-art automated analysis tools using two new datasets: i) a dataset of 69 annotated vulnerable smart contracts that can be used to evaluate the precision of analysis tools; and ii) a dataset with all the smart contracts in the Ethereum Blockchain that have Solidity source code available on Etherscan (a total of 47,518 contracts). The datasets are part of SmartBugs, a new extendable execution framework that we created to facilitate the integration and comparison between multiple analysis tools and the analysis of Ethereum smart contracts. We used SmartBugs to execute the 9 automated analysis tools on the two datasets. In total, we ran 428,337 analyses that took approximately 564 days and 3 hours, being the largest experimental setup to date both in the number of tools and in execution time. We found that only 42% of the vulnerabilities from our annotated dataset are detected by all the tools, with the tool Mythril having the higher accuracy (27%). When considering the largest dataset, we observed that 97% of contracts are tagged as vulnerable, thus suggesting a considerable number of false positives. Indeed, only a small number of vulnerabilities (and of only two categories) were detected simultaneously by four or more tools. |
|---|---|
| AbstractList | Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation of 9 state-of-the-art automated analysis tools using two new datasets: i) a dataset of 69 annotated vulnerable smart contracts that can be used to evaluate the precision of analysis tools; and ii) a dataset with all the smart contracts in the Ethereum Blockchain that have Solidity source code available on Etherscan (a total of 47,518 contracts). The datasets are part of SmartBugs, a new extendable execution framework that we created to facilitate the integration and comparison between multiple analysis tools and the analysis of Ethereum smart contracts. We used SmartBugs to execute the 9 automated analysis tools on the two datasets. In total, we ran 428,337 analyses that took approximately 564 days and 3 hours, being the largest experimental setup to date both in the number of tools and in execution time. We found that only 42% of the vulnerabilities from our annotated dataset are detected by all the tools, with the tool Mythril having the higher accuracy (27%). When considering the largest dataset, we observed that 97% of contracts are tagged as vulnerable, thus suggesting a considerable number of false positives. Indeed, only a small number of vulnerabilities (and of only two categories) were detected simultaneously by four or more tools. |
| Author | Durieux, Thomas Ferreira, Joao F. Cruz, Pedro Abreu, Rui |
| Author_xml | – sequence: 1 givenname: Thomas surname: Durieux fullname: Durieux, Thomas email: thomas@durieux.me organization: INESC-ID and IST, University of Lisbon,Portugal – sequence: 2 givenname: Joao F. surname: Ferreira fullname: Ferreira, Joao F. email: joao@joaoff.com organization: INESC-ID and IST, University of Lisbon,Portugal – sequence: 3 givenname: Rui surname: Abreu fullname: Abreu, Rui email: rui@computer.org organization: INESC-ID and IST, University of Lisbon,Portugal – sequence: 4 givenname: Pedro surname: Cruz fullname: Cruz, Pedro email: pedrocrvz@gmail.com organization: INESC-ID and IST, University of Lisbon,Portugal |
| BookMark | eNotjk1LAzEUAKMoaKtnD17yA9yal5e3SY6l1A8oFLSeS7r7Fld2N2WTKv33FvQ0MIdhJuJiiAMLcQdqBmDoEdFaBzBDdApLcyYmJ6vQggY8F9dA5ArQmq7EJKUvpVRpvL8W62W_b8e2Cp184--Wf2Rs5PyQYx8y13I-hO6Y2iQ3MXZJxkEa-0DOymX-5JEPvXzvw5jlIg55DFVON-KyCV3i239OxcfTcrN4KVbr59fFfFUEbWwu_K5xVHmlTEVcl6wtlRUhAOrdjoKxhlhTcI2vGwUKqGTHofbMFNgZxKm4_-u2zLzdj-1p47j12hmlEX8BlKFN-Q |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1145/3377811.3380364 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 1450371213 9781450371216 |
| EISSN | 1558-1225 |
| EndPage | 541 |
| ExternalDocumentID | 9284023 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: FCT, Fundação para a Ciência e a Tecnologia grantid: UIDB/50021/2020,PTDC/CCI-COM/29300/2017 funderid: 10.13039/501100001871 – fundername: Horizon 2020 grantid: 822404 funderid: 10.13039/100010661 |
| GroupedDBID | -~X .4S .DC 123 23M 29O 5VS 6IE 6IF 6IH 6IK 6IL 6IM 6IN 8US AAJGR AAWTH ABLEC ADZIZ AFFNX ALMA_UNASSIGNED_HOLDINGS APO ARCSS AVWKF BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO EDO FEDTE I-F I07 IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO RNS XOL |
| ID | FETCH-LOGICAL-a247t-9bf85c9004c5ed6e2756c531132bb5a4745e25a8f9df010156e8ead9ee5ae8433 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 269 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000652529800044&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:32:58 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a247t-9bf85c9004c5ed6e2756c531132bb5a4745e25a8f9df010156e8ead9ee5ae8433 |
| PageCount | 12 |
| ParticipantIDs | ieee_primary_9284023 |
| PublicationCentury | 2000 |
| PublicationDate | 2020-Oct. |
| PublicationDateYYYYMMDD | 2020-10-01 |
| PublicationDate_xml | – month: 10 year: 2020 text: 2020-Oct. |
| PublicationDecade | 2020 |
| PublicationTitle | 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) |
| PublicationTitleAbbrev | ICSE |
| PublicationYear | 2020 |
| Publisher | ACM |
| Publisher_xml | – name: ACM |
| SSID | ssj0006499 ssj0002870079 |
| Score | 2.6153824 |
| Snippet | Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 530 |
| SubjectTerms | Blockchain Debugging Ethereum Reproducible Bugs Smart contracts Solidity Testing |
| Title | Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts |
| URI | https://ieeexplore.ieee.org/document/9284023 |
| WOSCitedRecordID | wos000652529800044&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09b8IwED0B6tCJtlD1Wx46EqCxHdtjVYE6UaRSiQ3ZzllCKgmCpL-_dhKoKnXpZmWynPPdvTu_dwCPzDJqRGwjNFxFjKUuktp6zOMwGQslHKW6GjYhZjO5XKp5CwZHLgwiVo_PcBiWVS8_zW0ZSmUj5X2pjzFtaAuR1FytYz0lNOzGoePUeOHEp_KNlM8T4yNKReBUDj0iC523X7NUqlAy7f5vE2fQ_-Hkkfkx2pxDC7ML6B6GMpDmjvbgbbLZrivdD1LX_UnuyHNZ5D41xZQcREjIIs8_9yTPCBMDLgWZhEwQyw1533hrIkG1KhCo9n34mE4WL69RMzUh0jETRaSMk9wqb_yWY5pg0He3_qZ52GkM10wwjjHX0qnUBYE5nqD05qQQuUbJKL2ETpZneAWEGak9xJHCGMq8H1R2LBLrFPocRjPtrqEXzme1rYUxVs3R3Pz9-RZO4wBWq5dwd9ApdiXew4n9Ktb73UP1N78BmR2fIA |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEG0QTfSECsZve_DIwrptt-3RGAhGRBIx4Uba7jQhkV0Cu_5-22XBmHjx1uyp6U5n5s30vUHonhpKNI9MAJrJgNLEBkIZh3ksxCGX3BKiymETfDQS06kc11B7x4UBgPLxGXT8suzlJ5kpfKmsK50vdTFmD-0zSqNww9baVVR8yy70PafKD8cuma_EfB4o6xLCPauy4zCZ7739mqZSBpN-43_bOEatH1YeHu_izQmqQXqKGtuxDLi6pU301lss56XyB95U_nFm8WORZy45hQRvZUjwJMs-1zhLMeVtJjju-VwQigV-Xzh7wl63ylOo1i300e9NngZBNTchUBHleSC1FcxIZ_6GQRKDV3g37q454Kk1U5RTBhFTwsrEeok5FoNwBiUBmAJBCTlD9TRL4RxhqoVyIEdwrQl1nlCakMfGSnBZjKLKXqCmP5_ZciONMauO5vLvz3focDB5Hc6Gz6OXK3QUeehavou7RvV8VcANOjBf-Xy9ui3_7DcR-qJn |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=2020+IEEE%2FACM+42nd+International+Conference+on+Software+Engineering+%28ICSE%29&rft.atitle=Empirical+Review+of+Automated+Analysis+Tools+on+47%2C587+Ethereum+Smart+Contracts&rft.au=Durieux%2C+Thomas&rft.au=Ferreira%2C+Joao+F.&rft.au=Abreu%2C+Rui&rft.au=Cruz%2C+Pedro&rft.date=2020-10-01&rft.pub=ACM&rft.eissn=1558-1225&rft.spage=530&rft.epage=541&rft_id=info:doi/10.1145%2F3377811.3380364&rft.externalDocID=9284023 |