Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Making
Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data science - are asked to use ML predictions to make high-stakes decisions. Multiple ML usability challenges can appear as result, such as lack...
Uložené v:
| Vydané v: | IEEE transactions on visualization and computer graphics Ročník 28; číslo 1; s. 1161 - 1171 |
|---|---|
| Hlavní autori: | , , , |
| Médium: | Journal Article |
| Jazyk: | English |
| Vydavateľské údaje: |
United States
IEEE
01.01.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Predmet: | |
| ISSN: | 1077-2626, 1941-0506, 1941-0506 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data science - are asked to use ML predictions to make high-stakes decisions. Multiple ML usability challenges can appear as result, such as lack of user trust in the model, inability to reconcile human-ML disagreement, and ethical concerns about oversimplification of complex problems to a single algorithm output. In this paper, we investigate the ML usability challenges that present in the domain of child welfare screening through a series of collaborations with child welfare screeners. Following the iterative design process between the ML scientists, visualization researchers, and domain experts (child screeners), we first identified four key ML challenges and honed in on one promising explainable ML technique to address them (local factor contributions). Then we implemented and evaluated our visual analytics tool, Sibyl, to increase the interpretability and interactivity of local factor contributions. The effectiveness of our tool is demonstrated by two formal user studies with 12 non-expert participants and 13 expert participants respectively. Valuable feedback was collected, from which we composed a list of design implications as a useful guideline for researchers who aim to develop an interpretable and interactive visualization tool for ML prediction models deployed for child welfare screeners and other similar domain experts. |
|---|---|
| AbstractList | Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data science - are asked to use ML predictions to make high-stakes decisions. Multiple ML usability challenges can appear as result, such as lack of user trust in the model, inability to reconcile human-ML disagreement, and ethical concerns about oversimplification of complex problems to a single algorithm output. In this paper, we investigate the ML usability challenges that present in the domain of child welfare screening through a series of collaborations with child welfare screeners. Following the iterative design process between the ML scientists, visualization researchers, and domain experts (child screeners), we first identified four key ML challenges and honed in on one promising explainable ML technique to address them (local factor contributions). Then we implemented and evaluated our visual analytics tool, Sibyl, to increase the interpretability and interactivity of local factor contributions. The effectiveness of our tool is demonstrated by two formal user studies with 12 non-expert participants and 13 expert participants respectively. Valuable feedback was collected, from which we composed a list of design implications as a useful guideline for researchers who aim to develop an interpretable and interactive visualization tool for ML prediction models deployed for child welfare screeners and other similar domain experts.Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data science - are asked to use ML predictions to make high-stakes decisions. Multiple ML usability challenges can appear as result, such as lack of user trust in the model, inability to reconcile human-ML disagreement, and ethical concerns about oversimplification of complex problems to a single algorithm output. In this paper, we investigate the ML usability challenges that present in the domain of child welfare screening through a series of collaborations with child welfare screeners. Following the iterative design process between the ML scientists, visualization researchers, and domain experts (child screeners), we first identified four key ML challenges and honed in on one promising explainable ML technique to address them (local factor contributions). Then we implemented and evaluated our visual analytics tool, Sibyl, to increase the interpretability and interactivity of local factor contributions. The effectiveness of our tool is demonstrated by two formal user studies with 12 non-expert participants and 13 expert participants respectively. Valuable feedback was collected, from which we composed a list of design implications as a useful guideline for researchers who aim to develop an interpretable and interactive visualization tool for ML prediction models deployed for child welfare screeners and other similar domain experts. Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data science - are asked to use ML predictions to make high-stakes decisions. Multiple ML usability challenges can appear as result, such as lack of user trust in the model, inability to reconcile human-ML disagreement, and ethical concerns about oversimplification of complex problems to a single algorithm output. In this paper, we investigate the ML usability challenges that present in the domain of child welfare screening through a series of collaborations with child welfare screeners. Following the iterative design process between the ML scientists, visualization researchers, and domain experts (child screeners), we first identified four key ML challenges and honed in on one promising explainable ML technique to address them (local factor contributions). Then we implemented and evaluated our visual analytics tool, Sibyl, to increase the interpretability and interactivity of local factor contributions. The effectiveness of our tool is demonstrated by two formal user studies with 12 non-expert participants and 13 expert participants respectively. Valuable feedback was collected, from which we composed a list of design implications as a useful guideline for researchers who aim to develop an interpretable and interactive visualization tool for ML prediction models deployed for child welfare screeners and other similar domain experts. |
| Author | Vaithianathan, Rhema Liu, Dongyu Zytek, Alexandra Veeramachaneni, Kalyan |
| Author_xml | – sequence: 1 givenname: Alexandra surname: Zytek fullname: Zytek, Alexandra email: zyteka@mit.edu organization: MIT, United States – sequence: 2 givenname: Dongyu surname: Liu fullname: Liu, Dongyu email: dongyu@mit.edu organization: MIT, United States – sequence: 3 givenname: Rhema surname: Vaithianathan fullname: Vaithianathan, Rhema email: rhema.vaithianathan@aut.ac.nz organization: Auckland University of Technology, New Zealand – sequence: 4 givenname: Kalyan surname: Veeramachaneni fullname: Veeramachaneni, Kalyan email: kalyan@csail.mit.edu organization: MIT, United States |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/34587081$$D View this record in MEDLINE/PubMed |
| BookMark | eNp9kTtv2zAUhYkiQfNofkBRoBDQpYtcXpLio1vgtkkAFx0SZxUo6cpmIlMJSQ_-96VqN0OGTIcEv8N7cc4ZOfKjR0I-Ap0BUPPt7n5-NWOUwYwDCC3FO3IKRkBJKyqP8pkqVTLJ5Ak5i_GBUhBCm_fkhItKK6rhlIy3rtkN34ul7zDEZH3n_KrIUlx2XcAYp2taY7GMtnGDS7tivrbDgH6FsRj74rdt185jsUAb_ATf-OLardblbbKPGfmBrYtu9Bl8zM8fyHFvh4gXBz0ny18_7-bX5eLP1c38clG2XJhUtlR1nUZFTdVrDtIY1hgutGipRrS6F1LoXjOhgDELvepaLiWKpuuhaQTn5-Tr_t-nMD5vMaZ642KLw2A9jttYs0ppqKhQNKNfXqEP4zb4vF3NJHCeg_pHfT5Q22aDXf0U3MaGXf0_ygzAHmjDGGPA_gUBWk911VNd9VRXfagre9QrT-uSTTmuFKwb3nR-2jsdIr5MMlXFtDD8L7F4oIM |
| CODEN | ITVGEA |
| CitedBy_id | crossref_primary_10_3390_diagnostics14090917 crossref_primary_10_1109_TVCG_2024_3487974 crossref_primary_10_1016_j_inffus_2024_102412 crossref_primary_10_1016_j_techfore_2023_122824 crossref_primary_10_3390_systems13020131 crossref_primary_10_3390_drones8040155 crossref_primary_10_3390_informatics9010014 crossref_primary_10_1109_TVCG_2024_3456160 crossref_primary_10_1016_j_techfore_2024_123326 crossref_primary_10_3389_fcomp_2024_1521066 crossref_primary_10_1093_cjres_rsac025 crossref_primary_10_3390_app14198884 crossref_primary_10_1007_s13198_025_02712_9 crossref_primary_10_1016_j_compbiolchem_2025_108599 crossref_primary_10_3390_make7010013 crossref_primary_10_1109_TVCG_2023_3259341 crossref_primary_10_1109_TVCG_2024_3357065 crossref_primary_10_1145_3512950 crossref_primary_10_1177_14738716241240156 |
| Cites_doi | 10.1145/2858036.2858529 10.1109/TVCG.2018.2864499 10.1109/TVCG.2019.2934629 10.1038/s41551-018-0304-0 10.1007/978-3-319-90403-0_9 10.1109/TVCG.2009.111 10.1145/3290605.3300831 10.1109/TVCG.2018.2865027 10.1145/2851581.2856492 10.1038/s42256-019-0048-x 10.2105/AJPH.2016.303545 10.1145/3290605.3300809 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 |
| DBID | 97E RIA RIE AAYXX CITATION NPM 7SC 7SP 8FD JQ2 L7M L~C L~D 7X8 |
| DOI | 10.1109/TVCG.2021.3114864 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef PubMed Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional MEDLINE - Academic |
| DatabaseTitle | CrossRef PubMed Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic PubMed Technology Research Database |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher – sequence: 3 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1941-0506 |
| EndPage | 1171 |
| ExternalDocumentID | 34587081 10_1109_TVCG_2021_3114864 9552849 |
| Genre | orig-research Research Support, U.S. Gov't, Non-P.H.S Journal Article |
| GrantInformation_xml | – fundername: NSF grantid: 1761812 funderid: 10.13039/100000001 |
| GroupedDBID | --- -~X .DC 0R~ 29I 4.4 53G 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ H~9 IEDLZ IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P PQQKQ RIA RIE RNI RNS RZB TN5 VH1 AAYXX CITATION AAYOK NPM PKN RIC RIG Z5M 7SC 7SP 8FD JQ2 L7M L~C L~D 7X8 |
| ID | FETCH-LOGICAL-c349t-c07dd8e7095f8316992b93484c08eea8f4648f8247122a1f7dc366e4bdf1bb433 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 31 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000733959000109&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1077-2626 1941-0506 |
| IngestDate | Thu Oct 02 07:36:42 EDT 2025 Mon Jun 30 04:29:16 EDT 2025 Wed Feb 19 02:27:58 EST 2025 Tue Nov 18 22:27:42 EST 2025 Sat Nov 29 03:31:38 EST 2025 Wed Aug 27 02:49:29 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 1 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c349t-c07dd8e7095f8316992b93484c08eea8f4648f8247122a1f7dc366e4bdf1bb433 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23 |
| PMID | 34587081 |
| PQID | 2613370870 |
| PQPubID | 75741 |
| PageCount | 11 |
| ParticipantIDs | pubmed_primary_34587081 proquest_miscellaneous_2578150470 crossref_primary_10_1109_TVCG_2021_3114864 proquest_journals_2613370870 crossref_citationtrail_10_1109_TVCG_2021_3114864 ieee_primary_9552849 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-Jan. 2022-1-00 2022-01-00 20220101 |
| PublicationDateYYYYMMDD | 2022-01-01 |
| PublicationDate_xml | – month: 01 year: 2022 text: 2022-Jan. |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States – name: New York |
| PublicationTitle | IEEE transactions on visualization and computer graphics |
| PublicationTitleAbbrev | TVCG |
| PublicationTitleAlternate | IEEE Trans Vis Comput Graph |
| PublicationYear | 2022 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref12 ref15 ref11 ref10 doshi-velez (ref2) 2017 vaithianathan (ref23) 2019 fisher (ref3) 2019; 20 lundberg (ref14) 2017; 31 ref16 ref19 (ref5) 2020 kenton (ref9) 2021 vaithianathan (ref24) 2017 richardson (ref18) 1990; 2 ref25 ref20 xu (ref26) 2019 kahng (ref8) 2017; 24 strobelt (ref22) 2017; 24 ref21 ref27 lipton (ref13) 2016; 16 (ref1) 2020 (ref17) 2017 ref4 ref6 hurley (ref7) 2018 |
| References_xml | – ident: ref11 doi: 10.1145/2858036.2858529 – volume: 31 start-page: 10 year: 2017 ident: ref14 article-title: A Unified Approach to Interpreting Model Predictions publication-title: Advances in neural information processing systems – start-page: 30 year: 2018 ident: ref7 publication-title: Can an Algorithm Tell When Kids Are in Danger? – volume: 24 start-page: 10 year: 2017 ident: ref8 article-title: ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models publication-title: IEEE Conference on Visual Analytics Science and Technology (VAST) – volume: 16 start-page: 31 year: 2016 ident: ref13 article-title: The Mythos of Model Interpretability publication-title: 2016 ICML Workshop on Human Interpretability in Machine Learning – volume: 20 start-page: 1 year: 2019 ident: ref3 article-title: All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously publication-title: Journal of Machine Learning Research – volume: 2 start-page: 226 year: 1990 ident: ref18 article-title: The Effects of a False Allegation of Child Sexual Abuse on an Intact Middle Class Family publication-title: IPT – start-page: 60 year: 2017 ident: ref24 publication-title: Developing Predictive Models to Support Child Maltreatment Hotline Screening Decisions Allegheny County Methodology and Implementation – ident: ref27 doi: 10.1109/TVCG.2018.2864499 – year: 2020 ident: ref5 publication-title: Machine Learning Glossary Fairness – year: 2021 ident: ref9 article-title: How Cost-Benefit Analysis Process Is Performed publication-title: Investope-dia – ident: ref21 doi: 10.1109/TVCG.2019.2934629 – year: 2017 ident: ref17 article-title: National Highway Traffic Safety Administration (NHTSA) publication-title: Automated Vehicles for Safety – ident: ref15 doi: 10.1038/s41551-018-0304-0 – ident: ref19 doi: 10.1007/978-3-319-90403-0_9 – year: 2019 ident: ref23 publication-title: Implementing a Child Welfare Decision Aide in Douglas County – ident: ref16 doi: 10.1109/TVCG.2009.111 – year: 2020 ident: ref1 article-title: Children's Bureau publication-title: Child Maltreatment 2018 Summary of Key Findings – ident: ref25 doi: 10.1145/3290605.3300831 – year: 2017 ident: ref2 publication-title: Towards a rigorous science of interpretable machine learning – ident: ref12 doi: 10.1109/TVCG.2018.2865027 – ident: ref4 doi: 10.1145/2851581.2856492 – year: 2019 ident: ref26 article-title: Modeling tabular data using conditional GAN publication-title: Advances in neural information processing systems – ident: ref20 doi: 10.1038/s42256-019-0048-x – ident: ref10 doi: 10.2105/AJPH.2016.303545 – ident: ref6 doi: 10.1145/3290605.3300809 – volume: 24 start-page: 667 year: 2017 ident: ref22 article-title: LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks publication-title: Infovis |
| SSID | ssj0014489 |
| Score | 2.5427222 |
| Snippet | Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts - who often have no expertise in ML or data... |
| SourceID | proquest pubmed crossref ieee |
| SourceType | Aggregation Database Index Database Enrichment Source Publisher |
| StartPage | 1161 |
| SubjectTerms | Algorithms Child welfare Context modeling Decision making Domains Iterative methods Machine learning Pediatrics Prediction algorithms Prediction models Predictive models Subject specialists Usability Visualization XAI |
| Title | Sibyl: Understanding and Addressing the Usability Challenges of Machine Learning In High-Stakes Decision Making |
| URI | https://ieeexplore.ieee.org/document/9552849 https://www.ncbi.nlm.nih.gov/pubmed/34587081 https://www.proquest.com/docview/2613370870 https://www.proquest.com/docview/2578150470 |
| Volume | 28 |
| WOSCitedRecordID | wos000733959000109&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1941-0506 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014489 issn: 1077-2626 databaseCode: RIE dateStart: 19950101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Lb9QwELbaigMceJXHQqmMxAlh6ldim1u1UOBAhUQX7S1ynDGqqBLU3UXqv2fseKMiARKnJMrkoczn-BvPi5AX3pvWS6VZiNowHSvBbB01Uy4gmxAdD77NzSbM6aldLt3nHfJqyoUBgBx8Bq_Tbvbld0PYpKWyI1dV-Dd1u2TXGDPmak0eAzQz3BhfaJhEll48mIK7o7Ov8_doCUqBBiqy_zr14lG6QqRa8dt0lPur_J1q5inn5M7_vexdcrtQS3o8YuEe2YH-Prl1reDgPhm-nLdXF2_o4npKC8UNPe66HBGLh0gJ6WKsvLu-ovNtt5UVHSL9lGMvgZayrN_ox56mUBGGpPU7irwtLXtQMK3BPyCLk3dn8w-stFxgQWm3ZoGbrrNgkHhFq0TtnGyd0lYHbgG8jbrWNlqJU5qUXkTTBVXXoNsuirbVSj0ke_3Qw2NCoa0caOmVd5U2Ve0NqqUTUHEJNQeYEb798k0o9chTW4yLJtsl3DVJb03SW1P0NiMvp0t-jMU4_iW8n5QyCRZ9zMjBVr1NGa6rBs1IpRAOhs_I8-k0DrTkPfE9DBuUwX8bsmedZB6NsJjuvUXTkz8_8ym5KVPWRF65OSB768sNPCM3ws_1-eryENG8tIcZzb8Ae8PtVw |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1ba9RAFD7UKqgP9VKta6uO4JOYNnNJZqZvZbW22C6Cu9K3MJmcSGlJpLsr9N97ZpINLajgUxJyciHfmcx35twA3jmnSyekSnytdKLqjCcmr1UirSc2wavUuzI2m9CTiTk7s1_X4MOQC4OIMfgMd8Nu9OVXrV-GpbI9m2X0N7V34G6mlOBdttbgMyBDw3YRhjoRxNN7HyZP7d70-_gz2YKCk4lK_D8P3XikykhXDb81IcUOK38nm3HSOXz0f6_7GDZ6cskOOm14AmvYPIWHN0oObkL77by8vtxns5tJLYw27KCqYkwsHRIpZLOu9u7imo1X_VbmrK3ZaYy-RNYXZv3BjhsWgkUSoq0XJPKxb9pDgmEV_hnMDj9Nx0dJ33Qh8VLZReJTXVUGNVGv2kieWytKK5VRPjWIztQqV6Y2giY1IRyvdeVlnqMqq5qXpZLyOaw3bYMvgGGZWVTCSWczpbPcaYKl4pilAvMUcQTp6ssXvq9IHhpjXBbRMkltEXArAm5Fj9sI3g-X_OzKcfxLeDOAMgj2eIxgZwVv0Q_YeUGGpJSkDjodwdvhNA214D9xDbZLkqG_G_FnFWS2OrUY7r3Sppd_fuYbuH80PT0pTo4nX7bhgQg5FHEdZwfWF1dLfAX3_K_F-fzqddTp32KH77Y |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sibyl%3A+Understanding+and+Addressing+the+Usability+Challenges+of+Machine+Learning+In+High-Stakes+Decision+Making&rft.jtitle=IEEE+transactions+on+visualization+and+computer+graphics&rft.au=Zytek%2C+Alexandra&rft.au=Liu%2C+Dongyu&rft.au=Vaithianathan%2C+Rhema&rft.au=Veeramachaneni%2C+Kalyan&rft.date=2022-01-01&rft.issn=1077-2626&rft.eissn=1941-0506&rft.volume=28&rft.issue=1&rft.spage=1161&rft.epage=1171&rft_id=info:doi/10.1109%2FTVCG.2021.3114864&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TVCG_2021_3114864 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1077-2626&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1077-2626&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1077-2626&client=summon |