TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays
Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due...
Uloženo v:
| Vydáno v: | 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition s. 9049 - 9058 |
|---|---|
| Hlavní autoři: | , , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.06.2018
|
| Témata: | |
| ISSN: | 1063-6919 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI). |
|---|---|
| AbstractList | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI). |
| Author | Wang, Xiaosong Lu, Zhiyong Summers, Ronald M. Lu, Le Peng, Yifan |
| Author_xml | – sequence: 1 givenname: Xiaosong surname: Wang fullname: Wang, Xiaosong – sequence: 2 givenname: Yifan surname: Peng fullname: Peng, Yifan – sequence: 3 givenname: Le surname: Lu fullname: Lu, Le – sequence: 4 givenname: Zhiyong surname: Lu fullname: Lu, Zhiyong – sequence: 5 givenname: Ronald M. surname: Summers fullname: Summers, Ronald M. |
| BookMark | eNotjr1OwzAURg0CiVI6M7D4BVKu45_YbCgUqFQBigJiq-zkpjVt4iqORPv2FMF0hqPz6bskZ13okJBrBlPGwNzmH2_FNAWmpwBG8BMyMZlmkmulRArmlIwYKJ4ow8wFmcT4BQCp0lwLOSKb0uMLDne0xP2QzFu7QjprHda171b0aL5Dv6FN6Gke2jZ0tFyH3u7pg49oI9J8a2P0ja_s4I_WdjUtcBf64Tf3Hc3XGAf6mRT2EK_IeWO3ESf_HJP3x1mZPyeL16d5fr9IfCrYkNQZNsLK2gnuKp1Jx-sUpVBQc9BWZ8frFQPXyIprxhtXySplOhNaKGc4IB-Tm79dj4jLXe9b2x-WWmYGBOM_fytaKQ |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/CVPR.2018.00943 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) - NZ url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 9781538664209 1538664208 |
| EISSN | 1063-6919 |
| EndPage | 9058 |
| ExternalDocumentID | 8579041 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 368 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:52:16 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_8579041 |
| PublicationCentury | 2000 |
| PublicationDate | 2018-06 |
| PublicationDateYYYYMMDD | 2018-06-01 |
| PublicationDate_xml | – month: 06 year: 2018 text: 2018-06 |
| PublicationDecade | 2010 |
| PublicationTitle | 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition |
| PublicationTitleAbbrev | CVPR |
| PublicationYear | 2018 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0002683845 ssj0003211698 |
| Score | 2.586755 |
| Snippet | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 9049 |
| SubjectTerms | Biomedical imaging Diseases Feature extraction Task analysis Training Visualization X-rays |
| Title | TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays |
| URI | https://ieeexplore.ieee.org/document/8579041 |
| WOSCitedRecordID | wos000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5zePA0dRN_k4NH49KmaVKvc0NByhhVdhtN84pF7GTrRP9789oyPXjx1pQ-CHlp3pe878sj5Eo6COBgcM50rkMWWM9jkTWaGSlC8I11oKQWCj-qONbzeTTtkOutFgYAavIZ3OBjncu3y2yDR2VDLVXEUaW-o5RqtFrb8xQ_1EK3GTJsC7ezCSPd3ubj8Wg4ep7OkMuF5MkIVTq_yqnU0WTS-18_9sngR5ZHp9uAc0A6UB6SXosjafuXrvvkNSkghuqWJritfXhzSwYdvxmwaEbjhvhNHVqlKA9ZljR5cRPhk941yRpaF8pEClHtNZqWljY4Hc2Lko6wxhads1n6tR6Qp8k4Gd2ztqgCK1ywrphVkAeptCYQJnNeMsL6IIOQW8F16tZGLTKPm1xmLpaL3GQy87Fun8MpJhIcxBHplssSjglVVjhDZXwQPHDfmVBbnvJc5cC10fKE9HHsFu_NvRmLdthO_359RvbQOQ0N65x0q9UGLshu9lEV69Vl7exvEOmpnw |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PT8IwFG4ImugJFYy_7cGjk25dt84rQiDiQsg03Mi6vsXFMAwMo_-9fduCHrx4W5e9pOnr-r72fV8fITfCQAADg1NLptKzXG3bVqCVtJTgHjhKG1BSCoXHfhjK2SyYNMjtVgsDACX5DO7wsczl62WywaOyrhR-wFClviNc17Ertdb2RMXxJJd1jgzb3OxtvEDW9_nYLOj2XiZTZHMhfTJAnc6vgiplPBm0_teTA9L5EebRyTbkHJIG5EekVSNJWv-n6zZ5izIIobinEW5sRwuzaND-QoFGMxpW1G9q8CpFgcgyp9GrmQqf9KFK19CyVCaSiEq_0TjXtELqaJ7ltIdVtujMmsZf6w55HvSj3tCqyypYmQnXhaV9SN1YaOVylRg_Ka4dEK7HNGcyNquj5InNVCoSE815qhKROFi5zyAVFXAG_Jg082UOJ4T6mhtDXznAmWu-U57ULGapnwKTSopT0saxm79XN2fM62E7-_v1NdkbRk_j-XgUPp6TfXRURcq6IM1itYFLspt8FNl6dVU6_hsRhazm |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2018+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition&rft.atitle=TieNet%3A+Text-Image+Embedding+Network+for+Common+Thorax+Disease+Classification+and+Reporting+in+Chest+X-Rays&rft.au=Wang%2C+Xiaosong&rft.au=Peng%2C+Yifan&rft.au=Lu%2C+Le&rft.au=Lu%2C+Zhiyong&rft.date=2018-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=9049&rft.epage=9058&rft_id=info:doi/10.1109%2FCVPR.2018.00943&rft.externalDocID=8579041 |