TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays
Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due...
Uloženo v:
| Vydáno v: | 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition s. 9049 - 9058 |
|---|---|
| Hlavní autoři: | , , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.06.2018
|
| Témata: | |
| ISSN: | 1063-6919 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI). |
|---|---|
| AbstractList | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI). |
| Author | Wang, Xiaosong Lu, Zhiyong Summers, Ronald M. Lu, Le Peng, Yifan |
| Author_xml | – sequence: 1 givenname: Xiaosong surname: Wang fullname: Wang, Xiaosong – sequence: 2 givenname: Yifan surname: Peng fullname: Peng, Yifan – sequence: 3 givenname: Le surname: Lu fullname: Lu, Le – sequence: 4 givenname: Zhiyong surname: Lu fullname: Lu, Zhiyong – sequence: 5 givenname: Ronald M. surname: Summers fullname: Summers, Ronald M. |
| BookMark | eNotjr1OwzAURg0CiVI6M7D4BVKu45_YbCgUqFQBigJiq-zkpjVt4iqORPv2FMF0hqPz6bskZ13okJBrBlPGwNzmH2_FNAWmpwBG8BMyMZlmkmulRArmlIwYKJ4ow8wFmcT4BQCp0lwLOSKb0uMLDne0xP2QzFu7QjprHda171b0aL5Dv6FN6Gke2jZ0tFyH3u7pg49oI9J8a2P0ja_s4I_WdjUtcBf64Tf3Hc3XGAf6mRT2EK_IeWO3ESf_HJP3x1mZPyeL16d5fr9IfCrYkNQZNsLK2gnuKp1Jx-sUpVBQc9BWZ8frFQPXyIprxhtXySplOhNaKGc4IB-Tm79dj4jLXe9b2x-WWmYGBOM_fytaKQ |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/CVPR.2018.00943 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 9781538664209 1538664208 |
| EISSN | 1063-6919 |
| EndPage | 9058 |
| ExternalDocumentID | 8579041 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 368 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:52:16 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_8579041 |
| PublicationCentury | 2000 |
| PublicationDate | 2018-06 |
| PublicationDateYYYYMMDD | 2018-06-01 |
| PublicationDate_xml | – month: 06 year: 2018 text: 2018-06 |
| PublicationDecade | 2010 |
| PublicationTitle | 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition |
| PublicationTitleAbbrev | CVPR |
| PublicationYear | 2018 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0002683845 ssj0003211698 |
| Score | 2.586755 |
| Snippet | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 9049 |
| SubjectTerms | Biomedical imaging Diseases Feature extraction Task analysis Training Visualization X-rays |
| Title | TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays |
| URI | https://ieeexplore.ieee.org/document/8579041 |
| WOSCitedRecordID | wos000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5zePA0dRN_k4NH69JmbRKvc0NByhh17Daa5BWLrJOtE_3vzWvL9ODFW1MIhLw2772873sfITcyUKliqe9ZrcBzETh4WgXghRlonVmZ6kozcvYs4ljO52rSIrc7LgwAVOAzuMPHqpZvV2aLV2V9GQrFkKW-J4SouVq7-5Qgklw2FTIcc5fZREo23Xx8pvrD2WSKWC4ETypk6fySU6m8ybjzv3Uckt4PLY9Odg7niLSgOCadJo6kzV-66ZK3JIcYynuaYFr7tHRHBh0tNVicRuMa-E1dtEqRHrIqaPLqPoRP-lAXa2gllIkQospqNC0sreN0nJ4XdIgaW3TuTdOvTY-8jEfJ8NFrRBW83Dnr0rMCskEaWj3g2rhkRXMbQDiImOVMpu5slNz4TGehcb6cZ9qEJkDdPhenaMUZ8BPSLlYFnBLK_VBiP_lMWuayvFBanxmmhUgFM1GkzkgX927xXvfNWDTbdv736wtygMapYViXpF2ut3BF9s1HmW_W15WxvwEbkam- |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELYqQIKJVxFvPDAS6sRxYrOWVq0oUVUF1K2K7YuoECnqA8G_x5dEhYGFLY5kyfIlvjvf991HyLUMVKZY5ntWK_BcBA6eVgF4IgetcyszXWpGPg_iJJHjsRo2yM2aCwMAJfgMbvGxrOXbmVnhVVlLilgxZKlvijAM_Iqttb5RCSLJZV0jwzF3uU2kZN3Px2eq1X4ejhDNhfBJhTydX4IqpT_p7v5vJXuk-UPMo8O1y9knDSgOyG4dSdL6P10cktd0Cgks72iKiW3_zR0atPOmweI0mlTQb-riVYoEkVlB0xf3KXzS-6pcQ0upTAQRlXajWWFpFanj9GlB26iyRcfeKPtaNMlTt5O2e14tq-BNnbteejaGPMyE1SHXxqUrmtsARBgxy5nM3OkoufGZzoVx3pzn2ggToHKfi1S04gz4EdkoZgUcE8p9IbGjfC4tc3mekNZnhuk4zmJmokidkEPcu8l71TljUm_b6d-vr8h2L30cTAb95OGM7KChKlDWOdlYzldwQbbMx3K6mF-Whv8GH72tBQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2018+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition&rft.atitle=TieNet%3A+Text-Image+Embedding+Network+for+Common+Thorax+Disease+Classification+and+Reporting+in+Chest+X-Rays&rft.au=Wang%2C+Xiaosong&rft.au=Peng%2C+Yifan&rft.au=Lu%2C+Le&rft.au=Lu%2C+Zhiyong&rft.date=2018-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=9049&rft.epage=9058&rft_id=info:doi/10.1109%2FCVPR.2018.00943&rft.externalDocID=8579041 |