TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays
Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due...
Gespeichert in:
| Veröffentlicht in: | 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition S. 9049 - 9058 |
|---|---|
| Hauptverfasser: | , , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
01.06.2018
|
| Schlagworte: | |
| ISSN: | 1063-6919 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI). |
|---|---|
| AbstractList | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI). |
| Author | Wang, Xiaosong Lu, Zhiyong Summers, Ronald M. Lu, Le Peng, Yifan |
| Author_xml | – sequence: 1 givenname: Xiaosong surname: Wang fullname: Wang, Xiaosong – sequence: 2 givenname: Yifan surname: Peng fullname: Peng, Yifan – sequence: 3 givenname: Le surname: Lu fullname: Lu, Le – sequence: 4 givenname: Zhiyong surname: Lu fullname: Lu, Zhiyong – sequence: 5 givenname: Ronald M. surname: Summers fullname: Summers, Ronald M. |
| BookMark | eNotjr1OwzAURg0CiVI6M7D4BVKu45_YbCgUqFQBigJiq-zkpjVt4iqORPv2FMF0hqPz6bskZ13okJBrBlPGwNzmH2_FNAWmpwBG8BMyMZlmkmulRArmlIwYKJ4ow8wFmcT4BQCp0lwLOSKb0uMLDne0xP2QzFu7QjprHda171b0aL5Dv6FN6Gke2jZ0tFyH3u7pg49oI9J8a2P0ja_s4I_WdjUtcBf64Tf3Hc3XGAf6mRT2EK_IeWO3ESf_HJP3x1mZPyeL16d5fr9IfCrYkNQZNsLK2gnuKp1Jx-sUpVBQc9BWZ8frFQPXyIprxhtXySplOhNaKGc4IB-Tm79dj4jLXe9b2x-WWmYGBOM_fytaKQ |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/CVPR.2018.00943 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library (IEL) (UW System Shared) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 9781538664209 1538664208 |
| EISSN | 1063-6919 |
| EndPage | 9058 |
| ExternalDocumentID | 8579041 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 368 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:52:16 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_8579041 |
| PublicationCentury | 2000 |
| PublicationDate | 2018-06 |
| PublicationDateYYYYMMDD | 2018-06-01 |
| PublicationDate_xml | – month: 06 year: 2018 text: 2018-06 |
| PublicationDecade | 2010 |
| PublicationTitle | 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition |
| PublicationTitleAbbrev | CVPR |
| PublicationYear | 2018 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0002683845 ssj0003211698 |
| Score | 2.586755 |
| Snippet | Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 9049 |
| SubjectTerms | Biomedical imaging Diseases Feature extraction Task analysis Training Visualization X-rays |
| Title | TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays |
| URI | https://ieeexplore.ieee.org/document/8579041 |
| WOSCitedRecordID | wos000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELVKxcBUoEV8ywMjoU5sxzZraQVLVFUBdavij4gINUVtiuDf40uiwsBCJieSJcsX--587_khdBM5okNhWKD9qg6Y0jZQGQBdTchyLimjltRiEyJJ5Hyuph10u-PCOOdq8Jm7g2Zdy7crs4WjsqHkQhFgqe8JIRqu1u48JYollW2FDN6pz2xiJdvbfEKihqOX6QywXACeVMDS-SWnUnuTSe9_4zhEgx9aHp7uHM4R6rjyGPXaOBK3q3TTR29p4RJX3eMU0tqnpd8y8HipnYVuOGmA39hHqxjoIasSp6_-R_jED02xBtdCmQAhqq2Gs9LiJk6H7kWJR6CxhefBLPvaDNDzZJyOHoNWVCEovLOuAitczjJuNaPa-GRFUxs5zmJiKZGZ3xslNSHROTfel9NcG24i0O3zcYpWlDh6grrlqnSnCOex1MZo_xjBuA4zySLDYx_iZdLnLfIM9WHuFu_NvRmLdtrO__58gQ7AOA0M6xJ1q_XWXaF981EVm_V1bexvDX2qIw |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5jCnqauom_zcGjdW2TtInXubHhLGNU2W00P4pF1snWif735rVlevBiT2khEPKavPfyvi8fQje-caUXKupIu6odKqR2RAJAV-XRlHFCiXZLsYkwivhsJiYNdLvlwhhjSvCZuYNmWcvXS7WBo7IuZ6FwgaW-wyj1vYqttT1R8QNOeF0jg3dic5tA8Po-H88V3d7LZApoLoBPCuDp_BJUKf3JoPW_kRygzg8xD0-2LucQNUx-hFp1JInrdbpuo7c4M5Ep7nEMie1oYTcN3F9Io6EbjiroN7bxKgaCyDLH8av9FT7xQ1WuwaVUJoCISrvhJNe4itShe5bjHqhs4ZkzTb7WHfQ86Me9oVPLKjiZddeFo0OT0oRpSYlUNl2RRPuG0cDVxOWJ3R05UZ4rU6asNyepVEz5oNxnIxUpiGvIMWrmy9ycIJwGXCol7aNCyqSXcOorFtggL-E2c-GnqA1zN3-vbs6Y19N29vfna7Q3jJ_G8_EoejxH-2CoCpR1gZrFamMu0a76KLL16qo0_DcU_a1q |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2018+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition&rft.atitle=TieNet%3A+Text-Image+Embedding+Network+for+Common+Thorax+Disease+Classification+and+Reporting+in+Chest+X-Rays&rft.au=Wang%2C+Xiaosong&rft.au=Peng%2C+Yifan&rft.au=Lu%2C+Le&rft.au=Lu%2C+Zhiyong&rft.date=2018-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=9049&rft.epage=9058&rft_id=info:doi/10.1109%2FCVPR.2018.00943&rft.externalDocID=8579041 |