TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays

Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due...

Full description

Saved in:
Bibliographic Details
Published in:2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 9049 - 9058
Main Authors: Wang, Xiaosong, Peng, Yifan, Lu, Le, Lu, Zhiyong, Summers, Ronald M.
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2018
Subjects:
Biomedical imaging
Diseases
Feature extraction
Task analysis
Training
Visualization
X-rays
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI).
AbstractList Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task for radiologist trainees. Yet, reading a chest X-ray image remains a challenging job for learning-oriented machine intelligence, due to (1) shortage of large-scale machine-learnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human radiologists that requires years of knowledge accumulation and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori knowledge for tackling these two key problems. We propose a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multi-level attention models are integrated into an end-to-end trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet to classify the chest X-rays by using both image features and text embeddings extracted from associated reports. The proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease labels for our hand-label evaluation dataset. Furthermore, we transform the TieNet into a chest X-ray reporting system. It simulates the reporting process and can output disease classification and a preliminary report together. The classification results are significantly improved (6% increase on average in AUCs) compared to the state-of-the-art baseline on an unseen and hand-labeled dataset (OpenI).
Author Wang, Xiaosong
Lu, Zhiyong
Summers, Ronald M.
Lu, Le
Peng, Yifan
Author_xml – sequence: 1
  givenname: Xiaosong
  surname: Wang
  fullname: Wang, Xiaosong
– sequence: 2
  givenname: Yifan
  surname: Peng
  fullname: Peng, Yifan
– sequence: 3
  givenname: Le
  surname: Lu
  fullname: Lu, Le
– sequence: 4
  givenname: Zhiyong
  surname: Lu
  fullname: Lu, Zhiyong
– sequence: 5
  givenname: Ronald M.
  surname: Summers
  fullname: Summers, Ronald M.
BookMark eNotjr1OwzAURg0CiVI6M7D4BVKu45_YbCgUqFQBigJiq-zkpjVt4iqORPv2FMF0hqPz6bskZ13okJBrBlPGwNzmH2_FNAWmpwBG8BMyMZlmkmulRArmlIwYKJ4ow8wFmcT4BQCp0lwLOSKb0uMLDne0xP2QzFu7QjprHda171b0aL5Dv6FN6Gke2jZ0tFyH3u7pg49oI9J8a2P0ja_s4I_WdjUtcBf64Tf3Hc3XGAf6mRT2EK_IeWO3ESf_HJP3x1mZPyeL16d5fr9IfCrYkNQZNsLK2gnuKp1Jx-sUpVBQc9BWZ8frFQPXyIprxhtXySplOhNaKGc4IB-Tm79dj4jLXe9b2x-WWmYGBOM_fytaKQ
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR.2018.00943
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library (IEL) (UW System Shared)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 9781538664209
1538664208
EISSN 1063-6919
EndPage 9058
ExternalDocumentID 8579041
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3
IEDL.DBID RIE
ISICitedReferencesCount 368
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:52:16 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i241t-d7ef4a5db43bc875b3d2e5460d308a87683c10bf5c3813fbc5c21874846b930e3
PageCount 10
ParticipantIDs ieee_primary_8579041
PublicationCentury 2000
PublicationDate 2018-06
PublicationDateYYYYMMDD 2018-06-01
PublicationDate_xml – month: 06
  year: 2018
  text: 2018-06
PublicationDecade 2010
PublicationTitle 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
PublicationTitleAbbrev CVPR
PublicationYear 2018
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0002683845
ssj0003211698
Score 2.586755
Snippet Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an...
SourceID ieee
SourceType Publisher
StartPage 9049
SubjectTerms Biomedical imaging
Diseases
Feature extraction
Task analysis
Training
Visualization
X-rays
Title TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays
URI https://ieeexplore.ieee.org/document/8579041
WOSCitedRecordID wos000457843609023&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELZKxcBUoEW85YERUydOYpu1tAIJRVUVqm6VXxEVaoraFMG_x5dEhYGFLQ9ZinyO7873ffchdGOZSqDPE5FMUBLlKiCKJ4b41UMDHYUmqZr6TJ95morZTI5b6HbHhXHOVeAzdweXVS3frswWjsr6IuaSAkt9j3Nec7V25ylhIphoKmRwz3xmk0jRdPMJqOwPpuMJYLkAPCmBpfNLTqXyJqPO_77jEPV-aHl4vHM4R6jlimPUaeJI3Pylmy56yxYudeU9ziCtfVr6LQMPl9pZGIbTGviNfbSKgR6yKnD26hfCJ36oizW4EsoECFFlNawKi-s4HYYvCjwAjS08IxP1temhl9EwGzySRlSBLLyzLonlLo9UbHXEtPHJimY2dHGUUMuoUH5vFMwEVOex8b6c5drEJgTdPh-naMmoYyeoXawKd4qw9m9tonzCkeeRVlobbqNAxor6pC7IozPUhbmbv9d9M-bNtJ3__fgCHYBxahjWJWqX6627Qvvmo1xs1teVsb8BQRqp1A
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEF6KCnqq2opv9-DR2E1281ivtaXFGkqJpbeyr2CQptKmov_enSRUD1685cFC2NnszOx833wI3WoqAujz5HAaEYelwnVEGCjHrh7iSuapoGzqMx2FcRzNZnzcQHdbLowxpgSfmXu4LGv5eqk2cFTWifyQE2Cp7_qMeW7F1tqeqHhBRKO6Rgb31OY2AY_qfj4u4Z3udDwBNBfAJznwdH4JqpT-pN_835ccovYPMQ-Pty7nCDVMfoyadSSJ6_903UJvSWZiUzzgBBLb4cJuGri3kEbDMBxX0G9s41UMBJFljpNXuxQ-8WNVrsGlVCaAiEq7YZFrXEXqMDzLcRdUtvDMmYivdRu99HtJd-DUsgpOZt114ejQpEz4WjIqlU1XJNWe8VlANCWRsLtjRJVLZOor681pKpWvPFDus5GK5JQYeoJ28mVuThGW9q0OhE050pRJIaUKNXO5L4hN69yUnaEWzN38veqcMa-n7fzvxzdof5A8j-ajYfx0gQ7AUBUo6xLtFKuNuUJ76qPI1qvr0vDfQyitGw
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2018+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition&rft.atitle=TieNet%3A+Text-Image+Embedding+Network+for+Common+Thorax+Disease+Classification+and+Reporting+in+Chest+X-Rays&rft.au=Wang%2C+Xiaosong&rft.au=Peng%2C+Yifan&rft.au=Lu%2C+Le&rft.au=Lu%2C+Zhiyong&rft.date=2018-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=9049&rft.epage=9058&rft_id=info:doi/10.1109%2FCVPR.2018.00943&rft.externalDocID=8579041