Augmenting Colonoscopy Using Extended and Directional CycleGAN for Lossy Image Translation
Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors for colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from computed tomography scans) f...
Uloženo v:
| Vydáno v: | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) Ročník 2020; s. 4695 - 4704 |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Konferenční příspěvek Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
United States
IEEE
01.06.2020
|
| Témata: | |
| ISSN: | 1063-6919, 1063-6919 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors for colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from computed tomography scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC and VC with patient-specific textures, color and specular highlights from OC (e.g. for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. |
|---|---|
| AbstractList | Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors for colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from computed tomography scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC and VC with patient-specific textures, color and specular highlights from OC (e.g. for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models.Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. |
| Author | Kaufman, Arie Kumari, Sruti Nadeem, Saad Mathew, Shawn |
| AuthorAffiliation | 2 Memorial Sloan Kettering Cancer Center 1 Stony Brook University |
| AuthorAffiliation_xml | – name: 2 Memorial Sloan Kettering Cancer Center – name: 1 Stony Brook University |
| Author_xml | – sequence: 1 givenname: Shawn surname: Mathew fullname: Mathew, Shawn organization: Stony Brook University – sequence: 2 givenname: Saad surname: Nadeem fullname: Nadeem, Saad organization: Memorial Sloan Kettering Cancer Center – sequence: 3 givenname: Sruti surname: Kumari fullname: Kumari, Sruti organization: Stony Brook University – sequence: 4 givenname: Arie surname: Kaufman fullname: Kaufman, Arie organization: Stony Brook University |
| BackLink | https://www.ncbi.nlm.nih.gov/pubmed/33456298$$D View this record in MEDLINE/PubMed |
| BookMark | eNpVUMtO6zAQNQh0gdIvuFfISzYtHju2482VqvCUKkAIWLCJXGdajBK7xCmif08QD8FiHppzdM7M7JGtEAMS8g_YGICZo-L--ibjirExZ5yNGcu03CBDo3PQvA9Qudwku8CUGCkDZutHv0OGKT0xxgQHUCb_Q3aEyKTiJt8lD5PVosHQ-bCgRaxjiMnF5ZrepffJyWuHocKK2lDRY9-i63wMtqbF2tV4Nrmk89jSaUxpTS8au0B629qQavtO2yfbc1snHH7WAbk7PbktzkfTq7OLYjIdeQFZNwKuNUflEESluewz5HN0xs0M4xwhk4y7mbRCzfhco2KIGpg0BhVwC0IMyP8P3eVq1mDl-mtaW5fL1je2XZfR-vI3EvxjuYgvZf88AC17gcNPgTY-rzB1ZeOTw7q2AeMqlTzTudYm63cbkIOfXt8mXw_tCX8_CB4Rv2EDUkkD4g1RxYfL |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding Journal Article |
| DBID | 6IE 6IH CBEJK RIE RIO NPM 7X8 5PM |
| DOI | 10.1109/CVPR42600.2020.00475 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present PubMed MEDLINE - Academic PubMed Central (Full Participant titles) |
| DatabaseTitle | PubMed MEDLINE - Academic |
| DatabaseTitleList | PubMed MEDLINE - Academic |
| Database_xml | – sequence: 1 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 2 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher – sequence: 3 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences Computer Science |
| EISBN | 9781728171685 1728171687 |
| EISSN | 1063-6919 |
| EndPage | 4704 |
| ExternalDocumentID | PMC7811175 33456298 9156591 |
| Genre | orig-research Journal Article |
| GrantInformation_xml | – fundername: NHLBI NIH HHS grantid: U01 HL127522 – fundername: NCI NIH HHS grantid: P30 CA008748 |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO 23M 29F 29O 6IK ABDPE ACGFS IPLJI M43 NPM RIG RNS 7X8 5PM |
| ID | FETCH-LOGICAL-i314t-12772e6ce13d72513d18fec9cb9022e14502cb5a36b2f7e60ee710599e612a133 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 51 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000620679504097&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1063-6919 |
| IngestDate | Thu Aug 21 18:24:17 EDT 2025 Fri Sep 05 10:39:28 EDT 2025 Wed Feb 19 02:04:13 EST 2025 Wed Aug 27 02:30:35 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i314t-12772e6ce13d72513d18fec9cb9022e14502cb5a36b2f7e60ee710599e612a133 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 Equal Contribution |
| PMID | 33456298 |
| PQID | 2478779451 |
| PQPubID | 23479 |
| PageCount | 10 |
| ParticipantIDs | pubmed_primary_33456298 pubmedcentral_primary_oai_pubmedcentral_nih_gov_7811175 proquest_miscellaneous_2478779451 ieee_primary_9156591 |
| PublicationCentury | 2000 |
| PublicationDate | 20200601 |
| PublicationDateYYYYMMDD | 2020-06-01 |
| PublicationDate_xml | – month: 6 year: 2020 text: 20200601 day: 1 |
| PublicationDecade | 2020 |
| PublicationPlace | United States |
| PublicationPlace_xml | – name: United States |
| PublicationTitle | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) |
| PublicationTitleAbbrev | CVPR |
| PublicationTitleAlternate | Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit |
| PublicationYear | 2020 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0003211698 ssj0023720 |
| Score | 2.570898 |
| Snippet | Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing... |
| SourceID | pubmedcentral proquest pubmed ieee |
| SourceType | Open Access Repository Aggregation Database Index Database Publisher |
| StartPage | 4695 |
| SubjectTerms | Cancer Colon Endoscopes Gallium nitride Image reconstruction Machine learning Three-dimensional displays |
| Title | Augmenting Colonoscopy Using Extended and Directional CycleGAN for Lossy Image Translation |
| URI | https://ieeexplore.ieee.org/document/9156591 https://www.ncbi.nlm.nih.gov/pubmed/33456298 https://www.proquest.com/docview/2478779451 https://pubmed.ncbi.nlm.nih.gov/PMC7811175 |
| Volume | 2020 |
| WOSCitedRecordID | wos000620679504097&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3JTsMwELUK4sCJHcpSGYkjoY0dx_GxisoioapCUFVcIieeQA-kqAtS_56xE8KiXrhFiiNZ9mTmPfvNDCEXWmVB2NHM87WRSFCM8KI8zT1jWCpMYCS4dkDDe9nvR6ORGjTIZZ0LAwBOfAZX9tHd5ZtJtrBHZW2FZEPYVPU1KWWZq1Wfp3BkMqGKquw4v6Pa8XDw4OqvIwtkVsAVWDGh66GyCk7-VUX-CDPXW_-b4DbZ_87Xo4M6Eu2QBhS7ZKsCmLT6fWd75Lm7eHHyoOKFxuj1iolNSllSpxugveo8nOrC0MoVWpxO4yXa1k23TxHg0nuMqkt694Z-iLpAV4rp9snTde8xvvWq5gremPvB3PMZ4moIM_C5kQhyuPGjHDKVpQrDOviB6LAsFZqHKcslhB0AabGYAsREGpntAVkvJgUcEZrpgOU8yJG6IB7TUstIMKFNxA1XQSiaZM8uVfJe1s9IqlVqkvOvTUjQpu1FhS5gspglzFYMQkchcMxhuSn1x5xbzqaiJpG_tqseYOtl_35TjF9d3WybVIto6Xj1dE7IprWTUgZ2Stbn0wWckY3sYz6eTVtocqOo5UzuE8x-19c |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fT9swED5VMIk9AYNtZWN4Eo9kNP4Rx4-oglFRqmoChHiJnPjC-kCKaIvU_35nJws_xAtvkeJIln25-z77uzuAfWsKmfQsj2LrNBEUp6K0zMvIOZ4rJ53G0A7oaqhHo_T62ow7cNDmwiBiEJ_hL_8Y7vLdtFj4o7JDQ2RD-VT1VSUlj-tsrfZERRCXSUza5MfFPXPYvxr_CRXYiQdyL-GSXk4Yuqi8BShf6yKfBZqT9fdNcQO2nzL22LiNRZvQweoTrDcQkzU_8GwLbo4Wt0EgVN2yPvm9aurTUpYsKAfYcXMizmzlWOMMPVJn_SVZ1--jESOIy4YUV5dscEeeiIVQV8vptuHy5Piifxo17RWiiYjlPIo5IWtMCoyF0wRzhIvTEgtT5IYCO8ZS9XiRKyuSnJcakx6i9mjMIKEiS9z2M6xU0wq_Aius5KWQJZEXQmRWW50qrqxLhRNGJqoLW36psvu6gkbWrFIXfv7fhIys2l9V2Aqni1nGfc0gchWKxnypN6X9WAjP2kzaBf1iu9oBvmL2yzfV5G-onO3Tagkv7bw9nT1YO704H2bDwejsG3z0NlOLwr7DyvxhgbvwoXicT2YPP4Lh_QMR4do2 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=Augmenting+Colonoscopy+Using+Extended+and+Directional+CycleGAN+for+Lossy+Image+Translation&rft.au=Mathew%2C+Shawn&rft.au=Nadeem%2C+Saad&rft.au=Kumari%2C+Sruti&rft.au=Kaufman%2C+Arie&rft.date=2020-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=4695&rft.epage=4704&rft_id=info:doi/10.1109%2FCVPR42600.2020.00475&rft.externalDocID=9156591 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1063-6919&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1063-6919&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1063-6919&client=summon |