Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources
Misinformation is now a major problem due to its poten-tial high risks to our core democratic and societal values and orders. Out-of-context misinformation is one of the easiest and effective ways used by adversaries to spread vi-ral false stories. In this threat, a real image is re-purposed to supp...
Uloženo v:
| Vydáno v: | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) s. 14920 - 14929 |
|---|---|
| Hlavní autoři: | , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.06.2022
|
| Témata: | |
| ISSN: | 1063-6919 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Misinformation is now a major problem due to its poten-tial high risks to our core democratic and societal values and orders. Out-of-context misinformation is one of the easiest and effective ways used by adversaries to spread vi-ral false stories. In this threat, a real image is re-purposed to support other narratives by misrepresenting its context and/or elements. The internet is being used as the go-to way to verify information using different sources and modali-ties. Our goal is an inspectable method that automates this time-consuming and reasoning-intensive process by fact-checking the image-caption pairing using Web evidence. To integrate evidence and cues from both modalities, we intro-duce the concept of 'multi-modal cycle-consistency check' starting from the image/caption, we gather tex-tual/visual evidence, which will be compared against the other paired caption/image, respectively. Moreover, we propose a novel architecture, Consistency-Checking Network (CCN), that mimics the layered human reasoning across the same and different modalities: the caption vs. textual evidence, the image vs. visual evidence, and the image vs. caption. Our work offers the first step and bench-mark for open-domain, content-based, multi-modal fact-checking, and significantly outperforms previous baselines that did not leverage external evidence 1 1 For code, checkpoints, and dataset, check: https://s-abdelnabi.github.io/OoC-multi-modal-fc/. |
|---|---|
| AbstractList | Misinformation is now a major problem due to its poten-tial high risks to our core democratic and societal values and orders. Out-of-context misinformation is one of the easiest and effective ways used by adversaries to spread vi-ral false stories. In this threat, a real image is re-purposed to support other narratives by misrepresenting its context and/or elements. The internet is being used as the go-to way to verify information using different sources and modali-ties. Our goal is an inspectable method that automates this time-consuming and reasoning-intensive process by fact-checking the image-caption pairing using Web evidence. To integrate evidence and cues from both modalities, we intro-duce the concept of 'multi-modal cycle-consistency check' starting from the image/caption, we gather tex-tual/visual evidence, which will be compared against the other paired caption/image, respectively. Moreover, we propose a novel architecture, Consistency-Checking Network (CCN), that mimics the layered human reasoning across the same and different modalities: the caption vs. textual evidence, the image vs. visual evidence, and the image vs. caption. Our work offers the first step and bench-mark for open-domain, content-based, multi-modal fact-checking, and significantly outperforms previous baselines that did not leverage external evidence 1 1 For code, checkpoints, and dataset, check: https://s-abdelnabi.github.io/OoC-multi-modal-fc/. |
| Author | Fritz, Mario Hasan, Rakibul Abdelnabi, Sahar |
| Author_xml | – sequence: 1 givenname: Sahar surname: Abdelnabi fullname: Abdelnabi, Sahar email: sahar.abdelnabi@cispa.de organization: CISPA Helmholtz Center for Information Security – sequence: 2 givenname: Rakibul surname: Hasan fullname: Hasan, Rakibul email: rakibul.hasan@cispa.de organization: CISPA Helmholtz Center for Information Security – sequence: 3 givenname: Mario surname: Fritz fullname: Fritz, Mario email: fritz@cispa.de organization: CISPA Helmholtz Center for Information Security |
| BookMark | eNotjEtOwzAUAA0Cibb0BLDwAeriT-LPEgUKlYqCKmBbuclzMSR2FbsIbg8CVrOZmTE6CTEAQpeMzhmj5qp6eVyXXGo955TzOWVFyY_QmElZFtIUUhyjEaNSEGmYOUPTlN4opYIzJo0eoabeQyA3sbc-zHAVQ4aQydYmaGf44dBlT_rY2g4vbJNJ8wrNuw87HB2uD5lER36Tz4yXvd1Bwh_e4jp0PgBeQ4qHoYF0jk6d7RJM_zlBz4vbp-qerOq7ZXW9Ip5TkYlytFVaF1sHmrECpJGUglLAtVampYXRrBHqRxKWceFaYG3pnNNcczC6FRN08ff1ALDZD763w9fGaKWVMOIbaP1XQQ |
| CODEN | IEEPAD |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/CVPR52688.2022.01452 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Applied Sciences |
| EISBN | 1665469463 9781665469463 |
| EISSN | 1063-6919 |
| EndPage | 14929 |
| ExternalDocumentID | 9878739 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH 6IL 6IN AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
| ID | FETCH-LOGICAL-i203t-7f0d7884bfe8114e69600e77e28879d04981c370d73a123fde1d5fff8282e98d3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 54 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000870783000050&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:15:08 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i203t-7f0d7884bfe8114e69600e77e28879d04981c370d73a123fde1d5fff8282e98d3 |
| PageCount | 10 |
| ParticipantIDs | ieee_primary_9878739 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-June |
| PublicationDateYYYYMMDD | 2022-06-01 |
| PublicationDate_xml | – month: 06 year: 2022 text: 2022-June |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) |
| PublicationTitleAbbrev | CVPR |
| PublicationYear | 2022 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0003211698 |
| Score | 2.5361776 |
| Snippet | Misinformation is now a major problem due to its poten-tial high risks to our core democratic and societal values and orders. Out-of-context misinformation is... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 14920 |
| SubjectTerms | categorization Cognition Computer architecture Computer vision Computer vision for social good; Recognition: detection Machine vision Manuals MIMICs retrieval; Vision + language; Vision applications and systems; Visual reasoning Visualization |
| Title | Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources |
| URI | https://ieeexplore.ieee.org/document/9878739 |
| WOSCitedRecordID | wos000870783000050&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA7b8OBp6ib-JgePy5YmbZOcp0Mvc4jKbiNrEhi4VtZu-Of7ktaJ4MVbKAmFPPK-fC_vew-hW7iFLiWPEgKHSZE41pzolHKSycim4A0B9ExoNiGmUzmfq1kLDfZaGGttSD6zQz8Mb_mmyLY-VDYCfiwFV23UFiKttVr7eAoHJpMq2ajjIqpG47fZsy9m4hO4GBv69zP2q4dKgJBJ938_P0L9Hy0enu1R5hi1bH6Cus3lETdHs-yhkBpC7oo1UP0BDkWn8op4kDIDHGS2ZF0Y_Y4nOqsImCrzQXJcOPy0rUjhSFjyWeHHNbiYEu9WGtd1SPF3iL_so9fJ_cv4gTQdFMiKUV4R4agBjhsvnZVAfGwKfIVaISwD36IMsAMZZVzAJK4BwpyxkUmcc0DDmFXS8FPUyYvcniGsAOMSahKlVQwmpEvmIiMNMxEzOqP2HPX8ni0-6iIZi2a7Lv7-fIkOvVHqnKsr1Kk2W3uNDrJdtSo3N8GyXzeEoxY |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEG0QTfSEisZve_BIoduyu-0ZJRARiUHDjZRtm5DIrmEX4s93WlaMiRdvzabNJp10Xt903gxCd3ALnQkehAQOkyTttuJERZSTRAQmAm8IoKd9s4l4OBSTiRxVUGOrhTHG-OQz03RD_5avs2TlQmUt4Mci5nIH7brOWaVaaxtR4cBlIilKfVxAZavzNnpx5UxcChdjTfeCxn51UfEg0q397_eH6ORHjYdHW5w5QhWTHqNaeX3E5eHM68gnh5D7bAFkv4F92am0IA6mdAN7oS1ZZFq9465KCgLGSlyYHGcWP68Kklnil3wWuL8AJ5Pj9VzhTSVS_B3kz0_Qa_dh3OmRsocCmTPKCxJbqoHltmfWCKA-JgLGQk0cGwbeRWrgByJIeAyTuAIQs9oEOrTWAhFjRgrNT1E1zVJzhrAElAupDqWSbTAinTEbaKGZDphWCTXnqO72bPqxKZMxLbfr4u_Pt2i_N34aTAf94eMlOnAG2mRgXaFqsVyZa7SXrIt5vrzxVv4C5UumXw |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=Open-Domain%2C+Content-based%2C+Multi-modal+Fact-checking+of+Out-of-Context+Images+via+Online+Resources&rft.au=Abdelnabi%2C+Sahar&rft.au=Hasan%2C+Rakibul&rft.au=Fritz%2C+Mario&rft.date=2022-06-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=14920&rft.epage=14929&rft_id=info:doi/10.1109%2FCVPR52688.2022.01452&rft.externalDocID=9878739 |