Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary modul...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 989 - 998
Main Authors: Bhunia, Ayan Kumar, Koley, Subhadeep, Khilji, Abdullah Faiz Ur Rahman, Sain, Aneeshan, Chowdhury, Pinaki Nath, Xiang, Tao, Song, Yi-Zhe
Format: Conference Proceeding
Language:English
Published: IEEE 01.01.2022
Subjects:
Boosting
categorization
Computational modeling
Computer vision
grouping and shape analysis; Vision applications and systems
Head
Image retrieval
Pattern recognition
Recognition: detection
Reinforcement learning
retrieval; Segmentation
ISSN:1063-6919
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the "I can't sketch". We consequently design a stroke subset selector that detects noisy strokes, leaving only those which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.
AbstractList Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the "I can't sketch". We consequently design a stroke subset selector that detects noisy strokes, leaving only those which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.
Author Chowdhury, Pinaki Nath
Khilji, Abdullah Faiz Ur Rahman
Song, Yi-Zhe
Sain, Aneeshan
Bhunia, Ayan Kumar
Koley, Subhadeep
Xiang, Tao
Author_xml – sequence: 1
  givenname: Ayan Kumar
  surname: Bhunia
  fullname: Bhunia, Ayan Kumar
  email: a.bhunia@surrey.ac.uk
  organization: University of Surrey,SketchX, CVSSP,United Kingdom
– sequence: 2
  givenname: Subhadeep
  surname: Koley
  fullname: Koley, Subhadeep
  email: s.koley@surrey.ac.uk
  organization: University of Surrey,SketchX, CVSSP,United Kingdom
– sequence: 3
  givenname: Abdullah Faiz Ur Rahman
  surname: Khilji
  fullname: Khilji, Abdullah Faiz Ur Rahman
  organization: iFlyTek-Surrey Joint Research Centre on Artificial Intelligence
– sequence: 4
  givenname: Aneeshan
  surname: Sain
  fullname: Sain, Aneeshan
  email: a.sain@surrey.ac.uk
  organization: University of Surrey,SketchX, CVSSP,United Kingdom
– sequence: 5
  givenname: Pinaki Nath
  surname: Chowdhury
  fullname: Chowdhury, Pinaki Nath
  email: p.chowdhury@surrey.ac.uk
  organization: University of Surrey,SketchX, CVSSP,United Kingdom
– sequence: 6
  givenname: Tao
  surname: Xiang
  fullname: Xiang, Tao
  email: t.xiang@surrey.ac.uk
  organization: University of Surrey,SketchX, CVSSP,United Kingdom
– sequence: 7
  givenname: Yi-Zhe
  surname: Song
  fullname: Song, Yi-Zhe
  email: y.song@surrey.ac.uk
  organization: University of Surrey,SketchX, CVSSP,United Kingdom
BookMark eNotjsFKw0AURUdRsK39Al3kBya-N5NMMu601FooKjXosrxMX9rRNJFkVPr3FurqwuFwuENx1rQNC3GNECOCvZm8vSxTZfI8VqBUDICQnYghGpMmxiZGn4oBgtHSWLQXYtz3HwCgFaKx-UDMXj85uK1vNtGvD9v2O0TvbdftD-A2emp9z7Joa-6oCdFRlffU8zqa72jD0ZJD5_mH6ktxXlHd8_h_R6J4mBaTR7l4ns0ndwvplbFBZunhpUPMoSRd6TQlp0uoMlMROCqTjCCrFFqrHZYJgWFnNMG6JOIkZT0SV8esZ-bVV-d31O1XNs8BrdZ_oEFOlg
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/CVPR52688.2022.00107
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Applied Sciences
EISBN 1665469463
9781665469463
EISSN 1063-6919
EndPage 998
ExternalDocumentID 9880193
Genre orig-research
GroupedDBID 6IE
6IH
6IL
6IN
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IJVOP
OCL
RIE
RIL
RIO
ID FETCH-LOGICAL-i269t-75202c1180ba3f355ac3b0f76fa0cab47a07f21993c1b4a06ec63a0dbaae45e3
IEDL.DBID RIE
ISICitedReferencesCount 33
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000867754201024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:15:11 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i269t-75202c1180ba3f355ac3b0f76fa0cab47a07f21993c1b4a06ec63a0dbaae45e3
PageCount 10
ParticipantIDs ieee_primary_9880193
PublicationCentury 2000
PublicationDate 2022-01-01
PublicationDateYYYYMMDD 2022-01-01
PublicationDate_xml – month: 01
  year: 2022
  text: 2022-01-01
  day: 01
PublicationDecade 2020
PublicationTitle Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)
PublicationTitleAbbrev CVPR
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0003211698
Score 2.4406521
Snippet Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for...
SourceID ieee
SourceType Publisher
StartPage 989
SubjectTerms Boosting
categorization
Computational modeling
Computer vision
grouping and shape analysis; Vision applications and systems
Head
Image retrieval
Pattern recognition
Recognition: detection
Reinforcement learning
retrieval; Segmentation
Title Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval
URI https://ieeexplore.ieee.org/document/9880193
WOSCitedRecordID wos000867754201024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB7a4sFT1VZ8k4NHY7O7aR4eLVYFKaWW2lvJa6Fou7Ld-vtNdpeK4MVbEgIhk8c3mcw3A3BNjfbAZhhmIiWYOmax1lr4ap_ZcCHq8sd09sJHIzGfy3EDbnZcGOdc6XzmbkOx_Mu3mdkGU1lP-s3mFY4mNDlnFVdrZ09J_EuGSVGz4yIie4PZeBKCmQQHrjiE5YzI7xwqJYQM2_8b_AC6P1w8NN6hzCE03PoI2rXyiOqjuenA4-t7WAHfBwXbarYt0FuW54HFdIdG2XLj8DT7cB6aClR1xfcewSx6XvkrBU3KzFp-23VhOnyYDp5wnSUBL2MmC8z7fnomRHLTKkm9-qBMoknKWaqIUZpyRXgaBz89E2mqCHOGJYpYrZSjfZccQ2udrd0JICqUiFORSmP9M4tKran1CpVLVCK4ZdEpdIJYFp9VHIxFLZGzv5vPYT_IvTJXXECryLfuEvbMV7Hc5Ffl4n0DX9CbRw
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PT8MgFH6Z00RPUzfjbzl4FEdbSqlHF-cWZ7PMZu62AKXJoq6m6_z7hbaZMfHiDQgJgQd8j8f73gO4pkoaYFMMM54STDVLsJSSm6rPEnshyvLHdDoKoojPZuG4ATcbLozWunQ-07e2WP7lJ5laW1NZNzSbzSgcW7DtU-qSiq21sah45i3DQl7z4xwSdnvT8cSGM7EuXK4NzOmQ31lUShDpt_43_D50fth4aLzBmQNo6OUhtGr1EdWHc9WGx5c3KwPTB1nrarYu0GuW55bHdIeibLHSOM7etQGnAlVd8b3BsAQNP8ylgiZlbi2z8ToQ9x_i3gDXeRLwwmVhgQPfTE_ZWG5SeKlRIITyJEkDlgqihKSBIEHqWk895UgqCNOKeYIkUghNfe0dQXOZLfUxIMoFd1OehioxDy0aSkkTo1JpT3g8SJhzAm27LPPPKhLGvF6R07-br2B3ED-P5qNh9HQGe1YGlfHiHJpFvtYXsKO-isUqvywF-Q2jgZ6O
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=proceeding&rft.title=Proceedings+%28IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition.+Online%29&rft.atitle=Sketching+without+Worrying%3A+Noise-Tolerant+Sketch-Based+Image+Retrieval&rft.au=Bhunia%2C+Ayan+Kumar&rft.au=Koley%2C+Subhadeep&rft.au=Khilji%2C+Abdullah+Faiz+Ur+Rahman&rft.au=Sain%2C+Aneeshan&rft.date=2022-01-01&rft.pub=IEEE&rft.eissn=1063-6919&rft.spage=989&rft.epage=998&rft_id=info:doi/10.1109%2FCVPR52688.2022.00107&rft.externalDocID=9880193