Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective
Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity, and uncertainty, which lead the knowledge learning from them full of challenges. With the emergence of crowdsourcing, versatile information can be obtained on-demand so that the wisdom of crowds is easily...
Saved in:
| Published in: | IEEE/CAA journal of automatica sinica Vol. 9; no. 5; pp. 749 - 762 |
|---|---|
| Main Author: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
Piscataway
Chinese Association of Automation (CAA)
01.05.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China |
| Subjects: | |
| ISSN: | 2329-9266, 2329-9274 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity, and uncertainty, which lead the knowledge learning from them full of challenges. With the emergence of crowdsourcing, versatile information can be obtained on-demand so that the wisdom of crowds is easily involved to facilitate the knowledge learning process. During the past thirteen years, researchers in the AI community made great efforts to remove the obstacles in the field of learning from crowds. This concentrated survey paper comprehensively reviews the technical progress in crowdsourcing learning from a systematic perspective that includes three dimensions of data, models, and learning processes. In addition to reviewing existing important work, the paper places a particular emphasis on providing some promising blueprints on each dimension as well as discussing the lessons learned from our past research work, which will light up the way for new researchers and encourage them to pursue new contributions. |
|---|---|
| AbstractList | Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity, and uncertainty, which lead the knowledge learning from them full of challenges. With the emergence of crowdsourcing, versatile information can be obtained on-demand so that the wisdom of crowds is easily involved to facilitate the knowledge learning process. During the past thirteen years, researchers in the AI community made great efforts to remove the obstacles in the field of learning from crowds. This concentrated survey paper comprehensively reviews the technical progress in crowdsourcing learning from a systematic perspective that includes three dimensions of data, models, and learning processes. In addition to reviewing existing important work, the paper places a particular emphasis on providing some promising blueprints on each dimension as well as discussing the lessons learned from our past research work, which will light up the way for new researchers and encourage them to pursue new contributions. |
| Author | Zhang, Jing |
| AuthorAffiliation | School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China |
| AuthorAffiliation_xml | – name: School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China |
| Author_xml | – sequence: 1 givenname: Jing surname: Zhang fullname: Zhang, Jing email: jzhang@njust.edu.cn organization: School of Computer Science and Engineering, Nanjing University of Science and Technology,Nanjing,China,210094 |
| BookMark | eNp9kEtPGzEURq0KpFJg3UU3lrqrlMTjxxh3l0Ytr0hEpYilZXuug1HwBHtgCL8eR4OoxIKVr6zz3cf5gnZiGwGhrxUZVxVRk7Pp5ZgSSscVEZzxT2iPMqpGikq-81bX9Wd0mPMtIaSiQtaK76HFeWz7FTRLwHMwKYa4xNehu8Gz1PZNbh-SK18_8RT_SgE8_guPAXpsYoMvN7mDO9MFhxeQ8hpcFx7hAO16s8pw-Pruo6s_v__NTkbzi-PT2XQ-cozybuQ5V8QYyayXIEkjlbLGSQWV9JYfeavASOub2hIDzoBQVjDhqJWcM2g820c_hr69id7Epb4tu8YyUT83N09Wb3q7FUJEObbA3wd4ndr7B8jdf5rWQhzVRUhdKDFQLrU5J_Daha7c18YumbDSFdFb2brI1tveepBdcpN3uXUKdyZtPkh8GxIBAN5oJWvGpWQvn6-L2A |
| CODEN | IJASJC |
| CitedBy_id | crossref_primary_10_1016_j_patcog_2025_111962 crossref_primary_10_1109_JAS_2023_123960 crossref_primary_10_3390_math11051153 crossref_primary_10_1007_s11432_023_3854_7 crossref_primary_10_1057_s41599_025_05105_2 crossref_primary_10_1007_s11704_023_2751_3 crossref_primary_10_1007_s11704_022_2225_z crossref_primary_10_1109_ACCESS_2025_3598463 crossref_primary_10_1109_TPAMI_2024_3388209 crossref_primary_10_1177_22104968251361342 crossref_primary_10_1002_widm_70037 crossref_primary_10_1109_TSC_2023_3292498 crossref_primary_10_1007_s10479_023_05619_5 crossref_primary_10_1109_TETCI_2024_3423346 crossref_primary_10_1016_j_patcog_2023_109525 crossref_primary_10_1016_j_ins_2024_120702 crossref_primary_10_1109_TNNLS_2022_3194022 crossref_primary_10_3390_diagnostics13061070 crossref_primary_10_1016_j_patcog_2024_110325 crossref_primary_10_1109_JAS_2024_124860 crossref_primary_10_1016_j_ins_2022_05_066 crossref_primary_10_1016_j_patcog_2024_111034 crossref_primary_10_3390_s23073518 crossref_primary_10_1016_j_is_2023_102321 crossref_primary_10_1109_TSC_2024_3404353 crossref_primary_10_1007_s11704_022_2245_8 crossref_primary_10_1145_3712605 crossref_primary_10_1016_j_ijar_2025_109570 crossref_primary_10_1109_TMC_2025_3563345 crossref_primary_10_1109_TNNLS_2024_3438680 crossref_primary_10_1109_TASE_2024_3360476 crossref_primary_10_1016_j_patcog_2023_109316 crossref_primary_10_1287_ijoc_2023_0440 crossref_primary_10_3390_app14020530 crossref_primary_10_1080_03610918_2025_2479843 |
| Cites_doi | 10.24963/ijcai.2020/210 10.18653/v1/p17-5006 10.1109/TKDE.2016.2555805 10.24963/ijcai.2017/413 10.1145/2362456.2362479 10.1109/TKDE.2018.2857766 10.1609/hcomp.v1i1.13091 10.1109/TKDE.2019.2951668 10.1007/s10115-019-01386-7 10.1007/978-3-642-40988-2_18 10.1007/s11432-020-3118-7 10.1145/3159652.3159654 10.24963/ijcai.2017/324 10.3115/1613715.1613751 10.1145/3077136.3080679 10.1093/bioinformatics/btt333 10.1609/hcomp.v3i1.13238 10.2307/2346806 10.1609/aaai.v28i2.19016 10.1109/TNNLS.2020.2978386 10.1145/2431211.2431215 10.1609/aaai.v33i01.33019837 10.24963/ijcai.2020/402 10.1609/aaai.v34i01.5392 10.1109/ICCV.2013.373 10.24963/ijcai.2020/214 10.1145/2661829.2661946 10.1145/3394486.3403167 10.1145/3018661.3018688 10.1007/s10115-020-01475-y 10.1145/1401890.1401965 10.1145/3290605.3300773 10.1609/aaai.v29i1.9761 10.1109/ICDM.2017.78 10.1007/978-3-319-18038-0_31 10.1609/aaai.v32i1.11513 10.1145/2484028.2484199 10.1109/TKDE.2015.2504974 10.1609/aaai.v27i1.8456 10.1609/aaai.v36i11.21684 10.1016/j.eswa.2014.03.048 10.1609/hcomp.v2i1.13165 10.1145/1656274.1656278 10.1109/TKDE.2016.2535242 10.1109/ICDM.2011.133 10.1145/2488388.2488414 10.24963/ijcai.2017/261 10.1109/TNNLS.2021.3082496 10.1609/aaai.v26i1.8105 10.14778/2735496.2735505 10.1109/TKDE.2014.2327026 10.1145/3404835.3463081 10.1016/j.ins.2021.05.060 10.1609/aaai.v28i1.8993 10.1145/2187836.2187900 10.1609/hcomp.v3i1.13231 10.1609/aaai.v27i1.8530 10.1609/hcomp.v3i1.13256 10.1609/hcomp.v1i1.13088 10.1145/3148148 10.1145/2487575.2487708 10.1109/TNNLS.2017.2677468 10.1145/2566486.2567989 10.1109/CVPR.2013.81 10.1007/978-3-030-45442-5_26 10.1007/978-3-030-58607-2_17 10.1609/aaai.v30i1.10315 10.1609/hcomp.v2i1.13167 10.1145/2487575.2487593 10.1016/j.comnet.2021.108227 10.1109/Allerton.2011.6120180 10.1109/ICDM.2012.64 10.1109/TNNLS.2020.2984729 10.1109/CVPRW.2010.5543189 10.1109/TKDE.2014.2327039 10.1007/s10994-013-5412-1 10.24963/ijcai.2021/238 10.1109/TCYB.2014.2344674 10.1609/aaai.v32i1.11506 10.1109/TKDE.2018.2860992 10.1145/1993574.1993599 10.14778/3055540.3055547 10.1007/s10462-016-9491-9 10.24963/ijcai.2017/184 10.1109/TKDE.2017.2754499 10.1145/3219819.3219958 10.1561/0600000071 10.1145/1557019.1557053 10.1145/2588555.2610509 10.1609/aaai.v34i04.5898 10.1145/1458082.1458165 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022 – notice: Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 7TB 8FD FR3 JQ2 L7M L~C L~D 2B. 4A8 92I 93N PSX TCJ |
| DOI | 10.1109/JAS.2022.105434 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Mechanical & Transportation Engineering Abstracts Technology Research Database Engineering Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional Wanfang Data Journals - Hong Kong WANFANG Data Centre Wanfang Data Journals 万方数据期刊 - 香港版 China Online Journals (COJ) China Online Journals (COJ) |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Mechanical & Transportation Engineering Abstracts Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Engineering Research Database Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 2329-9274 |
| EndPage | 762 |
| ExternalDocumentID | zdhxb_ywb202205001 10_1109_JAS_2022_105434 9763477 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Key Research and Development Program of China grantid: 2018AAA0102002 funderid: 10.13039/501100012166 – fundername: National Natural Science Foundation of China grantid: 62076130,91846104 funderid: 10.13039/501100001809 |
| GroupedDBID | -0I -0Y -SI -S~ 0R~ 4.4 5VR 6IK 92M 97E 9D9 9DI AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACIWK AFUIB AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CAJEI EBS EJD IFIPE IPLJI JAVBF M43 O9- OCL PQQKQ Q-- RIA RIE RT9 T8Y TCJ TGT U1F U1G U5I U5S AAYXX CITATION 7SC 7SP 7TB 8FD FR3 JQ2 L7M L~C L~D 2B. 4A8 92I 93N PSX R-I RIG |
| ID | FETCH-LOGICAL-c324t-f4490aa73bf7e70d799bac79e17fb48fb9ea7bfd6b0aecae59b535c2b7443edf3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 45 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000794202700005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2329-9266 |
| IngestDate | Thu May 29 04:10:31 EDT 2025 Fri Jul 25 05:30:43 EDT 2025 Tue Nov 18 22:05:37 EST 2025 Sat Nov 29 03:31:06 EST 2025 Wed Aug 27 02:40:28 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 5 |
| Keywords | Crowdsourcing data fusion learning from crowds learning paradigms learning with uncertainty |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c324t-f4490aa73bf7e70d799bac79e17fb48fb9ea7bfd6b0aecae59b535c2b7443edf3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| PQID | 2655860126 |
| PQPubID | 2040495 |
| PageCount | 14 |
| ParticipantIDs | wanfang_journals_zdhxb_ywb202205001 proquest_journals_2655860126 ieee_primary_9763477 crossref_citationtrail_10_1109_JAS_2022_105434 crossref_primary_10_1109_JAS_2022_105434 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-05-01 |
| PublicationDateYYYYMMDD | 2022-05-01 |
| PublicationDate_xml | – month: 05 year: 2022 text: 2022-05-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | Piscataway |
| PublicationPlace_xml | – name: Piscataway |
| PublicationTitle | IEEE/CAA journal of automatica sinica |
| PublicationTitleAbbrev | JAS |
| PublicationTitle_FL | IEEE/CAA Journal of Automatica Sinica |
| PublicationYear | 2022 |
| Publisher | Chinese Association of Automation (CAA) The Institute of Electrical and Electronics Engineers, Inc. (IEEE) School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China |
| Publisher_xml | – name: Chinese Association of Automation (CAA) – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) – name: School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China |
| References | ref57 ref56 ref59 ref58 ref52 ref55 ref54 Zhou (ref29) 2012; 25 Bonald (ref64) 2017; 30 Tian (ref49) 2015; 28 Zhou (ref33) 2014 ref51 ref46 ref45 ref48 ref47 ref42 ref41 ref44 ref43 ref8 ref7 Zhang (ref116) 2015; 16 ref9 ref4 ref3 ref6 ref5 Yan (ref73) 2010 ref100 ref101 ref40 Rodrigues (ref106) 2014 ref35 ref34 ref37 ref36 ref30 Kim (ref26) 2012 ref38 Yan (ref74) 2011; 11 Ratner (ref97) 2016; 29 Li (ref32) 2019 ref24 Zhou (ref50) 2016 ref20 Zhao (ref76) 2015 ref22 Zhang (ref31) 2016; 17 ref28 ref27 Liu (ref23) 2012; 25 Whitehill (ref21) 2009; 22 Jung (ref39) 2011 Cao (ref78) 2019; abs/1905.13436 ref13 ref15 Welinder (ref25) 2010; 23 ref96 ref11 ref99 ref10 Kipf (ref94) 2017 Howe (ref1) 2006; 14 ref17 ref16 ref19 ref18 Khattak (ref63) 2011; 2 ref93 ref92 ref95 Hsieh (ref98) 2015 ref91 ref90 ref89 ref86 ref85 ref88 ref87 Bi (ref14) 2014 ref82 ref81 ref84 ref83 ref80 ref79 ref108 ref109 ref107 ref75 ref104 ref105 Raykar (ref12) 2010; 11 ref77 ref102 ref103 ref2 Oyama (ref65) 2013 Gaunt (ref53) 2016 ref71 ref111 ref70 ref112 ref72 ref110 ref68 ref67 ref117 ref69 ref118 ref115 ref66 ref113 ref114 ref60 ref62 ref61 |
| References_xml | – volume: 23 start-page: 2424 volume-title: Advances Neural Inf. Process. Syst. year: 2010 ident: ref25 article-title: The multidimensional wisdom of crowds – ident: ref56 doi: 10.24963/ijcai.2020/210 – start-page: 932 volume-title: Proc. 13 Int. Conf. Artif. Intell. Stat. year: 2010 ident: ref73 article-title: Modeling annotator expertise: Learning when everybody knows a bit of something – ident: ref6 doi: 10.18653/v1/p17-5006 – ident: ref19 doi: 10.1109/TKDE.2016.2555805 – ident: ref69 doi: 10.24963/ijcai.2017/413 – ident: ref2 doi: 10.1145/2362456.2362479 – ident: ref103 doi: 10.1109/TKDE.2018.2857766 – ident: ref35 doi: 10.1609/hcomp.v1i1.13091 – ident: ref38 doi: 10.1109/TKDE.2019.2951668 – ident: ref57 doi: 10.1007/s10115-019-01386-7 – ident: ref109 doi: 10.1007/978-3-642-40988-2_18 – ident: ref34 doi: 10.1007/s11432-020-3118-7 – ident: ref59 doi: 10.1145/3159652.3159654 – ident: ref66 doi: 10.24963/ijcai.2017/324 – volume-title: Proc. Int. Conf. Learn. Representation year: 2017 ident: ref94 article-title: Semi-supervised classification with graph convolution networks – volume: 28 start-page: 1621 volume-title: Advances Neural Inf. Process. Syst. year: 2015 ident: ref49 article-title: Max-margin majority voting for learning from crowds – ident: ref8 doi: 10.3115/1613715.1613751 – ident: ref67 doi: 10.1145/3077136.3080679 – ident: ref4 doi: 10.1093/bioinformatics/btt333 – ident: ref61 doi: 10.1609/hcomp.v3i1.13238 – ident: ref9 doi: 10.2307/2346806 – start-page: 2445 volume-title: Proc. 32nd Int. Conf. Mach. Learn. year: 2015 ident: ref98 article-title: PU learning for matrix completion – ident: ref45 doi: 10.1609/aaai.v28i2.19016 – ident: ref93 doi: 10.1109/TNNLS.2020.2978386 – start-page: 397 volume-title: Proc. 18th Int. Conf. Extending Database Tech. year: 2015 ident: ref76 article-title: Crowd-selection query processing in crowdsourcing databases: A task-driven approach – ident: ref86 doi: 10.1145/2431211.2431215 – ident: ref20 doi: 10.1609/aaai.v33i01.33019837 – ident: ref71 doi: 10.24963/ijcai.2020/402 – start-page: 433 volume-title: Proc. 31st Int. Conf. Mach. Learn. year: 2014 ident: ref106 article-title: Gaussian process classification and active learning with multiple annotators – ident: ref113 doi: 10.1609/aaai.v34i01.5392 – volume: 11 start-page: 1161 volume-title: Proc. 28th Int. Conf. Mach. Learn. year: 2011 ident: ref74 article-title: Active learning from crowds – ident: ref102 doi: 10.1109/ICCV.2013.373 – ident: ref80 doi: 10.24963/ijcai.2020/214 – ident: ref16 doi: 10.1145/2661829.2661946 – ident: ref88 doi: 10.1145/3394486.3403167 – ident: ref5 doi: 10.1145/3018661.3018688 – ident: ref46 doi: 10.1007/s10115-020-01475-y – ident: ref7 doi: 10.1145/1401890.1401965 – ident: ref62 doi: 10.1145/3290605.3300773 – ident: ref112 doi: 10.1609/aaai.v29i1.9761 – ident: ref95 doi: 10.1109/ICDM.2017.78 – ident: ref28 doi: 10.1007/978-3-319-18038-0_31 – ident: ref87 doi: 10.1609/aaai.v32i1.11513 – ident: ref114 doi: 10.1145/2484028.2484199 – start-page: 619 year: 2012 ident: ref26 article-title: Bayesian classifier combination publication-title: Artif. Intell. and Statistics – start-page: 262 volume-title: Proc. 31st Int. Conf. Mach. Learn. year: 2014 ident: ref33 article-title: Aggregating ordinal labels from crowds by minimax conditional entropy – volume: 25 start-page: 692 year: 2012 ident: ref23 article-title: Variational inference for crowdsourcing publication-title: Advances Neural Inf. Process. Syst. – ident: ref52 doi: 10.1109/TKDE.2015.2504974 – ident: ref83 doi: 10.1609/aaai.v27i1.8456 – volume: 29 start-page: 3567 volume-title: Advances Neural Inf. Process. Syst. year: 2016 ident: ref97 article-title: Data programming: Creating large training sets, quickly – ident: ref72 doi: 10.1609/aaai.v36i11.21684 – volume: 25 start-page: 2195 volume-title: Advances Neural Inf. Process. Syst. year: 2012 ident: ref29 article-title: Learning from the wisdom of crowds by minimax entropy – ident: ref36 doi: 10.1016/j.eswa.2014.03.048 – ident: ref107 doi: 10.1609/hcomp.v2i1.13165 – ident: ref117 doi: 10.1145/1656274.1656278 – ident: ref100 doi: 10.1109/TKDE.2016.2535242 – ident: ref85 doi: 10.1109/ICDM.2011.133 – ident: ref42 doi: 10.1145/2488388.2488414 – ident: ref17 doi: 10.24963/ijcai.2017/261 – ident: ref51 doi: 10.1109/TNNLS.2021.3082496 – start-page: 82 volume-title: Proc. 30th Conf. Uncertainty Artif. Intell. year: 2014 ident: ref14 article-title: Learning to predict from crowdsourced data – start-page: 2554 volume-title: Proc. 22th Int. Joint Conf. Artif. Intell. year: 2013 ident: ref65 article-title: Accurate integration of crowdsourced labels using workers self-reported confidence scores – ident: ref13 doi: 10.1609/aaai.v26i1.8105 – ident: ref47 doi: 10.14778/2735496.2735505 – volume: 22 start-page: 2035 year: 2009 ident: ref21 article-title: Whose vote should count more: Optimal integration of labels from labelers of unknown expertise publication-title: Advances Neural Inf. Process. Syst. – ident: ref22 doi: 10.1109/TKDE.2014.2327026 – ident: ref99 doi: 10.1145/3404835.3463081 – ident: ref70 doi: 10.1016/j.ins.2021.05.060 – ident: ref110 doi: 10.1609/aaai.v28i1.8993 – ident: ref27 doi: 10.1145/2187836.2187900 – ident: ref82 doi: 10.1609/hcomp.v3i1.13231 – ident: ref58 doi: 10.1609/aaai.v27i1.8530 – ident: ref118 doi: 10.1609/hcomp.v3i1.13256 – ident: ref115 doi: 10.1609/hcomp.v1i1.13088 – ident: ref18 doi: 10.1145/3148148 – ident: ref111 doi: 10.1145/2487575.2487708 – ident: ref68 doi: 10.1109/TNNLS.2017.2677468 – ident: ref30 doi: 10.1145/2566486.2567989 – ident: ref81 doi: 10.1109/CVPR.2013.81 – ident: ref60 doi: 10.1007/978-3-030-45442-5_26 – volume: 11 start-page: 1297 year: 2010 ident: ref12 article-title: Learning from crowds publication-title: J. Mach. Learn. Res. – ident: ref77 doi: 10.1007/978-3-030-58607-2_17 – ident: ref105 doi: 10.1609/aaai.v30i1.10315 – ident: ref90 doi: 10.1609/hcomp.v2i1.13167 – ident: ref108 doi: 10.1145/2487575.2487593 – start-page: 3886 volume-title: Int. Conf. Mach. Learn. year: 2019 ident: ref32 article-title: Exploiting worker correlation for label aggregation in crowdsourcing – ident: ref89 doi: 10.1016/j.comnet.2021.108227 – volume: 16 start-page: 2853 issue: 1 year: 2015 ident: ref116 article-title: CEKA: A tool for mining the wisdom of crowds publication-title: J. Mach. Learn. Res. – ident: ref40 doi: 10.1109/Allerton.2011.6120180 – ident: ref96 doi: 10.1109/ICDM.2012.64 – ident: ref104 doi: 10.1109/TNNLS.2020.2984729 – volume: 2 volume-title: Proc. NIPS 2nd Workshop on Comput. Social Science and the Wisdom of Crowds year: 2011 ident: ref63 article-title: Quality control of crowd labeling through expert evaluation – ident: ref15 doi: 10.1109/CVPRW.2010.5543189 – ident: ref43 doi: 10.1109/TKDE.2014.2327039 – start-page: 2435 volume-title: Proc. 25th Int. Joint Conf. Artif. Intell. year: 2016 ident: ref50 article-title: Crowdsourcing via tensor augmentation and completion – volume: 17 start-page: 3537 issue: 1 year: 2016 ident: ref31 article-title: Spectral methods meet EM: A provably optimal algorithm for crowdsourcing publication-title: J. Mach. Learn. Res. – ident: ref75 doi: 10.1007/s10994-013-5412-1 – volume: 14 start-page: 1 issue: 6 year: 2006 ident: ref1 article-title: The rise of crowdsourcing publication-title: Wired Magazine – ident: ref24 doi: 10.24963/ijcai.2021/238 – start-page: 242 volume-title: Proc. 32nd Conf. Uncertainty Artif. Intell. year: 2016 ident: ref53 article-title: Training deep neural nets to aggregate crowdsourced responses – ident: ref101 doi: 10.1109/TCYB.2014.2344674 – ident: ref55 doi: 10.1609/aaai.v32i1.11506 – ident: ref91 doi: 10.1109/TKDE.2018.2860992 – ident: ref41 doi: 10.1145/1993574.1993599 – ident: ref11 doi: 10.14778/3055540.3055547 – ident: ref10 doi: 10.1007/s10462-016-9491-9 – ident: ref54 doi: 10.24963/ijcai.2017/184 – volume: 30 start-page: 4355 volume-title: Advances Neural Inf. Process. Syst. year: 2017 ident: ref64 article-title: A minimax optimal algorithm for crowdsourcing – ident: ref92 doi: 10.1109/TKDE.2017.2754499 – ident: ref37 doi: 10.1145/3219819.3219958 – volume: abs/1905.13436 year: 2019 ident: ref78 article-title: Max-MIG: An information theoretic approach for joint learning from crowds publication-title: ArXiv – ident: ref3 doi: 10.1561/0600000071 – volume-title: Proc. 3rd Human Comput. Workshop year: 2011 ident: ref39 article-title: Improving consensus accuracy via Z-score and weighted voting – ident: ref44 doi: 10.1145/1557019.1557053 – ident: ref48 doi: 10.1145/2588555.2610509 – ident: ref79 doi: 10.1609/aaai.v34i04.5898 – ident: ref84 doi: 10.1145/1458082.1458165 |
| SSID | ssj0001257694 |
| Score | 2.4566057 |
| SecondaryResourceType | review_article |
| Snippet | Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity, and uncertainty, which lead the knowledge learning from them... |
| SourceID | wanfang proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 749 |
| SubjectTerms | Big Data Costs Crowdsourcing data fusion Data models Lead Learning learning from crowds learning paradigms learning with uncertainty Systematics Training Uncertainty |
| Title | Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective |
| URI | https://ieeexplore.ieee.org/document/9763477 https://www.proquest.com/docview/2655860126 https://d.wanfangdata.com.cn/periodical/zdhxb-ywb202205001 |
| Volume | 9 |
| WOSCitedRecordID | wos000794202700005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2329-9274 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001257694 issn: 2329-9266 databaseCode: RIE dateStart: 20140101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8NAEB5UFPTgW6wvFvTgwbR5bLpdb1UUUSiFKnoL-2wLkoptrfrrnWzSF-jBW0gmS5iZMN83uzMDcOYHwvixQJKjufVoIGNPYtT2NFcYfJSKNZVu2ARrNGovL7y5ABeTWhhjjDt8ZsrZpdvL1z01zFJlFQydEWVsERYZY3mt1kw-BZGzm3uIGIF7HANP0ckn8Hnlvt5CLhiG2VhbGtG5IOSmqswBzJWRSK1I2zOR5nbjf9-4CesFoiT13AW2YMGk27A202dwB5oP48wZKfqptslzd9Ah10jCtcvf461LUidXyJwtyTcMiEg1aU06PZPmtC5zF55ubx6v77xilIKnEDENPEsp94VgkbTMMF8zzqVQjJuAWUlrVnIjmLS6Kn1hlDAxl3EUq1AySiOjbbQHS2kvNftAmAgDKjXCIGRaYaRqwlhcQUhUr0J0V4LyWLeJKvqMZ-MuXhPHN3yeoDGSzBhJbowSnE9eeMtbbPwtupNpfCJWKLsER2PjJcUv2E_CahzXkG6G1RKcFgadPv3WnU-ZfI1k6EqN0X8Ofl_7EFYzkfyM4xEsDd6H5hiW1ceg238_cT74Azqe2c0 |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT-QwDLZ4CjiwD0AMsLuR4MBhC22aTCbcBgTiORoJVnCL8gQkVBAMz1-Pm3aGQdo97K1q3aiyXfn7nNgGWEsz7VOukeQ4GRKWGZ4YjNqJkxaDj7XcMROHTYhOp3VxIbsj8HtQC-O9j4fP_EZ5Gffy3a19LFNlmxg6cybEKIxzxmhWVWsNZVQQO8fJh4gSZCIx9NS9fLJUbh62T5ENUloOtmU5-xSG4lyVTxBz8lkXQReXQ7Fm78v_feVXmK0xJWlXTvANRnzxHWaGOg3OQfeonzsjdUfVS3J-3bsiO0jDXczg460t0ibbyJ0DqbYMiC4cOR30eibdj8rMefizt3u2s5_UwxQSi5iplwTGZKq1yE0QXqROSGm0FdJnIhjWCkZ6LUxwTZNqb7Xn0vCcW2oEY7l3IV-AseK28ItAhKYZMw6BEHItmtuW9gFX0AbVaxHfNWCjr1tl607j5cCLGxUZRyoVGkOVxlCVMRqwPnjhrmqy8W_RuVLjA7Fa2Q1Y6RtP1T_hg6JNzltIOGmzAau1QT-evrmrF6Nenw2NxcboP0t_X_sXTO2fnRyr44PO0TJMl-LViccVGOvdP_ofMGGfetcP9z-jP74DtWXdFA |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Knowledge+Learning+With+Crowdsourcing%3A+A+Brief+Review+and+Systematic+Perspective&rft.jtitle=IEEE%2FCAA+journal+of+automatica+sinica&rft.au=Zhang%2C+Jing&rft.date=2022-05-01&rft.issn=2329-9266&rft.eissn=2329-9274&rft.volume=9&rft.issue=5&rft.spage=749&rft.epage=762&rft_id=info:doi/10.1109%2FJAS.2022.105434&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_JAS_2022_105434 |
| thumbnail_s | http://cvtisr.summon.serialssolutions.com/2.0.0/image/custom?url=http%3A%2F%2Fwww.wanfangdata.com.cn%2Fimages%2FPeriodicalImages%2Fzdhxb-ywb%2Fzdhxb-ywb.jpg |