Empirical investigation of active learning strategies
Many predictive tasks require labeled data to induce classification models. The data labeling process may have a high cost. Several strategies have been proposed to optimize the selection of the most relevant examples, a process referred to as active learning. However, a lack of empirical studies co...
Saved in:
| Published in: | Neurocomputing (Amsterdam) Vol. 326-327; pp. 15 - 27 |
|---|---|
| Main Authors: | , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
Elsevier B.V
31.01.2019
|
| Subjects: | |
| ISSN: | 0925-2312, 1872-8286 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Many predictive tasks require labeled data to induce classification models. The data labeling process may have a high cost. Several strategies have been proposed to optimize the selection of the most relevant examples, a process referred to as active learning. However, a lack of empirical studies comparing different active learning approaches across multiple datasets makes it difficult identifying the most promising strategies, or even assessing the relative gain of active learning over the trivial random selection of instances. In this study, a comprehensive comparison of active learning strategies is presented, with various instance selection criteria, different classification algorithms and a large number of datasets. The experimental results confirm the effectiveness of active learning and provide insights about the relationship between classification algorithms and active learning strategies. Additionally, ranking curves with bands are introduced as a means to summarize in a single chart the performance of each active learning strategy for different classification algorithms and datasets. |
|---|---|
| AbstractList | Many predictive tasks require labeled data to induce classification models. The data labeling process may have a high cost. Several strategies have been proposed to optimize the selection of the most relevant examples, a process referred to as active learning. However, a lack of empirical studies comparing different active learning approaches across multiple datasets makes it difficult identifying the most promising strategies, or even assessing the relative gain of active learning over the trivial random selection of instances. In this study, a comprehensive comparison of active learning strategies is presented, with various instance selection criteria, different classification algorithms and a large number of datasets. The experimental results confirm the effectiveness of active learning and provide insights about the relationship between classification algorithms and active learning strategies. Additionally, ranking curves with bands are introduced as a means to summarize in a single chart the performance of each active learning strategy for different classification algorithms and datasets. |
| Author | de Carvalho, André C.P.L.F. Pereira-Santos, Davi Prudêncio, Ricardo Bastos Cavalcante |
| Author_xml | – sequence: 1 givenname: Davi orcidid: 0000-0002-3766-3137 surname: Pereira-Santos fullname: Pereira-Santos, Davi email: davips@icmc.usp.br, dpsabc@gmail.com organization: Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, Trabalhador São-carlense Av. 400, São Carlos, São Paulo 13560-970, Brazil – sequence: 2 givenname: Ricardo Bastos Cavalcante surname: Prudêncio fullname: Prudêncio, Ricardo Bastos Cavalcante email: rbcp@cin.ufpe.br organization: Centro de Informática, Universidade Federal de Pernambuco, Jornalista Aníbal Fernandes Av., Recife, Pernambuco 50740-560, Brazil – sequence: 3 givenname: André C.P.L.F. surname: de Carvalho fullname: de Carvalho, André C.P.L.F. email: andre@icmc.usp.br organization: Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, Trabalhador São-carlense Av. 400, São Carlos, São Paulo 13560-970, Brazil |
| BookMark | eNqFkEFLAzEQhYNUsK3-Aw_7B3ZNsptu4kGQUqtQ8KLnkE1my5RttiSx4L83tZ486GngzXyPN29GJn70QMgtoxWjbHG3qzx82HFfccraioqsigsyZbLlpeRyMSFTqrgoec34FZnFuKP5kHE1JWK1P2BAa4YC_RFiwq1JOPpi7AtjEx6hGMAEj35bxBRMgi1CvCaXvRki3PzMOXl_Wr0tn8vN6_pl-bgpbS14KqEFxVu-aKBTYJhTtu8c612nGmCUyt4JwykYC6pXTSeZy3vFJMhWslrZek7uz742jDEG6LXF9J0vR8FBM6pPBeidPhegTwVoKrIqMtz8gg8B9yZ8_oc9nDHIjx0Rgo4WwVtwGMAm7Ub82-ALKCl7TQ |
| CitedBy_id | crossref_primary_10_1016_j_ijmultiphaseflow_2025_105265 crossref_primary_10_1177_01655515231204802 crossref_primary_10_1016_j_addma_2021_102411 crossref_primary_10_1080_10494820_2023_2263043 crossref_primary_10_1111_coin_12217 crossref_primary_10_1016_j_neucom_2021_08_091 crossref_primary_10_1109_ACCESS_2021_3063002 crossref_primary_10_1021_acs_jchemed_4c00887 crossref_primary_10_1109_ACCESS_2022_3214544 crossref_primary_10_1371_journal_pone_0273725 crossref_primary_10_1109_TSE_2024_3376964 crossref_primary_10_3390_informatics7040050 crossref_primary_10_3233_IDA_226899 crossref_primary_10_3138_jvme_2022_0013 crossref_primary_10_1016_j_jocs_2023_101946 crossref_primary_10_1007_s00521_022_07812_2 crossref_primary_10_1080_0144929X_2022_2079560 crossref_primary_10_1038_s41524_022_00787_7 |
| Cites_doi | 10.1016/j.neucom.2013.01.051 10.1007/s10618-016-0469-7 10.1023/A:1010933404324 10.1109/JPROC.2012.2231951 10.1007/BF00993468 10.1016/j.tcs.2010.12.054 10.1007/BF00994018 10.1016/j.neucom.2015.03.056 10.1109/5254.708428 10.1016/j.neucom.2016.01.091 10.1016/j.neucom.2011.06.037 10.1145/219587.219592 10.1016/j.patcog.2011.08.009 10.1016/j.neucom.2013.01.008 10.1007/s10994-007-5019-5 10.1016/j.neucom.2014.05.075 10.1007/BF00993277 10.1145/1656274.1656278 10.1002/j.1538-7305.1949.tb00928.x 10.1109/TIT.1968.1054155 |
| ContentType | Journal Article |
| Copyright | 2017 Elsevier B.V. |
| Copyright_xml | – notice: 2017 Elsevier B.V. |
| DBID | AAYXX CITATION |
| DOI | 10.1016/j.neucom.2017.05.105 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 1872-8286 |
| EndPage | 27 |
| ExternalDocumentID | 10_1016_j_neucom_2017_05_105 S0925231217315357 |
| GroupedDBID | --- --K --M .DC .~1 0R~ 123 1B1 1~. 1~5 4.4 457 4G. 53G 5VS 7-5 71M 8P~ 9JM 9JN AABNK AACTN AADPK AAEDT AAEDW AAIAV AAIKJ AAKOC AALRI AAOAW AAQFI AAXLA AAXUO AAYFN ABBOA ABCQJ ABFNM ABJNI ABMAC ABYKQ ACDAQ ACGFS ACRLP ACZNC ADBBV ADEZE AEBSH AEKER AENEX AFKWA AFTJW AFXIZ AGHFR AGUBO AGWIK AGYEJ AHHHB AHZHX AIALX AIEXJ AIKHN AITUG AJOXV ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ AOUOD AXJTR BKOJK BLXMC CS3 DU5 EBS EFJIC EFLBG EJD EO8 EO9 EP2 EP3 F5P FDB FIRID FNPLU FYGXN G-Q GBLVA GBOLZ IHE J1W KOM LG9 M41 MO0 MOBAO N9A O-L O9- OAUVE OZT P-8 P-9 P2P PC. Q38 RIG ROL RPZ SDF SDG SDP SES SPC SPCBC SSN SSV SSZ T5K ZMT ~G- 29N 9DU AAQXK AATTM AAXKI AAYWO AAYXX ABWVN ABXDB ACLOT ACNNM ACRPL ACVFH ADCNI ADJOM ADMUD ADNMO AEIPS AEUPX AFJKZ AFPUW AGQPQ AIGII AIIUN AKBMS AKRWK AKYEP ANKPU APXCP ASPBG AVWKF AZFZN CITATION EFKBS FEDTE FGOYB HLZ HVGLF HZ~ R2- SBC SEW WUQ XPP ~HD |
| ID | FETCH-LOGICAL-c352t-e7e927264eb9ea1d9cfbd1fdb94e1008fd5a20eace9f94b81dfbd918e878139c3 |
| ISICitedReferencesCount | 33 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000452306700003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0925-2312 |
| IngestDate | Sat Nov 29 03:02:52 EST 2025 Tue Nov 18 22:38:10 EST 2025 Fri Feb 23 02:30:22 EST 2024 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Keywords | Partially labeled data Non-agnostic active learning Agnostic active learning Data labeling Active learning Data sampling |
| Language | English |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c352t-e7e927264eb9ea1d9cfbd1fdb94e1008fd5a20eace9f94b81dfbd918e878139c3 |
| ORCID | 0000-0002-3766-3137 |
| OpenAccessLink | http://dx.doi.org/10.1016/j.neucom.2017.05.105 |
| PageCount | 13 |
| ParticipantIDs | crossref_citationtrail_10_1016_j_neucom_2017_05_105 crossref_primary_10_1016_j_neucom_2017_05_105 elsevier_sciencedirect_doi_10_1016_j_neucom_2017_05_105 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-01-31 |
| PublicationDateYYYYMMDD | 2019-01-31 |
| PublicationDate_xml | – month: 01 year: 2019 text: 2019-01-31 day: 31 |
| PublicationDecade | 2010 |
| PublicationTitle | Neurocomputing (Amsterdam) |
| PublicationYear | 2019 |
| Publisher | Elsevier B.V |
| Publisher_xml | – name: Elsevier B.V |
| References | Abe, Mamitsuka (bib0037) 1998 Cherman (bib0040) 2013 Cohn, Atlas, Ladner (bib0025) 1994; 15 Settles (bib0012) 2010; 52 Xiong, Xie, Zhou, Guan (bib0004) 2014; 145 Guo, Schuurmans (bib0042) 2007 Freytag, Rodner, Denzler (bib0030) 2014 Zhao, Sukthankar, Sukthankar (bib0026) 2012 Kearns, Schapire, Sellie (bib0013) 1994; 17 Ross (bib0035) 2000 Hall, Frank, Holmes, Pfahringer, Reutemann, Witten (bib0051) 2009; 11 Maiora, Ayerdi, Graña (bib0005) 2014; 126 Settles, Craven (bib0008) 2008 Dasgupta (bib0016) 2005 Guo, Greiner (bib0020) 2007 Delgado, Cernadas, Barro, Amorim (bib0053) 2014; 15 Quinlan (bib0047) 1993 Zhang, Wang, Zhu, Yun, Wu, Wang (bib0029) 2015; 162 Lewis (bib0048) 1998; volume 1398 Miller (bib0055) 1968; volume 33 Mitchell (bib0024) 1997 Demsar (bib0043) 2006; 7 Bagaveyev, Cook (bib0023) 2014 Beygelzimer, Dasgupta, Langford (bib0027) 2009 Breiman (bib0050) 2001; 45 Ramirez-Loaiza, Sharma, Kumar, Bilgic (bib0010) 2017; 31 Raghavan, Madani, Jones (bib0041) 2006; 7 Feng, Xiao, Zha, Zhang, Yang (bib0003) 2012; 95 Hearst, Dumais, Osman, Platt, Scholkopf (bib0049) 1998; 13 dos Santos, de Carvalho (bib0014) 2015 Shannon (bib0032) 1949; 28 Fujii, Inui, Tokunaga, Tanaka (bib0034) 1998; 24 Bache, Lichman (bib0044) 2013 Batista, Campello, Sander (bib0021) 2016 Lewis (bib0031) 1995; 29 Körner, Wrobel (bib0009) 2006; volume 4212 Roy, McCallum (bib0019) 2001 Settles (bib0018) 2008 Nah (bib0054) 2003 Prieto, Ruiz, Hernández (bib0001) 2013; 121 Settles (bib0002) 2012 Lughofer (bib0036) 2012; 45 Santos, Carvalho (bib0017) 2014; volume 8480 Cortes, Vapnik (bib0052) 1995; 20 Settles, Craven, Ray (bib0033) 2007 Shah (bib0038) 2011; volume 6913 Hart (bib0046) 1968; 14 Beygelzimer, Hsu, Langford, Zhang (bib0022) 2016 Dasgupta (bib0015) 2011; 412 Evans, Adams, Anagnostopoulos (bib0011) 2013; volume 8207 Crawford, Tuia, Yang (bib0039) 2013; 101 Garcia (bib0045) 2015 Schein, Ungar (bib0007) 2007; 68 Ye, Liu, Liu, Tang, Zhao (bib0028) 2016; 196 Miller (10.1016/j.neucom.2017.05.105_bib0055) 1968; volume 33 Ross (10.1016/j.neucom.2017.05.105_bib0035) 2000 Settles (10.1016/j.neucom.2017.05.105_bib0012) 2010; 52 Batista (10.1016/j.neucom.2017.05.105_bib0021) 2016 dos Santos (10.1016/j.neucom.2017.05.105_bib0014) 2015 Breiman (10.1016/j.neucom.2017.05.105_bib0050) 2001; 45 Roy (10.1016/j.neucom.2017.05.105_bib0019) 2001 Shah (10.1016/j.neucom.2017.05.105_bib0038) 2011; volume 6913 Nah (10.1016/j.neucom.2017.05.105_bib0054) 2003 Zhao (10.1016/j.neucom.2017.05.105_bib0026) 2012 Garcia (10.1016/j.neucom.2017.05.105_sbref0045) 2015 Bache (10.1016/j.neucom.2017.05.105_bib0044) 2013 Quinlan (10.1016/j.neucom.2017.05.105_bib0047) 1993 Kearns (10.1016/j.neucom.2017.05.105_bib0013) 1994; 17 Guo (10.1016/j.neucom.2017.05.105_bib0020) 2007 Lewis (10.1016/j.neucom.2017.05.105_bib0031) 1995; 29 Hearst (10.1016/j.neucom.2017.05.105_bib0049) 1998; 13 Shannon (10.1016/j.neucom.2017.05.105_bib0032) 1949; 28 Freytag (10.1016/j.neucom.2017.05.105_bib0030) 2014 Guo (10.1016/j.neucom.2017.05.105_bib0042) 2007 Abe (10.1016/j.neucom.2017.05.105_bib0037) 1998 Evans (10.1016/j.neucom.2017.05.105_bib0011) 2013; volume 8207 Beygelzimer (10.1016/j.neucom.2017.05.105_bib0027) 2009 Xiong (10.1016/j.neucom.2017.05.105_bib0004) 2014; 145 Hall (10.1016/j.neucom.2017.05.105_bib0051) 2009; 11 Settles (10.1016/j.neucom.2017.05.105_bib0008) 2008 Schein (10.1016/j.neucom.2017.05.105_bib0007) 2007; 68 Lewis (10.1016/j.neucom.2017.05.105_bib0048) 1998; volume 1398 Settles (10.1016/j.neucom.2017.05.105_bib0033) 2007 Lughofer (10.1016/j.neucom.2017.05.105_bib0036) 2012; 45 Feng (10.1016/j.neucom.2017.05.105_bib0003) 2012; 95 Zhang (10.1016/j.neucom.2017.05.105_bib0029) 2015; 162 Körner (10.1016/j.neucom.2017.05.105_bib0009) 2006; volume 4212 Ramirez-Loaiza (10.1016/j.neucom.2017.05.105_bib0010) 2017; 31 Demsar (10.1016/j.neucom.2017.05.105_bib0043) 2006; 7 Ye (10.1016/j.neucom.2017.05.105_bib0028) 2016; 196 Santos (10.1016/j.neucom.2017.05.105_bib0017) 2014; volume 8480 Mitchell (10.1016/j.neucom.2017.05.105_bib0024) 1997 Dasgupta (10.1016/j.neucom.2017.05.105_bib0016) 2005 Crawford (10.1016/j.neucom.2017.05.105_bib0039) 2013; 101 Cortes (10.1016/j.neucom.2017.05.105_bib0052) 1995; 20 Delgado (10.1016/j.neucom.2017.05.105_bib0053) 2014; 15 Fujii (10.1016/j.neucom.2017.05.105_bib0034) 1998; 24 Prieto (10.1016/j.neucom.2017.05.105_bib0001) 2013; 121 Maiora (10.1016/j.neucom.2017.05.105_bib0005) 2014; 126 Bagaveyev (10.1016/j.neucom.2017.05.105_bib0023) 2014 Cherman (10.1016/j.neucom.2017.05.105_bib0040) 2013 Settles (10.1016/j.neucom.2017.05.105_bib0002) 2012 Beygelzimer (10.1016/j.neucom.2017.05.105_bib0022) 2016 Settles (10.1016/j.neucom.2017.05.105_bib0018) 2008 Raghavan (10.1016/j.neucom.2017.05.105_bib0041) 2006; 7 Dasgupta (10.1016/j.neucom.2017.05.105_bib0015) 2011; 412 Cohn (10.1016/j.neucom.2017.05.105_bib0025) 1994; 15 Hart (10.1016/j.neucom.2017.05.105_bib0046) 1968; 14 |
| References_xml | – year: 2008 ident: bib0018 publication-title: Curious Machines: Active Learning with Structured Instances – start-page: 823 year: 2007 end-page: 829 ident: bib0020 article-title: Optimistic active-learning using mutual information publication-title: Proceedings of International Joint Conference on Artificial Intelligence IJCAI – volume: 7 start-page: 1655 year: 2006 end-page: 1686 ident: bib0041 article-title: Active learning with feedback on features and instances publication-title: J. Mach. Learn. Res. – year: 1993 ident: bib0047 publication-title: C4.5: Programs for Machine Learning – volume: 68 start-page: 235 year: 2007 end-page: 265 ident: bib0007 article-title: Active learning for logistic regression: an evaluation publication-title: Mach. Learn. – year: 2000 ident: bib0035 publication-title: Introduction to Probability and Statistics for Engineers and Scientists (2nd ed.) – volume: 11 start-page: 10 year: 2009 end-page: 18 ident: bib0051 article-title: The WEKA data mining software: an update publication-title: SIGKDD Explor. – volume: 95 start-page: 54 year: 2012 end-page: 59 ident: bib0003 article-title: Active learning for social image retrieval using locally regressive optimal design publication-title: Neurocomputing – volume: 52 start-page: 55 year: 2010 end-page: 66 ident: bib0012 publication-title: Active Learning Literature Survey – volume: 20 start-page: 273 year: 1995 end-page: 297 ident: bib0052 article-title: Support-vector networks publication-title: Mach. Learn. – start-page: 1 year: 1998 end-page: 9 ident: bib0037 article-title: Query learning strategies using boosting and bagging publication-title: Proceedings of International Conference on Machine Learning, ICML – volume: 24 start-page: 573 year: 1998 end-page: 597 ident: bib0034 article-title: Selective sampling for example-based word sense disambiguation publication-title: Comput. Linguist. – volume: 121 start-page: 1 year: 2013 end-page: 4 ident: bib0001 article-title: Advances in artificial neural networks and machine learning publication-title: Neurocomputing – volume: 162 start-page: 163 year: 2015 end-page: 170 ident: bib0029 article-title: Update vs. upgrade: modeling with indeterminate multi-class active learning publication-title: Neurocomputing – year: 2007 ident: bib0033 article-title: Multiple-Instance Active Learning publication-title: NIPS – volume: volume 8207 start-page: 174 year: 2013 end-page: 185 ident: bib0011 article-title: When does active learning work? publication-title: Advances in Intelligent Data Analysis – start-page: 235 year: 2005 end-page: 242 ident: bib0016 article-title: Coarse sample complexity bounds for active learning publication-title: Proceedings of Advances in Neural Information Processing Systems – volume: 14 start-page: 515 year: 1968 end-page: 516 ident: bib0046 article-title: The condensed nearest neighbor rule (corresp.) publication-title: IEEE Trans. Inf. Theory – volume: volume 8480 start-page: 618 year: 2014 end-page: 629 ident: bib0017 article-title: Comparison of active learning strategies and proposal of a multiclass hypothesis space search publication-title: Proceedings of Hybrid Artificial Intelligent Systems HAIS – volume: volume 6913 start-page: 191 year: 2011 end-page: 206 ident: bib0038 article-title: Generalized agreement statistics over fixed group of experts publication-title: Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases ECMLPKDD (3) – volume: 45 start-page: 884 year: 2012 end-page: 896 ident: bib0036 article-title: Hybrid active learning for reducing the annotation effort of operators in classification systems publication-title: Pattern Recogn. – start-page: 469 year: 2014 end-page: 478 ident: bib0023 article-title: Designing and evaluating active learning methods for activity recognition publication-title: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication – volume: 7 start-page: 1 year: 2006 end-page: 30 ident: bib0043 article-title: Statistical comparisons of classifiers over multiple data sets publication-title: J. Mach. Learn. Res. – year: 2007 ident: bib0042 article-title: Discriminative Batch Mode Active Learning publication-title: NIPS – year: 2015 ident: bib0045 publication-title: A Huge Collection of Preprocessed ARFF Datasets for Supervised Classification Problems – volume: 15 start-page: 3133 year: 2014 end-page: 3181 ident: bib0053 article-title: Do we need hundreds of classifiers to solve real world classification problems? publication-title: J. Mach. Learn. Res. – volume: 196 start-page: 95 year: 2016 end-page: 106 ident: bib0028 article-title: Practice makes perfect: an adaptive active learning framework for image classification publication-title: Neurocomputing – year: 2012 ident: bib0002 publication-title: Active Learning – volume: 145 start-page: 44 year: 2014 end-page: 52 ident: bib0004 article-title: Active learning for protein function prediction in protein-protein interaction networks publication-title: Neurocomputing – start-page: 62 year: 2015 end-page: 67 ident: bib0014 article-title: Selectively inhibiting learning bias for active sampling publication-title: Proceedings of Brazilian Conference on Intelligent Systems – volume: 101 start-page: 593 year: 2013 end-page: 608 ident: bib0039 article-title: Active learning: any value for classification of remotely sensed data? publication-title: Proc. IEEE – volume: 28 start-page: 656 year: 1949 end-page: 715 ident: bib0032 article-title: Communication theory of secrecy systems publication-title: Bell Syst. Tech. J. – volume: volume 1398 start-page: 4 year: 1998 end-page: 15 ident: bib0048 article-title: Naive (bayes) at forty: the independence assumption in information retrieval publication-title: Proceedings of European Conference on Machine Learning ECML – volume: 17 start-page: 115 year: 1994 end-page: 141 ident: bib0013 article-title: Toward efficient agnostic learning publication-title: Mach. Learn. – start-page: 3476 year: 2012 end-page: 3479 ident: bib0026 article-title: Importance-weighted label prediction for active learning with noisy annotations publication-title: Proceedings of the 21st International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan, November 11–15, 2012 – start-page: 1070 year: 2008 end-page: 1079 ident: bib0008 article-title: An analysis of active learning strategies for sequence labeling tasks publication-title: Empirical Methods in Natural Language Processing – volume: volume 4212 start-page: 687 year: 2006 end-page: 694 ident: bib0009 article-title: Multi-class ensemble-based active learning publication-title: Proceedings of European Conference on Machine Learning – year: 1997 ident: bib0024 publication-title: Machine Learning – volume: 126 start-page: 71 year: 2014 end-page: 77 ident: bib0005 article-title: Random forest active learning for AAA thrombus segmentation in computed tomography angiography images publication-title: Neurocomputing – volume: 29 start-page: 13 year: 1995 end-page: 19 ident: bib0031 article-title: A sequential algorithm for training text classifiers: corrigendum and additional data publication-title: SIGIR Forum – volume: volume 33 start-page: 267 year: 1968 end-page: 277 ident: bib0055 article-title: Response time in man-computer conversational transactions publication-title: Proceedings of Fall Joint Computing Conference – volume: 15 start-page: 201 year: 1994 end-page: 221 ident: bib0025 article-title: Improving generalization with active learning publication-title: Mach. Learn. – start-page: 11 year: 2016 end-page: 20 ident: bib0021 article-title: Active semi-supervised classification based on multiple clustering hierarchies publication-title: Proceedings of IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2016 – year: 2013 ident: bib0044 article-title: UCI Repository of Machine Learning Databases, Machine-Readable Data Repository – year: 2013 ident: bib0040 publication-title: Aprendizado de máquina Multirrótulo: Explorando a Dependência de róTulos e o Aprendizado ativo – volume: 412 start-page: 1767 year: 2011 end-page: 1781 ident: bib0015 article-title: Two faces of active learning publication-title: Theor. Comput. Sci. – volume: 13 start-page: 18 year: 1998 end-page: 28 ident: bib0049 article-title: Support vector machines publication-title: Intell. Syst. Appl. IEEE – volume: 31 start-page: 287 year: 2017 end-page: 313 ident: bib0010 article-title: Active learning: an empirical study of common baselines publication-title: Data Min. Knowl. Discov. – start-page: 3342 year: 2016 end-page: 3350 ident: bib0022 article-title: Search improves label for active learning publication-title: Proceedings of Advances in Neural Information Processing Systems – start-page: 49 year: 2009 end-page: 56 ident: bib0027 article-title: Importance weighted active learning publication-title: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14–18, 2009 – start-page: 285 year: 2003 ident: bib0054 article-title: A study on tolerable waiting time: how long are web users willing to wait? publication-title: Proceedings of Americas Conference on Information Systems, AMCIS – start-page: 441 year: 2001 end-page: 448 ident: bib0019 article-title: Toward optimal active learning through sampling estimation of error reduction publication-title: ICML – volume: 45 start-page: 5 year: 2001 end-page: 32 ident: bib0050 article-title: Random forests publication-title: Mac. Learn. – start-page: 562 year: 2014 end-page: 577 ident: bib0030 article-title: Selecting influential examples: active learning with expected model output changes publication-title: Proceedings of European Conference on Computer Vision – start-page: 823 year: 2007 ident: 10.1016/j.neucom.2017.05.105_bib0020 article-title: Optimistic active-learning using mutual information – start-page: 3342 year: 2016 ident: 10.1016/j.neucom.2017.05.105_bib0022 article-title: Search improves label for active learning – start-page: 62 year: 2015 ident: 10.1016/j.neucom.2017.05.105_bib0014 article-title: Selectively inhibiting learning bias for active sampling – volume: 126 start-page: 71 year: 2014 ident: 10.1016/j.neucom.2017.05.105_bib0005 article-title: Random forest active learning for AAA thrombus segmentation in computed tomography angiography images publication-title: Neurocomputing doi: 10.1016/j.neucom.2013.01.051 – volume: 31 start-page: 287 issue: 2 year: 2017 ident: 10.1016/j.neucom.2017.05.105_bib0010 article-title: Active learning: an empirical study of common baselines publication-title: Data Min. Knowl. Discov. doi: 10.1007/s10618-016-0469-7 – start-page: 3476 year: 2012 ident: 10.1016/j.neucom.2017.05.105_bib0026 article-title: Importance-weighted label prediction for active learning with noisy annotations – volume: volume 4212 start-page: 687 year: 2006 ident: 10.1016/j.neucom.2017.05.105_bib0009 article-title: Multi-class ensemble-based active learning – start-page: 469 year: 2014 ident: 10.1016/j.neucom.2017.05.105_bib0023 article-title: Designing and evaluating active learning methods for activity recognition – year: 1993 ident: 10.1016/j.neucom.2017.05.105_bib0047 – year: 2000 ident: 10.1016/j.neucom.2017.05.105_bib0035 – year: 2007 ident: 10.1016/j.neucom.2017.05.105_bib0042 article-title: Discriminative Batch Mode Active Learning – start-page: 1070 year: 2008 ident: 10.1016/j.neucom.2017.05.105_bib0008 article-title: An analysis of active learning strategies for sequence labeling tasks – volume: volume 1398 start-page: 4 year: 1998 ident: 10.1016/j.neucom.2017.05.105_bib0048 article-title: Naive (bayes) at forty: the independence assumption in information retrieval – start-page: 49 year: 2009 ident: 10.1016/j.neucom.2017.05.105_bib0027 article-title: Importance weighted active learning – year: 2007 ident: 10.1016/j.neucom.2017.05.105_bib0033 article-title: Multiple-Instance Active Learning – volume: 45 start-page: 5 issue: 1 year: 2001 ident: 10.1016/j.neucom.2017.05.105_bib0050 article-title: Random forests publication-title: Mac. Learn. doi: 10.1023/A:1010933404324 – start-page: 285 year: 2003 ident: 10.1016/j.neucom.2017.05.105_bib0054 article-title: A study on tolerable waiting time: how long are web users willing to wait? – volume: 101 start-page: 593 issue: 3 year: 2013 ident: 10.1016/j.neucom.2017.05.105_bib0039 article-title: Active learning: any value for classification of remotely sensed data? publication-title: Proc. IEEE doi: 10.1109/JPROC.2012.2231951 – volume: 17 start-page: 115 issue: 2–3 year: 1994 ident: 10.1016/j.neucom.2017.05.105_bib0013 article-title: Toward efficient agnostic learning publication-title: Mach. Learn. doi: 10.1007/BF00993468 – volume: 412 start-page: 1767 issue: 19 year: 2011 ident: 10.1016/j.neucom.2017.05.105_bib0015 article-title: Two faces of active learning publication-title: Theor. Comput. Sci. doi: 10.1016/j.tcs.2010.12.054 – volume: 20 start-page: 273 issue: 3 year: 1995 ident: 10.1016/j.neucom.2017.05.105_bib0052 article-title: Support-vector networks publication-title: Mach. Learn. doi: 10.1007/BF00994018 – year: 1997 ident: 10.1016/j.neucom.2017.05.105_bib0024 – volume: 162 start-page: 163 year: 2015 ident: 10.1016/j.neucom.2017.05.105_bib0029 article-title: Update vs. upgrade: modeling with indeterminate multi-class active learning publication-title: Neurocomputing doi: 10.1016/j.neucom.2015.03.056 – volume: 13 start-page: 18 issue: 4 year: 1998 ident: 10.1016/j.neucom.2017.05.105_bib0049 article-title: Support vector machines publication-title: Intell. Syst. Appl. IEEE doi: 10.1109/5254.708428 – start-page: 1 year: 1998 ident: 10.1016/j.neucom.2017.05.105_bib0037 article-title: Query learning strategies using boosting and bagging – volume: 7 start-page: 1655 year: 2006 ident: 10.1016/j.neucom.2017.05.105_bib0041 article-title: Active learning with feedback on features and instances publication-title: J. Mach. Learn. Res. – volume: 196 start-page: 95 year: 2016 ident: 10.1016/j.neucom.2017.05.105_bib0028 article-title: Practice makes perfect: an adaptive active learning framework for image classification publication-title: Neurocomputing doi: 10.1016/j.neucom.2016.01.091 – volume: 95 start-page: 54 year: 2012 ident: 10.1016/j.neucom.2017.05.105_bib0003 article-title: Active learning for social image retrieval using locally regressive optimal design publication-title: Neurocomputing doi: 10.1016/j.neucom.2011.06.037 – volume: 15 start-page: 3133 issue: 1 year: 2014 ident: 10.1016/j.neucom.2017.05.105_bib0053 article-title: Do we need hundreds of classifiers to solve real world classification problems? publication-title: J. Mach. Learn. Res. – volume: volume 6913 start-page: 191 year: 2011 ident: 10.1016/j.neucom.2017.05.105_bib0038 article-title: Generalized agreement statistics over fixed group of experts – start-page: 11 year: 2016 ident: 10.1016/j.neucom.2017.05.105_bib0021 article-title: Active semi-supervised classification based on multiple clustering hierarchies – volume: 7 start-page: 1 year: 2006 ident: 10.1016/j.neucom.2017.05.105_bib0043 article-title: Statistical comparisons of classifiers over multiple data sets publication-title: J. Mach. Learn. Res. – year: 2013 ident: 10.1016/j.neucom.2017.05.105_bib0044 – start-page: 235 year: 2005 ident: 10.1016/j.neucom.2017.05.105_bib0016 article-title: Coarse sample complexity bounds for active learning – volume: 29 start-page: 13 issue: 2 year: 1995 ident: 10.1016/j.neucom.2017.05.105_bib0031 article-title: A sequential algorithm for training text classifiers: corrigendum and additional data publication-title: SIGIR Forum doi: 10.1145/219587.219592 – volume: volume 8480 start-page: 618 year: 2014 ident: 10.1016/j.neucom.2017.05.105_bib0017 article-title: Comparison of active learning strategies and proposal of a multiclass hypothesis space search – volume: 45 start-page: 884 issue: 2 year: 2012 ident: 10.1016/j.neucom.2017.05.105_bib0036 article-title: Hybrid active learning for reducing the annotation effort of operators in classification systems publication-title: Pattern Recogn. doi: 10.1016/j.patcog.2011.08.009 – start-page: 562 year: 2014 ident: 10.1016/j.neucom.2017.05.105_bib0030 article-title: Selecting influential examples: active learning with expected model output changes – volume: 121 start-page: 1 year: 2013 ident: 10.1016/j.neucom.2017.05.105_bib0001 article-title: Advances in artificial neural networks and machine learning publication-title: Neurocomputing doi: 10.1016/j.neucom.2013.01.008 – volume: 68 start-page: 235 issue: 3 year: 2007 ident: 10.1016/j.neucom.2017.05.105_bib0007 article-title: Active learning for logistic regression: an evaluation publication-title: Mach. Learn. doi: 10.1007/s10994-007-5019-5 – volume: 145 start-page: 44 year: 2014 ident: 10.1016/j.neucom.2017.05.105_bib0004 article-title: Active learning for protein function prediction in protein-protein interaction networks publication-title: Neurocomputing doi: 10.1016/j.neucom.2014.05.075 – volume: 15 start-page: 201 issue: 2 year: 1994 ident: 10.1016/j.neucom.2017.05.105_bib0025 article-title: Improving generalization with active learning publication-title: Mach. Learn. doi: 10.1007/BF00993277 – year: 2015 ident: 10.1016/j.neucom.2017.05.105_sbref0045 – year: 2012 ident: 10.1016/j.neucom.2017.05.105_bib0002 – start-page: 441 year: 2001 ident: 10.1016/j.neucom.2017.05.105_bib0019 article-title: Toward optimal active learning through sampling estimation of error reduction – volume: volume 8207 start-page: 174 year: 2013 ident: 10.1016/j.neucom.2017.05.105_bib0011 article-title: When does active learning work? – volume: 24 start-page: 573 issue: 4 year: 1998 ident: 10.1016/j.neucom.2017.05.105_bib0034 article-title: Selective sampling for example-based word sense disambiguation publication-title: Comput. Linguist. – volume: 11 start-page: 10 issue: 1 year: 2009 ident: 10.1016/j.neucom.2017.05.105_bib0051 article-title: The WEKA data mining software: an update publication-title: SIGKDD Explor. doi: 10.1145/1656274.1656278 – volume: 28 start-page: 656 issue: 4 year: 1949 ident: 10.1016/j.neucom.2017.05.105_bib0032 article-title: Communication theory of secrecy systems publication-title: Bell Syst. Tech. J. doi: 10.1002/j.1538-7305.1949.tb00928.x – year: 2008 ident: 10.1016/j.neucom.2017.05.105_bib0018 – volume: 14 start-page: 515 issue: 3 year: 1968 ident: 10.1016/j.neucom.2017.05.105_bib0046 article-title: The condensed nearest neighbor rule (corresp.) publication-title: IEEE Trans. Inf. Theory doi: 10.1109/TIT.1968.1054155 – volume: volume 33 start-page: 267 year: 1968 ident: 10.1016/j.neucom.2017.05.105_bib0055 article-title: Response time in man-computer conversational transactions – year: 2013 ident: 10.1016/j.neucom.2017.05.105_bib0040 – volume: 52 start-page: 55 year: 2010 ident: 10.1016/j.neucom.2017.05.105_bib0012 |
| SSID | ssj0017129 |
| Score | 2.45383 |
| Snippet | Many predictive tasks require labeled data to induce classification models. The data labeling process may have a high cost. Several strategies have been... |
| SourceID | crossref elsevier |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 15 |
| SubjectTerms | Active learning Agnostic active learning Data labeling Data sampling Non-agnostic active learning Partially labeled data |
| Title | Empirical investigation of active learning strategies |
| URI | https://dx.doi.org/10.1016/j.neucom.2017.05.105 |
| Volume | 326-327 |
| WOSCitedRecordID | wos000452306700003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVESC databaseName: Elsevier SD Freedom Collection Journals 2021 customDbUrl: eissn: 1872-8286 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0017129 issn: 0925-2312 databaseCode: AIEXJ dateStart: 19950101 isFulltext: true titleUrlDefault: https://www.sciencedirect.com providerName: Elsevier |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1ba9swFBZZuoe97D7a7oIf9hZkbMWypccspGyjlMK6kTcjSzJzae3gJKX_YH-7R5Zkp-vo1oe9mCBF8uV8HH06OheEPpaE6FRQjqlMGU50QrBQJcdpAXyXpFRHXSzMj-Ps5IQtl_x0NPrlY2GuLrK6ZtfXfPVfRQ1tIGwTOvsAcfeTQgP8BqHDFcQO138S_OJyVdm8H9WQQ8OyQtEpN18pwtgRfKKIXY7a5euQXbUHZ0eYXZp0Cspghw-6tNVVK_A3U4V47d3j-952q7oz-Fktq8ZF8AMYm8knAXRzPZkLeOdOrH6I0sb7BFp_Nt7T0p7iT-bhaXgcHoW7FgoTFBV71e5MjYRi4JG3tC7QRjy1aQGc7rRhnW4Vtj139Ls1NZyHtd4aZx-4W5d4NY7osJ75M_zflrne-dD7tZ3ndpbczJJHFFrpI7RHMsrZGO3NviyWX_sDqSwmNm2jexUfhdm5Ct59mj-znB3mcvYcPXVbjmBmofICjXT9Ej3z5TwCp91fIdojJ7iFnKApA4ucwCMnGJDzGn0_WpzNP2NXVQNLINsbrDPNSQY8WBdci1hxWRYqLlXBE20yPZWKChLBeqx5yZMC9jPQz2OmWcZguyCnb9C4bmq9j4IYmjTniUw1SyIhWVHGBSGqTGhBiyQ6QFP_FXLpUs6byicX-X0yOEC4H7WyKVf-8v_Mf-Dc0UZLB3NAzb0jDx94p7foyYDvd2i8abf6PXosrzbVuv3gIHMDebGP7g |
| linkProvider | Elsevier |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Empirical+investigation+of+active+learning+strategies&rft.jtitle=Neurocomputing+%28Amsterdam%29&rft.au=Pereira-Santos%2C+Davi&rft.au=Prud%C3%AAncio%2C+Ricardo+Bastos+Cavalcante&rft.au=de+Carvalho%2C+Andr%C3%A9+C.P.L.F.&rft.date=2019-01-31&rft.issn=0925-2312&rft.volume=326-327&rft.spage=15&rft.epage=27&rft_id=info:doi/10.1016%2Fj.neucom.2017.05.105&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_neucom_2017_05_105 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0925-2312&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0925-2312&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0925-2312&client=summon |