Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: The case of drug–drug interactions

Purpose To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug–drug interactions (DDIs). Methods Starting from a large and diversified reference set for two‐way DDIs...

Full description

Saved in:

Bibliographic Details
Published in:	Pharmacoepidemiology and drug safety Vol. 32; no. 8; pp. 832 - 844
Main Authors:	Kontsioti, Elpida, Maskell, Simon, Pirmohamed, Munir
Format:	Journal Article
Language:	English
Published:	Chichester, UK John Wiley & Sons, Inc 01.08.2023 Wiley Subscription Services, Inc
Subjects:	Adverse Drug Reaction Reporting Systems Adverse events Algorithms Business metrics Databases, Factual Drug development Drug interaction Drug Interactions Drug-Related Side Effects and Adverse Reactions - diagnosis Drug-Related Side Effects and Adverse Reactions - epidemiology drug–drug interactions Humans Original performance metrics Pharmacovigilance postmarketing surveillance signal detection spontaneous reports data United States United States Food and Drug Administration United States postmarketing surveillance drug-drug interactions signal detection performance metrics spontaneous reports data adverse events pharmacovigilance
ISSN:	1053-8569, 1099-1557, 1099-1557
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Purpose To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug–drug interactions (DDIs). Methods Starting from a large and diversified reference set for two‐way DDIs, we generated custom‐made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data. Results For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates. Conclusions The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by “favoring” some algorithms while penalizing others.
AbstractList	PurposeTo evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug–drug interactions (DDIs).MethodsStarting from a large and diversified reference set for two‐way DDIs, we generated custom‐made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data.ResultsFor some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates.ConclusionsThe relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by “favoring” some algorithms while penalizing others. To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs). Starting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data. For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates. The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favoring" some algorithms while penalizing others. To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs).PURPOSETo evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs).Starting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data.METHODSStarting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data.For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates.RESULTSFor some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates.The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favoring" some algorithms while penalizing others.CONCLUSIONSThe relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favoring" some algorithms while penalizing others. Purpose To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug–drug interactions (DDIs). Methods Starting from a large and diversified reference set for two‐way DDIs, we generated custom‐made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data. Results For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates. Conclusions The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by “favoring” some algorithms while penalizing others.
Author	Kontsioti, Elpida Pirmohamed, Munir Maskell, Simon
AuthorAffiliation	1 Department of Electrical Engineering and Electronics University of Liverpool Liverpool UK 2 The Wolfson Center for Personalized Medicine, Center for Drug Safety Science, Department of Pharmacology and Therapeutics Institute of Systems, Molecular and Integrative Biology, University of Liverpool Liverpool UK
AuthorAffiliation_xml	– name: 2 The Wolfson Center for Personalized Medicine, Center for Drug Safety Science, Department of Pharmacology and Therapeutics Institute of Systems, Molecular and Integrative Biology, University of Liverpool Liverpool UK – name: 1 Department of Electrical Engineering and Electronics University of Liverpool Liverpool UK
Author_xml	– sequence: 1 givenname: Elpida orcidid: 0000-0002-4053-0220 surname: Kontsioti fullname: Kontsioti, Elpida email: e.kontsioti@liverpool.ac.uk organization: University of Liverpool – sequence: 2 givenname: Simon orcidid: 0000-0003-1917-2913 surname: Maskell fullname: Maskell, Simon organization: University of Liverpool – sequence: 3 givenname: Munir orcidid: 0000-0002-7534-7266 surname: Pirmohamed fullname: Pirmohamed, Munir organization: Institute of Systems, Molecular and Integrative Biology, University of Liverpool
BackLink	https://www.ncbi.nlm.nih.gov/pubmed/36916014$$D View this record in MEDLINE/PubMed
BookMark	eNp1kstu1TAQhiNURC8g8QTIEhs2Odhx7MRsECrlIlUCibK2fJxJjivHDrZT6I53YMXr8SQ4aSlQwcrWzDf_P_bMYbHnvIOieEjwhmBcPZ26uGEcizvFAcFClISxZm-5M1q2jIv94jDGc4xzTtT3in3KBeGY1AfF95Mvk_XBuAGlHSAzTkon5HvUQTSDQzqYBMEo1PuAAvQQwGlAEVJE3qEJQk6MaonBhbKzSiaHc_1SrWyWSaDXmLJD9km7MT5DZ9lKqwirUZiHH1-_LQcyLpuplY_3i7u9shEeXJ9HxcdXJ2fHb8rTd6_fHr84LTXDTJSaVtBpzDgj2y00RIgt7biuVYfrBvO-oowTQbWomwY0qEq3Tdf3XHQMKiVqelQ8v9Kd5u2YpcCloKycghlVuJReGfl3xpmdHPyFzD9dN1UjssKTa4XgP80QkxxN1GCtcuDnKKum5S2hnC3o41vouZ9D_qhMtZS2PE-RZerRny3d9PJrbL8ddfAx5rHcIATLZSNk3gi5bERGN7dQbdI6pfwYY_9VUF4VfDYWLv8rLN-__LDyPwE4cMtY
CitedBy_id	crossref_primary_10_1002_pds_5731 crossref_primary_10_3390_ph18071062 crossref_primary_10_1007_s43441_024_00729_z crossref_primary_10_1002_pds_5722
Cites_doi	10.1002/pds.4275 10.2165/11534410-000000000-00000 10.1126/scitranslmed.3003377 10.1002/pds.4970 10.1007/s40264‐016‐0409‐x 10.1007/s40264‐013‐0053‐7 10.1093/oxfordjournals.aje.a009758 10.1002/sim 10.1111/j.1365‐2125.2007.02900.x 10.1007/s40264‐015‐0388‐3 10.1016/j.ijmedinf.2005.04.006 10.1007/s40264‐012‐0002‐x 10.1038/sdata.2014.43 10.1002/pds.1742 10.1007/s40264‐014‐0198‐z 10.1038/clpt.2013.24.Performance 10.1007/s40264‐014‐0187‐2 10.1007/S40264‐016‐0392‐2/TABLES/7 10.1007/s40264‐014‐0246‐8 10.1016/j.ijmedinf.2009.01.001 10.1007/s40264‐013‐0097‐8 10.1136/amiajnl‐2013‐001612 10.2165/11586990-000000000-00000 10.1007/s40264‐014‐0245‐9 10.1007/s40264‐016‐0405‐1 10.1038/s41597-022-01159-y 10.1038/sdata.2016.26 10.1007/s40264‐020‐00939‐y 10.1007/s40264‐014‐0189‐0 10.2165/00002018‐200831110‐00008 10.1002/pds.885 10.1186/1471‐2105‐11‐S9‐S7 10.1007/s40264‐017‐0618‐y 10.2165/00002018-200528010-00002 10.1101/068692 10.2165/00002018‐200932060‐00007
ContentType	Journal Article
Copyright	2023 The Authors. published by John Wiley & Sons Ltd. 2023 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons Ltd. 2023. This article is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml	– notice: 2023 The Authors. published by John Wiley & Sons Ltd. – notice: 2023 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons Ltd. – notice: 2023. This article is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID	24P AAYXX CITATION CGR CUY CVF ECM EIF NPM 7TK K9. 7X8 5PM
DOI	10.1002/pds.5609
DatabaseName	Wiley Online Library Open Access CrossRef Medline MEDLINE MEDLINE (Ovid) MEDLINE MEDLINE PubMed Neurosciences Abstracts ProQuest Health & Medical Complete (Alumni) MEDLINE - Academic PubMed Central (Full Participant titles)
DatabaseTitle	CrossRef MEDLINE Medline Complete MEDLINE with Full Text PubMed MEDLINE (Ovid) ProQuest Health & Medical Complete (Alumni) Neurosciences Abstracts MEDLINE - Academic
DatabaseTitleList	ProQuest Health & Medical Complete (Alumni) MEDLINE MEDLINE - Academic
Database_xml	– sequence: 1 dbid: 24P name: Wiley Online Library Open Access url: https://authorservices.wiley.com/open-science/open-access/browse-journals.html sourceTypes: Publisher – sequence: 2 dbid: NPM name: PubMed url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database – sequence: 3 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database
DeliveryMethod	fulltext_linktorsrc
Discipline	Pharmacy, Therapeutics, & Pharmacology
DocumentTitleAlternate	Kontsioti et al
EISSN	1099-1557
EndPage	844
ExternalDocumentID	PMC10947279 36916014 10_1002_pds_5609 PDS5609
Genre	researchArticle Research Support, Non-U.S. Gov't Journal Article
GeographicLocations	United States
GeographicLocations_xml	– name: United States
GrantInformation_xml	– fundername: AstraZeneca – fundername: Engineering and Physical Sciences Research Council
GroupedDBID	--- .3N .GA .Y3 05W 0R~ 10A 123 1L6 1OB 1OC 1ZS 24P 31~ 33P 3SF 3WU 4.4 50Y 50Z 51W 51X 52M 52N 52O 52P 52R 52S 52T 52U 52V 52W 52X 53G 5VS 66C 702 7PT 8-0 8-1 8-3 8-4 8-5 8UM 930 A01 A03 AAESR AAEVG AAHHS AAHQN AAIPD AAMNL AANHP AANLZ AAONW AASGY AAXRX AAYCA AAZKR ABCQN ABCUV ABEML ABIJN ABJNI ABPVW ABQWH ABXGK ACAHQ ACBWZ ACCFJ ACCZN ACFBH ACGFO ACGFS ACGOF ACMXC ACPOU ACPRK ACRPL ACSCC ACXBN ACXQS ACYXJ ADBBV ADBTR ADEOM ADIZJ ADKYN ADMGS ADNMO ADOZA ADXAS ADZMN AEEZP AEGXH AEIGN AEIMD AENEX AEQDE AEUQT AEUYR AFBPY AFFPM AFGKR AFPWT AFRAH AFWVQ AFZJQ AHBTC AHMBA AIACR AIAGR AITYG AIURR AIWBW AJBDE ALAGY ALMA_UNASSIGNED_HOLDINGS ALUQN ALVPJ AMBMR AMYDB ASPBG ATUGU AVWKF AZBYB AZFZN AZVAB BAFTC BDRZF BFHJK BHBCM BMXJE BROTX BRXPI BY8 C45 CS3 D-6 D-7 D-E D-F DCZOG DPXWK DR2 DRFUL DRMAN DRSTM DU5 EBD EBS EJD EMOBN F00 F01 F04 F5P FEDTE FUBAC G-S G.N GNP GODZA GWYGA H.X HF~ HGLYW HHZ HVGLF HZ~ IX1 J0M JPC KBYEO KQQ LATKE LAW LC2 LC3 LEEKS LH4 LITHE LOXES LP6 LP7 LSO LUTES LW6 LYRES M6Q MEWTI MK4 MRFUL MRMAN MRSTM MSFUL MSMAN MSSTM MXFUL MXMAN MXSTM N04 N05 N9A NF~ NNB O66 O9- OIG OVD P2P P2W P2X P2Z P4B P4D PALCI PQQKQ Q.N Q11 QB0 QRW R.K RIWAO RJQFR ROL RWI RX1 RYL SAMSI SUPJJ SV3 TEORI UB1 V8K W8V W99 WBKPD WHWMO WIB WIH WIJ WIK WJL WOHZO WQJ WRC WUP WVDHM WWP WXI WXSBR XG1 XV2 YCJ ZZTAW ~IA ~WT AAMMB AAYXX AEFGJ AEYWJ AGHNM AGQPQ AGXDD AGYGG AIDQK AIDYY AIQQE CITATION O8X CGR CUY CVF ECM EIF NPM 7TK K9. 7X8 5PM
ID	FETCH-LOGICAL-c5059-c32edc05651bbe7199b3d6c4ad04706f2356193c9477ecea2c87dff69d5e2a943
IEDL.DBID	24P
ISICitedReferencesCount	4
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000958729600001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1053-8569 1099-1557
IngestDate	Tue Nov 04 02:06:11 EST 2025 Thu Oct 02 09:46:06 EDT 2025 Sat Nov 29 14:49:22 EST 2025 Mon Jul 21 06:02:04 EDT 2025 Tue Nov 18 22:11:25 EST 2025 Sat Nov 29 06:35:21 EST 2025 Wed Jan 22 16:20:07 EST 2025
IsDoiOpenAccess	true
IsOpenAccess	true
IsPeerReviewed	true
IsScholarly	true
Issue	8
Keywords	postmarketing surveillance drug-drug interactions signal detection performance metrics spontaneous reports data adverse events pharmacovigilance
Language	English
License	Attribution 2023 The Authors. Pharmacoepidemiology and Drug Safety published by John Wiley & Sons Ltd. This is an open access article under the terms of the http://creativecommons.org/licenses/by/4.0/ License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c5059-c32edc05651bbe7199b3d6c4ad04706f2356193c9477ecea2c87dff69d5e2a943
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 content type line 23
ORCID	0000-0003-1917-2913 0000-0002-7534-7266 0000-0002-4053-0220
OpenAccessLink	https://onlinelibrary.wiley.com/doi/abs/10.1002%2Fpds.5609
PMID	36916014
PQID	2833865605
PQPubID	105383
PageCount	13
ParticipantIDs	pubmedcentral_primary_oai_pubmedcentral_nih_gov_10947279 proquest_miscellaneous_2786813659 proquest_journals_2833865605 pubmed_primary_36916014 crossref_primary_10_1002_pds_5609 crossref_citationtrail_10_1002_pds_5609 wiley_primary_10_1002_pds_5609_PDS5609
PublicationCentury	2000
PublicationDate	August 2023
PublicationDateYYYYMMDD	2023-08-01
PublicationDate_xml	– month: 08 year: 2023 text: August 2023
PublicationDecade	2020
PublicationPlace	Chichester, UK
PublicationPlace_xml	– name: Chichester, UK – name: England – name: Bethesda
PublicationTitle	Pharmacoepidemiology and drug safety
PublicationTitleAlternate	Pharmacoepidemiol Drug Saf
PublicationYear	2023
Publisher	John Wiley & Sons, Inc Wiley Subscription Services, Inc
Publisher_xml	– name: John Wiley & Sons, Inc – name: Wiley Subscription Services, Inc
References	2010; 11 2015; 38 2015; 33 2013; 93 2018; 41 2011; 34 2008; 31 2012; 35 2016; 39 2005; 28 1999; 149 2014; 23 2014; 21 2003; 12 2009; 78 2014; 1 2013; 36 2009; 32 2016; 3 2022; 9 2008; 27 2005; 74 2014; 37 2018 2020; 43 2007; 64 2012; 4 2009; 18 2020; 29 e_1_2_12_4_1 e_1_2_12_3_1 e_1_2_12_5_1 e_1_2_12_19_1 e_1_2_12_18_1 e_1_2_12_2_1 e_1_2_12_17_1 e_1_2_12_16_1 e_1_2_12_38_1 e_1_2_12_39_1 e_1_2_12_20_1 e_1_2_12_21_1 e_1_2_12_22_1 e_1_2_12_23_1 e_1_2_12_24_1 e_1_2_12_25_1 e_1_2_12_26_1 Hill R (e_1_2_12_10_1) 2012; 35 e_1_2_12_27_1 e_1_2_12_28_1 Catalog of Bias Collaboration (e_1_2_12_6_1) 2018 e_1_2_12_29_1 e_1_2_12_30_1 e_1_2_12_31_1 e_1_2_12_32_1 e_1_2_12_33_1 e_1_2_12_34_1 e_1_2_12_35_1 e_1_2_12_36_1 e_1_2_12_37_1 e_1_2_12_15_1 e_1_2_12_14_1 e_1_2_12_13_1 e_1_2_12_12_1 e_1_2_12_8_1 e_1_2_12_11_1 e_1_2_12_7_1 e_1_2_12_9_1 38013249 - Pharmacoepidemiol Drug Saf. 2024 Jan;33(1):e5731. doi: 10.1002/pds.5731. 37946583 - Pharmacoepidemiol Drug Saf. 2024 Jan;33(1):e5722. doi: 10.1002/pds.5722.
References_xml	– volume: 38 start-page: 113 year: 2015 end-page: 114 article-title: Comment on: “Zoo or Savannah? choice of training ground for evidence‐based pharmacovigilance publication-title: Drug Saf – volume: 41 start-page: 377 year: 2018 end-page: 387 article-title: An automated system combining safety signal detection and prioritization from healthcare databases: a pilot study publication-title: Drug Saf – volume: 35 start-page: 1194 year: 2012 end-page: 1195 article-title: An attempt to expedite signal detection by grouping related adverse reaction terms publication-title: Drug Saf – volume: 39 start-page: 355 year: 2016 end-page: 364 article-title: Performance of stratified and subgrouped disproportionality analyzes in spontaneous databases publication-title: Drug Saf – volume: 28 start-page: 19 year: 2005 end-page: 34 article-title: Appraisal of the MedDRA conceptual structure for describing and grouping adverse drug reactions publication-title: Drug Saf – volume: 64 start-page: 489 year: 2007 end-page: 495 article-title: Detecting signals of drug‐drug interactions in a spontaneous reports database publication-title: Br J Clin Pharmacol – volume: 93 start-page: 539 year: 2013 end-page: 546 article-title: Performance of pharmacovigilance signal detection algorithms for the FDA adverse event reporting system publication-title: Clin Pharmacol Ther – volume: 27 start-page: 3057 year: 2008 end-page: 3070 article-title: A statistical methodology for drug–drug interaction surveillance publication-title: Stat Med – volume: 38 start-page: 115 year: 2015 end-page: 116 article-title: Authors' reply to harpaz comment on: “zoo or savannah? choice of training ground for evidence‐based pharmacovigilance publication-title: Drug Saf – volume: 43 start-page: 775 year: 2020 end-page: 785 article-title: A feasibility study of drug–drug interaction signal detection in regular pharmacovigilance publication-title: Drug Saf – volume: 39 start-page: 469 year: 2016 end-page: 490 article-title: Good signal detection practices: evidence from IMI PROTECT publication-title: Drug Saf – volume: 3 start-page: 3 year: 2016 article-title: Data descriptor: a curated and standardized adverse drug event resource to accelerate drug safety research publication-title: Sci Data – volume: 39 start-page: 421 year: 2016 end-page: 432 article-title: Evidence of misclassification of drug–event associations classified as gold standard ‘negative controls’ by the observational medical outcomes partnership (OMOP) publication-title: Drug Saf – volume: 36 start-page: S36 issue: suppl 1 year: 2013 end-page: S47 article-title: Defining a reference set to support methodological research in drug safety publication-title: Drug Saf – volume: 74 start-page: 563 year: 2005 end-page: 571 article-title: Implementation of automated signal generation in pharmacovigilance using a knowledge‐based approach publication-title: Int J Med Inform – year: 2018 – volume: 23 start-page: 294 year: 2014 end-page: 295 article-title: Evaluation of statistical measures for adverse drug interaction surveillance publication-title: Pharmacoepidemiol Drug Saf – volume: 36 start-page: 371 year: 2013 end-page: 388 article-title: The development and evaluation of triage algorithms for early discovery of adverse drug interactions publication-title: Drug Saf – volume: 4 year: 2012 article-title: Data‐driven prediction of drug effects and interactions publication-title: Sci Transl Med – volume: 12 start-page: 517 year: 2003 end-page: 521 article-title: Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post‐marketing setting publication-title: Pharmacoepidemiol Drug Saf – volume: 36 start-page: 13 year: 2013 end-page: 23 article-title: A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases publication-title: Drug Saf – volume: 21 start-page: 353 year: 2014 end-page: 362 article-title: Mining clinical text for signals of adverse drug‐drug interactions publication-title: J Am Med Inform Assoc – volume: 31 start-page: 1035 year: 2008 end-page: 1048 article-title: Impact of stratification on adverse drug reaction surveillance publication-title: Drug Saf – volume: 78 start-page: 97 year: 2009 end-page: 103 article-title: Influence of the MedDRA® hierarchy on pharmacovigilance data mining results publication-title: Int J Med Inform – volume: 9 start-page: 72 year: 2022 article-title: A reference set of clinically relevant adverse drug‐drug interactions publication-title: Sci Data – volume: 37 start-page: 537 year: 2014 end-page: 542 article-title: Performance of the standardised MedDRA® queries for case retrieval in the French spontaneous reporting database publication-title: Drug Saf – volume: 37 start-page: 655 year: 2014 end-page: 659 article-title: Zoo or Savannah? Choice of training ground for evidence‐based pharmacovigilance publication-title: Drug Saf – volume: 1 year: 2014 article-title: A time‐indexed reference standard of adverse drug reactions publication-title: Sci Data – volume: 29 start-page: 396 year: 2020 end-page: 403 article-title: Adverse drug reaction or innocent bystander? A systematic comparison of statistical discovery methods for spontaneous reporting systems publication-title: Pharmacoepidemiol Drug Saf – volume: 32 start-page: 509 year: 2009 end-page: 525 article-title: An evaluation of three signal‐detection algorithms using a highly inclusive reference event database publication-title: Drug Saf – volume: 18 start-page: 427 year: 2009 end-page: 436 article-title: Quantitative signal detection using spontaneous ADR reporting publication-title: Pharmacoepidemiol Drug Saf – volume: 37 start-page: 557 year: 2014 end-page: 567 article-title: Bridging islands of information to establish an integrated knowledge base of drugs and health outcomes of interest publication-title: Drug Saf – volume: 33 start-page: 475 year: 2015 end-page: 487 article-title: Validation of statistical signal detection procedures in eudravigilance post‐authorization data a retrospective evaluation of the potential for earlier signalling publication-title: Drug Saf – volume: 39 start-page: 561 year: 2016 end-page: 575 article-title: A pharmacovigilance signaling system based on FDA regulatory action and post‐marketing adverse event reports publication-title: Drug Saf – volume: 11 start-page: S7 year: 2010 article-title: Mining multi‐item drug adverse effect associations in spontaneous reporting systems publication-title: BMC Bioinform – volume: 34 year: 2011 article-title: Reporting patterns indicative of adverse drug interactions: a systematic evaluation in VigiBase publication-title: Drug Saf – volume: 149 start-page: 981 year: 1999 end-page: 983 article-title: Confounding by indication: an example of variation in the use of epidemiologic terminology publication-title: Am J Epidemiol – ident: e_1_2_12_20_1 doi: 10.1002/pds.4275 – ident: e_1_2_12_36_1 doi: 10.2165/11534410-000000000-00000 – ident: e_1_2_12_3_1 doi: 10.1126/scitranslmed.3003377 – volume: 35 start-page: 1194 year: 2012 ident: e_1_2_12_10_1 article-title: An attempt to expedite signal detection by grouping related adverse reaction terms publication-title: Drug Saf – ident: e_1_2_12_4_1 doi: 10.1002/pds.4970 – ident: e_1_2_12_24_1 doi: 10.1007/s40264‐016‐0409‐x – ident: e_1_2_12_38_1 doi: 10.1007/s40264‐013‐0053‐7 – ident: e_1_2_12_7_1 doi: 10.1093/oxfordjournals.aje.a009758 – ident: e_1_2_12_33_1 doi: 10.1002/sim – ident: e_1_2_12_34_1 doi: 10.1111/j.1365‐2125.2007.02900.x – ident: e_1_2_12_26_1 doi: 10.1007/s40264‐015‐0388‐3 – ident: e_1_2_12_8_1 doi: 10.1016/j.ijmedinf.2005.04.006 – ident: e_1_2_12_27_1 doi: 10.1007/s40264‐012‐0002‐x – ident: e_1_2_12_32_1 doi: 10.1038/sdata.2014.43 – ident: e_1_2_12_2_1 doi: 10.1002/pds.1742 – ident: e_1_2_12_15_1 doi: 10.1007/s40264‐014‐0198‐z – ident: e_1_2_12_19_1 doi: 10.1038/clpt.2013.24.Performance – ident: e_1_2_12_12_1 doi: 10.1007/s40264‐014‐0187‐2 – ident: e_1_2_12_17_1 doi: 10.1007/S40264‐016‐0392‐2/TABLES/7 – volume-title: Confounding by Indication year: 2018 ident: e_1_2_12_6_1 – ident: e_1_2_12_35_1 doi: 10.1007/s40264‐014‐0246‐8 – ident: e_1_2_12_9_1 doi: 10.1016/j.ijmedinf.2009.01.001 – ident: e_1_2_12_23_1 doi: 10.1007/s40264‐013‐0097‐8 – ident: e_1_2_12_29_1 doi: 10.1136/amiajnl‐2013‐001612 – ident: e_1_2_12_21_1 doi: 10.2165/11586990-000000000-00000 – ident: e_1_2_12_16_1 doi: 10.1007/s40264‐014‐0245‐9 – ident: e_1_2_12_13_1 doi: 10.1007/s40264‐016‐0405‐1 – ident: e_1_2_12_31_1 doi: 10.1038/s41597-022-01159-y – ident: e_1_2_12_30_1 doi: 10.1038/sdata.2016.26 – ident: e_1_2_12_39_1 doi: 10.1007/s40264‐020‐00939‐y – ident: e_1_2_12_14_1 doi: 10.1007/s40264‐014‐0189‐0 – ident: e_1_2_12_5_1 doi: 10.2165/00002018‐200831110‐00008 – ident: e_1_2_12_18_1 doi: 10.1002/pds.885 – ident: e_1_2_12_28_1 doi: 10.1186/1471‐2105‐11‐S9‐S7 – ident: e_1_2_12_25_1 doi: 10.1007/s40264‐017‐0618‐y – ident: e_1_2_12_11_1 doi: 10.2165/00002018-200528010-00002 – ident: e_1_2_12_37_1 doi: 10.1101/068692 – ident: e_1_2_12_22_1 doi: 10.2165/00002018‐200932060‐00007 – reference: 38013249 - Pharmacoepidemiol Drug Saf. 2024 Jan;33(1):e5731. doi: 10.1002/pds.5731. – reference: 37946583 - Pharmacoepidemiol Drug Saf. 2024 Jan;33(1):e5722. doi: 10.1002/pds.5722.
SSID	ssj0009994
Score	2.3956957
Snippet	Purpose To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance... To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal... PurposeTo evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal...
SourceID	pubmedcentral proquest pubmed crossref wiley
SourceType	Open Access Repository Aggregation Database Index Database Enrichment Source Publisher
StartPage	832
SubjectTerms	Adverse Drug Reaction Reporting Systems Adverse events Algorithms Business metrics Databases, Factual Drug development Drug interaction Drug Interactions Drug-Related Side Effects and Adverse Reactions - diagnosis Drug-Related Side Effects and Adverse Reactions - epidemiology drug–drug interactions Humans Original performance metrics Pharmacovigilance postmarketing surveillance signal detection spontaneous reports data United States United States Food and Drug Administration
Title	Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: The case of drug–drug interactions
URI	https://onlinelibrary.wiley.com/doi/abs/10.1002%2Fpds.5609 https://www.ncbi.nlm.nih.gov/pubmed/36916014 https://www.proquest.com/docview/2833865605 https://www.proquest.com/docview/2786813659 https://pubmed.ncbi.nlm.nih.gov/PMC10947279
Volume	32
WOSCitedRecordID	wos000958729600001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVWIB databaseName: Wiley Online Library Full Collection 2020 customDbUrl: eissn: 1099-1557 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0009994 issn: 1053-8569 databaseCode: DRFUL dateStart: 19960101 isFulltext: true titleUrlDefault: https://onlinelibrary.wiley.com providerName: Wiley-Blackwell
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV3NbtQwELag5cCFf8qWUg0SWi4NXcdJbHNDlBUHVEXQSnuLHMdpK5XsapNF6q3vwInX40mYibMJq4KExCWJlPGP1jOez9mZbxh7laOTlgl9qOJOB5FxZWDK2ATKcps4ogsxvtiEPD5Ws5lOu6hKyoXx_BD9BzeyjHa_JgM3eX04kIYuivoNumt9m21zLiRpdBilA-GubosgInwQgYoTvSaenYSH65abrugGvrwZJvk7fG39z_T-_8z8AbvXoU5459XkIbvlqkdsnHra6qsDOBmysOoDGEM6EFpfPWY_-kA9QLgIPrES5iUUbfgH4MZDjM8GEABDX7cEatfUMK9gMeQmwEAuTu2pNU6rcE0bEFaBuTzDcZrzr_VbmhNY9LHtQMvV2c_r73QDIrhY-nSM-gk7nX44ef8x6Eo6BBahlg6sCPH3RNAV8zx3kmudiyKxkSkmkZwkZSgQz2lhdSSls86EVsmiLBNdxC40OhJP2VY1r9wzBrGwMndOFdbwyHF8wH1b5WoSOpmr2I7Y6_XqZrbjO6eyG5eZZ2oOM1yHjNZhxF72kgvP8fEHmb21gmSdldcZQjMqmYonQuyif432SX-6mMrNVygjVaIolhC72PH61A8iEgTneEYdMbWhab0AcX9vvqkuzlsOcI7HcoSe2Om4VbW_TjxLj77QffdfBZ-zuyECOR_kuMe2muXKvWB37Lfmol7ut0aGVzlT-2z76PP09NMv4YIz9A
linkProvider	Wiley-Blackwell
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1Lb9NAEB6VFAkulDeBAoOEwqVu4-fulhOiREWEKIJU6s1ar9dtpeJEsYPUG_-BE3-vv6SzXj-IChISJ0fy7EPZmZ1v1zPfALxOyEmzyFxUuVo4gdSZI7NQOly5KtKGLkTaYhNsMuHHx2K6AW-bXBjLD9FeuBnLqPZrY-DmQnqvYw1dpMUu-WtxAzYD0qKwB5sHX0ZH445zV1R1EAlB-A4PI9Fwzw69vabtuje6BjGvR0r-jmArFzTa-q_J34U7NfLEd1ZV7sGGzu_DYGqpqy92cNZlYhU7OMBpR2p98QB-tcF6SJARbXIlzjNMqxAQpM3HsD5LJBCMbe0SLHRZ4DzHRZefgB3BuGlvWtO0Ul1WQWE5yvMTGqc8_VbsmzmhIj9bDbRcnVz--GkeaEguljYlo3gIR6MPs_eHTl3WwVEEt4SjfI_-UAJeoZskmrlCJH4aqUCmw4ANo8zzCdMJX4mAMa209BRnaZZFIg21J0XgP4JePs_1E8DQVyzRmqdKuoF26Qft3TzhQ0-zhIeqD2-a5Y1VzXluSm-cx5at2YtpHWKzDn141UouLM_HH2S2Gw2Ja0svYoJnpmwqnQqpi_Y12aj58CJzPV-RDOMRN_GE1MVjq1DtIH5EAJ3OqX3ga6rWChj-7_U3-dlpxQPu0tGc4Cd1Oqh07a8Tj6cHX83z6b8KvoRbh7PP43j8cfLpGdz2CNjZoMdt6JXLlX4ON9X38qxYvqht7gqXhDbp
linkToPdf	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV1Lb9QwELZKQYgL78dCgUFCy6VpN0_bcEKUFQi0ikSReosc22krlexqk0Xqjf_Aib_HL2HG3iSsChISp0TK-KF4xvM5mfmGseclOmme0Yeq0MogUbYKVJWqQOhQZ5boQpQvNsFnM3F0JPMt9qrLhfH8EP0HN7IMt1-TgduFqfYH1tCFafbQX8tL7HKS4hZLtM5JPjDuSlcFEfFDHIg0kx3z7CTa71pu-qILAPNinOTv-NU5oOmN_5r6TXZ9jTvhtVeUW2zL1rfZOPfE1ee7cDjkYTW7MIZ8oLQ-v8N-9KF6gIARfGolzCswLgAEcOshzmcFCIGhr1wCjW0bmNewGLITYKAXp_bUGqdlbOtCwmpQZ8c4TnvypXlJcwKNXtYNtFwd__z2nS5AFBdLn5DR3GWfp28P37wL1kUdAo1gSwY6jvCFIuxKw7K0PJSyjE2mE2UmCZ9kVRQjopOxlgnnVlsVacFNVWXSpDZSMonvse16XtsHDNJY89JaYbQKExviDe7cohSTyPJSpHrEXnTLW-g14zkV3jgrPFdzVOA6FLQOI_asl1x4lo8_yOx0GlKs7bwpEJxR0VQ8E2IX_WO0UPrtomo7X6EMF5mgaELs4r5XqH6QOEN4jqfUERMbqtYLEPv35pP69MSxgId4MEfwiZ2Ona79deJFfvCJrg__VfApu5ofTIuP72cfHrFrEaI6H_G4w7bb5co-Zlf01_a0WT5xBvcLb7M00g
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+the+impact+of+design+criteria+for+reference+sets+on+performance+evaluation+of+signal+detection+algorithms%3A+The+case+of+drug%E2%80%93drug+interactions&rft.jtitle=Pharmacoepidemiology+and+drug+safety&rft.au=Kontsioti%2C+Elpida&rft.au=Maskell%2C+Simon&rft.au=Pirmohamed%2C+Munir&rft.date=2023-08-01&rft.pub=Wiley+Subscription+Services%2C+Inc&rft.issn=1053-8569&rft.eissn=1099-1557&rft.volume=32&rft.issue=8&rft.spage=832&rft.epage=844&rft_id=info:doi/10.1002%2Fpds.5609&rft.externalDBID=NO_FULL_TEXT
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1053-8569&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1053-8569&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1053-8569&client=summon