Extracting Knowledge From Scientific Texts on Patient-Derived Cancer Models Using Large Language Models: Algorithm Development and Validation Study

Abstract BackgroundPatient-derived cancer models (PDCMs) have become essential tools in cancer research and preclinical studies. Consequently, the number of publications on PDCMs has increased significantly over the past decade. Advances in artificial intelligence, particularly in large language mod...

Full description

Saved in:
Bibliographic Details
Published in:JMIR bioinformatics and biotechnology Vol. 6; p. e70706
Main Authors: Yao, Jiarui, Perova, Zinaida, Mandloi, Tushar, Lewis, Elizabeth, Parkinson, Helen, Savova, Guergana
Format: Journal Article
Language:English
Published: Toronto, Canada JMIR Publications 30.06.2025
Subjects:
AI applications in genomics and pathology informatics, including ‘Generative AI’
Applications of AI
Artificial Intelligence
Cancer Prognosis Models and Machine Learning
Clinical Cancer Research
Original Paper
Theme Issue 2024: Artificial Intelligence in Oncology
Tools, Programs and Algorithms
ISSN:2563-3570, 2563-3570
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Abstract BackgroundPatient-derived cancer models (PDCMs) have become essential tools in cancer research and preclinical studies. Consequently, the number of publications on PDCMs has increased significantly over the past decade. Advances in artificial intelligence, particularly in large language models (LLMs), offer promising solutions for extracting knowledge from scientific literature at scale. ObjectiveThis study aims to investigate LLM-based systems, focusing specifically on prompting techniques for the automated extraction of PDCM-related entities from scientific texts. MethodsWe explore 2 LLM-prompting approaches. The classic method, direct prompting, involves manually designing a prompt. Our direct prompt consists of an instruction, entity-type definitions, gold examples, and a query. In addition, we experiment with a novel and underexplored prompting strategy—soft prompting. Unlike direct prompting, soft prompts are trainable continuous vectors that learn from provided data. We evaluate both prompting approaches across state-of-the-art proprietary and open LLMs. ResultsWe manually annotated 100 abstracts of PDCM-relevant papers, focusing on PDCM papers with data deposited in the CancerModels.Org platform. The resulting gold annotations span 15 entity types for a total 3313 entity mentions, which we split across training (2089 entities), development (542 entities) and held-out, eye-off test (682 entities) sets. Evaluation includes the standard metrics of precision or positive predictive value, recall or sensitivity, and F1F1F1 ConclusionsWe investigated LLM-prompting techniques for the automatic extraction of PDCM-relevant entities from scientific texts, comparing the traditional direct prompting approach with the emerging soft prompting method. In our experiments, GPT4-o demonstrated strong performance with direct prompting, maintaining competitive results. Meanwhile, soft prompting significantly enhanced the performance of smaller open LLMs. Our findings suggest that training soft prompts on smaller open models can achieve performance levels comparable to those of proprietary very large language models.
Bibliography:these authors contributed equally
ISSN:2563-3570
2563-3570
DOI:10.2196/70706