Splitwise: Efficient Generative LLM Inference Using Phase Splitting

Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory inten...

Full description

Bibliographic Details

Published in:	2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) pp. 118 - 132
Main Authors:	Patel, Pratyush, Choukse, Esha, Zhang, Chaojie, Shah, Aashaka, Goiri, Inigo, Maleki, Saeed, Bianchini, Ricardo
Format:	Conference Proceeding
Language:	English
Published:	IEEE 29.06.2024
Subjects:	Cluster deployments Computational modeling Computer architecture Costs GPUs Graphics processing units Inference efficiency Large language models Machine learning Processor scheduling Resource management Scheduling Throughput
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!