Splitwise: Efficient Generative LLM Inference Using Phase Splitting

Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory inten...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) S. 118 - 132
Hauptverfasser:	Patel, Pratyush, Choukse, Esha, Zhang, Chaojie, Shah, Aashaka, Goiri, Inigo, Maleki, Saeed, Bianchini, Ricardo
Format:	Tagungsbericht
Sprache:	Englisch
Veröffentlicht:	IEEE 29.06.2024
Schlagworte:	Cluster deployments Computational modeling Computer architecture Costs GPUs Graphics processing units Inference efficiency Large language models Machine learning Processor scheduling Resource management Scheduling Throughput
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!