Splitwise: Efficient Generative LLM Inference Using Phase Splitting

Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory inten...

Celý popis

Podrobná bibliografia

Vydané v:	2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) s. 118 - 132
Hlavní autori:	Patel, Pratyush, Choukse, Esha, Zhang, Chaojie, Shah, Aashaka, Goiri, Inigo, Maleki, Saeed, Bianchini, Ricardo
Médium:	Konferenčný príspevok..
Jazyk:	English
Vydavateľské údaje:	IEEE 29.06.2024
Predmet:	Cluster deployments Computational modeling Computer architecture Costs GPUs Graphics processing units Inference efficiency Large language models Machine learning Processor scheduling Resource management Scheduling Throughput
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!