MILLION: MasterIng Long-Context LLM Inference Via Outlier-Immunized KV Product QuaNtization

Large language models (LLMs) are increasingly utilized for complex tasks requiring longer context lengths, with some models supporting up to 128 K or 1 M tokens. This trend, however, presents significant challenges in inference speed and memory management. The primary bottleneck in long-context LLM...

Ausführliche Beschreibung

Bibliographische Detailangaben

Veröffentlicht in:	2025 62nd ACM/IEEE Design Automation Conference (DAC) S. 1 - 7
Hauptverfasser:	Wang, Zongwu, Xu, Peng, Liu, Fangxin, Hu, Yiwei, Sun, Qingxiao, Li, Gezi, Li, Cheng, Wang, Xuan, Jiang, Li, Guan, Haibing
Format:	Tagungsbericht
Sprache:	Englisch
Veröffentlicht:	IEEE 22.06.2025
Schlagworte:	Accuracy Inference algorithms Kernel Large language models Low latency communication Memory management Performance gain Pipelines Quantization (signal)
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!