ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding

Recent advancements in generative large language models (LLMs) have significantly boosted the performance in natural language processing tasks. However, their efficiency is constrained by the inherent limitations in autoregressive token generation. While parallel decoding with token tree verificatio...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Digest of technical papers - IEEE/ACM International Conference on Computer-Aided Design S. 1 - 8
Hauptverfasser: Zhong, Shuzhang, Yang, Zebin, Gong, Ruihao, Wang, Runsheng, Huang, Ru, Li, Meng
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: ACM 27.10.2024
Schlagworte:
Batch production systems
Computational efficiency
Decoding
Design automation
Heuristic algorithms
Large language models
Natural language processing
Parallel processing
Real-time systems
ISSN:1558-2434
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Recent advancements in generative large language models (LLMs) have significantly boosted the performance in natural language processing tasks. However, their efficiency is constrained by the inherent limitations in autoregressive token generation. While parallel decoding with token tree verification, e.g., Medusa, has been proposed to improve decoding parallelism and efficiency, it often struggles with maintaining contextual relationships due to its independent token prediction approach and incurs significant verification overhead, especially with large tree sizes and batch processing. In this paper, we propose ProPD, an efficient LLM parallel decoding framework based on dynamic token tree pruning and generation. ProPD features an advanced early pruning mechanism to efficiently eliminate unpromising token sequences to improve verification efficiency. Additionally, it introduces a dynamic token tree generation algorithm to balance the computation and parallelism of the verification phase in real-time and maximize the overall efficiency across different batch sizes, sequence lengths, and tasks, etc. We verify ProPD across a diverse set of datasets, LLMs, and batch sizes and demonstrate ProPD consistently outperforms existing decoding algorithms by 1.1-3.2 ×.
ISSN:1558-2434
DOI:10.1145/3676536.3676695