ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding

Recent advancements in generative large language models (LLMs) have significantly boosted the performance in natural language processing tasks. However, their efficiency is constrained by the inherent limitations in autoregressive token generation. While parallel decoding with token tree verificatio...

Full description

Saved in:
Bibliographic Details
Published in:Digest of technical papers - IEEE/ACM International Conference on Computer-Aided Design pp. 1 - 8
Main Authors: Zhong, Shuzhang, Yang, Zebin, Gong, Ruihao, Wang, Runsheng, Huang, Ru, Li, Meng
Format: Conference Proceeding
Language:English
Published: ACM 27.10.2024
Subjects:
Batch production systems
Computational efficiency
Decoding
Design automation
Heuristic algorithms
Large language models
Natural language processing
Parallel processing
Real-time systems
ISSN:1558-2434
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Recent advancements in generative large language models (LLMs) have significantly boosted the performance in natural language processing tasks. However, their efficiency is constrained by the inherent limitations in autoregressive token generation. While parallel decoding with token tree verification, e.g., Medusa, has been proposed to improve decoding parallelism and efficiency, it often struggles with maintaining contextual relationships due to its independent token prediction approach and incurs significant verification overhead, especially with large tree sizes and batch processing. In this paper, we propose ProPD, an efficient LLM parallel decoding framework based on dynamic token tree pruning and generation. ProPD features an advanced early pruning mechanism to efficiently eliminate unpromising token sequences to improve verification efficiency. Additionally, it introduces a dynamic token tree generation algorithm to balance the computation and parallelism of the verification phase in real-time and maximize the overall efficiency across different batch sizes, sequence lengths, and tasks, etc. We verify ProPD across a diverse set of datasets, LLMs, and batch sizes and demonstrate ProPD consistently outperforms existing decoding algorithms by 1.1-3.2 ×.
ISSN:1558-2434
DOI:10.1145/3676536.3676695