Dual-hierarchical knowledge distillation for video captioning

•Introduce a dual-hierarchical distillation framework for video captioning.•Propose a caption quality grading method to reduce annotation noise.•Distill semantic features into a lightweight student model for efficiency.•Improve captioning accuracy while lowering computational cost.•Outperform prior...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Pattern recognition Jg. 171; S. 112192
Hauptverfasser: Luo, HuiLan, Wan, SiQi, Cai, Xia, Wang, ChanJuan, Chen, HongKun
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Elsevier Ltd 01.03.2026
Schlagworte:
Encoder-decoder architecture
Hierarchical semantic representation
Knowledge distillation
Teacher-student network
Video captioning
Video captioning
Encoder-decoder architecture
Knowledge distillation
Teacher-student network
Hierarchical semantic representation
ISSN:0031-3203
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:•Introduce a dual-hierarchical distillation framework for video captioning.•Propose a caption quality grading method to reduce annotation noise.•Distill semantic features into a lightweight student model for efficiency.•Improve captioning accuracy while lowering computational cost.•Outperform prior methods on MSVD and MSR-VTT benchmarks. Video captioning remains a challenging task due to the tradeoff between accuracy and computational efficiency. We propose CapDistill, a dual hierarchical distillation framework that transfers semantic knowledge from a powerful teacher model to a lightweight student model. CapDistill captures object-level and action-level semantics from captions and transfers multilevel knowledge including object features, action features, and word-level predictions through a hierarchical strategy. To reduce the impact of noisy annotations, we introduce a caption quality grading mechanism that assigns quality-based weights to training captions. Experiments on MSR-VTT and MSVD demonstrate that CapDistill achieves state-of-the-art accuracy while significantly reducing inference cost. Code is available at: https://github.com/ccc000-png/SFTCap.
ISSN:0031-3203
DOI:10.1016/j.patcog.2025.112192