Distributed Semantic Communications for Multimodal Audio-Visual Parsing Tasks

Semantic communication has significantly improved in single-modal single-task scenarios, but its progress is limited in multimodal and multi-task transmission contexts. To address this issue, this paper investigates a distributed semantic communication system for audio-visual parsing (AVP) task. The...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on green communications and networking Vol. 8; no. 4; pp. 1707 - 1716
Main Authors: Wang, Penghong, Li, Jiahui, Liu, Chen, Fan, Xiaopeng, Ma, Mengyao, Wang, Yaowei
Format: Journal Article
Language:English
Published: Piscataway IEEE 01.12.2024
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Audio data
audio-visual parsing
auxiliary information feedback
Bandwidth
Bandwidths
Communication
Communications systems
Consumption
deep joint source-channel coding
Distributed semantic communication
Feature extraction
Noise measurement
Performance evaluation
Semantics
Servers
Task analysis
Visual tasks
Visualization
ISSN:2473-2400, 2473-2400
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Semantic communication has significantly improved in single-modal single-task scenarios, but its progress is limited in multimodal and multi-task transmission contexts. To address this issue, this paper investigates a distributed semantic communication system for audio-visual parsing (AVP) task. The system acquires audio-visual information from distributed terminals and conducts multi-task analysis on the far-end server, which involves event categorization and boundary recording. We propose a distributed deep joint source-channel coding scheme with auxiliary information feedback to implement this system, aiming to enhance parsing performance and reduce bandwidth consumption during communication. Specifically, the server initially receives the audio feature from the audio terminal and then sends the semantic information extracted from the audio feature back to the visual terminal. The received semantic and visual information are interactively processed by the visual terminal before being encoded and transmitted. The audio and visual semantic information received is processed and parsed on the far-end server. The experimental results demonstrate a significant reduction in transmission bandwidth consumption and notable performance improvements across various evaluation metrics for distributed AVP task compared to current state-of-the-art methods.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2473-2400
2473-2400
DOI:10.1109/TGCN.2024.3374700