Harnessing the Power of Pre-Trained Models for Efficient Semantic Communication of Text and Images

This paper investigates point-to-point multimodal digital semantic communications in a task-oriented setup, where messages are classified at the receiver. We employ a pre-trained transformer model to extract semantic information and propose three methods for generating semantic codewords. First, we...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Entropy (Basel, Switzerland) Ročník 27; číslo 8; s. 813
Hlavní autoři: Kutay, Emrecan, Yener, Aylin
Médium: Journal Article
Jazyk:angličtina
Vydáno: Switzerland MDPI AG 29.07.2025
MDPI
Témata:
Accuracy
Analysis
Classification
Coding
Communication
Communications equipment
Distortion
Efficiency
Large language models
multimodal communication
Neural networks
Random variables
semantic communication
semantic compression
Semantics
Source code
task-oriented efficiency
Transition probabilities
semantic communication
6G
multimodal communication
semantic compression
task-oriented efficiency
ISSN:1099-4300, 1099-4300
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:This paper investigates point-to-point multimodal digital semantic communications in a task-oriented setup, where messages are classified at the receiver. We employ a pre-trained transformer model to extract semantic information and propose three methods for generating semantic codewords. First, we propose semantic quantization that uses quantized embeddings of source realizations as a codebook. We investigate the fixed-length coding, considering the source semantic structure and end-to-end semantic distortion. We propose a neural network-based codeword assignment mechanism incorporating codeword transition probabilities to minimize the expected semantic distortion. Second, we present semantic compression that clusters embeddings, exploiting the inherent semantic redundancies to reduce the codebook size, i.e., further compression. Third, we introduce a semantic vector-quantized autoencoder (VQ-AE) that learns a codebook through training. In all cases, we follow this semantic source code with a standard channel code to transmit over the wireless channel. In addition to classification accuracy, we assess pre-communication overhead via a novel metric we term system time efficiency. Extensive experiments demonstrate that our proposed semantic source-coding approaches provide comparable accuracy and better system time efficiency compared to their learning-based counterparts.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
This paper is an extended version of our paper published in Kutay, E.; Yener, A. Semantic Text Compression for Classification. In Proceedings of the 2023 IEEE International Conference on Communications Workshops (ICC Workshops), Rome, Italy, 28 May–1 June 2023; pp. 1368–1373. doi:10.1109/ICCWorkshops57953.2023.10283705; and Kutay, E.; Yener, A. Classification-Oriented Semantic Wireless Communications. In Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Korea, Republic of Korea, 14–19 April 2024; pp. 9096–9100. doi:10.1109/ICASSP48485.2024.10447549.
ISSN:1099-4300
1099-4300
DOI:10.3390/e27080813