BL-JUNIPER: A CNN-Assisted Framework for Perceptual Video Coding Leveraging Block-Level JND

Just Noticeable Distortion (JND) finds the minimum distortion level perceivable by humans. This can be a natural solution for setting the compression for each video region in perceptual video coding. However, existing JND-based solutions estimate JND levels for each video frame and ignore the fact t...

Celý popis

Uložené v:

Podrobná bibliografia
Vydané v:	IEEE transactions on multimedia Ročník 25; s. 5077 - 5092
Hlavní autori:	Nami, Sanaz, Pakdaman, Farhad, Hashemi, Mahmoud Reza, Shirmohammadi, Shervin
Médium:	Journal Article
Jazyk:	English
Vydavateľské údaje:	Piscataway IEEE 2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:	Algorithms Bit rate Coding Control algorithms Control theory Convolutional neural network (cnn) Distortion Encoding just noticeable distortion (jnd) Measurement perceptual video coding (pvc) Predictive models quantization control algorithm Robustness (mathematics) Streaming media Video coding Video compression visual attention Visualization
ISSN:	1520-9210, 1941-0077
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!

Abstract	Just Noticeable Distortion (JND) finds the minimum distortion level perceivable by humans. This can be a natural solution for setting the compression for each video region in perceptual video coding. However, existing JND-based solutions estimate JND levels for each video frame and ignore the fact that different video regions have different perceptual importance. To address this issue, we propose a Block-Level Just Noticeable Distortion-based Perceptual (BL-JUNIPER) framework for video coding. The proposed four-stage framework combines different perceptual information to further improve the prediction accuracy. The JND mapping in the first stage derives block-level JNDs from frame-level information without the need to collect a new bock-level JND dataset. In the second stage, an efficient CNN-based model is proposed to predict JND levels for each block according to spatial and temporal characteristics. Unlike existing methods, BL-JUNIPER works on raw video frames and avoids re-encoding each frame several times, making it computationally practical. Third, the visual importance of each block is measured using a visual attention model. Finally, a proposed quantization control algorithm uses both JND levels and visual importance to adjust the Quantization Parameter (QP) for each block. The specific algorithm for each stage of the proposed framework can be changed, as long as the input and output formats of each block are followed, without the need to change other stages, based on any current or future methods, providing a flexible and robust solution. Extensive experimental results demonstrate that BL-JUNIPER achieves a mean bitrate reduction of 27.75% with a Delta Mean Opinion Score (DMOS) close to zero and BD-Rate gains of 25.44% based on MOS, compared to the baseline encoding, and also gains a better performance compared to competing methods.
AbstractList	Just Noticeable Distortion (JND) finds the minimum distortion level perceivable by humans. This can be a natural solution for setting the compression for each video region in perceptual video coding. However, existing JND-based solutions estimate JND levels for each video frame and ignore the fact that different video regions have different perceptual importance. To address this issue, we propose a Block-Level Just Noticeable Distortion-based Perceptual (BL-JUNIPER) framework for video coding. The proposed four-stage framework combines different perceptual information to further improve the prediction accuracy. The JND mapping in the first stage derives block-level JNDs from frame-level information without the need to collect a new bock-level JND dataset. In the second stage, an efficient CNN-based model is proposed to predict JND levels for each block according to spatial and temporal characteristics. Unlike existing methods, BL-JUNIPER works on raw video frames and avoids re-encoding each frame several times, making it computationally practical. Third, the visual importance of each block is measured using a visual attention model. Finally, a proposed quantization control algorithm uses both JND levels and visual importance to adjust the Quantization Parameter (QP) for each block. The specific algorithm for each stage of the proposed framework can be changed, as long as the input and output formats of each block are followed, without the need to change other stages, based on any current or future methods, providing a flexible and robust solution. Extensive experimental results demonstrate that BL-JUNIPER achieves a mean bitrate reduction of 27.75% with a Delta Mean Opinion Score (DMOS) close to zero and BD-Rate gains of 25.44% based on MOS, compared to the baseline encoding, and also gains a better performance compared to competing methods.
Author	Hashemi, Mahmoud Reza Shirmohammadi, Shervin Nami, Sanaz Pakdaman, Farhad
Author_xml	– sequence: 1 givenname: Sanaz orcidid: 0000-0002-4826-1168 surname: Nami fullname: Nami, Sanaz email: snami@ut.ac.ir organization: School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran – sequence: 2 givenname: Farhad orcidid: 0000-0001-6526-3811 surname: Pakdaman fullname: Pakdaman, Farhad email: farhad.pakdaman@umz.ac.ir organization: Faculty of Engineering and Technology, University of Mazandaran, Babolsar, Iran – sequence: 3 givenname: Mahmoud Reza orcidid: 0000-0002-3518-9195 surname: Hashemi fullname: Hashemi, Mahmoud Reza email: rhashemi@ut.ac.ir organization: School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran – sequence: 4 givenname: Shervin orcidid: 0000-0002-3973-4445 surname: Shirmohammadi fullname: Shirmohammadi, Shervin email: shervin@eecs.uottawa.ca organization: DISCOVER Laboratory, School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON, Canada
BookMark	eNp9kDtPwzAUhS0EEuWxI7FYYk65dh622UpogSotCLUsDJHr3qC0IS52CuLfk6iIgYHpHl2dh_Qdkf3a1kjIGYM-Y6AuZ5NJnwPn_ZBJwWO1R3pMRSwAEGK_1TGHQHEGh-TI-xUAi2IQPfJynQXj-fT-cfh0RQc0nU6Dgfelb3BJR06_4ad1a1pYRx_RGdw0W13R53KJlqZ2WdavNMMPdPq1k9eVNeuge1R0PL05IQeFrjye_txjMh8NZ-ldkD3c3qeDLDBhLJrA8MhoFbKowFDrREUcQmNAcC3QaKaUiFFIBcsYWcJlseASFwp4ESZJggbDY3Kx6904-75F3-Qru3V1O5lzKSPZdcvWBTuXcdZ7h0W-ceWbdl85g7xDmLcI8w5h_oOwjSR_IqZsdFPaunG6rP4Lnu-CJSL-7ijJoIUefgOLc311
CODEN	ITMUF8
CitedBy_id	crossref_primary_10_1109_TCSVT_2024_3402363 crossref_primary_10_1145_3571727 crossref_primary_10_1016_j_engappai_2024_109806 crossref_primary_10_1109_JETCAS_2024_3524260 crossref_primary_10_3390_electronics13244977 crossref_primary_10_3390_s23052634 crossref_primary_10_1109_TMM_2023_3340882 crossref_primary_10_1109_TCE_2025_3526479 crossref_primary_10_1109_TCSVT_2024_3389988
Cites_doi	10.1109/ACCESS.2019.2901342 10.1109/TIP.2017.2685682 10.1145/3304112.3325604 10.1007/s11045-016-0395-2 10.1109/TCSVT.2016.2539862 10.1109/TBC.2020.2977542 10.1109/ICIP.2019.8803454 10.1109/TIP.2013.2279934 10.1109/ACCESS.2021.3050489 10.1109/TIP.2018.2818439 10.1109/ICCV.2013.370 10.1109/TIP.2019.2933743 10.1109/JSTSP.2020.3034501 10.2352/ISSN.2470-1173.2016.13.IQSP-222 10.1109/TIP.2003.819861 10.1145/3394171.3413536 10.1109/ICMEW46912.2020.9106036 10.1109/CVPR.2018.00068 10.1109/ICIP.2016.7532610 10.1016/j.jvcir.2017.04.009 10.1109/TMM.2013.2268053 10.1109/TIP.2020.3029428 10.1109/ICIP40778.2020.9190983 10.1109/TIP.2016.2568459 10.1109/LSP.2016.2641456 10.1109/ACCESS.2019.2910245 10.1109/TBC.2018.2823909 10.1109/TIP.2017.2760518 10.1109/TIP.2018.2815842 10.1109/TCSVT.2009.2013518 10.1109/ICASSP40776.2020.9053580 10.1007/s00530-014-0381-1 10.1145/3397227 10.1016/j.jvcir.2012.04.010 10.1109/PCS.2018.8456297 10.1109/WPMC.2014.7014800 10.3390/e21020165 10.1016/j.neucom.2017.08.054 10.1016/j.neucom.2020.06.003 10.1109/TMM.2019.2931807 10.1145/3386292.3397120 10.1109/TIP.2016.2573597
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/TMM.2022.3187259
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Xplore CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Xplore url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Computer Science
EISSN	1941-0077
EndPage	5092
ExternalDocumentID	10_1109_TMM_2022_3187259 9810507
Genre	orig-research
GroupedDBID	-~X 0R~ 29I 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACGFS ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD HZ~ H~9 IFIPE IFJZH IPLJI JAVBF LAI M43 O9- OCL P2P PQQKQ RIA RIE RNS TN5 VH1 ZY4 AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c357t-c24ca9314fe3aa694203cc072a7eca19975e7890d5e1628fb28eb902f3666ece3
IEDL.DBID	RIE
ISICitedReferencesCount	17
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001097340300034&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1520-9210
IngestDate	Wed Nov 26 03:47:41 EST 2025 Sat Nov 29 03:10:09 EST 2025 Tue Nov 18 21:43:03 EST 2025 Wed Aug 27 02:24:47 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c357t-c24ca9314fe3aa694203cc072a7eca19975e7890d5e1628fb28eb902f3666ece3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-3518-9195 0000-0001-6526-3811 0000-0002-4826-1168 0000-0002-3973-4445
PQID	2884893148
PQPubID	75737
PageCount	16
ParticipantIDs	crossref_primary_10_1109_TMM_2022_3187259 proquest_journals_2884893148 ieee_primary_9810507 crossref_citationtrail_10_1109_TMM_2022_3187259
PublicationCentury	2000
PublicationDate	20230000 2023-00-00 20230101
PublicationDateYYYYMMDD	2023-01-01
PublicationDate_xml	– year: 2023 text: 20230000
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE transactions on multimedia
PublicationTitleAbbrev	TMM
PublicationYear	2023
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref12 ref15 ref14 (ref48) 2002 ref11 ref10 achanta (ref45) 2010 nair (ref43) 2010 ref17 ref16 ref19 ref18 ref50 (ref42) 2021 ref41 (ref1) 2019 ref49 ref8 ref7 ref9 ref4 ref3 ref6 ref5 ref40 ref35 ref34 ref37 kingma (ref44) 2015 ref36 bossen (ref47) 2012 ref31 ref30 ref33 ref32 ref2 ref39 ref38 wei (ref29) 2009; 19 rosewarne (ref46) 2016 ref24 ref23 ref26 ref25 ref20 ref22 ref21 ref28 ref27 bjontegaard (ref51) 2001
References_xml	– ident: ref14 doi: 10.1109/ACCESS.2019.2901342 – ident: ref25 doi: 10.1109/TIP.2017.2685682 – ident: ref23 doi: 10.1145/3304112.3325604 – ident: ref37 doi: 10.1007/s11045-016-0395-2 – ident: ref31 doi: 10.1109/TCSVT.2016.2539862 – year: 2015 ident: ref44 article-title: Adam: A method for stochastic optimization publication-title: Proc Int Conf Learn Representations – ident: ref2 doi: 10.1109/TBC.2020.2977542 – ident: ref11 doi: 10.1109/ICIP.2019.8803454 – ident: ref26 doi: 10.1109/TIP.2013.2279934 – ident: ref5 doi: 10.1109/ACCESS.2021.3050489 – ident: ref12 doi: 10.1109/TIP.2018.2818439 – ident: ref41 doi: 10.1109/ICCV.2013.370 – ident: ref3 doi: 10.1109/TIP.2019.2933743 – ident: ref15 doi: 10.1109/JSTSP.2020.3034501 – ident: ref34 doi: 10.2352/ISSN.2470-1173.2016.13.IQSP-222 – ident: ref49 doi: 10.1109/TIP.2003.819861 – ident: ref16 doi: 10.1145/3394171.3413536 – start-page: 807 year: 2010 ident: ref43 article-title: Rectified linear units improve restricted Boltzmann machines publication-title: Proc Int Conf Mach Learn – ident: ref10 doi: 10.1109/ICMEW46912.2020.9106036 – year: 2010 ident: ref45 article-title: SLIC superpixels – ident: ref50 doi: 10.1109/CVPR.2018.00068 – ident: ref35 doi: 10.1109/ICIP.2016.7532610 – ident: ref36 doi: 10.1016/j.jvcir.2017.04.009 – ident: ref24 doi: 10.1109/TMM.2013.2268053 – start-page: 1 year: 2001 ident: ref51 article-title: Calculation of average PSNR differences between RD curves publication-title: Proc 13th Video Coding Expert Gr Meeting – year: 2021 ident: ref42 article-title: Subjective video quality assessment methods for multimedia applications – ident: ref38 doi: 10.1109/TIP.2020.3029428 – ident: ref40 doi: 10.1109/ICIP40778.2020.9190983 – ident: ref30 doi: 10.1109/TIP.2016.2568459 – ident: ref39 doi: 10.1109/LSP.2016.2641456 – ident: ref17 doi: 10.1109/ACCESS.2019.2910245 – ident: ref7 doi: 10.1109/TBC.2018.2823909 – ident: ref19 doi: 10.1109/TIP.2017.2760518 – ident: ref20 doi: 10.1109/TIP.2018.2815842 – volume: 19 start-page: 337 year: 2009 ident: ref29 article-title: Spatio-temporal just noticeable distortion profile for grey scale image/video in DCT domain publication-title: IEEE Trans Circuits Syst Video Technol doi: 10.1109/TCSVT.2009.2013518 – ident: ref32 doi: 10.1109/ICASSP40776.2020.9053580 – ident: ref4 doi: 10.1007/s00530-014-0381-1 – ident: ref21 doi: 10.1145/3397227 – year: 2002 ident: ref48 article-title: Methodology for the subjective assessment of the quality of television pictures – year: 2012 ident: ref47 article-title: Common test conditions and software reference configurations – ident: ref28 doi: 10.1016/j.jvcir.2012.04.010 – ident: ref13 doi: 10.1109/PCS.2018.8456297 – ident: ref33 doi: 10.1109/WPMC.2014.7014800 – ident: ref8 doi: 10.3390/e21020165 – year: 2016 ident: ref46 article-title: High efficiency video coding (HEVC) test model 16 (HM16) improved encoder description update 5 – year: 2019 ident: ref1 article-title: Cisco visual networking index: Global mobile data traffic forecast – ident: ref9 doi: 10.1016/j.neucom.2017.08.054 – ident: ref18 doi: 10.1016/j.neucom.2020.06.003 – ident: ref6 doi: 10.1109/TMM.2019.2931807 – ident: ref22 doi: 10.1145/3386292.3397120 – ident: ref27 doi: 10.1109/TIP.2016.2573597
SSID	ssj0014507
Score	2.4594483
Snippet	Just Noticeable Distortion (JND) finds the minimum distortion level perceivable by humans. This can be a natural solution for setting the compression for each...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	5077
SubjectTerms	Algorithms Bit rate Coding Control algorithms Control theory Convolutional neural network (cnn) Distortion Encoding just noticeable distortion (jnd) Measurement perceptual video coding (pvc) Predictive models quantization control algorithm Robustness (mathematics) Streaming media Video coding Video compression visual attention Visualization
Title	BL-JUNIPER: A CNN-Assisted Framework for Perceptual Video Coding Leveraging Block-Level JND
URI	https://ieeexplore.ieee.org/document/9810507 https://www.proquest.com/docview/2884893148
Volume	25
WOSCitedRecordID	wos001097340300034&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Xplore customDbUrl: eissn: 1941-0077 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014507 issn: 1520-9210 databaseCode: RIE dateStart: 19990101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8QwEB5W8aAH3-L6IgcvgnG7ado03nR1kUXLIiqCh9KmU1hcdmUf_n4zabsIiuCtlISWfklnJjPzfQCnEpXyRS54JtKASymJA9LLOImUe6HBXAfGiU2oOI5eX3W_AeeLXhhEdMVneEGXLpefj82cjspaOrLeALWOLykVlr1ai4yBDFxrtDVHHtc2jqlTkp5uPT082EBQCBufRkoQK-k3E-Q0VX78iJ116W787702Yb3yItlVCfsWNHC0DRu1QgOrNuw2rH2jG9yBt-t73rMBX__28ZJdsU4cc4sO4Zyzbl2kxawXy_plucvcPuNlkOOYdcZk49g92pXvdI3YtbWC75xuDFkvvtmF5-7tU-eOV-IK3PiBmnEjpEm135YF-mkaaik83xhPiVShSan8JEBqks0DbIciKjIRYaY9Ufg24EGD_h4sj8Yj3AcWpH4YKY2hypQMiV5G50XYNtp6d3agaEKr_t6JqZjHSQBjmLgIxNOJRSghhJIKoSacLWZ8lKwbf4zdIUQW4yowmnBUQ5pU23KaiCgish0bAh78PusQVklPvjxjOYLl2WSOx7BiPmeD6eTErbgvHE7PXg
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fS8MwED5EBfXB3-J0ah58EYzr0rRpfNPp8MdWhkwRfChteoOhbOI2_35zWTsERfCtlISWfknvLnf3fQDHEpXyRS54JtKASymJA9LLOImUe6HBXAfGiU2oOI6en3VnDk5nvTCI6IrP8IwuXS4_H5oJHZXVdGS9AWodXyDlrKJba5YzkIFrjrYGyePaRjJlUtLTtW67bUNBIWyEGilBvKTfjJBTVfnxK3b2pbn2vzdbh9XCj2QXU-A3YA4Hm7BWajSwYstuwso3wsEteLls8Tsb8nWuH87ZBWvEMbf4ENI5a5ZlWsz6sawzLXiZ2Gc89XMcssaQrBxroV37TtmIXVo7-Mrpxhu7i6-24bF53W3c8EJegRs_UGNuhDSp9uuyh36ahloKzzfGUyJVaFIqQAmQ2mTzAOuhiHqZiDDTnuj5NuRBg_4OzA-GA9wFFqR-GCmNocqUDIlgRue9sG609e_sQFGBWvm9E1Nwj5MExlviYhBPJxahhBBKCoQqcDKb8T7l3fhj7BYhMhtXgFGBaglpUmzMUSKiiOh2bBC49_usI1i66bZbSes2vt-HZVKXn564VGF-_DHBA1g0n-P-6OPQrb4vFx7Spw
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BL-JUNIPER%3A+A+CNN-Assisted+Framework+for+Perceptual+Video+Coding+Leveraging+Block-Level+JND&rft.jtitle=IEEE+transactions+on+multimedia&rft.au=Nami%2C+Sanaz&rft.au=Pakdaman%2C+Farhad&rft.au=Hashemi%2C+Mahmoud+Reza&rft.au=Shirmohammadi%2C+Shervin&rft.date=2023&rft.issn=1520-9210&rft.eissn=1941-0077&rft.volume=25&rft.spage=5077&rft.epage=5092&rft_id=info:doi/10.1109%2FTMM.2022.3187259&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TMM_2022_3187259
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1520-9210&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1520-9210&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1520-9210&client=summon