A Real-Time CPU-GPU Embedded Implementation of a Tightly-Coupled Visual-Inertial Navigation System

In autonomous navigation technologies, the Multi-State Constraint Kalman Filter (MSCKF) is one of the most accurate and robust tightly-coupled fusion frameworks for Visual-Inertial Navigation (VIN). However, the adoption of the MSCKF VIN system in real-time embedded applications depends heavily on a...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE access Jg. 10; S. 86384 - 86394
Hauptverfasser: Sheikhpour, K. Soroush, Atia, Mohamed
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Piscataway IEEE 2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:
Algorithms
Autonomous navigation
Autonomous systems
Central processing units
CPUs
Embedded software
Embedded systems
Graphics processing units
Inertial navigation
kalman filter
Kalman filters
Localization
Navigation systems
Optimization
Parallel processing
Pipelining (computers)
Real time
Real-time systems
sensor fusion
Sensors
visual-inertial navigation
Visualization
ISSN:2169-3536, 2169-3536
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Abstract In autonomous navigation technologies, the Multi-State Constraint Kalman Filter (MSCKF) is one of the most accurate and robust tightly-coupled fusion frameworks for Visual-Inertial Navigation (VIN). However, the adoption of the MSCKF VIN system in real-time embedded applications depends heavily on an efficient implementation of its tangled pipeline. This work initially proposes a novel parallel multi-thread implementation of the MSCKF VIN pipeline on an embedded CPU-enabled hardware that has speeded up the per-epoch processing time of the pipeline by 41% compared to the conventional sequential implementation. The heart of the MSCKF pipeline's visual backend is an inertially-aided 3D localization of visual feature points that are reduced to a set of nonlinear optimization problems which were conventionally solved in a serial fashion using the single-objective Gauss-Newton optimization algorithm. This work leveraged the parallel architecture of an embedded GPU and further proposes an efficient parallel implementation of a multi-objective Gauss-Newton algorithm. Integration of the proposed GPU-accelerated feature localization technique in the MSCKF parallel pipeline has resulted in 33% faster per-epoch processing time and consequently, the satisfaction of strict real-time constraints. The proposed parallel MSCKF VIN pipelines have been developed using C++ and CUDA on the NVIDIA Jetson TX2 embedded board. Experimental evaluations on a real visual-inertial odometry dataset have been provided to validate the efficacy and real-time performance enhancement of the proposed parallel implementation.
AbstractList In autonomous navigation technologies, the Multi-State Constraint Kalman Filter (MSCKF) is one of the most accurate and robust tightly-coupled fusion frameworks for Visual-Inertial Navigation (VIN). However, the adoption of the MSCKF VIN system in real-time embedded applications depends heavily on an efficient implementation of its tangled pipeline. This work initially proposes a novel parallel multi-thread implementation of the MSCKF VIN pipeline on an embedded CPU-enabled hardware that has speeded up the per-epoch processing time of the pipeline by 41% compared to the conventional sequential implementation. The heart of the MSCKF pipeline's visual backend is an inertially-aided 3D localization of visual feature points that are reduced to a set of nonlinear optimization problems which were conventionally solved in a serial fashion using the single-objective Gauss-Newton optimization algorithm. This work leveraged the parallel architecture of an embedded GPU and further proposes an efficient parallel implementation of a multi-objective Gauss-Newton algorithm. Integration of the proposed GPU-accelerated feature localization technique in the MSCKF parallel pipeline has resulted in 33% faster per-epoch processing time and consequently, the satisfaction of strict real-time constraints. The proposed parallel MSCKF VIN pipelines have been developed using C++ and CUDA on the NVIDIA Jetson TX2 embedded board. Experimental evaluations on a real visual-inertial odometry dataset have been provided to validate the efficacy and real-time performance enhancement of the proposed parallel implementation.
Author Atia, Mohamed
Sheikhpour, K. Soroush
Author_xml – sequence: 1
  givenname: K. Soroush
  orcidid: 0000-0001-5555-5626
  surname: Sheikhpour
  fullname: Sheikhpour, K. Soroush
  email: soroushsheikhpourkou@cmail.carleton.ca
  organization: Department of Electronics, Carleton University, Ottawa, Canada
– sequence: 2
  givenname: Mohamed
  orcidid: 0000-0002-4272-7019
  surname: Atia
  fullname: Atia, Mohamed
  organization: Department of Systems and Computer Engineering, Carleton University, Ottawa, Canada
BookMark eNpNUU1P4zAQtVastMDyC7hY4pzi-KOxj1VU2EoIEG33ak3icddVEpckReq_X0MQYi7z9d6bkd4FOetih4Rc52yW58zcLspyuV7POON8JnJjhJY_yDnP5yYTSszPvtW_yNUw7FkKnUaqOCfVgr4gNNkmtEjL5212_7yly7ZC59DRVXtosMVuhDHEjkZPgW7C7t_YnLIyHtPS0b9hOCaBVYf9GKChj_AWdhN-fRpGbH-Tnx6aAa8-8yXZ3i035Z_s4el-VS4espprPWZaescLrLzLRQX1vEBT145LZgrhvZQAQgNCVefSgDZSaO515bV3Drx0TFyS1aTrIuztoQ8t9CcbIdiPQex3FtKLdYOWSSm1lEoBd1K5uWFOyCI1tVHCVT5p3Uxahz6-HnEY7T4e-y69b3nBlFbcSJNQYkLVfRyGHv3X1ZzZd2_s5I1998Z-epNY1xMrIOIXw2ilmeLiP9UejCw
CODEN IAECCG
Cites_doi 10.1109/cvpr.2012.6248074
10.5244/C.2.23
10.1109/tro.2018.2853729
10.1109/robot.2007.364024
10.1109/icra.2018.8460664
10.1115/1.3662552
10.33012/2019.16918
10.3390/rs9010003
10.1109/ismar.2008.4637316
10.1109/iros45743.2020.9340851
10.1007/978-3-642-15561-1_56
10.1109/icra.2016.7487290
10.1109/iccv.2011.6126544
10.1109/icra.2014.6906892
10.1109/tpami.2019.2909895
10.1109/icomet.2018.8346440
10.1137/1.9781611971484
10.1177/0278364914554813
10.3390/robotics7030045
10.1109/lra.2018.2793349
10.1109/robot.2002.1014441
10.1007/11744023_34
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
DBID 97E
ESBDL
RIA
RIE
AAYXX
CITATION
7SC
7SP
7SR
8BQ
8FD
JG9
JQ2
L7M
L~C
L~D
DOA
DOI 10.1109/ACCESS.2022.3199384
DatabaseName IEEE Xplore (IEEE)
IEEE Xplore Open Access (Activated by CARLI)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Xplore
CrossRef
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Engineered Materials Abstracts
METADEX
Technology Research Database
Materials Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
DOAJ Directory of Open Access Journals
DatabaseTitle CrossRef
Materials Research Database
Engineered Materials Abstracts
Technology Research Database
Computer and Information Systems Abstracts – Academic
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Advanced Technologies Database with Aerospace
METADEX
Computer and Information Systems Abstracts Professional
DatabaseTitleList

Materials Research Database
Database_xml – sequence: 1
  dbid: DOA
  name: Open Access资源_DOAJ
  url: https://www.doaj.org/
  sourceTypes: Open Website
– sequence: 2
  dbid: RIE
  name: IEEE Xplore
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
EISSN 2169-3536
EndPage 86394
ExternalDocumentID oai_doaj_org_article_044484455a2d45d690d347a2dc953dbf
10_1109_ACCESS_2022_3199384
9858052
Genre orig-research
GrantInformation_xml – fundername: Natural Sciences and Engineering Research Council (NSERC) of Canada
  grantid: RGPIN-2017-06261
  funderid: 10.13039/501100000038
GroupedDBID 0R~
4.4
5VS
6IK
97E
AAJGR
ABAZT
ABVLG
ACGFS
ADBBV
AGSQL
ALMA_UNASSIGNED_HOLDINGS
BCNDV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
EJD
ESBDL
GROUPED_DOAJ
IPLJI
JAVBF
KQ8
M43
M~E
O9-
OCL
OK1
RIA
RIE
RNS
AAYXX
CITATION
7SC
7SP
7SR
8BQ
8FD
JG9
JQ2
L7M
L~C
L~D
ID FETCH-LOGICAL-c288t-84fd27ebfd13bac67e9ccd240973ff44aa38aeabc149a894382f8bf8fddaf4d03
IEDL.DBID DOA
ISICitedReferencesCount 1
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000844069500001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 2169-3536
IngestDate Fri Oct 03 12:52:17 EDT 2025
Mon Jun 30 06:22:36 EDT 2025
Sat Nov 29 06:32:20 EST 2025
Wed Aug 27 02:28:35 EDT 2025
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Language English
License https://creativecommons.org/licenses/by-nc-nd/4.0
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c288t-84fd27ebfd13bac67e9ccd240973ff44aa38aeabc149a894382f8bf8fddaf4d03
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0001-5555-5626
0000-0002-4272-7019
OpenAccessLink https://doaj.org/article/044484455a2d45d690d347a2dc953dbf
PQID 2705852949
PQPubID 4845423
PageCount 11
ParticipantIDs proquest_journals_2705852949
ieee_primary_9858052
crossref_primary_10_1109_ACCESS_2022_3199384
doaj_primary_oai_doaj_org_article_044484455a2d45d690d347a2dc953dbf
PublicationCentury 2000
PublicationDate 20220000
2022-00-00
20220101
2022-01-01
PublicationDateYYYYMMDD 2022-01-01
PublicationDate_xml – year: 2022
  text: 20220000
PublicationDecade 2020
PublicationPlace Piscataway
PublicationPlace_xml – name: Piscataway
PublicationTitle IEEE access
PublicationTitleAbbrev Access
PublicationYear 2022
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref15
ref14
ref31
(ref28) 2021
ref11
ref10
ref32
ref2
ref1
G (ref26) 2010
ref17
ref16
ref19
ref18
Maybeck (ref20) 1982
Bradski (ref24) 2000; 25
ref23
Groups (ref9) 2020
ref25
ref21
Weidendorfer (ref29) 2020
Nguyen (ref12) 2019
Groves (ref22) 2013
ref8
ref7
ref4
(ref30) 2020
ref3
ref6
ref5
Stallman (ref27) 2003
References_xml – ident: ref10
  doi: 10.1109/cvpr.2012.6248074
– ident: ref31
  doi: 10.5244/C.2.23
– volume-title: CUDA C++ Programming Guide Design Guide v11.2.0
  year: 2020
  ident: ref30
– volume-title: KCachegrind
  year: 2020
  ident: ref29
– ident: ref5
  doi: 10.1109/tro.2018.2853729
– year: 2019
  ident: ref12
  article-title: Computationally-efficient visual inertial odometry for autonomous vehicle
– ident: ref3
  doi: 10.1109/robot.2007.364024
– ident: ref8
  doi: 10.1109/icra.2018.8460664
– ident: ref19
  doi: 10.1115/1.3662552
– volume: 25
  start-page: 120
  issue: 11
  year: 2000
  ident: ref24
  article-title: The OpenCV library
  publication-title: Dr. Dobb’s J. Softw. Tools Prof. Program.
– ident: ref7
  doi: 10.33012/2019.16918
– ident: ref2
  doi: 10.3390/rs9010003
– ident: ref4
  doi: 10.1109/ismar.2008.4637316
– ident: ref15
  doi: 10.1109/iros45743.2020.9340851
– ident: ref32
  doi: 10.1007/978-3-642-15561-1_56
– ident: ref13
  doi: 10.1109/icra.2016.7487290
– ident: ref25
  doi: 10.1109/iccv.2011.6126544
– ident: ref14
  doi: 10.1109/icra.2014.6906892
– volume-title: Principles of GNSS, Inertial, and Multisensor Integrated Navigation Systems
  year: 2013
  ident: ref22
– ident: ref18
  doi: 10.1109/tpami.2019.2909895
– ident: ref17
  doi: 10.1109/icomet.2018.8346440
– volume-title: Stochastic Models, Estimation, and Control
  year: 1982
  ident: ref20
– volume-title: Google Performance Tool
  year: 2020
  ident: ref9
– ident: ref23
  doi: 10.1137/1.9781611971484
– ident: ref6
  doi: 10.1177/0278364914554813
– volume-title: Eigen v3
  year: 2010
  ident: ref26
– ident: ref1
  doi: 10.3390/robotics7030045
– volume-title: CUDA Compiler Driver NVCC Reference Guide
  year: 2021
  ident: ref28
– ident: ref11
  doi: 10.1109/lra.2018.2793349
– ident: ref21
  doi: 10.1109/robot.2002.1014441
– ident: ref16
  doi: 10.1007/11744023_34
– volume-title: Using the GNU Compiler Collection (GCC)
  year: 2003
  ident: ref27
SSID ssj0000816957
Score 2.2355094
Snippet In autonomous navigation technologies, the Multi-State Constraint Kalman Filter (MSCKF) is one of the most accurate and robust tightly-coupled fusion...
SourceID doaj
proquest
crossref
ieee
SourceType Open Website
Aggregation Database
Index Database
Publisher
StartPage 86384
SubjectTerms Algorithms
Autonomous navigation
Autonomous systems
Central processing units
CPUs
Embedded software
Embedded systems
Graphics processing units
Inertial navigation
kalman filter
Kalman filters
Localization
Navigation systems
Optimization
Parallel processing
Pipelining (computers)
Real time
Real-time systems
sensor fusion
Sensors
visual-inertial navigation
Visualization
SummonAdditionalLinks – databaseName: IEEE Xplore
  dbid: RIE
  link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Nb9QwEB21FQc48FUQWwrygWNDHcfZ2MclagEJrVaoW_Vm2R5bqkSzqLtbiX_fseOuQHDhlg_HdvwyHj_H8wzwQXGUItSy0ih0JWUgm0v22NZBtiG6qLMS0-W3bj5XV1d6sQcnu1iYEEJefBY-psP8Lx9Xfpumyk61apMC_z7sd910jNXazaekDSR02xVhoZrr01nf0zsQBRSCmCn5YSX_cD5Zo79sqvJXT5zdy_mz_6vYc3hahpFsNuL-AvbC8BKe_CYueAhuxr7TKLBKQR6sXyyrz4slO7txgboaZFkV-KYEHg1sFZllF4mo__hV9ast3UR2eb3eUgZfh7T2mkqb27ssyEHpR6HzV7A8P7vov1RlR4XKC6U2BENE0QUXsW6c9dMuaO9RJM2rJkYprW2UDdZ54k02KbMrEZWLKiLaKJE3r-FgWA3hDTDOrfIOybn5IOto9VQJh5oGRMhFRD6Bk4emNj9H4QyTCQfXZkTGJGRMQWYCnxIcu6RJ9TpfoHY2xYhM0rZTUratFShbJGKPjezoxOu2QRcncJiw2WVSYJnA8QO4pljo2oiOE1MSWuqjfz_1Fh6nCo7TLcdwsLndhnfwyN9trte37_PHdw_apdiE
  priority: 102
  providerName: IEEE
Title A Real-Time CPU-GPU Embedded Implementation of a Tightly-Coupled Visual-Inertial Navigation System
URI https://ieeexplore.ieee.org/document/9858052
https://www.proquest.com/docview/2705852949
https://doaj.org/article/044484455a2d45d690d347a2dc953dbf
Volume 10
WOSCitedRecordID wos000844069500001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVAON
  databaseName: Open Access资源_DOAJ
  customDbUrl:
  eissn: 2169-3536
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0000816957
  issn: 2169-3536
  databaseCode: DOA
  dateStart: 20130101
  isFulltext: true
  titleUrlDefault: https://www.doaj.org/
  providerName: Directory of Open Access Journals
– providerCode: PRVHPJ
  databaseName: ROAD: Directory of Open Access Scholarly Resources
  customDbUrl:
  eissn: 2169-3536
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0000816957
  issn: 2169-3536
  databaseCode: M~E
  dateStart: 20130101
  isFulltext: true
  titleUrlDefault: https://road.issn.org
  providerName: ISSN International Centre
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1La9wwEBYl9NAcSh4t2Waz6JBjTGRZXkvHrdk8IFmWshtyE5JGgkDiLdkH5JLf3pHshC095NKLwbbQ4xtrRp-RviHkVDIQ3OciU8BVJoTHORfnY5l7Ufpgg0pKTHc31WQi7-_VdCvVV9wT1soDt8CdRz0zKURZGg6iBCRzUIgKb5wqC7Ahel9WqS0ylXywzIeqrDqZoZyp81Fd44iQEHKOPBWjshR_haKk2N-lWPnHL6dgc7FHvnarRDpqe7dPPvnmgOxuaQceEjuiv3CRl8UzHLSezrPL6ZyOn6xHTwI0if4-deeKGroI1NBZ5OGPL1m9WONLoHcPyzVWcN3ErdXY2sRskt4Glm91zL-R-cV4Vl9lXcKEzHEpV4hyAF55GyAvrHHDyivngEdJqyIEIYwppPHGOqRFJgqvSx6kDTIAmCCAFd_JTrNo_BGhjBnpLGDscl7kwaih5BYUrneA8QCsR87esNO_W10MnfgEU7qFWkeodQd1j_yM-L4XjaLW6QGaWnem1h-ZukcOo3XeK1GyjAkZeqT_Zi3dTcCl5hVDIsSVUD_-R9PH5EscTvvvpU92Vs9rf0I-u83qYfk8SN8eXm9fx4N0gvAP8BPc6w
linkProvider Directory of Open Access Journals
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Nb9QwEB2VggQc-GoRCwV84NjQxHEa-7hELa1YViu0W_Vm2R5bqkSzqLtbiX_P2HFXILhwy4eTOH4Zj5_jeQPwQZYouK9EoZCrQghPNhftsam8aHywQSUlpotJO53Ky0s124HDbSyM9z4tPvMf42b6l49Lt4lTZUdKNlGB_x7cj5mzcrTWdkYlppBQTZulhapSHY27jt6CSCDnxE3JE0vxh_tJKv05rcpffXFyMKdP_69qz-BJHkiy8YD8c9jx_Qt4_Ju84B7YMftG48AihnmwbrYoPs8W7OTaeupskCVd4OscetSzZWCGzSNV__6z6JYbOons4mq1oRuc93H1NT1tam6TJAeVH6TO92FxejLvzoqcU6FwXMo1ARGQt94GrGpr3HHrlXPIo-pVHYIQxtTSeGMdMScTtdklD9IGGRBNEFjWL2G3X_b-FbCyNNJZJPfmvKiCUceSW1Q0JMKSByxHcHjX1PrHIJ2hE-UolR6Q0REZnZEZwacIx7Zo1L1OB6iddTYjHdXtpBBNYziKBonaYy1a2nGqqdGGEexFbLY3ybCM4OAOXJ1tdKV5WxJX4kqo1_--6j08PJt_nejJ-fTLG3gUKztMvhzA7vpm49_CA3e7vlrdvEsf4i9Hn9vN
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Real-Time+CPU-GPU+Embedded+Implementation+of+a+Tightly-Coupled+Visual-Inertial+Navigation+System&rft.jtitle=IEEE+access&rft.au=Sheikhpour%2C+K.+Soroush&rft.au=Atia%2C+Mohamed&rft.date=2022&rft.pub=IEEE&rft.eissn=2169-3536&rft.volume=10&rft.spage=86384&rft.epage=86394&rft_id=info:doi/10.1109%2FACCESS.2022.3199384&rft.externalDocID=9858052
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2169-3536&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2169-3536&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2169-3536&client=summon