2 Framework: A Software Framework for Multimodal Human-Computer Interaction Systems

This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed software framework, which is called the HCI ∧ 2 Framework, is built upon publish/subscribe (P/S) architecture. It implements a shared-memory...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on cybernetics Vol. 43; no. 6; pp. 1593 - 1606
Main Authors: Jie Shen, Pantic, Maja
Format: Journal Article
Language:English
Published: IEEE 01.12.2013
Subjects:
Computer architecture
Human computer interaction
Middleware
Multimodal human-computer interface
Protocols
publish/subscribe architecture
Reliability
Runtime
software framework
ISSN:2168-2267, 2168-2275
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed software framework, which is called the HCI ∧ 2 Framework, is built upon publish/subscribe (P/S) architecture. It implements a shared-memory-based data transport protocol for message delivery and a TCP-based system management protocol. The latter ensures that the integrity of system structure is maintained at runtime. With the inclusion of bridging modules, the HCI ∧ 2 Framework is interoperable with other software frameworks including Psyclone and ActiveMQ. In addition to the core communication middleware, we also present the integrated development environment (IDE) of the HCI ∧ 2 Framework. It provides a complete graphical environment to support every step in a typical MHCI system development process, including module development, debugging, packaging, and management, as well as the whole system management and testing. The quantitative evaluation indicates that our framework outperforms other similar tools in terms of average message latency and maximum data throughput under a typical single PC scenario. To demonstrate HCI ∧ 2 Framework's capabilities in integrating heterogeneous modules, we present several example modules working with a variety of hardware and software. We also present an example of a full system developed using the proposed HCI ∧ 2 Framework, which is called the CamGame system and represents a computer game based on hand-held marker(s) and low-cost camera(s).
AbstractList This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed software framework, which is called the HCI ∧ 2 Framework, is built upon publish/subscribe (P/S) architecture. It implements a shared-memory-based data transport protocol for message delivery and a TCP-based system management protocol. The latter ensures that the integrity of system structure is maintained at runtime. With the inclusion of bridging modules, the HCI ∧ 2 Framework is interoperable with other software frameworks including Psyclone and ActiveMQ. In addition to the core communication middleware, we also present the integrated development environment (IDE) of the HCI ∧ 2 Framework. It provides a complete graphical environment to support every step in a typical MHCI system development process, including module development, debugging, packaging, and management, as well as the whole system management and testing. The quantitative evaluation indicates that our framework outperforms other similar tools in terms of average message latency and maximum data throughput under a typical single PC scenario. To demonstrate HCI ∧ 2 Framework's capabilities in integrating heterogeneous modules, we present several example modules working with a variety of hardware and software. We also present an example of a full system developed using the proposed HCI ∧ 2 Framework, which is called the CamGame system and represents a computer game based on hand-held marker(s) and low-cost camera(s).
Author Jie Shen
Pantic, Maja
Author_xml – sequence: 1
  surname: Jie Shen
  fullname: Jie Shen
  email: maja@doc.ic.ac.uk
  organization: Imperial Coll. London, London, CA, USA
– sequence: 2
  givenname: Maja
  surname: Pantic
  fullname: Pantic, Maja
  email: js1907@doc.ic.ac.uk
  organization: Imperial Coll. London, London, CA, USA
BookMark eNp9kM1Kw0AQxxepWK19APGyL5C6u0l2E281WFuoeGgvnsJkMwvRJFt2U0rf3oSWCh6cw3zBfz5-d2TU2hYJeeBsxjlLn7bZ58tMMB7OhFA8luEVuRVcJkFfxqNLLtWYTL3_Yr0lfStNbshYRCKMRZzcko2gCwcNHqz7fqZzurGmO4DD3y411tH3fd1VjS2hpst9A22Q2Wa379DRVdt70F1lW7o5-g4bf0-uDdQep-c4IdvF6zZbBuuPt1U2Xwe6f0AFMk00cFYWRQkYo9JSpTGXrDS8iCKZMAlFJFJujBQgtZAIRuk44ig0AwwnhJ_Game9d2jynasacMecs3xAlA-I8gFRfkbUa9Qfja46GI7vHFT1v8rHk7JCxMsmKcNIMBX-AAckdGU
CODEN ITCEB8
CitedBy_id crossref_primary_10_1109_TCYB_2015_2506985
crossref_primary_10_1109_TVCG_2017_2657098
crossref_primary_10_1016_j_patrec_2018_04_009
crossref_primary_10_1109_TPAMI_2019_2944808
Cites_doi 10.1098/rstb.2009.0135
10.1109/T-AFFC.2011.34
10.1023/B:VISI.0000013087.49260.fb
10.1109/34.824823
10.1109/CVPR.2000.854761
10.1504/IJAACS.2008.019799
10.1007/978-3-540-72348-6_3
10.1109/TSMCB.2012.2200675
10.1145/1177352.1177355
10.1007/978-3-540-72348-6_13
10.1162/014892600559182
10.1016/j.imavis.2011.07.004
10.1109/TPAMI.2008.52
10.1016/j.neucom.2011.07.024
10.1109/JPROC.2003.817122
10.1016/j.cviu.2006.10.019
ContentType Journal Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/TCYB.2013.2271563
DatabaseName IEEE Xplore (IEEE)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Sciences (General)
EISSN 2168-2275
EndPage 1606
ExternalDocumentID 10_1109_TCYB_2013_2271563
6634207
Genre orig-research
GroupedDBID 0R~
4.4
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABQJQ
ABVLG
ACIWK
AENEX
AGQYO
AGSQL
AHBIQ
AKJIK
AKQYR
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
EJD
HZ~
IFIPE
IPLJI
JAVBF
M43
O9-
OCL
PQQKQ
RIA
RIE
RNS
AAYXX
CITATION
ID FETCH-LOGICAL-c1107-698ca10dbbdae5e7c6795160df1b446806ab4291ff62a6c26eaf7c541e2c0ae3
IEDL.DBID RIE
ISSN 2168-2267
IngestDate Tue Nov 18 22:27:54 EST 2025
Sat Nov 29 06:48:28 EST 2025
Tue Aug 26 16:43:10 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 6
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c1107-698ca10dbbdae5e7c6795160df1b446806ab4291ff62a6c26eaf7c541e2c0ae3
PMID 24235258
PageCount 14
ParticipantIDs crossref_citationtrail_10_1109_TCYB_2013_2271563
crossref_primary_10_1109_TCYB_2013_2271563
ieee_primary_6634207
PublicationCentury 2000
PublicationDate 2013-Dec.
2013-12-00
PublicationDateYYYYMMDD 2013-12-01
PublicationDate_xml – month: 12
  year: 2013
  text: 2013-Dec.
PublicationDecade 2010
PublicationTitle IEEE transactions on cybernetics
PublicationTitleAbbrev TCYB
PublicationYear 2013
Publisher IEEE
Publisher_xml – name: IEEE
References stevens (ref28) 1999
ref34
ref31
shen (ref12) 2009
(ref24) 2012
ref30
(ref10) 2009
ref32
bradski (ref22) 1998
(ref33) 2008
ref2
ref1
ref17
ref16
lawson (ref11) 2008
thorisson (ref14) 2004; 25
(ref20) 2010
meyers (ref15) 1997
(ref21) 2012
(ref8) 2008
(ref23) 2012
ref25
ref27
(ref18) 2009
dabak (ref19) 1999
ref29
ref7
ref4
schr der (ref26) 2010; 2010
ref3
ref6
(ref9) 2007
ref5
pantic (ref13) 2004
References_xml – year: 2009
  ident: ref10
– year: 2012
  ident: ref23
  publication-title: MSDN Mutex Objects
– ident: ref31
  doi: 10.1098/rstb.2009.0135
– ident: ref27
  doi: 10.1109/T-AFFC.2011.34
– year: 2012
  ident: ref21
– ident: ref16
  doi: 10.1023/B:VISI.0000013087.49260.fb
– year: 1998
  ident: ref22
  article-title: Computer vision face tracking for use in a perceptual user interface
  publication-title: Intel Technol J
– start-page: 2038
  year: 2009
  ident: ref12
  article-title: A Software framework for multimodal human-computer interaction systems
  publication-title: Proc IEEE Int Conf Syst Man Cybern
– start-page: 303
  year: 1999
  ident: ref28
  publication-title: UNIX Network Programming Interprocess Communications
– ident: ref6
  doi: 10.1109/34.824823
– year: 2010
  ident: ref20
– ident: ref17
  doi: 10.1109/CVPR.2000.854761
– year: 2012
  ident: ref24
– year: 2008
  ident: ref8
– volume: 25
  start-page: 77
  year: 2004
  ident: ref14
  article-title: Constructionist design methodology for interactive intelligences
  publication-title: AI Mag
– start-page: 525
  year: 2004
  ident: ref13
  article-title: Fleeble agent framework for teaching an introductory course in AI
  publication-title: Proc IADIS Int Conf Cogn Exploratory Learn Digit Age
– ident: ref4
  doi: 10.1504/IJAACS.2008.019799
– ident: ref3
  doi: 10.1007/978-3-540-72348-6_3
– start-page: 143
  year: 1999
  ident: ref19
  publication-title: Undocumented Windows NT
– year: 2007
  ident: ref9
– ident: ref32
  doi: 10.1109/TSMCB.2012.2200675
– ident: ref29
  doi: 10.1145/1177352.1177355
– ident: ref7
  doi: 10.1007/978-3-540-72348-6_13
– ident: ref25
  doi: 10.1162/014892600559182
– start-page: 140
  year: 1997
  ident: ref15
  publication-title: Effective C++ 55 Specific Ways to Improve Your Programs and Designs
– start-page: 41
  year: 2008
  ident: ref11
  article-title: Rapid prototyping of multimodal interactive applications based on off-the-shelf heterogeneous components
  publication-title: Proc Adjunct 21st Ann ACM Symp User Interface Softw Technol
– ident: ref34
  doi: 10.1016/j.imavis.2011.07.004
– ident: ref5
  doi: 10.1109/TPAMI.2008.52
– ident: ref30
  doi: 10.1016/j.neucom.2011.07.024
– year: 2008
  ident: ref33
– ident: ref1
  doi: 10.1109/JPROC.2003.817122
– ident: ref2
  doi: 10.1016/j.cviu.2006.10.019
– volume: 2010
  start-page: 21
  year: 2010
  ident: ref26
  article-title: The SEMAINE API: Toward a standards-based framework for building emotion-oriented systems
  publication-title: Advances in human-computer interaction 1
– year: 2009
  ident: ref18
SSID ssj0000816898
Score 1.9672259
Snippet This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed...
SourceID crossref
ieee
SourceType Enrichment Source
Index Database
Publisher
StartPage 1593
SubjectTerms Computer architecture
Human computer interaction
Middleware
Multimodal human-computer interface
Protocols
publish/subscribe architecture
Reliability
Runtime
software framework
Title 2 Framework: A Software Framework for Multimodal Human-Computer Interaction Systems
URI https://ieeexplore.ieee.org/document/6634207
Volume 43
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 2168-2275
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0000816898
  issn: 2168-2267
  databaseCode: RIE
  dateStart: 20130101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELZKxcAClIIoL3lgAERa20nsmK1UVEwVUjuUKbKdi4QELeoD_j624wYGhMRmWXYU3Tm578539yF0yZlQKpVplEmtHYWZiKQ0KoLY2uqECkVK7ckmxGiUTafyqYFu61oYAPDJZ9B1Q3-XX8zN2oXKetY6JsyVjm8JwatarTqe4gkkPPUts4PIogoRLjEpkb3J4Pne5XHFXcaEdVkcfY5DEilzXO8_LNIPihVvYYZ7_3u3fbQbkCTuV6pvoQbMDlArfKtLfBUaSl-30Zjh4SYH6w738dj-ej_VAr5nsYWu2Nfivs0L-1Af2o82jA_Yhw2rCggcWpwfosnwYTJ4jAKZQmScixdxmRlFSaF1oSAFYbiw4IqToqTauoQZ4Upb20TLkjPFDeOgSmHShAIzREF8hJqz-QyOEY5pYUFHoQQtTWIykpUpEwaEtq6Xddegg8hGnrkJjcYd38Vr7h0OInOnjdxpIw_a6KCbest71WXjr8VtJ_56YZD8ye_Tp2jHba4yUM5Qc7VYwznaNh-rl-Xiwp-hL7-qwMc
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFA9DBb2oc4rzMwcPKmZL0zZpvM3hmDiHsB3mqSRpCoJusg_9903SrO4ggrdHSEPJS_t-7-W99wPgghImRMxjlHApLYUZQ5wrgXRobHUUMIFz6cgmWL-fjEb8uQJuyloYrbVLPtMNK7q7_GyiFjZU1jTWMSK2dHw9joxQVGuVERVHIeHIb4kRkMEVzF9jBpg3h-2XO5vJFTYIYcZpsQQ6FkvExLK9r9ikFZIVZ2M6O_97u12w7bEkbBXKr4KKHu-Bqv9aZ_DSt5S-qoEBgZ1lFtYtbMGB-fl-ian-GYUGvEJXjfs-ycyiLriPlpwP0AUOixoI6Juc74Nh537Y7iJPp4CUdfIQ5YkSAc6kzISONVOUGXhFcZYH0jiFCaZCGusU5DklgipCtciZiqNAE4WFDg_A2ngy1ocAhkFmYEcmWJCrSCU4yWPClGbSOF_GYdN1gJf7mSrfatwyXrylzuXAPLXaSK02Uq-NOrguH_ko-mz8Nblmt7-c6Hf-6Pfhc7DZHT710t5D__EYbNmFinyUE7A2ny70KdhQn_PX2fTMnadv9f7EDg
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%7B%5Crm+HCI%7D%5E%7B%5Cwedge%7D2%24+Framework%3A+A+Software+Framework+for+Multimodal+Human-Computer+Interaction+Systems&rft.jtitle=IEEE+transactions+on+cybernetics&rft.au=Shen%2C+Jie&rft.au=Pantic%2C+Maja&rft.date=2013-12-01&rft.issn=2168-2267&rft.eissn=2168-2275&rft.volume=43&rft.issue=6&rft.spage=1593&rft.epage=1606&rft_id=info:doi/10.1109%2FTCYB.2013.2271563&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TCYB_2013_2271563
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2168-2267&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2168-2267&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2168-2267&client=summon