2 Framework: A Software Framework for Multimodal Human-Computer Interaction Systems
This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed software framework, which is called the HCI ∧ 2 Framework, is built upon publish/subscribe (P/S) architecture. It implements a shared-memory...
Uloženo v:
| Vydáno v: | IEEE transactions on cybernetics Ročník 43; číslo 6; s. 1593 - 1606 |
|---|---|
| Hlavní autoři: | , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
01.12.2013
|
| Témata: | |
| ISSN: | 2168-2267, 2168-2275 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed software framework, which is called the HCI ∧ 2 Framework, is built upon publish/subscribe (P/S) architecture. It implements a shared-memory-based data transport protocol for message delivery and a TCP-based system management protocol. The latter ensures that the integrity of system structure is maintained at runtime. With the inclusion of bridging modules, the HCI ∧ 2 Framework is interoperable with other software frameworks including Psyclone and ActiveMQ. In addition to the core communication middleware, we also present the integrated development environment (IDE) of the HCI ∧ 2 Framework. It provides a complete graphical environment to support every step in a typical MHCI system development process, including module development, debugging, packaging, and management, as well as the whole system management and testing. The quantitative evaluation indicates that our framework outperforms other similar tools in terms of average message latency and maximum data throughput under a typical single PC scenario. To demonstrate HCI ∧ 2 Framework's capabilities in integrating heterogeneous modules, we present several example modules working with a variety of hardware and software. We also present an example of a full system developed using the proposed HCI ∧ 2 Framework, which is called the CamGame system and represents a computer game based on hand-held marker(s) and low-cost camera(s). |
|---|---|
| AbstractList | This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed software framework, which is called the HCI ∧ 2 Framework, is built upon publish/subscribe (P/S) architecture. It implements a shared-memory-based data transport protocol for message delivery and a TCP-based system management protocol. The latter ensures that the integrity of system structure is maintained at runtime. With the inclusion of bridging modules, the HCI ∧ 2 Framework is interoperable with other software frameworks including Psyclone and ActiveMQ. In addition to the core communication middleware, we also present the integrated development environment (IDE) of the HCI ∧ 2 Framework. It provides a complete graphical environment to support every step in a typical MHCI system development process, including module development, debugging, packaging, and management, as well as the whole system management and testing. The quantitative evaluation indicates that our framework outperforms other similar tools in terms of average message latency and maximum data throughput under a typical single PC scenario. To demonstrate HCI ∧ 2 Framework's capabilities in integrating heterogeneous modules, we present several example modules working with a variety of hardware and software. We also present an example of a full system developed using the proposed HCI ∧ 2 Framework, which is called the CamGame system and represents a computer game based on hand-held marker(s) and low-cost camera(s). |
| Author | Jie Shen Pantic, Maja |
| Author_xml | – sequence: 1 surname: Jie Shen fullname: Jie Shen email: maja@doc.ic.ac.uk organization: Imperial Coll. London, London, CA, USA – sequence: 2 givenname: Maja surname: Pantic fullname: Pantic, Maja email: js1907@doc.ic.ac.uk organization: Imperial Coll. London, London, CA, USA |
| BookMark | eNp9kM1Kw0AQxxepWK19APGyL5C6u0l2E281WFuoeGgvnsJkMwvRJFt2U0rf3oSWCh6cw3zBfz5-d2TU2hYJeeBsxjlLn7bZ58tMMB7OhFA8luEVuRVcJkFfxqNLLtWYTL3_Yr0lfStNbshYRCKMRZzcko2gCwcNHqz7fqZzurGmO4DD3y411tH3fd1VjS2hpst9A22Q2Wa379DRVdt70F1lW7o5-g4bf0-uDdQep-c4IdvF6zZbBuuPt1U2Xwe6f0AFMk00cFYWRQkYo9JSpTGXrDS8iCKZMAlFJFJujBQgtZAIRuk44ig0AwwnhJ_Game9d2jynasacMecs3xAlA-I8gFRfkbUa9Qfja46GI7vHFT1v8rHk7JCxMsmKcNIMBX-AAckdGU |
| CODEN | ITCEB8 |
| CitedBy_id | crossref_primary_10_1109_TCYB_2015_2506985 crossref_primary_10_1109_TVCG_2017_2657098 crossref_primary_10_1016_j_patrec_2018_04_009 crossref_primary_10_1109_TPAMI_2019_2944808 |
| Cites_doi | 10.1098/rstb.2009.0135 10.1109/T-AFFC.2011.34 10.1023/B:VISI.0000013087.49260.fb 10.1109/34.824823 10.1109/CVPR.2000.854761 10.1504/IJAACS.2008.019799 10.1007/978-3-540-72348-6_3 10.1109/TSMCB.2012.2200675 10.1145/1177352.1177355 10.1007/978-3-540-72348-6_13 10.1162/014892600559182 10.1016/j.imavis.2011.07.004 10.1109/TPAMI.2008.52 10.1016/j.neucom.2011.07.024 10.1109/JPROC.2003.817122 10.1016/j.cviu.2006.10.019 |
| ContentType | Journal Article |
| DBID | 97E RIA RIE AAYXX CITATION |
| DOI | 10.1109/TCYB.2013.2271563 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Sciences (General) |
| EISSN | 2168-2275 |
| EndPage | 1606 |
| ExternalDocumentID | 10_1109_TCYB_2013_2271563 6634207 |
| Genre | orig-research |
| GroupedDBID | 0R~ 4.4 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACIWK AENEX AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD HZ~ IFIPE IPLJI JAVBF M43 O9- OCL PQQKQ RIA RIE RNS AAYXX CITATION |
| ID | FETCH-LOGICAL-c1107-698ca10dbbdae5e7c6795160df1b446806ab4291ff62a6c26eaf7c541e2c0ae3 |
| IEDL.DBID | RIE |
| ISSN | 2168-2267 |
| IngestDate | Tue Nov 18 22:27:54 EST 2025 Sat Nov 29 06:48:28 EST 2025 Tue Aug 26 16:43:10 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 6 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c1107-698ca10dbbdae5e7c6795160df1b446806ab4291ff62a6c26eaf7c541e2c0ae3 |
| PMID | 24235258 |
| PageCount | 14 |
| ParticipantIDs | crossref_citationtrail_10_1109_TCYB_2013_2271563 crossref_primary_10_1109_TCYB_2013_2271563 ieee_primary_6634207 |
| PublicationCentury | 2000 |
| PublicationDate | 2013-Dec. 2013-12-00 |
| PublicationDateYYYYMMDD | 2013-12-01 |
| PublicationDate_xml | – month: 12 year: 2013 text: 2013-Dec. |
| PublicationDecade | 2010 |
| PublicationTitle | IEEE transactions on cybernetics |
| PublicationTitleAbbrev | TCYB |
| PublicationYear | 2013 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| References | stevens (ref28) 1999 ref34 ref31 shen (ref12) 2009 (ref24) 2012 ref30 (ref10) 2009 ref32 bradski (ref22) 1998 (ref33) 2008 ref2 ref1 ref17 ref16 lawson (ref11) 2008 thorisson (ref14) 2004; 25 (ref20) 2010 meyers (ref15) 1997 (ref21) 2012 (ref8) 2008 (ref23) 2012 ref25 ref27 (ref18) 2009 dabak (ref19) 1999 ref29 ref7 ref4 schr der (ref26) 2010; 2010 ref3 ref6 (ref9) 2007 ref5 pantic (ref13) 2004 |
| References_xml | – year: 2009 ident: ref10 – year: 2012 ident: ref23 publication-title: MSDN Mutex Objects – ident: ref31 doi: 10.1098/rstb.2009.0135 – ident: ref27 doi: 10.1109/T-AFFC.2011.34 – year: 2012 ident: ref21 – ident: ref16 doi: 10.1023/B:VISI.0000013087.49260.fb – year: 1998 ident: ref22 article-title: Computer vision face tracking for use in a perceptual user interface publication-title: Intel Technol J – start-page: 2038 year: 2009 ident: ref12 article-title: A Software framework for multimodal human-computer interaction systems publication-title: Proc IEEE Int Conf Syst Man Cybern – start-page: 303 year: 1999 ident: ref28 publication-title: UNIX Network Programming Interprocess Communications – ident: ref6 doi: 10.1109/34.824823 – year: 2010 ident: ref20 – ident: ref17 doi: 10.1109/CVPR.2000.854761 – year: 2012 ident: ref24 – year: 2008 ident: ref8 – volume: 25 start-page: 77 year: 2004 ident: ref14 article-title: Constructionist design methodology for interactive intelligences publication-title: AI Mag – start-page: 525 year: 2004 ident: ref13 article-title: Fleeble agent framework for teaching an introductory course in AI publication-title: Proc IADIS Int Conf Cogn Exploratory Learn Digit Age – ident: ref4 doi: 10.1504/IJAACS.2008.019799 – ident: ref3 doi: 10.1007/978-3-540-72348-6_3 – start-page: 143 year: 1999 ident: ref19 publication-title: Undocumented Windows NT – year: 2007 ident: ref9 – ident: ref32 doi: 10.1109/TSMCB.2012.2200675 – ident: ref29 doi: 10.1145/1177352.1177355 – ident: ref7 doi: 10.1007/978-3-540-72348-6_13 – ident: ref25 doi: 10.1162/014892600559182 – start-page: 140 year: 1997 ident: ref15 publication-title: Effective C++ 55 Specific Ways to Improve Your Programs and Designs – start-page: 41 year: 2008 ident: ref11 article-title: Rapid prototyping of multimodal interactive applications based on off-the-shelf heterogeneous components publication-title: Proc Adjunct 21st Ann ACM Symp User Interface Softw Technol – ident: ref34 doi: 10.1016/j.imavis.2011.07.004 – ident: ref5 doi: 10.1109/TPAMI.2008.52 – ident: ref30 doi: 10.1016/j.neucom.2011.07.024 – year: 2008 ident: ref33 – ident: ref1 doi: 10.1109/JPROC.2003.817122 – ident: ref2 doi: 10.1016/j.cviu.2006.10.019 – volume: 2010 start-page: 21 year: 2010 ident: ref26 article-title: The SEMAINE API: Toward a standards-based framework for building emotion-oriented systems publication-title: Advances in human-computer interaction 1 – year: 2009 ident: ref18 |
| SSID | ssj0000816898 |
| Score | 1.9672259 |
| Snippet | This paper presents a novel software framework for the development and research in the area of multimodal human-computer interface (MHCI) systems. The proposed... |
| SourceID | crossref ieee |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 1593 |
| SubjectTerms | Computer architecture Human computer interaction Middleware Multimodal human-computer interface Protocols publish/subscribe architecture Reliability Runtime software framework |
| Title | 2 Framework: A Software Framework for Multimodal Human-Computer Interaction Systems |
| URI | https://ieeexplore.ieee.org/document/6634207 |
| Volume | 43 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2168-2275 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0000816898 issn: 2168-2267 databaseCode: RIE dateStart: 20130101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePDiq4r1RQ4eVEybZHeTrLdaLB6kCC1ST0s2OwuCttKH_n2TbFp7EMFbCMmyZJLMN5OZ-RC6AFVquxUUMRAJEsfUEBUZZs-VkjouE019JabnR9nvq9Eofaqhm1UuDAD44DNouaZ_yy8mZuFcZW2rHWPuUsc3pBRVrtbKn-IJJDz1LbcNYlGFDI-YjKbtYfflzsVxRS3OpTVZHH2OQxIJd1zvaxppjWLFa5jezv_-bRdtBySJO5Xo91ANxvtoL5zVGb4MBaWvGmjAcW8Zg3WLO3hgr94vPYWfXmyhK_a5uO-Twn7Uu_bJkvEBe7dhlQGBQ4nzAzTs3Q-7DySQKRDjTDwiUmU0o0WeFxoSkEZIC64ELUqWW5NQUaFzq5tYWQquheECdClNEjPghmqIDlF9PBnDEcI514yL3AIDZeGA0akEw1msJRdFBNQ0EV2uZ2ZCoXHHd_GWeYODppmTRuakkQVpNNH1aspHVWXjr8ENt_yrgWHlj3_vPkFbbnIVgXKK6vPpAs7Qpvmcv86m534PfQObhsCo |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwEA9jCvqizinOzzz4oGK3NG2T1rc5HBPnEFZkPpU0vYKgm-xD_32TNKt7EMG3EJJQcknvd5e7-yF0DmEu1FEIHQkec3yfSCf0pKvuVciFnweCmEpMz30-GISjUfRUQddlLgwAmOAzaOqmecvPJnKhXWUtpR19qlPH1wJfNYpsrdKjYigkDPktVQ1H4QpunzFdErXizsutjuTympRyZbRoAh2NJQKq2d5XdNIKyYrRMd3t_33dDtqyWBK3C-HXUAXGu6hmb-sMX9iS0pd1NKS4u4zCusFtPFQ_3y8xhZ9erMArNtm475NMLWqc-86S8wEbx2GRA4FtkfM9FHfv4k7PsXQKjtRGnsOiUAqXZGmaCQiAS8YVvGIky91UGYUhYSJV2snNc0YFk5SByLkMfBeoJAK8fVQdT8ZwgHBKhUtZqqBBqACBFBEHSV1fcMoyD4hsILLcz0TaUuOa8eItMSYHiRItjURLI7HSaKCrcspHUWfjr8F1vf3lQLvzh793n6GNXvzYT_r3g4cjtKkXKuJRjlF1Pl3ACVqXn_PX2fTUnKdv0FTD7w |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=2+Framework%3A+A+Software+Framework+for+Multimodal+Human-Computer+Interaction+Systems&rft.jtitle=IEEE+transactions+on+cybernetics&rft.au=Jie+Shen&rft.au=Pantic%2C+Maja&rft.date=2013-12-01&rft.pub=IEEE&rft.issn=2168-2267&rft.volume=43&rft.issue=6&rft.spage=1593&rft.epage=1606&rft_id=info:doi/10.1109%2FTCYB.2013.2271563&rft_id=info%3Apmid%2F24235258&rft.externalDocID=6634207 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2168-2267&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2168-2267&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2168-2267&client=summon |