A modular and scalable computational framework for interactive immersion into imaging data with a holographic augmented reality interface

•Framework for interactive immersion into imaging data.•3D/4D holographic visualization and interaction with a head mounted display.•Modular, expandable, scalable, and customizable framework.•Real-time segmentation of 4D MRI CINE with machine learning. Modern imaging scanners produce an ever-growing...

Full description

Saved in:
Bibliographic Details
Published in:Computer methods and programs in biomedicine Vol. 198; p. 105779
Main Authors: Velazco-Garcia, Jose D., Shah, Dipan J., Leiss, Ernst L., Tsekos, Nikolaos V.
Format: Journal Article
Language:English
Published: Ireland Elsevier B.V 01.01.2021
Subjects:
Augmented Reality
Computational framework
Human-cyber Interface
Image Processing, Computer-Assisted
Imaging, Three-Dimensional
Immersion
Interactive Immersion
Medical imaging
Rendering and visualization
User-Computer Interface
Human-cyber Interface
Computational framework
Interactive Immersion
Medical imaging
Augmented reality
Rendering and visualization
ISSN:0169-2607, 1872-7565, 1872-7565
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:•Framework for interactive immersion into imaging data.•3D/4D holographic visualization and interaction with a head mounted display.•Modular, expandable, scalable, and customizable framework.•Real-time segmentation of 4D MRI CINE with machine learning. Modern imaging scanners produce an ever-growing body of 3D/4D multimodal data requiring image analytics and visualization of fused images, segmentations, and information. For the latter, augmented reality (AR) with head-mounted displays (HMDs) has shown potential. This work describes a framework (FI3D) for interactive immersion with data, integration of image processing and analytics, and rendering and fusion with an AR interface. The FI3D was designed and endowed with modules to communicate with peripherals, including imaging scanners and HMDs, and to provide computational power for data acquisition and processing. The core of FI3D is deployed to a dedicated computational unit that performs the computationally demanding processes in real-time, and the HMD is used as a display output peripheral and an input peripheral through gestures and voice commands. FI3D offers user-made processing and analysis dedicated modules. Users can customize and optimize these for a particular workflow while incorporating current or future libraries. The FI3D framework was used to develop a workflow for processing, rendering, and visualization of CINE MRI cardiac sets. In this version, the data were loaded from a remote database, and the endocardium and epicardium of the left ventricle (LV) were segmented using a machine learning model and transmitted to a HoloLens HMD to be visualized in 4D. Performance results show that the system is capable of maintaining an image stream of one image per second with a resolution of 512 × 512. Also, it can modify visual properties of the holograms at 1 update per 16 milliseconds (62.5 Hz) while providing enough resources for the segmentation and surface reconstruction tasks without hindering the HMD. We provide a system design and framework to be used as a foundation for medical applications that benefit from AR visualization, removing several technical challenges from the developmental pipeline.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0169-2607
1872-7565
1872-7565
DOI:10.1016/j.cmpb.2020.105779