DeepTaste: Augmented Reality Gustatory Manipulation with GAN-Based Real-Time Food-to-Food Translation

We have been studying augmented reality (AR)-based gustatory manipulation interfaces and previously proposed a gustatory manipulation interface using generative adversarial network (GAN)-based real time image-to-image translation. Unlike three-dimensional (3D) food model-based systems that only chan...

Full description

Saved in:
Bibliographic Details
Published in:2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) pp. 212 - 223
Main Authors: Nakano, Kizashi, Horita, Daichi, Sakata, Nobuchika, Kiyokawa, Kiyoshi, Yanai, Keiji, Narumi, Takuji
Format: Conference Proceeding
Language:English
Published: IEEE 01.10.2019
Subjects:
Human centered computing Human computer interaction (HCI) Interaction paradigms Mixed / augmented reality Computing methodologies Computer graphics Graphics systems and interfaces Perception Computing methodologies Machine learning Machine learning approaches Neural networks
Image color analysis
Modulation
Real-time systems
Resists
Solid modeling
Three-dimensional displays
Visualization
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We have been studying augmented reality (AR)-based gustatory manipulation interfaces and previously proposed a gustatory manipulation interface using generative adversarial network (GAN)-based real time image-to-image translation. Unlike three-dimensional (3D) food model-based systems that only change the color or texture pattern of a particular type of food in an inflexible manner, our GAN-based system changes the appearance of food into multiple types of food in real time flexibly, dynamically, and interactively. In the present paper, we first describe in detail a user study on a vision-induced gustatory manipulation system using a 3D food model and report its successful experimental results. We then summarize identified problems of the 3D model-based system and describe implementation details of the GAN-based system. We finally report in detail the main user study in which we investigated the impact of the GAN-based system on gustatory sensations and food recognition when somen noodles were turned into ramen noodles or fried noodles, and steamed rice into curry and rice or fried rice. The experimental results revealed that our system successfully manipulates gustatory sensations to some extent and that the effectiveness seems to depend on the original and target types of food as well as the experience of each individual with the food.
DOI:10.1109/ISMAR.2019.000-1