Fusion of children’s speech and 2D gestures when conversing with 3D characters
Article dans une revue avec comité de lecture
Most existing multi-modal prototypes enabling users to combine 2D gestures and speech input are task-oriented. They help adult users solve particular information tasks often in 2D standard Graphical User Interfaces. This paper describes the NICE Andersen system, which aims at demonstrating multi-modal conversation between humans and embodied historical and literary characters. The target users are 10–18 years old children and teenagers. We discuss issues in 2D gesture recognition and interpretation as well as temporal and semantic dimensions of input fusion, ranging from systems and component design through technical evaluation and user evaluation with two different user groups. We observed that recognition and understanding of spoken deictics were quite robust and that spoken deictics were always used in multimodal input. We identified the causes of the most frequent failures of input fusion and suggest possible improvements for removing these errors. The concluding discussion summarises the knowledge provided by the NICE Andersen system on how children gesture and combine their 2D gestures with speech when conversing with a 3D character, and looks at some of the challenges facing theoretical solutions aimed at supporting unconstrained speech/2D gesture fusion.
Files in this item
Showing items related by title, author, creator and subject.
Impact of Expressive Wrinkles on Perception of a Virtual Character’s Facial Expressions of Emotions Communication avec acteCOURGEON, Matthieu; BUISINE, Stéphanie; MARTIN, Jean-Claude (Springer, 2009)Facial animation has reached a high level of photorealism. Skin is rendered with grain and translucency, wrinkles are accurate and dynamic. These recent visual improvements are not fully tested for their contribution to ...
Article dans une revue avec comité de lectureMARTIN, Jean-Claude; NIEWIADOMSKI, Radoslaw; DEVILLERS, Laurence; BUISINE, Stéphanie; PELACHAUD, Catherine (World Scientific Publishing, 2006)One of the challenges of designing virtual humans is the definition of appropriate models of the relation between realistic emotions and the coordination of behaviors in several modalities. In this paper, we present the ...
The effects of speech–gesture cooperation in animated agents’ behavior in multimedia presentations Article dans une revue avec comité de lectureBUISINE, Stéphanie; MARTIN, Jean-Claude (Oxford University Press (OUP), 2007)Until now, research on arrangement of verbal and non-verbal information in multimedia presentations has not considered multimodal behavior of animated agents. In this paper, we will present an experiment exploring the ...
Communication avec acteLANG, Armand; GAZO, Claude; SEGONDS, Frédéric; MANTELET, Fabrice; JEAN, Camille; GUEGAN, Jérôme; BUISINE, Stéphanie (Springer, 2019)The industry has quickly realized the importance of bringing creativity into product design. The industrial context requires robust and efficient methods and tools to access untapped sources of ideas. Furthermore, Additive ...
Augmented Design with Additive Manufacturing Methodology: Tangible Object-Based Method to Enhance Creativity in Design for Additive Manufacturing Article dans une revue avec comité de lectureLANG, Armand; SEGONDS, Frédéric; JEAN, Camille; GAZO, Claude; GUEGAN, Jérôme; BUISINE, Stéphanie; MANTELET, Fabrice (Mary Ann Liebert Inc, 2021)Additive manufacturing (AM) brings new design potential compared with traditional manufacturing. Nevertheless, traditional manufacturing knowledge remains embedded in the minds of designers and is a real cognitive barrier ...