Integration of Gesture Generation System Using Gesture Library with DIY Robot Design Kit

Conversational agents are expected to improve the quality of communication by adding gestures to the speech, and are considered to be a promising tool. Recent data-driven methods are capable of attaching gestures to arbitrary speech, but the output is still not in line with human intuition. Therefore, we propose a gesture transformation system that utilizes gesture types as intermediate information, based on the theory of psycholinguistics. We employ the gesture-first principle to create gesture clusters based on gesture similarities among imagistic gestures, one type of gesture to represent image-like motions, which are considered to represent important concepts in conversations. Since this system explicitly takes into account the gesture types recognized by a deep neural network (DNN) and the semantics of the sentence to select gestures, it is expected to output gestures that are more in line with human intuition than existing end-to-end systems that do not place these intermediate states. We prepared a DIY robot kit consisting of cheap parts so that conversational agents at home become available to ordinary users, and implemented the proposed gesture generation system on this robot. In order to evaluate the effectiveness of the conversational agent, we evaluated user impression when using various media for conversation and confirmed the advantage of using our agent.