Intent and Emotions in Image Search and Viewing

The talk is divided into two sections. In the first section, I will talk about behavioral and physiological responses associated with interest, curiosity, and novelty in image viewing. We recorded facial expressions, eye gaze and electrodermal responses of 50 participants viewing 80 images. Participants self-reported their level of interest, perceived novelty, complexity and comprehensibility of the images. We found that not everybody perceived interest similarly and pleasantness is the strongest indicator of interest. In the second study, we aimed at automatically recognizing a user’s intent for image search in the early stage of a search session. We designed seven different search scenarios under the intent conditions of finding items, re-finding items, and entertainment. We collected the same behavioral and physiological responses in addition to implicit user interactions from the same participants. Participants performed seven different search tasks on a custom-built image retrieval platform. Finally, we trained machine learning models to predict users’ search intentions from the visual content of the visited images, the user interactions, and the spontaneous responses. After fusing the visual and user interaction features, our system achieved the F-1 score of 0.722 for classifying three classes in a user-independent cross-validation. We found that eye gaze and implicit user interactions, including mouse movements and keystrokes are the most informative features in search intent prediction.