Eye Tracking and Sentiment Analysis to evaluate user behavior and opinion

This dissertation concerns the use and integration of two different techniques, eye tracking and sentiment analysis, to improve our ability to extract information about human behavior and opinion. These techniques can be applied in several different fields in which it is desirable to be able to model behavior patterns or to conduct opinion mining using automated methods. While for some applications it is possible to ask information directly (for instance through a survey or an interview), in several situations this could either be not feasible or involve a high risk that questions could be misinterpreted, the answers may be deceptive, or the subject might not even know the answer. Eye tracking and sentiment analysis allow to obtain knowledge from different types of raw data, i.e. gaze position coordinates during visualization of a stimulus (eye tracking) and texts (sentiment analysis). However, there are several challenges related to the way in which data are collected, processed and analyzed. The main problem this thesis aims to address is how we can improve our ability to obtain knowledge on human behavior and opinion using eye tracking and sentiment analysis, and how these two methods can be integrated to address this task. Besides illustrating different studies in which we applied these two techniques to study the behavior of different types of users, we describe a new method to improve performance of sentiment analysis by leveraging eye tracking data. First, we focus on eye tracking and show how this technique can be used to identify aspects of web pages or digital flyers that might benefit of improvement, in order to provide a better user experience. We also show how eye tracking data can be useful to accomplish image classification tasks. Next, we apply sentiment analysis to understand how sentiment towards Italy shifted during the first phases of the COVID-19 outbreak by analyzing a large data set of tweets. We compare different sentiment analysis tools, identify a common breakpoint corresponding to the shift of sentiment scores and show that this change can serve as an early predictor of the evolution of stock exchange values. Finally, based on the hypothesis that the eye tracking technology can provide a substantial contribution to identify words that are able to attract more attention, and are thus potentially more relevant, we present a new dictionary that allows to perform sentiment analysis leveraging eye tracking data. We apply the Eye dictionary to the classification of different types of texts, showing that this tool is able to achieve a good performance, even when compared with dictionaries implementing a much higher number of words.