Leveraging cognitive computing for gender and emotion detection

In this paper we present a tool that performs two tasks: given an input image, first, (i) it detects whether the image corresponds to a male or female person and then (ii), it further recognizes which emotion the face expression of the detected person is conveying. We mapped the two tasks as multi-label classifications. The first one aims at identifying if the input image contains one of the following four categories: one male person, one female person, a group of both male and female persons or if the image does not contain any person. The second task is triggered whether the image has been recognized to belong to one of the first two classes and aims at detecting whether that image is conveying one among six different emotions: sadness, anger, surprise, happiness, disgust, fear. For both the problems, Microsoft Cognitive Services have been leveraged to extract tags from the input image. Tags are text elements that have been adopted to form the vectorial space model, using the bag of words model, that has been fed to the machine learning classifiers for the prediction tasks. For both tasks, we manually annotated 3000 images, which have been extracted from students who agreed using our system and providing their Facebook pictures for our analysis. Our evaluation uses Naive Bayes and Random Forest classifiers and with a 10-fold cross-validation reached satisfactory accuracies both for the two tasks and for the combination of them. Finally, our system works online and has been integrated with social media. In that way, any visitors logged in to Facebook through its APIs, is allowed to quickly classify any of their photos.