Automatic facial expression recognition (FER) and emotion recognition have aroused many researchers' interest in a variety of research fields because of an important role in human centered interfaces and the advent of cheap and powerful computer and video camera in the last decade. In addition, the emergence of the smartphones era has aroused considerable interest in the mobile application development in connection with facial expression and emotion recognition. However, in spite of the enhanced hardware of recent smartphones, mobile applications for processing real-time video should always consider limited resources available in smartphones. The limited processing resources in smartphones still make it difficult to directly adopt the existing facial expression and emotion recognition system from desktops. Most studies for FER have been carried out and evaluated under restricted experimental environment. For instance, some approaches deal with only static images or work with video sequences manually pre-segmented (temporally) for each expression. However, the temporal segmentation of expressions is the most essential element in automatic FER systems as real world applications for real-time video. Also, the real world dataset for FER is different from most conventional datasets which are mainly collected in a limited experimental environment. It is hard to apply models made with datasets collected under lab environment to real world application. The automatic FER should be capable of satisfying these various types of noisy datasets. We address several problems for real-time FER on low-power smartphones. First, we presents a real-time FER effectively running on smartphones. The system employs a set of Support Vector Machines (SVM) for neutral expression and 6 basic emotions with 13D geometric facial features including temporal information. We evaluated the performance of the proposed system in terms of speed and accuracy on offline dataset and commercial off-the-shelf smartphones. Second, we present a real-time temporal video segmenting approach for automatic FER applicable in a smartphone. The proposed system uses a Finite State Machine (FSM) for segmenting real time video into temporal phases from neutral expression to the peak of an expression. The system performs FER with SVM on every apex state after automatic temporal segmentation, without any sampling time delay. Third, we present gender-driven ensemble models for FER on smartphones working with a context-sensitive multimedia content recommendation system. Based on the fact that male and female express an emotion with a distinct difference in the horizontal and vertical facial movements, we employ the ensemble model with three weak classifiers trained by gender-specific subsets and a general dataset of facial expression. In the system, users receive feedback by links to multimedia contents such as videos, photos and e-books regarding a current user's emotion. Last, we present an approach using CNN model for FER to accommodate noisy images and videos dataset in real world environment. We adopt FER2013 dataset for training CNN model. We show the CNN model is able to work very well for expression recognition even with real, noisy data that is not used for training.

Master's Thesis from the year 2017 in the subject Engineering - Computer Engineering, grade: 10, , course: M.Tech-ECE, language: English, abstract: Images containing faces are essential to intelligent vision-based human computer interaction, and research efforts in face processing include face recognition, face tracking, pose estimation, and expression recognition. The rapidly expanding research in face processing is based on the premise that information about a user’s identity, state, and intent can be extracted from images and that computers can then react accordingly, e.g., by knowing person’s identity, person may be authenticated to utilize a particular service or not. A first step of any face processing system is registering the locations in images where faces are present. The local binary pattern is a simple yet very efficient texture operator which labels the pixels of an image by thresholding the neighborhood of each pixel and considers the result as a binary number. The LBP method can be seen as a unifying approach to the traditionally divergent statistical and structural models of texture analysis. Perhaps the most important property of the LBP operator in real-world applications is its invariance against monotonic gray level changes caused, e.g., by illumination variations. Another equally important is its computational simplicity, which makes it possible to analyze images in challenging real-time settings. The success of LBP in face description is due to the discriminative power and computational simplicity of the LBP operator, and the robustness of LBP to mono-tonic gray scale changes caused by, for example, illumination variations. The use of histograms as features also makes the LBP approach robust to face misalignment and pose variations. For these reasons, the LBP methodology has already attained an established position in face analysis research. Because finding an efficient spatiotemporal representation for face analysis from videos is challenging, most of the existing works limit the scope of the problem by discarding the facial dynamics and only considering the structure. Motivated by the psychophysical findings which indicate that facial movements can provide valuable information to face analysis, spatiotemporal LBP approaches for face, facial expression and gender recognition from videos were described.

This book constitutes the proceedings of the Third Workshop on Video Analytics for Audience Measurement, VAAM 2016, and the Second International Workshop on Face and Facial Expression Recognition from Real World Videos, FFER 2016, held at the 23rd International Conference on Pattern Recognition, ICPR 2016, in Cancun, Mexico, in December 2016. The 11 papers presented in this volume were carefully reviewed and selected from 13 submissions. They deal with: re-identification; consumer behavior analysis; utilizing pupillary response for task difficulty measurement; logo detection; saliency prediction; classification of facial expressions; face recognition; face verification; age estimation; super resolution; pose estimation; and pain recognition.

This book constitutes the thoroughly refereed conference proceedings of the International Workshop on Face and Facial Expression Recognition from Real World Videos in conjunction with the 22nd International Conference on Pattern Recognition held in Stockholm, Sweden, in August 2014. The 11 revised full papers were carefully reviewed and selected from numerous submissions and cover topics such as Face Recognition, Face Alignment, Facial Expression Recognition and Facial Images.

Computer vision algorithms for the analysis of video data are obtained from a camera aimed at the user of an interactive system. It is potentially useful to enhance the interface between users and machines. These image sequences provide information from which machines can identify and keep track of their users, recognize their facial expressions and gestures, and complement other forms of human-computer interfaces. Facial Analysis from Continuous Video with Applications to Human-Computer Interfaces presents a learning technique based on information-theoretic discrimination which is used to construct face and facial feature detectors. This book also describes a real-time system for face and facial feature detection and tracking in continuous video. Finally, this book presents a probabilistic framework for embedded face and facial expression recognition from image sequences. Facial Analysis from Continuous Video with Applications to Human-Computer Interfaces is designed for a professional audience composed of researchers and practitioners in industry. This book is also suitable as a secondary text for graduate-level students in computer science and engineering.

This book constitutes the proceedings of the Third Workshop on Face and Facial Expression Recognition from Real World Videos, FFER 2018, and the Second International Workshop on Deep Learning for Pattern Recognition, DLPR 2018, held at the 24th International Conference on Pattern Recognition, ICPR 2018, in Beijing, China, in August 2018. The 7 papers presented in this volume were carefully reviewed and selected from 9 submissions. They deal with topics such as histopathological images, action recognition, scene text detection, speech recognition, object classification, presentation attack detection, and driver drowsiness detection.

"In computer vision and image processing, object detection algorithms are used to detect semantic objects of certain classes of images and videos. Object detector algorithms use deep learning networks to classify detected regions. Unprecedented advancements in Convolutional Neural Networks (CNN) have led to new possibilities and implementations for object detectors. An object detector which uses a deep learning algorithm detect objects through proposed regions, and then classifies the region using a CNN. Object detectors are computationally efficient unlike a typical CNN which is computationally complex and expensive. Object detectors are widely used for face detection, recognition, and object tracking. In this thesis, deep learning based object detection algorithms are implemented to classify facially expressed emotions in real-time captured through a webcam. A typical CNN would classify images without specifying regions within an image, which could be considered as a limitation towards better understanding the network performance which depend on different training options. It would also be more difficult to verify whether a network have converged and is able to generalize, which is the ability to classify unseen data, data which was not part of the training set. Fast Region-based Convolutional Neural Network, an object detection algorithm; used to detect facially expressed emotion in real-time by classifying proposed regions. The Fast R-CNN is trained using a high-quality video database, consisting of 24 actors, facially expressing eight different emotions, obtained from images which were processed from 60 videos per actor. An object detector’s performance is measured using various metrics. Regardless of how an object detector performed with respect to average precision or miss rate, doing well on such metrics would not necessarily mean that the network is correctly classifying regions. This may result from the fact that the network model has been over-trained. In our work we showed that object detector algorithm such as Fast R-CNN performed surprisingly well in classifying facially expressed emotions in real-time, performing better than CNN."--Abstract.