Being able to predict and recognize human activities is an essential element for us to effectively communicate with other humans during our day to day activities. A system that is able to do this has a number of appealing applications, from assistive robotics to health care and preventative medicine. Previous work in supervised video-based human activity prediction and detection fails to capture the richness of spatiotemporal data that these activities generate. Convolutional Long short-term memory (Convolutional LSTM) networks are a useful tool in analyzing this type of data, showing good results in many other areas. This thesis’ focus is on utilizing RGB-D Data to improve human activity prediction and recognition. A modified Convolutional LSTM network is introduced to do so. Experiments are performed on the network and are compared to other models in-use as well as the current state-of-the-art system. We show that our proposed model for human activity prediction and recognition outperforms the current state-of-the-art models in the CAD-120 dataset without giving bounding frames or ground-truths about objects.

This book provides a unique view of human activity recognition, especially fine-grained human activity structure learning, human-interaction recognition, RGB-D data based action recognition, temporal decomposition, and causality learning in unconstrained human activity videos. The techniques discussed give readers tools that provide a significant improvement over existing methodologies of video content understanding by taking advantage of activity recognition. It links multiple popular research fields in computer vision, machine learning, human-centered computing, human-computer interaction, image classification, and pattern recognition. In addition, the book includes several key chapters covering multiple emerging topics in the field. Contributed by top experts and practitioners, the chapters present key topics from different angles and blend both methodology and application, composing a solid overview of the human activity recognition techniques.

This book offers a systematic, comprehensive, and timely review on V-HAR, and it covers the related tasks, cutting-edge technologies, and applications of V-HAR, especially the deep learning-based approaches. The field of Human Activity Recognition (HAR) has become one of the trendiest research topics due to the availability of various sensors, live streaming of data and the advancement in computer vision, machine learning, etc. HAR can be extensively used in many scenarios, for example, medical diagnosis, video surveillance, public governance, also in human–machine interaction applications. In HAR, various human activities such as walking, running, sitting, sleeping, standing, showering, cooking, driving, abnormal activities, etc., are recognized. The data can be collected from wearable sensors or accelerometer or through video frames or images; among all the sensors, vision-based sensors are now the most widely used sensors due to their low-cost, high-quality, and unintrusive characteristics. Therefore, vision-based human activity recognition (V-HAR) is the most important and commonly used category among all HAR technologies. The addressed topics include hand gestures, head pose, body activity, eye gaze, attention modeling, etc. The latest advancements and the commonly used benchmark are given. Furthermore, this book also discusses the future directions and recommendations for the new researchers.

Learn How to Design and Implement HAR Systems The pervasiveness and range of capabilities of today’s mobile devices have enabled a wide spectrum of mobile applications that are transforming our daily lives, from smartphones equipped with GPS to integrated mobile sensors that acquire physiological data. Human Activity Recognition: Using Wearable Sensors and Smartphones focuses on the automatic identification of human activities from pervasive wearable sensors—a crucial component for health monitoring and also applicable to other areas, such as entertainment and tactical operations. Developed from the authors’ nearly four years of rigorous research in the field, the book covers the theory, fundamentals, and applications of human activity recognition (HAR). The authors examine how machine learning and pattern recognition tools help determine a user’s activity during a certain period of time. They propose two systems for performing HAR: Centinela, an offline server-oriented HAR system, and Vigilante, a completely mobile real-time activity recognition system. The book also provides a practical guide to the development of activity recognition applications in the Android framework.

Traditional computer vision algorithms try to understand the world using visible light cameras. However, there are inherent limitations of this type of data source. First, visible light images are sensitive to illumination changes and background clutter. Second, the 3D structural information of the scene is lost when projecting the 3D world to 2D images. Recovering the 3D information from 2D images is a challenging problem. Range sensors have existed for over thirty years, which capture 3D characteristics of the scene. However, earlier range sensors were either too expensive, difficult to use in human environments, slow at acquiring data, or provided a poor estimation of distance. Recently, the easy access to the RGBD data at real-time frame rate is leading to a revolution in perception and inspired many new research using RGBD data. I propose algorithms to detect persons and understand the activities using RGBD data. I demonstrate the solutions to many computer vision problems may be improved with the added depth channel. The 3D structural information may give rise to algorithms with real-time and view-invariant properties in a faster and easier fashion. When both data sources are available, the features extracted from the depth channel may be combined with traditional features computed from RGB channels to generate more robust systems with enhanced recognition abilities, which may be able to deal with more challenging scenarios. As a starting point, the first problem is to find the persons of various poses in the scene, including moving or static persons. Localizing humans from RGB images is limited by the lighting conditions and background clutter. Depth image gives alternative ways to find the humans in the scene. In the past, detection of humans from range data is usually achieved by tracking, which does not work for indoor person detection. In this thesis, I propose a model based approach to detect the persons using the structural information embedded in the depth image. I propose a 2D head contour model and a 3D head surface model to look for the head-shoulder part of the person. Then, a segmentation scheme is proposed to segment the full human body from the background and extract the contour. I also give a tracking algorithm based on the detection result. I further research on recognizing human actions and activities. I propose two features for recognizing human activities. The first feature is drawn from the skeletal joint locations estimated from a depth image. It is a compact representation of the human posture called histograms of 3D joint locations (HOJ3D). This representation is view-invariant and the whole algorithm runs at real-time. This feature may benefit many applications to get a fast estimation of the posture and action of the human subject. The second feature is a spatio-temporal feature for depth video, which is called Depth Cuboid Similarity Feature (DCSF). The interest points are extracted using an algorithm that effectively suppresses the noise and finds salient human motions. DCSF is extracted centered on each interest point, which forms the description of the video contents. This descriptor can be used to recognize the activities with no dependence on skeleton information or pre-processing steps such as motion segmentation, tracking, or even image de-noising or hole-filling. It is more flexible and widely applicable to many scenarios. Finally, all the features herein developed are combined to solve a novel problem: first-person human activity recognition using RGBD data. Traditional activity recognition algorithms focus on recognizing activities from a third-person perspective. I propose to recognize activities from a first-person perspective with RGBD data. This task is very novel and extremely challenging due to the large amount of camera motion either due to self exploration or the response of the interaction. I extracted 3D optical flow features as the motion descriptor, 3D skeletal joints features as posture descriptors, spatio-temporal features as local appearance descriptors to describe the first-person videos. To address the ego-motion of the camera, I propose an attention mask to guide the recognition procedures and separate the features on the ego-motion region and independent-motion region. The 3D features are very useful at summarizing the discerning information of the activities. In addition, the combination of the 3D features with existing 2D features brings more robust recognition results and make the algorithm capable of dealing with more challenging cases.

Defines the notion of an activity model learned from sensor data and presents key algorithms that form the core of the field Activity Learning: Discovering, Recognizing and Predicting Human Behavior from Sensor Data provides an in-depth look at computational approaches to activity learning from sensor data. Each chapter is constructed to provide practical, step-by-step information on how to analyze and process sensor data. The book discusses techniques for activity learning that include the following: Discovering activity patterns that emerge from behavior-based sensor data Recognizing occurrences of predefined or discovered activities in real time Predicting the occurrences of activities The techniques covered can be applied to numerous fields, including security, telecommunications, healthcare, smart grids, and home automation. An online companion site enables readers to experiment with the techniques described in the book, and to adapt or enhance the techniques for their own use. With an emphasis on computational approaches, Activity Learning: Discovering, Recognizing, and Predicting Human Behavior from Sensor Data provides graduate students and researchers with an algorithmic perspective to activity learning.

The book reports on the author’s original work to address the use of today’s state-of-the-art smartphones for human physical activity recognition. By exploiting the sensing, computing and communication capabilities currently available in these devices, the author developed a novel smartphone-based activity-recognition system, which takes into consideration all aspects of online human activity recognition, from experimental data collection, to machine learning algorithms and hardware implementation. The book also discusses and describes solutions to some of the challenges that arose during the development of this approach, such as real-time operation, high accuracy, low battery consumption and unobtrusiveness. It clearly shows that it is possible to perform real-time recognition of activities with high accuracy using current smartphone technologies. As well as a detailed description of the methods, this book also provides readers with a comprehensive review of the fundamental concepts in human activity recognition. It also gives an accurate analysis of the most influential works in the field and discusses them in detail. This thesis was supervised by both the Universitat Politècnica de Catalunya (primary institution) and University of Genoa (secondary institution) as part of the Erasmus Mundus Joint Doctorate in Interactive and Cognitive Environments.