Sensors based Human Activity Recognition (HAR) have numerous applications in eHeath, sports, fitness assessments, ambient assisted living (AAL), human-computer interaction and many more. The human physical activity can be monitored by using wearable sensors or external devices. The usage of external devices has disadvantages in terms of cost, hardware installation, storage, computational time and lighting conditions dependencies. Therefore, most of the researchers used smart devices like smart phones, smart bands and watches which contain various sensors like accelerometer, gyroscope, GPS etc., and adequate processing capabilities. For the task of recognition, human activities can be broadly categorized as basic and complex human activities. Recognition of complex activities have received very less attention of researchers due to difficulty of problem by using either smart phones or smart watches. Other reasons include lack of sensor-based labeled dataset having several complex human daily life activities. Some of the researchers have worked on the smart phone’s inertial sensors to perform human activity recognition, whereas a few of them used both pocket and wrist positions. In this research, we have proposed a novel framework which is capable to recognize both basic and complex human activities using built-in-sensors of smart phone and smart watch. We have considered 25 physical activities, including 20 complex ones, using smart device’s built-in sensors. To the best of our knowledge, the existing literature consider only up to 15 activities of daily life.
Sensor based Human Activity Recognition (HAR) is an emerging field of machine learning, having several advantages as compared to the vision based human activity recognition [
The proposed sensor based human physical activity recognition is divided into four main steps. (1) data acquisition, (2) pre-processing, (3) feature extraction (4) classification of various activities. Firstly, we have formulated a dataset for more than 20 complex human physical activities using built-in sensors of smart phone and smart watch. The sensor data fusion is followed by the pre-processing stage which removes noise from raw data and divides it into windows or segments. Further the dataset is divided into test and train data. After this, feature extraction is performed by using the proposed feature extraction method. At the final stage, classification of various activities is done on the basis of feature extraction. The main contribution of this work is listed below:
The effect of two devices A dataset for several overlapped human basic and complex physical activities is formulated. These activities are recognized by using the proposed methodology. A machine learning approach is presented that provide better performance for the recognition of more than 25 human physical activities.
Human activity recognition is an important research area in computer vision [
Complex human activities overlapped with basic human activities leads towards the complexity of daily life activity recognition system. In order to overcome these complications, we have designed a mathematical model for multisensory learning and multilocational placement human activity recognition. The designed approach efficiently uses the data from smart phone and smart watch. This system provides advantages in terms of less computational time, reduction in installation cost, power efficiency, also the system is more appealing and contemporary than other conventional methods. The system effectively monitors the human activities for 24-h interval. The main objective of proposed system is to classify various complex human activities using built-in sensor of smartphone and smartwatch with better accuracy and less computational requirements. Sensing of human activities Pre-processing of sensor raw data Features extraction and selection Classification Algorithms.
For complex human activity recognition, selection and positioning of a sensor is a most challenging step. The appropriate placement of sensor on human body leads toward the better performance of human activity recognition system. The basic daily life activities
Multisensory approach Multilocational sensor placement Sensor data fusion
Data for 13 basic human activities from [
The application easily interfaces with the sensor for the recording and collection of data. Before recording sensor data, the application required the name and ID of participants in order to keep track of different participants. During the activity, the android device interacting with the application is placed at specific body position. The data has been collected as comma-separated values (CSV) file on the smart device which can be processed later. The system has also been tested for 14 overlapped basic and complex activities. Hence, final dataset comprises of more than 25 human activities. Each physical activity has been performed by multiple individuals for more than 60 s with known labels and timestamp. Fifty samples (50 Hz of each sensor output) have been collected for different human physical activities performed by multiple individuals. Thus, for each device that consist of four triaxial sensor, total (4 ∗ 3 = 12) dimensional data is collected at a single instant. Particularly, for one second time period against a single human activity, 50 sample of each 12-dimensional data is collected from each device. We have collected hundreds of samples against each human activity performed by single participant and finally series concatenate the samples of all participants. A big dataset of similar complex human activity leads to complexity and misclassification of recognition system. To overcome these barriers and forming an accurate activity recognition system, preprocessing and feature extraction techniques are used.
12-dimensional data has been collected from each device (smart phone, smart watch) with frequency of 50 Hz, where each sample of data is labeled and time stamp. For better understanding of complex human activities, we have collected the data from smart phone and smart watch at the same instant. Afterwards, concatenate the data from both devices in parallel. Hence, our sample space dimensions become 24 against each each human activity.
The human physical activity recognition system has not been classified directly using the raw data of sensor. The classification task has been performed by using structural data representation (features vector) obtained from several pre-processing and feature extraction techniques. Each sensor generates three time series, along x-axis, y-axis and z-axis. After preprocessing, the tri-dimensional (
The instantaneous rotational feature derived from orientation of device, like pitch (
Let
where β can be, θ, φ, α or or ū, the mean of the input feature over window, and n are number of sensors. Features 7–9 are components of the fast Fourier transform (FFT) over the window. The preceding steps provide us all the nine features of each sensor, hence, a set of 72 features for each time sample has been obtained. Feature space information has been given below. To find the time series components, the data from four sensors of smartphone
The vector dimension of X1 (t) is 12 in every sample of each human activity. Similarly, smartwatch sensors time series data is represented as;
After combining the data of sensors using both devices in time series, we obtain X(t).
where X (t) is 24-dimensional raw data in every sample of each human activity. Further, we have calculated rotational feature V (t)
Next 48 statistical features are obtained using windowing method over a small defined period we calculate variance in pitch
In next step we have calculated 24 frequency domain features
Smartphone sensor smart watch sensor | Smartphone sensor smart watch sensor |
---|---|
Accelerometer (x, y, z) Accelerometer (x, y, z) | Accelerometer (x, y, z) Accelerometer (x, y, z) |
Linear acceleration sensor (x, y, z) Linear acceleration sensor (x, y, z) | Linear acceleration sensor (x, y, z) Linear acceleration sensor (x, y, z) |
Gyroscope (x, y, z) Gyroscope (x, y, z) | Gyroscope (x, y, z) Gyroscope (x, y, z) |
Magneto meter (x, y, z) Magneto meter |
Magneto meter (x, y, z) Magneto meter |
4 Triaxial sensor of smartphone 4 Triaxial sensor of smartwatch | 4 Triaxial sensor of smartphone 4 Triaxial sensor of smartwatch |
1 Sample: 4 × 3 = 12-dimensional Data 1 Sample: 4 × 3 = 12-dimensional Data | 1 Sample: 4 × 3 = 12-dimensional Data 1 Sample: 4 × 3 = 12-dimensional Data |
· Both devices sensor data fusion: 12 × 2 = 24-dimensional (column) data | |
Add Label and Time Stamp: 24 + 2 = 26 column in one sample | |
—Total number of human physical activity classes: 26 | |
· Total number of samples: 23, 75, 713 | |
· Total sample space dimension is: [23, 75, 713 × 26] | |
· Calculate three orientation or rotational feature (Pitch (θ), Roll(φ), Magnitude of acceleration(α)) from each sensor | |
· Number of triaxial sensor = 4 (smartphone) + 4 (smartwatch) = 8(s1, s2, s3,…, s8) | |
· 8 (Sensor) ∗ 3 (features(θ, φ, α)) = 24 rotational features | |
· Calculate variance over Pitch (σ2θ), Roll (σ2φ), Magnitude of acceleration (σ2α) data against each sensor | |
· Number of triaxial sensor = 4(smartphone) + 4(smartwatch) = 8(s1, s2, s3,…, s8) | |
· 8(Sensor) × 3(variance features (σ2θ, σ2φ, σ2α)) = 24 Time domain statistical features | |
· Calculate absolute Dc component of FFT over (Pitch (fθ), Roll (fφ), Magnitude of acceleration (fα)) data against each sensor | |
· Number of triaxial sensor = 4(smartphone) + 4(smartwatch) = 8(s1, s2, s3,…, s8) | |
· 8 (Sensor) × 3 (basic component of FFT features (fθ, fφ, fα)) = 24 Frequency domain statistical features | |
Finally, the whole sensor data has been fused using
Hence, the whole dataset has been formed which is then utilized for classification process by splitting the data for training and testing process with ratio 8:2. Finally, different classification algorithms applied which classify the several human physical activities.
In literature, researchers have used different approaches to classify the human activities using sensor-based data. In this work, we have used Naive Bayes (NB), K-Nearest Neighbors (KNN) and Neural Network (NN) for classification.
To perform Naive-Bayes Classification, we split the data into two groups: 80% of the data is used for training of the classifier and remaining 20% is reserved as testing data. Let
If
This gives overall classes being compared with
Similarly, for multipoint, we have repeated the above procedure. We have calculated KNN Euclidean distance from mean value of each neighbor group. After re-assigning each datapoint to new class of minimum distance, we have calculated centroid of these neighbor groups. We have repeated the procedure until datapoints left with only few fixed numbers. The parameters of neural network (NN)
As we have explained earlier, four sensors of each device
Neural network having 2 hidden layers is implemented for activity recognition. The MATLAB pattern recognition (PR) neural network (NN) tool has been used. The configuration set as the number of given inputs and classified outputs are 72 and 26 respectively. The total of 72 inputs passed through first hidden layer of 75 neurons and second hidden layer of 50 neurons and lastly classify the 26 human activities. For classification purposes, Sigmoid activation function has been used which led to conjugate gradient training function and back propagation for error minimization.
The NN gather results for 500 epochs and it has been proven that NN with 2 hidden layer and KNN perform better than other algorithms. Accuracy of classifier in term of percentage is calculated as: Percentage Correct Classification = 100 ∗ (1 −
The reliable recognition of several human physical activities of daily life can be very helpful for many applications like eHealth, remote monitoring and tracking of human for awareness feedback, coaching, human machine interaction, bad habits motoring. etc. In this paper, a multisensory learning approach of complex human activity recognition is proposed that provide better recognition performance for a large number of complex human physical activities of daily life. Neural network (NN), Naive Bayes (NB), K-Nearest Neighbor (KNN) and Ensemble method AdaBoost classifiers are used along a proposed mathematical model of preprocessing and feature extraction. Neural network and KNN classifiers outperformed other classifiers. It is further concluded that the smart device based multisensory based human activity recognition is a cost effective and more practical solution rather than vision based or dedicated sensor-based approaches. Furthermore, in this work a new data set against 26 human physical activities of daily life is formulated using built-in sensors of smart phone and smart watch, that will be helpful for future research in this field. The data of smartphone and smartwatch for a large number of complex human physical activity will serve as a benchmark.