DOI:10.32604/iasc.2023.026051
| Intelligent Automation & Soft Computing DOI:10.32604/iasc.2023.026051 | |
| Article |
Student’s Health Exercise Recognition Tool for E-Learning Education
1Department of Humanities and Social Science, Al Ain University, Al Ain, 15551, UAE
2Department of Computer Science, Air University, Islamabad, 44000, Pakistan
3Department of Computer Science and Software Engineering, United Arab Emirates University, Al Ain, 15551, UAE
4Department of Computer Science, College of Computer, Qassim University, Buraydah, 51452, Saudi Arabia
5Department of Computer Science and Software Engineering, Al Ain University, Al Ain, 15551, UAE
6Department of Computer Engineering, Korea Polytechnic University, Siheung-si, Gyeonggi-do, 237, Korea
*Corresponding Author: Jeongmin Park. Email: jmpark@kpu.ac.kr
Received: 14 December 2021; Accepted: 27 January 2022
Abstract: Due to the recently increased requirements of e-learning systems, multiple educational institutes such as kindergarten have transformed their learning towards virtual education. Automated student health exercise is a difficult task but an important one due to the physical education needs especially in young learners. The proposed system focuses on the necessary implementation of student health exercise recognition (SHER) using a modified Quaternion-based filter for inertial data refining and data fusion as the pre-processing steps. Further, cleansed data has been segmented using an overlapping windowing approach followed by patterns identification in the form of static and kinematic signal patterns. Furthermore, these patterns have been utilized to extract cues for both patterned signals, which are further optimized using Fisher’s linear discriminant analysis (FLDA) technique. Finally, the physical exercise activities have been categorized using extended Kalman filter (EKF)-based neural networks. This system can be implemented in multiple educational establishments including intelligent training systems, virtual mentors, smart simulations, and interactive learning management methods.
Keywords: E-learning; exercise recognition; online physical education; student’s healthcare
E-learning has become a necessary part of our daily lives in this digital era, where information is flowing from minds and books to virtually everywhere through online systems. Automated systems have become more progressive in the last couple of decades as compared to offline learning methods. Given the demands of online education, physical education has also become an important aspect of young learners’ health training. Machine learning is a field altering the e-learning domain by providing adaptive e-learning algorithms and techniques for robust learning outcomes [1]. SHER is a part of the e-learning domain and has also become a challenging task to be investigated by researchers [2]. SHER consists of different physical actions including daily routine work, walking, jogging, exercising, etc.
The focus of this research article is on the physical exercises conducted online for young students that have a few factors to be taken care of such as alike exercise postures can cause analogous signal peaks, low-quality systems giving poor results, and precisions are not up to the mark [3]. Therefore, this research idea has gained popularity and valuable utilization in many practical applications such as e-learning [4], human activity recognition [5], patterns recognition [6], smart physical education [7], and intelligent mentors [8]. In the online physical education (OPE) domain, a few researchers have contributed their work based on a variety of features and machine learning techniques being utilized. In this article, a unique method has been proposed for SHER using a modified Quaternion-based filter for raw data cleansing and EKF-based neural networks for classification. The modified Quaternion-based filter has been designed and proposed for raw inertial measurement unit (IMU) data cleansing. Further, a robust system has been designed for static and kinematic patterns identification and cues extraction based on two different domains. Moreover, FLDA technique has been utilized for the cues optimization and an EKF-based neural networks approach has been proposed for the classification of online physical activities. A publicly available dataset named enabling real-time mistake detection & corrective feedback for free-weights exercises (ERICA) [9] has been utilized as a benchmark for the proposed system and attained significant improvement in SHER over the other state-of-the-art methods.
The article has been organized as follows. Section 2 gives a brief description of related work been accomplished in the topic on hand. Then, Section 3 specifies the architecture flow of the proposed SHER system along with the details of each phase. Section 4 presents the performance assessment of the proposed process in the previous section. Additionally, Section 5 contributes towards a brief discussion about the proposed health exercise tool and its shortcomings. Lastly, Section 6 concludes the subject matter and provides a couple of forthcoming directions to get an improved system.
In this research, the SHER tool has been proposed over an exercise dataset. A large number of studies have already been carried to propose methodologies for OPE improvements. A prime concern is to detect the exercise type, intensity, and associated parameters. However, this section discusses the drawbacks of traditional systems and the benefits of well-designed systems.
2.1 OPE via Traditional Systems
Typically, the researchers proposed to place sensors at multiple parts of the human body, extract distinguished cues, and train those cues over some machine learning algorithm. For instance, authors in [10] proposed a similar model to acquire data from physical activities, pre-process it, and extract features. Then, they used features selection and a genetic algorithm to classify the physical activities. Reference [11] describes a similar method to gain raw sensor signals from human motion and heart. Next, multiple filters and sliding windows had been applied to segment filtered data. Then, multiple features were extracted and a feature-to-feature fusion had been applied to combine the extracted features. Multi-layer sequential forward selection had been proposed for further features dimensionality reduction followed by a Gaussian mixture model and Gaussian mixture regression to extract the keywords. These keywords were further provided to deep belief networks using restricted Boltzmann machines in order to classify physical activities. Another similar conventional model had been proposed in [12] by Khan et al. It extracted acceleration signals followed by noise reduction. Then, spectral entropy along with linear discriminant analysis had been calculated and neural networks were utilized for the classification. However, these customary techniques failed to provide desired performance, which will be addressed using the proposed SHER system.
2.2 OPE via Well-Designed Systems
There are also a few well-structured methods present in the literature that helped in achieving better OPE activity recognition than the conventional methods mentioned above. Morillo et al. in [13] explained about physical activity recognition method using smartphones, where they used an accelerometer to gain data and temporal windows followed by noise removal via filters. Next, they extracted features from time, Fourier transform, and frequency domains. Then, they utilized the Ameva algorithm for each variable in order to get intervals of each feature and its associated activity. Further, their system retrieved the relative probabilities of each activity to associate it with each interval’s class matrix. The system was sophisticated enough to help in attaining higher accuracies. Another well-designed model had been proposed in [14]. It included data acquisition, framing, and pre-processing stages. Next, a mixture of features had been extracted from the filtered data and combined together followed by a series of steps for features selection and pre-classification via grey wolf optimization. Finally, the decision trees, genetic algorithms, and support vector machines had been utilized for classification. The unique process aided in accomplishing good results. Therefore, it can be observed that a well-designed methodology is important to achieve good results for OPE. Although these systems provided good results but they lacked in a few perspectives, which is the focus of this research.
The proposed system comprises filtering, sliding windows segmentation [15], and patterns identification as initial phases of the pre-processing stage. After the patterns have been identified as static and kinematic, the cues for both patterns have been computed using cepstral coefficients and time domains. Next, FDA [16] has been applied to symbolize the optimized cues from both patterned data together. Lastly, an EKF using neural networks [17] has been applied to classify the OPE exercises. An overview of the proposed system has been presented in Fig. 1 via multiple stages.
Figure 1: The architecture flow diagram of the proposed SHER for OPE
3.1 Pre-Processing Students’ Exercise Data
The proposed model consists of three phases to pre-process raw data. A modified Quaternion-based IMU filter has been proposed for cleansing raw data. Further, the filtered data has been segmented and patterns have been identified as described below.
3.1.1 Modified Quaternion-Based IMU Filter
This study proposed a modified Quaternion-based filter for IMU data. Acceleration and gyroscope data have been taken from the ERICA dataset and a calibration stage has been proposed to remove missing values and biasness from the signals [18]. Then, an error correction phase has been introduced in order to get errors removed from the signals. Next, the data has been normalized using gradient descent and Quaternion techniques. Finally, the IMU sensors’ signals have been fused to get the normalized data. Tab. 1 elucidates the Quaternion-based IMU filter algorithm (QIFA) in detail.
To further pre-process the fused data, this article has proposed to utilize sliding overlapping windows [19] for the filtered data. Overlapping windows have been proposed to make sure there are no missed data. It helped in dividing the data into the windows of five seconds data each. Fig. 2 presents the sliding overlapping windows for IMU fused signal.
Figure 2: Sliding overlapping windows for IMU fused data
The five seconds segmented IMU data has further been used to get two types of patterns in order to pre-classify OPE activities of young learners. The two kinds of patterns are based on the activity signal strength i.e., static patterns and kinematic patterns. Pearson correlation coefficient has been utilized to determine the relationship between two variables [20] as;
where ai are the values of the first variable in a window,
Figure 3: Static and kinematic patterns based on pearson correlation coefficients (threshold = 0.1)
Static patterned category activities are those actions, which have a Pearson correlation coefficient below a defined level such as 0.1. The physical actions performed by learners include those actions that have few variations in the motion signals like IMU. Examples of such physical activities are exercises done while sitting or lying on the floor.
An activity is considered to be in the kinematic patterned category if the window’s Pearson correlation coefficient is above a certain threshold i.e., 0.1. Kinematic patterned activities are generally those physical actions, whose signals have more variations due to movement diversity such as running, jogging, jumping, etc.
Multi-domain cues have been extracted from the patterned activity signals. For kinematic patterned signals, Hilbert-Huang transform (HHT) and Teager energy cepstral coefficients (TECCs) have been extracted. However, for static patterned signals, linear prediction cepstral coefficients (LPCCs) and synchro-squeezing transform (SST) have been mined.
HHT is the new theory for IMU signal analysis, which helps to decompose data into multiple intrinsic mode functions by using the empirical mode decomposition method. It also represents the signals into energy distribution in the time-frequency domain known as the Hilbert spectrum. HHT works wonders for the non-stationary and non-linear data [21]. The raw data D(t) can be calculated as;
where, fi represents the ith IMF and rm is the residue. Furthermore, it can also measure the changes in signals using instantaneous frequencies and helps in calculating the cues of the OPE actions [22]. Fig. 4 explains the movement variations calculated through HHT.
Figure 4: HHT movement variations for a) Lateral raises and b) Biceps curls
3.3.2 Teager Energy Cepstral Coefficients
Teager energy operator is quite efficient in detecting disturbances in signals by calculating the energy of the signal [23]. It is used to estimate the true total energy of the signal that is utilized to extract TECCs. Teager energy operator (TEO) is defined as;
where, x is the sample window and n is the number of windows. TECCs are similar to mel-frequency cepstral coefficients [24] and consists of pre-processing, Gabor filterbank, TEO, framing, averaging, log, and discrete cosine transform along with cepstral mean subtraction phases [25]. Fig. 5 displays the effects of a TEO over the patterned signal window.
Figure 5: Effect of TEO over a patterned signal window
3.3.3 Linear Prediction Cepstral Coefficients
Linear prediction cepstral coefficients have been utilized to get the complex action patterns from the signal and its transfer function. The cepstrum has been able to show the rate of change for a variety of bands. The LPCCs have been extracted from linear prediction coefficients cnby using the recursive relationship between the predictor coefficients and the cepstral coefficients as;
where σ2 is the gain in linear prediction analysis, d is the number of LPCCs, and am is the linear prediction coefficient [26].
3.3.4 Synchro-Squeezing Transform
The SST is an analysis method in the time-frequency domain. It decomposes complex motion signals into time-varying oscillatory components [27]. First, the continuous wavelet transform has been calculated as;
where, WTx denotes the continuous wavelet transform of a signal window x(t), a is the scale factor, and b is the time translation factor. φ (t) is a mother wavelet and further instantaneous frequency has also been calculated. Finally, the SST has been represented as;
where, ai is the ith scale value and Δai = ai − ai−1. ωl represents the lth frequency point [28]. Fig. 6 gives a SST for a window of filtered and patterned inertial data.
Figure 6: SST for a window of inertial data
Codebook generation is important to symbolize the cues [29] further in order to maximize the between-activity variance and minimize the within-activity variance. This research method proposed FLDA technique for cues symbolization. FLDA is employed to extract the linear discriminating cues that categorized the physical actions performed in an e-learning environment [16]. Fig. 7 shows that by applying FLDA over the ERICA dataset, the system has been successful in finding the discriminating cues for all three OPE actions.
Figure 7: Optimized and segregated cues using FLDA technique
This model has proposed a unique method for the OPE activities classification. EKF has been used as an effective learning mechanism for training neural networks in a supervised learning environment. EKF is more efficient than other fine-tuning methods such as backpropagation because it considers the second-order information about the error surface that is important for the covariance matrix training [17]. Although EKF is more costly than backpropagation in terms of orders of magnitude, EKF keeps on updating the covariance matrix with the second-order information, which leads to chances of the training process coming to a full stop at a local minima point. Eq. (9) explains the EKF implementation reduced to formula as:
where, w is the vector for weights and thresholds, z is the output of the system at m window, (dm − zm) is the residual as the difference between expected and actual observation, and
Figure 8: Classification method using EKF [31] and neural networks
This section is further structured into three sub-sections. First, a benchmark dataset has been described in detail. Second, the experimental results and the analysis have been evaluated. Third, the state-of-the-art OPE healthcare tools have been compared to our proposed system’s performance.
In this paper, physical exercise data has been utilized so that the variety of physical actions have been taken into consideration for the system to be more precise. For effective results, a 5-fold cross-validation method has been utilized over the exercise data. The number of training samples have been set according to cross-validation in order to minimize reconstruction errors [32].
4.1.1 Experimental Data Setup: ERICA
For the experimentation of the proposed method, this research selected ERICA dataset. The dataset has been collected from the real-time online users who have used ERICA as a digital personal trainer. It consists of 33 participants between the ages of 20 to 40 years old. The self-rated expertise level varied from novice to expert where mostly participants rated themselves as intermediate leveled experts. There were four different sessions in the dataset. The session duration varied from 22 to 48 min each. The participants performed multiple exercises but the available dataset consists of biceps curls, lateral raises, and triceps extension [9]. Fig. 9 shows the exercises performed in the dataset.
Figure 9: Bottom and top positions for a) Biceps curls, b) Lateral raises, and c) Triceps extension
4.1.2 Classification Results and Analysis
The goal of this research work is to enhance the e-learning domain by proposing a state-of-the-art OPE system for young students. The system should be able to recognize the OPE actions performed correctly and point out the mistakes. Hence, neural networks using EKF technique for the fine-tuning has been proposed, which is computationally expensive but results are enhanced when compared to other well-structured systems. The proposed system achieved 86% accuracy rate giving 7% false-negative rate for Lateral raises, 17% for biceps curls, and 26% for triceps extension. Tab. 2 gives the details of the confusion matrix retrieved from the neural network and EKF implementation.
4.1.3 Proposed System Comparison with State-of-the-Art Methods
Other state-of-the-art models have been selected for peer comparison in order to prove the effectiveness and proficiency of the proposed OPE system. According to Tab. 3, Radhakrishnan et al. in [9] achieved 70% accuracy by using frequency-domain cues with temporal features. The authors have tried 2-s and 5-s windows in [33], where the 5-s windows gave the recognition accuracy of 61% only. Smartphone-based exercise training system COPDTrainer showed the lowest recognition rate for exercising as 77.5% in [34], which is below any other recognition rates in the proposed system. Leg muscles have been monitored using a pressure sensing mechanism in [35] that achieved an accuracy rate of 81.7% for gym exercises recognition. In a filter-based sensor fusion activity recognition system [36], the accuracy achieved using the Kalman filter is 84%. In [37], Crema et al. proposed a work consisting of a well-structured strategy for exercise classification achieving an 85% accuracy rate.
The research works described in Tab. 3 achieved good accuracies but there is a need for improvement for real-time OPE. The proposed technique achieves state-of-the-art performance by comparing it to the existing methods of sophisticated SHER systems. The system is well-designed for healthcare and can support multiple applications of the method such as live gym classes and kids’ OPE. However, the proposed system lacks performance evaluation over larger datasets. Moreover, other machine learning and deep learning techniques can help get better results for the proposed OPE activities recognition system. Also, the one serious issue is the lack of interaction between students and teacher in OPE classes, which is causing a deficiency in conveying the value of physical education among students. A solution is to provide live feedback to students using a sophisticated system. Fig. 10 shows a system providing live feedback.
Figure 10: Live feedback provided by the system for OPE
An innovative system for OPE activities has been proposed in this research article. The proposed approach refers to OPE recognition as a fundamental problem of this research work that is performed using multiple algorithms. Our proposed system is efficient in detecting the physical education exercises for e-learning, which is important for the OPE of students such as kindergarten kids. The proposed methodology consists of pre-processing stage via QIFA and sliding windowing approach. The system further identified static and kinematic patterns followed by cues extraction using HHT, TECCs, LPCCs, and SST. The cues are optimized using FLDA and OPE activities have been classified via EKF and neural network. The system can be applied to OPE classes of young learners, intelligent mentors, and interactive learning systems. Quick reaction and interaction are missing in online sessions, which is important for students in order to get more attentive and concerned about their health. Therefore, we will extend the proposed system in the future in order to provide live feedback to the learners. The students will get motivated and be able to grasp the value of physical education in an e-learning environment more effectively when they get live feedback for their OPE.
Acknowledgement: This research was supported by a Grant (2021R1F1A1063634) of the Basic Science Research Program through the National Research Foundation (NRF) funded by the Ministry of Education, Republic of Korea.
Funding Statement: The authors received no specific funding for this study.
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
 | This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |