Fire Hawk Optimizer with Deep Learning Enabled Human Activity Recognition

1  Introduction

Usability and Human-Computer Interaction (HCI) are main aspects of the system development processes for enhancing and improving system facilities and for satisfying necessities and needs of users [1]. HCI would support users, designers, and analysts in identifying the requirements of system from graphics, text style, color, fonts, and layout whereas usability would verify if the mechanism was easy to use, efficient, easy to learn, utility, easy to evaluate, easy to remember, and safe practical visible and offer job satisfaction to users [2]. HCI becomes a sub-field in computer science that deals with the study of the communication among people computers and users and the implementation, design, and evaluation of user interfaces for computer systems that were receptive to the user’s habits and needs. It was a multidisciplinary domain, that involves design, computer science, and behavioral sciences. The main goal of HCI was to make computer systems user-friendly and highly usable [3]. Users communicate with computer system via user interfaces, that has software and hardware which offers means of input, permitting users for manipulating the output and system, permitting the system for providing data to the end-user [4]. The evaluation, design, and application of interfaces was been main aim of HCI. It can be identified in HCI whose good interface model presupposes a better theory or method of HCI, and that’s why a theory must depend in large part on a theory of human cognition for modelling the cognitive process of users communicating with computer systems [5]. Different fields of HCI are displayed in Fig. 1.

Figure 1: Different fields involved in HCI

Human activity recognition (HAR) serves an important role in human-to-human communication and inter-personal relationships. The reason behind offering information regarding the identity of an individual, their psychological state, and personality, it becomes tough for extraction [6]. The human capacity for recognizing another individual’s actions becomes one chief subjects of research in the scientific regions of machine learning (ML) and computer vision (CV). So, in this research, many applications, which include robotics, video surveillance systems, and human-computer interaction for human behavior characterization, need multiple activity recognition mechanisms [7].

This article develops a Fire Hawk Optimizer with Deep Learning Enabled Activity Recognition (FHODL-AR) on HCI driven usability. In the presented FHODL-AR technique, the input images are investigated for the identification of different human activities. For feature extraction, a modified SqueezeNet model is introduced by the inclusion of few bypass connections to the SqueezeNet among Fire modules. Besides, the FHO algorithm is utilized as a hyperparameter optimization algorithm, which in turn boosts the classification performance. To detect and categorize different kinds of activities, probabilistic neural network (PNN) classifier is applied. The experimental validation of the FHODL-AR technique is tested using benchmark datasets.

2  Related Works

Much prevailing research concentrates on feature extraction approaches due to the discriminative characteristics being significant to ensuring the generalizing ability of the HAR mechanism. There were 2 principal ways for extracting features from sensor-related data. One uses hand-crafted features related to the statistical knowledge; another one automatically derives features utilizing neural networks (NN) [8]. The derivation of meaningful hand-crafted features from time and frequency fields depends heavily on domain knowledge and human experience. Moreover, hand-crafted features were generally devised for a particular task and were not appropriate for more general tasks and atmospheres [9]. Deep learning (DL) advancements were broadly implemented due to DL methods could automatically extract high dimensional features and were not relies on field knowledge [10].

Ronald et al. [11] devised a convolutional neural architecture by ensembling transfer learning (TL) related multi-channel attention network. In this study, four convolutional neural networks (CNN) branches are utilized to make feature fusion related ensembling and in every branch, an attention method has been employed for extracting the contextual data from the feature map generated by prevailing pretrained methods. Lastly, the feature maps which is extracted from 4 branches concatenated, and put into fully-connected network for producing the final recognition output. Hirooka et al. [12] offer a new hybrid DL network for HAR which uses multi-modal sensor data, but this presented method was ConvLSTM pipeline which completely uses the data in every layer derived from the temporal domain. At last, a fully-connected layer and a softmax function were utilized for computing the probability of every class. Lv et al. [13] introduce a technique to recognize human actions utilizing skeleton data by RGB-D camera, called Kinect device. The HAR was learning in the CV field. In its application, the recognition of human activities is employed for image processing, sign language learning, surveillance of the elderly, and HCI. This technique depends upon skeleton data having coordinate value of every joint in human body, which is categorized utilizing SVM method when executing a movement for predicting the name of activities.

Komang et al. [14] modelled a new DL-related architecture for prediction and recognition of human activities related to a hybrid method. The foremost contribution of this work was to devise a novel hybrid architecture, compiling 4 wide-ranging pretrained network methods in an optimized way, utilizing a meta-heuristic technique. Yilmaz et al. [15] project a robust classifier method for HAR utilizing bidirectional long short-term memory (BiLSTM), wearable sensor data, and a hybrid of CNN can be leveraged. The devised multibranch CNN-BiLSTM network performs automated feature extraction from raw sensor data having minimum data preprocessing. The utility of BiLSTM and CNN forms the model to learn local features along with long-term dependencies in sequential data.

3  The Proposed Model

In this article, a new FHODL-AR technique has been developed for activity recognition on HCI driven usability. In the presented FHODL-AR technique, the input images are investigated for the identification of different human activities. The overall block diagram is shown in Fig. 2. Initially, the input data is pre-processed and then the features are derived using the modified SqueezeNet model. Next, the recognition process take place using the PNN. Lastly, the FHO algorithm is used for parameter optimization.

Figure 2: Block diagram of FHODL-AR model

3.1 Feature Extraction Using Modified SqueezeNet

For feature extraction, a modified SqueezeNet model is introduced by the inclusion of few bypass connections to the SqueezeNet among Fire modules [16]. The SqueezeNet is a small CNN structure that employs less parameter when preserving competitive performance. Various approaches are applied based on CNN to develop the SqueezeNet: (1) substitute 3 × 3 filters with 1 × 1 filter, (2) reduce the input channel number to 3 × 3 filters, and (3) down-sample in the network such that the convolutional layer has larger activation map. The SqueezeNet chiefly encompassed of Fire module that is squeeze convolutional layer with 1 × 1 filter. Then, these layer is fed into an extended layer that has a mixture of 1 × 1 and 3 × 3 convolutions. Here, a small CNN named SqueezeNet is applied that could accomplish more efficiency than other CNN architectures namely AlexNet when requiring less parameter that is useful in realtime scenarios. The fundamental SqueezeNet initiates by a convolutional layer and eight Fire modules that end with other convolutional layers. The filter count of each Fire module is progressively improved from the start to the ending of the network. The max pooling with a stride of 2 is implemented afterward layers conv1, Fire4, Fire8, and conv10. Rectified linear unit (ReLU) is adapted as the activation function and Dropout using a ratio of 0.5 is utilized afterward the Fire9 modules.

To increase the detection performance, an adapted SqueezeNet is developed by including bypass connection to the SqueezeNet among Fire modules. In this framework, bypass connection is additional nearby Fire modules 3, 5, 7, and 9, necessitating the module to learn a residual function among inputs and outputs. To perform a bypass connection around Fire3, we fixed the input to Fire4 equivalent to the output of Fire2 + output of Fire3, whereby the + operator is a component-wise calculation. These variations of the regularization employed to the parameter of the Fire module and, according to ResNet, might enhance the concluding accurateness or trainability.

3.2 Hyperparameter Tuning Using FHO Algorithm

Here, the FHO algorithm is utilized as a hyperparameter optimization algorithm, which in turn boosts the classification performance [17]. The FHO meta-heuristic approach stimulates the foraging behaviors of fire hawks that consider the procedure of catching prey, and setting and spreading fires. Firstly, a solution candidate (X) is defined by the location vector of the prey and fire hawks. An initial process is applied for identifying the first location of this vector in the searching region.

x=[X1X2⋮Xi⋮XN]

[x11x12…x1j…x1dx21x22…x2J…x2d⋮⋮…⋱…⋮xi1xi2…xij…xid⋮⋮…⋱…⋮xN1xN2…xNj…xNd],{i=1,2,…N.j=1,2,…d. (1)

xij(0)=xi,minj+rand.(xi,maxj−xi,minj),{i=1,2,…N.j=1,2,…d. (2)

Now, Xi denotes the i - th candidate solution in the searching region; d indicates the dimension vector; N denotes the overall amount of candidate solution in the searching region; xij represent the j - th decision parameter of the i - th candidate solution; xij(0) characterizes the initialized location of the candidate solution; xi,minj and xi,maxj represents the minimal and maximal limits of the j - th decision parameter for the i - th candidate solution; rand is a random number distributed uniformly within zero and one. Here, global optimal solution is considered as the major fire that is firstly applied by the Fire Hawks to b , the schematic representation of the aspect is given that is arithmetically given in the following:

PR=[PR1PR2⋮PRk⋮PRm]k=1,2,…m, (3)

FH=[FH1FH2⋮FHl⋮FHn]L=1,2,…n, (4)

Let PRk be the k - th prey in the searching space w.r.t. overall amount of m preys; and FH1 denotes the l - th fire hawk considered an overall amount of n fire hawks in the searching region. Then, the overall distance between the prey and the Fire Hawks is evaluated.

Dk1=(x2−x1)2+(y2−y1)2,{l=1,2,……n.k=1,2,……m., (5)

In Eq. (5), Dk1 denotes the overall distance between l - th fire hawk and the k - th prey; m indicates the overall amount of prey in the searching domain; n represents the overall amount of fire hawks in the searching domain, and (x1,y1) and (x2,y2) characterize the coordinate of Fire Hawks and prey in the searching domain. In the meantime, birds are eager to apply the burning sticks from Fire Hawk territory; for that reason, those behaviors are applied as location update procedure in the major searching loop of FHO, as follows:

FH1new=FH1+(r1×GB−r2×FHNear),l=1,2,,n, (6)

In Eq. (6), FH1new indicates the novel location vector of l - th Fire Hawks (FH1) ; GB denotes the global optimal solution in the searching domain assumed as the main fire; FHNear denotes the other Fire Hawks in the searching domain; and r1 and r2 are random numbers distributed uniformly within (0,1) to define the movement of Fire Hawk towards the major fire and the other Fire Hawks’ territory.

Then, the motion of prey inside the territory of Fire Hawk assumed a major aspect of animal activities for the location updating method.

PRqnew=PRq+(r3×FH1−r4×SP1),{l=1,2,…n.q=1,2,…r. (7)

In Eq. (7), PRqnew indicates the novel location vector of qth prey (PRq) enclosed by l - th Fire Hawk (FH1) ; GB shows the global optimal solution in the searching domain assumed as major fire; SP1 indicate a safer position under l - th Fire Hawk territory; and r3 and r4 are random numbers distributed uniformly with (0,1) to define the motion of prey towards the Fire Hawk and the safer position.

In addition, the prey motion toward the other Fire Hawk territories while there is a chance the prey might be approaching the Fire Hawk in the nearby ambushes or hide in a safe position outside the Fire Hawk territories where they are trapped:

PRqnew=PRq+(r5×FHAlter−r6×SP),{l=1,2,…r.q=1,2,…n. (8)

In Eq. (8), PRqnew denotes the novel location vector of qth prey (PRq) enclosed by the l - th fire hawk (FH1);FHAlter denotes the fire hawk in the searching domain; SP indicates a safer potion outside the l - th Fire Hawk territories; r5 and r6 are random numbers distributed uniformly within (0,1) to define the movement of prey towards the Fire Hawk and the safer location outside the territory and it is mathematically expressed in the following:

SP1=∑q=1r⁡PRqr,{q=1,2,…r.l=1,2,…n. (9)

SP=∑k=1m⁡PRkm,k=1,2,…m. (10)

Now, PRq indicate the qth prey bounded by the l - th fire hawk (FH1);PRk denotes the k - th prey in the searching domain.

3.3 Activity Recognition Using PNN

To detect and categorize different kinds of activities, PNN classifier is applied [18]. The PNN is an effective neural network that is commonly employed for object classification. Usually, PNN is more accurate and faster when compared to the multiple layer perceptron network, and is comparatively impervious to the outlier. PNN makes use of the Parzen estimator to estimate the probability distribution function (PDF) of all the classes. The multi-variate Bayesian rule is employed to distribute the class with the maximum posterior possibility to original input dataset. The architecture of PNN is discussed in the following. Given that f1(x) and f2(x) are PDF related to p-dimension input vector X for π1 and π2 , the previous probability and misclassification cost ratios are formulated below

X∈{π1f1(x)/f2(x)≥{[C(1|2)/C(2|1)]×[P2/P1]}π2 (11)

In Eq. (5), C(i|j) denotes the misclassification cost (single object is categorized into πi however, it belongs to πj. Pj denotes the previous possibility of existence of population πi . The PDF is applied for estimating the posterior possibility that x belongs to class πi . The PDF in PNN can be resolved using the Bayesian classification with Parzen estimator and it is formulated in the following equation:

fA(X)=1(2π)p/2bp×1m×∑i=1m⁡exp[−(X−XAi)T(X−XAi)2b2] (12)

In Eq. (6), i indicates the pattern count, m shows the overall amount of training patterns, XAi denotes i - th training pattern from πA, and p denotes the dimension of input space. fA(X) be able to estimate smooth density function. There exist 4 layers in PNN: output, input, pattern, and summation layers.

The input unit is distribution unit that supplies a similar input value to the pattern unit and creates a dot product of pattern vector X using weight vector Wi(Zi=X∗Wi) . A non-linear function (exp[(Zi−1)/b2]) has been implemented on Z beforehand transmitting the activation level towards the summation layer. Consider that X and Wi are standardized towards a unit length, the non-linear function is formulated below

NO∼exp[−(X−Wi)T(X−Wi)2b2] (13)

Then, the non-linear function is a similar method to a Parzen estimator with a Gaussian kernel. The summation units sum the output for pattern unit respective to the class and evaluate the PDF. The output pattern applied the maximum vote to forecast the target class. Because the input layer is applied to the connection weight, PNN doesn’t require altering the connection weight. As a result, the training speed is quicker when compared to the conventional backpropagation neural network (BP-NN).

4  Results and Discussion

This section inspects the HAR outcomes of the FHODL-AR model on two datasets. The first KTH dataset (http://www.nada.kth.se/cvap/actions/) comprises 600 samples under six class labels. Table 1 reports the details related to the KTH dataset. Next, the UCF Sports dataset comprises 1000 samples under ten classes (http://crcv.ucf.edu/data/UCF_Sports_Action.php). The details relevant to the UCF Sports dataset are given in Table 2.

The confusion matrices produced by the FHODL-AR model on KTH dataset are portrayed in Fig. 3. The results indicated that the FHODL-AR model has proficiently recognized all the class labels.

Figure 3: Confusion matrices of FHODL-AR model on KTH dataset

Table 3 exhibits detailed HAR outcomes of the FHODL-AR model under distinct aspects of KTH dataset. The results implied that the FHODL-AR model has shown enhanced outcomes. With 80% of TR data, the FHODL-AR model has obtained average accuy of 99.31%, precn of 98%, recal of 97.90%, specy of 99.58%, and Fscore of 97.91%. Likewise, With 20% of TR dataset, the FHODL-AR method has attained average accuy of 99.44%, precn of 98.52%, recal of 98.45%, specy of 99.66%, and Fscore of 98.47%. Ultimately, With 70% of TR data, the FHODL-AR technique has achieved average accuy of 97.86%, precn of 93.54%, recal of 93.52%, specy of 98.72%, and Fscore of 93.47%. Meanwhile, With 30% of TR data, the FHODL-AR approach has gained average accuy of 99.07%, precn of 97.11%, recal of 97.10%, specy of 99.45%, and Fscore of 97.08%.

The training accuracy (TA) and validation accuracy (VA) achieved by the FHODL-AR algorithm on KTH dataset is displayed in Fig. 4. The experimental outcome denoted the DSOCDBN-STC system has established highest values of TA and VA. Especially, the VA is larger than TA.

Figure 4: TA and VA of FHODL-AR model on KTH dataset

The training loss (TL) and validation loss (VL) acquired by FHODL-AR process on KTH dataset are shown in Fig. 5. The experimental outcome implicit the FHODL-AR technique has attained minimum values of TL and VL. Especially, the VL is smaller than TL.

Figure 5: TL and VL of FHODL-AR model on KTH dataset

Table 4 and Fig. 6 exhibit a comprehensive accuy examination of the FHODL-AR model with other HAR models on KTH dataset. The experimental outcomes implied that the DBN and GRU models have shown poor performance with lower accuy values of 97.72% and 97.05% respectively. Next, the Bi-LSTM model has attained slightly enhanced accuy of 98.19%. Then, the RNN and CNN models demonstrated reasonable accuy of 98.62% and 98.60% respectively. However, the FHODL-AR model has accomplished superior accuy of 99.44%.

Figure 6: Acccy analysis of FHODL-AR model with existing models on KTH dataset

The confusion matrices produced by the FHODL-AR technique on UCF Sports data are indicated in Fig. 7. The outcomes portrayed that the FHODL-AR approach has proficiently identified each class label.

Figure 7: Confusion matrices of FHODL-AR model on UCF Sports dataset

Table 5 displays comprehensive HAR outcomes of the FHODL-AR method under distinct aspects of UCF sports dataset. The results inferred that the FHODL-AR technique has shown better results. With 80% of TR dataset, the FHODL-AR approach has acquired average accuy of 99.22%, precn of 96.10%, recal of 96.10%, specy of 99.57%, and Fscore of 96.09%. Likewise, With 20% of TR dataset, the FHODL-AR algorithm has attained average accuy of 99.10%, precn of 94.68%, recal of 95.33%, specy of 99.51%, and Fscore of 94.94%. Eventually, With 70% of TR dataset, the FHODL-AR process has achieved average accuy of 98.63%, precn of 93.31%, recal of 93.11%, specy of 93.24%, and Fscore of 93.14%. Meanwhile, With 30% of TR dataset, the FHODL-AR system has acquired average accuy of 98.67%, precn of 93.62%, recal of 93.70%, specy of 99.26%, and Fscore of 93.56%.

Table 6 and Fig. 8 show a comprehensive accuy inspection of the FHODL-AR with other HAR techniques on UCF sports data. The experimental outcome implies that the Bi-LSTM and RNN approaches have shown poor performance with lesser accuy values of 97.01% and 97.60% correspondingly.

Figure 8: Acccy analysis of FHODL-AR model with existing models on UCF Sport dataset

Then, the DBN technique has accomplished somewhat better accuy of 97.74%. Next, the GRU and CNN algorithms have established reasonable accuy of 98.21% and 98.26% correspondingly. However, the FHODL-AR system has obtained greater accuy of 99.10%. Therefore, the FHODL-AR method can be employed as a powerful mechanism for activity recognition.

5  Conclusion

In this article, a new FHODL-AR technique has been developed for activity recognition on HCI driven usability. In the presented FHODL-AR technique, the input images are investigated for the identification of different human activities. For feature extraction, a modified SqueezeNet model is introduced by the inclusion of few bypass connections to the SqueezeNet among Fire modules. Besides, the FHO algorithm is utilized as a hyperparameter optimization algorithm, which in turn boosts the classification performance. To detect and categorize different kinds of activities, PNN classifier is applied. The experimental validation of the FHODL-AR technique is tested using benchmark datasets, and the outcomes reported the improvements of the FHODL-AR technique over other recent approaches with maximum accuracy of 99.10%. In the future, the detection performance of the FHODL-AR technique can be boosted by the use of ensemble fusion approaches.

Funding Statement: The authors received no specific funding for this study.

Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.