Breast Cancer Diagnosis Using Feature Selection Approaches and Bayesian Optimization

1  Introduction

Breast cancer (BC) have been considered as the most diagnosed malignant disease among females in recent years [1]. Abnormal growths of some cell in the breast tissue can cause breast cancer. Benign and malign tumors are abnormally growing cells in the breast tissue. Benign tumor is noncancerous which can be treated with medicine or surgery. However, malignant tumor shows the characteristics of cancer. When malignant tumor is not treated, it can rapidly spread to organs and increase the mortality rates. Therefore, early diagnosis of BC is a necessary step to reduce the mortality rates [2]. Methods such as radiologic imaging and pathological examinations have been used for early diagnosis [3]. Mammography is the most preferred imaging tool for early diagnosis [4]. However, the detection of suspicious breast tumors on mammogram images can be challenging. When mammography detects a suspicious breast tumor, breast biopsy is generally performed [5]. Although breast biopsies provide a definite diagnosis, the statistics show that most biopsies are benign [6,7]. Thus, many studies have been carried out for efficient early diagnosis and to minimize the unnecessary biopsy rates in the literature. Due to the high successful classification rates, machine learning algorithms have been promising approach for the prediction of BC in recent years [8,9]. Machine learning (ML) approaches are a type of artificial intelligence that allow software applications to obtain higher accuracy of predicting outcomes. These algorithms use past values as input to predict new outputs [10]. Feature selection (FS) and hyperparameter optimization (HO) are two critical issues to improve the classification rates of ML. FS is the process of determining the suitable features to be used in machine learnings approaches. These methods decrease overfitting, improve the performance of classification process [11,12]. HO is a process in finding optimal hyperparameters for the training of ML. Such hyperparameters need to be set carefully to provide excellent classification performance. These hyperparameters are very important when creating and evaluating machine learning algorithms [13]. Although using ML to predict BC has been an active field of research, there are a few studies combining FS and HO [14–16]. In view of this, this manuscript proposes a new BC classification model based on improved ML algorithms integrating FS and HO. First, this paper is designed to explore the effects of FS and HO on ML algorithms. Second, the best combined algorithm identified in this manuscript can be used in future studies for efficient diagnosis of BC.

The summary of this article and its contribution to science is given below:

1.    Decision Tree (DT), Naive Bayes (NB), Support Vector Machine (SVM), K-Nearest Neighbor (K-NN) and Ensemble Learners (EL) were used to classify malign and benign breast lesions.

2.    Three different feature selection methods namely, Relief (RF), LASSO and Sequential Forward Selection (SFS) were used to determine the most selective and discriminative features for effective identification of BC.

3.    Bayesian optimization (BO) algorithm was utilized for optimizing the classification algorithms.

4.    The statistical measures (accuracy, precision, recall and F1-Score) were used to measure the performance of suggested classification model which was implemented in MATLAB software.

5.    A series of comparative analyses are performed on the Wisconsin Breast Cancer Dataset (WBCD) which was retrieved from the UCI machine learning repository and Mammographic Breast Cancer Dataset (MBCD) which has never been used before.

The rest of the study is presented as follow: The literature studies are summarized in Section 2. The general structure of methods and methods are given in Section 3. The results are demonstrated in Section 4. The results are discussed in Section 5 and Section 6 presents the conclusion.

2  Literature Survey

Numerous studies have been investigated to provide early diagnosis of BC in recent years. There are various ML algorithms that are being used for this purpose [17,18]. Studies of BC prediction, SVM, DT, NB, K-NN and EL algorithms were comparatively used [10]. Asri et al. [19] compared several classification algorithms such as K-NN, DT, NB and SVM. The experiments were conducted on WBCD. The study achieved the highest accuracy of 97,13% using SVM. Khan et al. [20] proposed comparative analysis based on machine learning models for BC prediction. The experiments were conducted on WBCD. Among the various classifiers, Logistic regression achieved the best accuracy of 98%.

In the literature, many feature selection methods used to select optimal features. Three techniques are used in FS: filter, wrapper and embedded. [11]. Filter approaches use the statistical functions to choose and rank the feature subsets. Recently, Relief, Minimum-redundancy maximum relevancy and Fisher have been suggested as the most convenient filter methods due to simple ranking strategies of these algorithms [21]. Tian et al. [22] compared ten filter feature selection methods for BC prediction. They used the digital database for screening mammography where Relief with 6 features (AUC:0.855) showed good classification rates. Wrapper method uses a specific selection criterion to determine the quality of various feature subsets. Sequential forward and backward feature selection [23] and metaheuristic algorithms such as genetic algorithm [24] and particle swarm optimization [25] have been extensively used as wrapper algorithm for BC prediction. Dhanya et al. [26] compared the FS and ML algorithms for BC prediction. The experiments were conducted on WBCD. The experiments showed SFS-NB showed the highest accuracy with 98.24%. Embedded feature selection utilized a learning algorithm in order to select features [11]. One representative embedded feature selection method is the LASSO [27]. Albaldawi et al. [27] proposed hybrid model ANOVA-LASSO methods and classification algorithms for microarray data classification. Three different classification algorithms like Linear Support Vector Classifier (LSVC), Random Forest (RF) and Multilayer Perceptron Classifier (MLP) were used. The proposed model demonstrated accuracy of 100% when using all classifiers.

Machine learning contains many hyperparameters and these hyperparameters need to be set automatically to optimize the performance. HO is an approach that chooses a set optimal hyperparameters for a machine learning algorithm. In the literature, grid search, random search, and Bayesian optimization have been used to automatically set hyperparameters in machine learning [28]. Compared to hyperparameter optimization models, BO uses less time with smaller evaluations to find the best hyperparameter values. Kumar et al. [29] proposed an efficient model for BC using Bayesian hyperparameter tuned RF. The study used WBCD and Wisconsin Prognostic Breast Cancer Dataset (WPBC). Their proposed approach achieved an accuracy of 96.4% for WBCD and 97.9% for WPBC. Mate et al. [30] presented a hybrid model that combined FS and BO with machine learning for BC prediction. Extra Tree Classifier algorithms was the best classification method with accuracy of 96,2%. Bensaoucha et al. [31] compared several classification algorithms for BC prediction. The hyperparameters of the algorithms were determined using BO approach. SVM showed the highest accuracy with 96.52% for WDBC dataset.

3  Materials and Methods

The suggested classification model is illustrated for prediction of BC in Fig. 1.

Figure 1: Overall workflow of proposed classification model

The working principle of suggestion model is shown Algorithm 1.

3.1 Description of the Dataset

The first dataset is the WBCD that consists of 569 instances with 32 features. The dataset contains 32 features (ID Number, Diagnosis and 30 input features). The features extracted from images of cell nuclei. Each instance is labeled as benign and malign. There are 356 benign and 213 instances [32]. The feature details are shown in Tab. 1.

The second dataset is the MBCD which includes a total of 195 breast tumors (116 images (59%) for malign, 79 (41%) images for benign). This dataset was a retrospective study, and it was retrieved from Ankara Training and Research Hospital. This retrospective study was approved by the Institutional Ethics Committee of Ankara Training and Research Hospital (319/E-20). All patients who underwent digital mammography between April 2015 and April 2020 were retrieved from the Picture Archiving and Communication System (PACS). All patients underwent mammography using IMS Giotto (Bologna-Italy). Patient consent was obtained on the condition that all data were anonymized. The mammogram images were subject to segmentation process to determine the region of interest (ROI) which represents breast tumors. The process of extraction of ROI is shown in Fig. 2. First, the mammogram images were retrieved (Fig. 2a). Then, the ROIs were defined manually by a physician with green contour (Fig. 2b). Finally, using the gray level thresholding and morphological operations, the ROIs were extracted from the image (Fig. 2c).

Figure 2: (a) Original mammogram image, (b) marking of ROI by a physician with green contours, (c) extracted ROI.

A total of 54 shape and texture features were calculated for each ROI. Intensity, Grey-level Co-occurrence Matrix (GLCM) and Gray Level Run Matrix (GLRM) were used to generate texture features. 16 shape features (F1–F16), 15 intensity-based features (F17–F31), 13 GLCM (F32–F43) and 11 GLRM (F44–F54) features were calculated [33–36]. The features are demonstrated in Tab. 2. These, Mean Absolute Deviation (MAD), Inter Quartile Range (IQR), Root Mean Square (RMS), Short Run Emphasis (SRE), Long Run Emphasis (LRE), Gray Level Non-Uniformity (GLNU), Run Length Nonuniformity (RLN), Run Percentage (RP), Low Gray-Level Run Emphasis (LGRE), High Gray-Level Run Emphasis (HGRE), Short Run Low Gray-Level Emphasis (SRLGE), Short Run High Gray-Level Emphasis (SRHGE), Long Run Low Gray-Level Run Emphasis (LGRE), Long Run Low Gray-Level Run Emphasis (LGHE) can be expressed.

3.2 Data Normalization

Data normalization is a preprocessing technique that aims to identify numeric values in the datasets within a fixed range. In this study, z-score normalization method was used. Z-score is a technique that represents the number of standard deviations away from the mean [37].

3.3 Feature Selection

FS is used to eliminate redundant and irrelevant features. Removing the irrelevant features improves the machine learning classification performance and reduces the computational cost of modeling [11]. RF is a filter approach that weighs feature according to their relationships. This algorithm uses similar methodology as in K-NN to determine the weights. The most important features get high weights while the remaining features get small weights. All features were ranked according to this measure. Here we selected 0 as a threshold value. If the weights of the features were higher than 0, the feature was selected, otherwise the feature is ignored [38,39]. LASSO is an embedded feature selection approach where the selected feature depends on modifying the absolute coefficient values. These features coefficient values set to zero and zero coefficient features are removed from the feature set. The selected features have high coefficient [40]. SFS is a wrapper-based feature selection method it adds features from the dataset sequentially to an empty candidate set. This process continues until further addition features does not reduce the criterion. The candidate feature subsets were evaluated by using 10-fold cross validation [41].

3.4 Cross Validation

Cross validation is a resampling model that divides into two groups; training and testing. 10-fold cross validation is used for evaluation of models. 90% of data were used for training, while 10% of data were used for testing purpose [42].

3.5 Machine Learning Algorithms

In this study, DT, NB, SVM, K-NN and EL algorithms were used for classification. In the DT approach, simple decision rules are used to estimate the value of target. This simple decision rules are extracted from the data. DT is generally used classification and regression process [43]. NB algorithm tries to estimate the class of the data by using “Bayes Theorem”. NB algorithm uses a series of probability principles to determine the class of the data [44]. SVM builds a line or a hyperplane that splits the data into classes. It provides a good accuracy value while performing prediction of large datasets [45]. K-NN uses feature similarity to estimate the values of new sample points [46]. EL is a machine learning model that combines two or more configuration algorithms such as bagging and boosting techniques. Bagging and boosted approaches are generally based on decision tree learners. Bagging model aims to achieve better success by using and combining a set of classifiers. Boosting algorithm aims to obtain strong classifiers from the weak classifiers with a low training error [47].

3.6 Hyperparameter Optimization

Hyperparameters are very important effect on performance of ML because they directly affect the training process. For example, box-constraint, kernel parameter and kernel scale are very important for SVM. Moreover, maximum number of splits affect the performance of decision tree. These hyperparameters are essential to be set to obtain excellent result. HO provides automation of the selection of hyperparameter values [48].

In this manuscript, BO was used to select automatically hyperparameters for machine learning algorithm. This algorithm is an effective approach for parameter search and is a black-box optimization technique. Algorithm builds a probabilistic model by setting a prior probability distribution over the function being optimized. Then, it combines with sample information to obtain a posterior function [49,50]. Hyperparameters and search ranges of ML are demonstrated in Tab. 3.

3.7 Performance Evaluation Metrics

Confusion matrix is used to visualize the success of ML models. True positive (TP), True negative (TN), False positive (FP) and False negative (FN) values need to be identified to measure the confusion matrix. TP means that malign cases are properly recognized as malign. TN means that benign cases are properly identified as benign. FP means that benign cases are mistakenly identified as malign. FN means that benign cases are mistakenly recognized as malign. Accuracy is the proportion of the cases correctly identified to entire cases. Precision is calculated by dividing true positive by overall positives. Recall is defined that the percentages of true positive among the real positive cases. F1-Score demonstrates the harmonic mean of precision and recall values [51].

Accuracy=((TP+TN)/(TP+TN+FP+FN))∗100 (1)

Recall=((TP)/(TP+FN))∗100 (2)

Precision=((TP)/(TP+FP))∗100 (3)

F1−Score=2∗(Precision∗Recall)/(Precision+Recall)) (4)

4  Experimental Results

Different experiments were implemented on machine learning methods to achieve the best classification rates for BC datasets. Experiments first started by using the functions developed by the MATLAB 2020a program. Then, by using all the features, MATLAB Statistics and Machine Learning Toolbox program [52] was used and the classification process was implemented by optimizing the features of machine learning methods with Bayesian Optimization technique. Finally, distinctive features were determined in the datasets using RF, LASSO and SFS methods, and the classification process was implemented by optimizing the features of machine learning methods with Bayesian Hyper Optimization technique. To apply FS, relief, lasso and sequential functions developed by MATLAB were used [53]. The selecting features for the BC datasets are demonstrated in Tab. 4. After applying FS, the selecting features were 16, 8 and 3 for MBCD, and they were 12, 10 and 4 for RF, LASSO and SFS, for WBCD, respectively.

The classification results based on accuracy for WBCD and MBCD are shown in Tabs. 5 and 6. The results are presented as ALL, BO, RF-BO, LASSO-BO and SFS-BO in ML algorithms in Tabs. 5 and 6. When the results in the Tab. 5 are analyzed, it is seen that RF-BO method showed the highest accuracy rates of 96,66% in NB. The LASSO-BO method resulted in better accuracy rates 95,43%, 98,95% and 98,24% in both DT, SVM and EL algorithms. In K-NN algorithm, the SFS-BO method showed the highest accuracy rate of 98.06%.

As observed in Tab. 6, the LASSO-BO method was the best performer in both SVM and EL (97,95% and 97,43%). The SFS-BO method gave the highest classification results with accuracy rates of 95,9% and 96,92% in NB and K-NN algorithms. The RF-BO method showed the highest classification results with accuracy rates of 95,38% in DT algorithm.

5  Discussion

For efficient early diagnosis of BC, a classification model based on improved machine learning algorithms presented in this study. The proposed classification model was tested on two different BC datasets. Initially, all features were given directly as input to machine learning algorithms. Then, machine learning methods were optimized with the help of Bayesian optimization method to improve the classification performance and all features were given to ML as input. Finally, we combined RF, LASSO and SFS and Bayesian optimization approach to improve ML and guarantee the performance efficiency of ML. In Figs. 3 and 4, how the performance of each ML was changed all the experiments are demonstrated for WBCD and MBCD. The classification results in the Figs. 3 and 4 are demonstrated in terms of all features and the FS-BO with the best classification rates. Based on the results in Figs. 3 and 4 when machine learning algorithms were optimized with help of Bayesian optimization, the optimization process were slightly increased the accuracy rates of all machine learning algorithms. However, when the combination of BO and FS method was used, the accuracy rates of all ML algorithms were significant increased. In the light of this information, it can be said that FS-BO approach significantly increased the classification rates of machine learning algorithms.

Figure 3: The accuracy values of for WBCD

Figure 4: The accuracy values for MBCD

The Tab. 7 depicts the best classification rates of each algorithm for WBCD. The performance like accuracy, precision, recall and F1-score is compared with various classifiers in Tab. 7. Here, LASSO-DT, RF-BO-NB, LASSO-BO-SVM, SFS-BO-K-NN methods are compared in Tab. 7. As a result of comparisons, LASSO-BO-SVM method showed higher accuracy of 98,95%, precision of 97,17%, recall of 100% and F1-score of 98,57% than other methods.

The best classification rates for MBCD are shown in Tab. 8. RF-BO-DT, SFS-BO-NB, LASSO-BO-SVM, SFS-BO-K-NN and LASSO-BO-EL methods are compared in Tab. 8. As observed in Tab. 8, LASSO-BO-SVM method gave the highest performance (accuracy of 97,95%, precision of 98,28%, recall of 98,28% and F1-score of 98,28%).

The confusion of matrices of LASSO-BO-SVM methods for WBCD and MBCD are showed in Fig. 5. While “0” represents benign status, “1” represents malign status.

Figure 5: The confusion matrices of WBCD and MBCD

The proposed method (LASSO-BO-SVM) are compared to six recent studies with using WBCD in Tab. 9. It was noticed that LASSO-BO-SVM method achieved higher accuracy values compared to previous works. LASSO-BHO-SVM method may be used an effective approach in the early diagnosis of BC with the high classification rates.

6  Conclusion and Future Works

Recent years witnessed many studies toward the diagnosis of BC in its initial stage. Although much effort has been directed to this field, it is still very challenging for researchers to choose the right method for an effective diagnostic model. The study proposes a novel classification model based on improved ML algorithms combining RF, LASSO and SFS methods and Bayesian optimization for efficient diagnosis of BC. Among the many variations, the LASSO-BO-SVM method depicted the highest accuracy, sensitivity, precision and F1-score for BC datasets. With these high classification rates, LASSO-BO-SVM technique has a potential to help radiologists for making more accurate BC diagnosis decisions. In the future works, we will use deep learning models for early diagnosis of BC and compare machine learning algorithms and deep learning models.

Acknowledgement: The authors would like to acknowledge the Department of Radiology of Ankara Training and Research Hospital, Ankara for their support in this work.

Funding Statement: The authors received no specific funding for this study.

Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.