DOI:10.32604/iasc.2021.019058
| Intelligent Automation & Soft Computing DOI:10.32604/iasc.2021.019058 | |
| Article |
Liver Lesions and Acute Intracerebral Hemorrhage Detection Using Multimodal Fusion
1Department of Information Technology, College of Computers and Information Technology, Taif University, P.O. Box 11099, Taif 21944, Saudi Arabia
2Department of Computer Science and Engineering, Faculty of Electronic Engineering, Menoufia University, Menouf 32952, Egypt
3Department of Electronics and Electrical Communications Engineering, Faculty of Electronic Engineering, Menoufia University, Menouf 32952, Egypt
4Department of Electrical Engineering, Faculty of Engineering, Benha University, Benha 13512, Egypt
5Department of Biomedical Engineering, College of Engineering and Computer Sciences, Marshall University, Huntington, WV 25755, USA
*Corresponding Author: Osama S. Faragallah. Email: o.salah@tu.edu.sa
Received: 29 March 2021; Accepted: 30 April 2021
Abstract: Medical image fusion is designed to help physicians in their decisions by providing them with a preclinical image with enough information. Accurate assessment and effective treatment of the disease reduce the time it takes to relieve the symptoms of the disease. This article utilizes an effective data fusion approach to work on two different imaging modalities; computed tomography (CT) and magnetic resonance imaging (MRI). The data fusion approach is based on the combination of singular value decomposition (SVD) and the Fast Discrete Curvelet Transform (FDCT) techniques to reduce processing time during the fusion process. The SVD-FDCT data fusion approach is being tested with two multimodal medical image fusion applications. The first application concerns the detection of liver lesions and the second application concerns the early detection of acute intracerebral hemorrhage. Experimental tests demonstrate that not only the SVD-FDCT data fusion algorithm can treat the curved objects and edges effectively as the FDCT do. But also, the throughput of the fusion algorithm is comparable to related fusion algorithms such as principal component analysis (PCA), Transform and Discrete Wavelet Transform (DWT), dual-tree complex wavelet transform (DT-CWT), and Curvelet fusion algorithms.
Keywords: Liver lesions; acute intracerebral hemorrhage; image fusion; CT; MRI; FDCT; SVD
Medical imaging plays a primary role in numerous medical therapy and diagnosis [1,2]. This necessitates a lot of precise images with significantly further detailed data for accurate medical diagnosis applications. The fusion process of medical images is one solution for achieving high spectral and spatial contents in a singular medical image [3]. In this regard, various approaches have been introduced in the works, for example, the high-pass filtering technique [4], Principal Component Analysis (PCA) [5] and Intensity Hue and Saturation (IHS) [6]. MRI and CT medical images are widely used in medical diagnosis. MRI images are beneficial in soft tissues such as the brain, spinal cord, and nerves than CT. On the other hand, the CT image provides better details in the case of the bones than the MRI image. In some cases, such as brain tumours, especially Gliomas, Skull base tumours, and bone tumours, complementary information about the tumour mass and its surroundings is very important to apply a successful removal.
Although these spatial domain approaches [7,8] provide enhanced graphical feature descriptions, several examination studies show the graphical distortion of colours in combination images.
The high-pass filter technique conserves spatial detail, but there is noise in the fused image. The IHS transformation precisely changes the intensity elements with images that have high resolution panchromatic. Nevertheless, the spatial resolution enhances the supernatural artifacts of the resulting image due to the exchange of the intensity element [9].
Lately, transformation domain methods using multi-resolution analysis have been developed, such as wavelet transformation. Wavelet transformations can store and preserve the time and frequency details. The entered input image is broken down to various frequency levels, and mixing schemes are then applied, changing the element coefficients with the detail components of the added input image. Although wavelet fusion gives high spectral quality, the spatial declaration is reduced because of subsampling when the remotely sensed images are used for land cover and use [1,5,6].
Due to the complex nature of land use characteristics, most applications require spatial information. Although many image fusion techniques are available in the literature for integrating remotely recognized data, not all methods meet the necessity for mapping the properties of the earth's surface [8].
In recent years, researchers have used other multi-resolution analyzes, including wavelet packet transform (WPT), multi-wavelet transform (MWT), (DWT), (DT-CWT), curvelet transform, atrous wavelet transform (ATWT), Contourlet transform, Framelet transform (FRT), and non-sampled Contourlet transform (NSCT). Significant advances have been produced in fusing images knowledge, but a little investigation is employed on other purposes like image classification and feature extraction [10].
One of the limitations of the simple fusion methods such as PCA [11], DWT [12], and DT-CWT [13] fusion methods is that they can’t handle the curved objects in the images to be fused efficiently. Curvelet fusion was introduced as a solution for that problem, but unfortunately, it takes a very long time to apply the fusion process [14]. The FDCT is the second Curvelet transform generation introduced to efficiently handle the curved objects and edges in the images to be fused and provide a short processing time, avoiding the time in the Curvelet transform. One of the most considerable limitations of the FDCT is its computational complexity due to its mathematical modeling and extensive processing capabilities.
In this paper, a SVD-FDCT fusion approach is utilized to overcome the limitation of all the last mentioned methods. The SVD-FDCT fusion approach is effectively treats the curved objects and edges with a low processing time slower than the Curvelet approach and lower computational complexity than the FDCT fusion approach.
The remainder of the paper is arranged as follows. Section 2 discusses and explores the CT and MRI medical imaging modalities. Section 3 gives the essential preliminaries regarding the FDCT and SVD techniques. Section 4 explores the SVD-FDCT image fusion algorithm. Section 5 presents the examined test results and provides discussions about these results. Finally, Section 6 gives the conclusions.
2 CT and MRI Medical Imaging Modalities
The medical image fusion process collects features of various images into a singular fused image. The fusion process is applied to improve the quality and reduce the redundant data in images before applying the fusion. Fusion also increases the clinical usage of the resulted image for diagnosing and assessing medical problems. The multiple medical images to be fused may be obtained from a single sensor or modality, and the fusion aims to increase the reliability of the features obtained from fused images. Also, medical images can be obtained from multiple sensors or multiple imaging modalities. In this case, the fusion methods assist in extracting features and revealing information that may be unseen to the human eye in a single imaging modality. These revealed features can be used for more localization of abnormalities. For example, MRI and CT show the anatomical structures.
In contrast, other modalities show functional and metabolic activity like SPECT and PET [15,16]. Medical image fusion is employed between different imaging modalities to combine all the complementary features into one image to have more information and features. The advantages of the fusion, in this case, are to assist in diagnosing diseases and reducing the storage needed to store the images before applying the fusion.
Practically, to apply the fusion between both images obtained from the last two modalities, there are two choices. The first is to have a hybrid scanner that can apply both the MRI and CT imaging to the patient and fuse the two resulted images and provide the final fused image with the complementary needed information. Still, unfortunately, this scanner is costly to be generalized, especially in developing nations. The second choice is to have each image alone from a separate scanner (which will be cheaper than the Hybrid scanner), and then by using specific software, the fusion process is done.
CT is a type of medical imaging modalities in which a computer and an array of X-ray sensors create CT images [15]. CT imaging is frequently used in looking at the bone and other hard tissues. The CT images are widely employed in various medical applications depending on the clinical condition. Image fusion with CT is employed in several applications such as 3D tumour simulations, brain diagnostics and more. One of the most important CT scan advantages is the short scan time and higher imaging resolution. But CT scan has limitations such as limited tissue characterization. Fusion between CT and other medical imaging modalities is common in the medical image fusion branch, which helps in many situations.
The MRI is another class of medical imaging modalities in which radio waves and a potential magnet are connected to a computer and utilized for creating images of regions within the body [15,16]. One of the advantages of the MRI is its safety for babies and pregnant women because it doesn’t use any exposure such as X-rays or any other radiation but only radio waves which lie in the FM range. Also, it helps look at the non-bony parts or soft tissues structures like the brain, heart, spinal cord, nerves and eyes, which are imaged with high accuracy than CT scans. For the same last reason, MRI scans are widely utilized for looking at shoulders and Knees after injuries. On the other hand, the main problem MRI images rely on their proportional sensibility to the motion, making it difficult to imaging the organs involving movements, such as mouth tumours. These limitations can be overcome using image fusion in a multimodal imaging environment. MRI images have been fused with other modalities such as CT; for example, that is one of the most common fusion combinations due to its practical usability in clinical issues.
This section explains and discusses the main preliminaries regarding the FDCT, and SVD approaches.
3.1 Fast Discrete Curvelet Transform (FDCT)
An anisotropic geometric wavelet transform named the ridgelet transform may be considered optimal for expressing the straight line singularities. However, the global line singularities are not often detected with real applications [12,17]. For solving either the singularity of local lines or curves, a partition of the image can be taken. After that, the transform can be applied to the resulted sub-images. Candès and Donoho have proposed the ridgelet block-based transformation known as the curvelet transformation in Xinchun et al. [17].
The curvelet approach efficiency results from its power to build strong approximation and operator theorems. The discrete curvelet transform can be considered powerful in describing curve-like edges in the image and analysing its features. The implementation of first-generation curvelet transform is restricted due to redundancy caused by the ridgelet analysis. The ridgelet geometry is unclear since they are not valid ridge functions in digital images [18]. The FDCT is considered the second curvelet transform generation that depends on a frequency partition method [19,20]. It is proposed with the potential of decreasing the amount of processing time by reducing redundancy caused by the Ridgelet transform [18]. The FDCT can be employed using two methods; the first method depends on Unequally-Spaced Fast Fourier Transform (USFFT), while the other depends on the selected Fourier sample wrapping [19]. Here, we are using the FDCT that depends on the selected Fourier sample wrapping. The FDCT coefficients C(j, l, b) in the selected Fourier sample wrapping version of data
where
3.2 Singular Value Decomposition
The SVD scheme may be considered a dimensional reduction scheme applied to the matrixes to reduce matrixes dimensionality [20,21]. If the SVD is applied to a matrix A of
where S defines the diagonal matrix that is termed the singular matrix. The S diagonal entries represent the Eigenvalues of matrix A with the Si > 0 and Si > Si+1. The other entries have a value of zero. The U and V matrixes define orthogonal matrixes, and their first
4 The SVD-FDCT Fusion Algorithm
The SVD-FDCT Fusion scheme presented in Mohammed et al. [22] is examined on two vital multimodal medical image fusion applications. The first application considers the detection of hepatic liver lesions. The second application considers the detecting of acute intracerebral hemorrhage.
The SVD-FDCT Fusion scheme is depicted in Fig. 1. As depicted in Fig. 1 and the detailed steps can be summed up as follows:
1. Both CT and MRI images that are registered are subjected to SVD to obtain the components Uct, Sct, and Vct for the CT image and Umri, Smri, and Vmri for the MRI image.
2. The FDCT is applied to the obtained S matrices (Sct and Smri), and the FDCT coefficients are obtained for each S matrix (Cct and Cmri).
3. The maximum fusion rule is used in fusing the obtained coefficients (Cct and Cmri coefficients), and a fused coefficient Cf is obtained.
4. Apply the inverse FDCT on the Cf coefficient to have the Sf matrix, representing the fusion result between the two matrices Sct and Smri using FDCT.
5. Fusion is applied on the (Uct, Umri) and (Vct, Vmri) utilizing a simple maximum fusion rule to obtain the two fused matrices Uf and Vf.
6. Finally, the Inverse SVD is applied on the obtained fused matrices Uf, Sf, and Vf to have the fused image.
In the last steps, only the S matrix obtained from applying the SVD on the input images (CT and MRI images) are subjected to the FDCT and not the two other matrices (U and V matrices). The reason for choosing only one matrix (S) instead of the three matrices (U, S, and V), which represents the whole image, is to decrease the algorithm complexity and reduce fusion processing time. The S matrix had been chosen and not the U or V matrix because it contains the most important features in the image. Also, it represents the diagonal matrix (all the values in it except the diagonal are zeros) and minimizes the processing time during computing the coefficients of the FDCT and so minimizing the total fusion processing time.
Figure 1: Block Diagram of the SVD-FDCT data Fusion Approach [22]
The SVD-FDCT fusion algorithm and four fusion schemes, including PCA [11], DWT [12], DT-CWT [13] and Curvelet [14], are applied to two vital multimodal medical image fusion applications. The first application utilizes image fusion techniques in detecting hepatic lesions. For hepatic lesions, CT images can show some lesions. Still, not all of them and the same thing is about MRI images, so CT/MRI image fusion is introduced in this application to show all the lesions that are not seen in one of the images alone. The second application utilizes image fusion techniques in detecting the acute intracerebral hemorrhage. The most important factor, in this case, is the time consumed during the fusion process, which represents the time of the detection of the acute intracerebral. For the above mentioned medical applications, CT and MRI images had been fused using Matlab 7.11.0 (R2017b) on a Personal Computer with an Intel Core 7 Due processor and 8 Gigabyte of RAM. The fusion key performance metrics including the average gradient (g), the local contrast (Clocal), the mutual information (MI), standard deviation (STD), entropy (E), and the edge intensity [23,24], were used to evaluate the produced fused images using the SVD-FDCT and the other four fusion schemes, including PCA [11], DWT [12], DT-CWT [13] and Curvelet [14]. The fused images produced by the different fusion algorithms were also shown for more subjective evaluation. The following discussion shows the results of applying the SVD-FDCT fusion approach and the other four fusion schemes, including PCA [11], DWT [12], DT-CWT [13] and Curvelet [14] in the two above mentioned medical applications. Fig. 2 and Tab. 1 show the experimental results of hepatic lesions detection of case 1 using the SVD-FDCT fusion algorithm and the other four fusion schemes, including PCA [11], DWT [12], DT-CWT [13] and Curvelet [14]. Medical image fusion techniques had been to facilitate hepatic lesions detection. CT images alone can’t provide all the lesions in the liver nor the MRI images alone, so CT/MRI image fusion is a solution for more accurate detection of the hepatic lesions. In this case, two 128 x 128 pixel size images, one of them is a CT image, and the other is a MRI image of a 52-year older woman with colon cancer, had been fused using the SVD-FDCT fusion algorithm. The CT image shows a focal lesion interpreted by the specialists as possible metastasis. Surgery, in this case, showed no metastatic lesions in the liver at those locations suspected in the CT image. As seen, neither the CT image nor the MRI image alone is sufficient for detecting hepatic lesions. The SVD-FDCT fusion approach is comparable to the other traditional fusion algorithms. The SVD-FDCT fusion algorithm has the advantage of handling the curved objects in the fused images more efficiently than the PCA [11], DWT [12], DT-CWT [13] and Curvelet [14], with a short processing time that is comparable to them. Figs. 3 and 4 present the obtained results for two cases 2–3 of a human acute intracerebral hemorrhage. The time used in the fusion process to detect acute intracerebral hemorrhage using the above mentioned four different image fusion techniques and the SVD-FDCT image fusion approach is listed in Tabs. 2 and 3. The SVD-FDCT fusion algorithm shows an efficient and small-time in detecting the acute intracerebral hemorrhage compared with the other four different traditional fusion approaches as shown in Tabs. 2 and 3. This small fusion processing time increases the chances of saving human life and brain cells from damage. The observed results prove the efficiency of the SVD-FDCT image fusion over the PCA algorithm in the values of standard deviation, entropy, edge intensity, local contrast, and the average gradient. But PCA is better than the SVD-FDCT image fusion in two metrics; the mutual information and the processing time. The obtained results were expected because the SVD-FDCT image fusion algorithm deals more efficiently with the curved objects in the images by using the FDCT during the fusion. So, it takes more time in computing the coefficients to apply the fusion on them, and the SVD-FDCT image fusion algorithm will be slower than the PCA algorithm. Comparing the SVD-FDCT image fusion algorithm with the DWT shows better entropy and the local contrast results for the SVD-FDCT image fusion algorithm. The DWT algorithm is better in standard deviation, edge intensity, average gradient, mutual information and processing time. By comparing the DT-CWT fusion algorithm with the SVD-FDCT image fusion, it can be noticed that the SVD-FDCT image fusion algorithm is better in some metrics, and DT-CWT is better in others. The same thing is also noticed when comparing the Curvelet algorithm with the SVD-FDCT image fusion. Finally, the SVD-FDCT image fusion algorithm can be considered a comparable algorithm with the other traditional fusion algorithms. It has the advantages of saving the processing time compared to the Curvelet algorithm. That is expected due to two reasons: using FDCT instead of the traditional Curvelet transform saves some time. The other reason is that using the SVD to choose only the most important features in the image represented in the S matrix and applying the FDCT on it saves more time than applying FDCT on the whole image in the traditional FDCT fusion approach. The SVD-FDCT image fusion algorithm also deals efficiently with the curved objects in the images to be fused with the same efficiency of the curvelet transform because of using FDCT.
Figure 2: Fusion results for the original images (CT and MRI) of liver lesions using five different fusion approaches for Case 1
Figure 3: Fusion results for the original images (CT and MRI) of a human brain hemorrhage using five fusion approaches for Case 2
Figure 4: Fusion results for the original images (CT and MRI) of a human brain hemorrhage using five fusion approaches for Case 3
This paper considered the idea of utilizing multimodal image fusion techniques for detecting liver lesions and acute intracerebral hemorrhage. The SVD-FDCT image fusion algorithm is a hybrid fusion algorithm based on combining the SVD and the FDCT. The SVD-FDCT image fusion algorithm is designed with the potential of handling the curved objects in the image and minimizing the processing time during the fusion process, especially with large sizes of the images to be fused. Results obtained by applying the SVD-FDCT image fusion algorithm and other four different fusion algorithms demonstrated that the SVD-FDCT image fusion algorithm could be considered as a comparable algorithm with the other traditional fusion algorithms. It has the advantages of saving the processing time than the curvelet algorithm and dealing with the curved objects in the images with the same efficiency.
Acknowledgement: This study was funded by the Deanship of Scientific Research, Taif University Researchers Supporting Project number (TURSP-2020/08), Taif University, Taif, Saudi Arabia.
Funding Statement: This study was funded by the Deanship of Scientific Research, Taif University Researchers Supporting Project number (TURSP-2020/08), Taif University, Taif, Saudi Arabia.
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
1. H. M. El-Hoseny, Z. Z. El Kareh, W. A. Mohamed, G. M. El-Banby, K. R. Mahmoud et al., “An optimal wavelet-based multi-modality medical image fusion approach based on modified central force optimization and histogram matching,” Multimedia Tools and Applications, vol. 78, no. 18, pp. 26373–26397, 2019. [Google Scholar]
2. M. B. A. Haghighat, A. Aghagolzadeh and H. Seyedarabi, “A non-reference image fusion metric based on mutual information of image features,” Computers & Electrical Engineering, vol. 37, no. 5, pp. 744–756, 2011. [Google Scholar]
3. H. M. El-Hoseny, W. A. El-Rahman, W. El-Shafai, S. El-Rabaie, K. R. Mahmoud et al., “Optimal multi-scale geometric fusion based on non-subsampled contourlet transform and modified central force optimization,” International Journal of Imaging Systems and Technology, vol. 29, no. 1, pp. 4–18, 2019. [Google Scholar]
4. B. Lalotra, R. Vig and S. Budhiraja, “Multimodal medical image fusion using butterworth high pass filter and cross bilateral filter,” MATEC Web of Conferences, vol. 57, pp. 1–6, 2016. [Google Scholar]
5. R. P. Desale and S. V. Verma, “Study and analysis of PCA, DCT & DWT based image fusion techniques,” in Int. Conf. on Signal Processing, Image Processing & Pattern Recognition, Coimbatore, India, pp. 66–69, 2013. [Google Scholar]
6. S. Nizam and Z. Telatar, “Multispectral image fusion based on the multiwavelet and IHS transforms,” in Signal Processing and Communications Applications Conf., Muga, pp. 1–4, 2012. [Google Scholar]
7. J. S. D. Río, C. Conde, A. Tsitiridis, J. Raúl Gómez, I. M. D. Diego et al., “Face-based recognition systems in the ABC e-gates,” in Annual IEEE Systems Conf. (SysCon) Proc., Vancouver, BC, Canada, pp. 340–346, 2015. [Google Scholar]
8. G. Shruti, D. Singh and S. Kumar, “Fusion of texture and wavelet features of PALSAR image using LDA and PCA for land cover classification,” International Journal of Image and Data Fusion, vol. 8, no. 4, pp. 354–374, 2017. [Google Scholar]
9. M. N. Aktar, A. J. Lambert and M. Pickering, “An automatic fusion algorithm for multi-modal medical images,” Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, vol. 6, no. 5, pp. 584–598, 2017. [Google Scholar]
10. Z. Chengquan, D. Liang, X. Yang, B. Xu and G. Yang, “Recognition of wheat spike from field based phenotype platform using multi-sensor fusion and improved maximum entropy segmentation algorithms,” Remote Sensing, vol. 10, no. 2, pp. 246, 2018. [Google Scholar]
11. S. Sulochana, R. Vidhya, K. Mohanraj and D. Vijayasekaran, “Effect of wavelet based image fusion techniques with principal component analysis (PCA) and singular value decomposition (SVD) in supervised classification,” Indian Journal of Geo Marine Sciences, vol. 46, no. 2, pp. 338–348, 2017. [Google Scholar]
12. Y. Yang, “Multimodal medical image fusion through a new DWT based technique,” in 4th Int. Conf. on Bioinformatics and Biomedical Engineering, Chengdu, China, pp. 1–4, 2010. [Google Scholar]
13. H. M. El-Hoseny, W. A. Abd El-Rahman, S. El-Rabaie, F. E. Abd El-Samie and O. S. Faragallah, “An efficient DT-CWT medical image fusion system based on modified central force optimization and histogram matching,” Infrared Physics & Technology, vol. 94, pp. 223–231, 2018. [Google Scholar]
14. O. A. Pappas, A. M. Achim and D. R. Bull, “Curvelet fusion of panchromatic and SAR satellite imagery using fractional lower order moments,” in 10th IEEE Int. Conf. on Advanced Video and Signal Based Surveillance, Krakow, Poland, pp. 342–346, 2013. [Google Scholar]
15. O. S. Faragallah, H. M. El-Hoseny, W. El-Shafai, W. A. El-Rahman, H. S. El-Sayed et al., “A Comprehensive survey analysis for present solutions of medical image fusion and future directions,” IEEE Access, vol. 9, pp. 11358–11371, 2021. [Google Scholar]
16. G. Wen, G. Li-qun and G. Wen, “A CT/MRI image fusion algorithm combined non-separable wavelet transform and regional priority,” in Int. Sym. on Computer Science and Computational Technology, Shanghai, China, pp. 137–140, 2008. [Google Scholar]
17. W. Xinchun, Y. Kaihua, L. Yuming and Y. Qing, “Palmprint recognition based on curvelet transform decision fusion,” Procedia Engineering, vol. 23, pp. 303–309, 2011. [Google Scholar]
18. S. Palakkal and K. M. M. Prabhu, “Poisson image denoising using fast discrete curvelet transform and wave atom,” Signal Processing, vol. 92, no. 9, pp. 2002–2017, 2012. [Google Scholar]
19. K. Jemseera and P. Noufal, “Satellite image fusion based on improved fast discrete curvelet transforms,” in Fifth Int. Conf. on Advances in Computing and Communications (ICACCKochi, India, pp. 430–433, 2015. [Google Scholar]
20. B. Sarwar, G. Karypis, J. Konstan and J. Riedl, “Incremental singular value decomposition algorithms for highly scalable recommender systems,” in Fifth Int. Conf. on Computer and Information Science, Minneapolis, USA, pp. 27–28, 2002. [Google Scholar]
21. A. P. James and B. V. Dasarathy, “Medical image fusion: A survey of the state of the art,” Information Fusion, vol. 19, pp. 4–19, 2014. [Google Scholar]
22. A. N. Mohammed, T. E. Taha and O. S. Faragallah, “Image fusion using FDCT based on SVD for CT/MRI medical images,” MJEER, vol. 23, pp. 37–51, 2014. [Google Scholar]
23. H. M. El-Hoseny, W. Abd El-Rahman, W. El-Shafai, G. M. El-Banby, S. El-Rabaie et al., “Efficient multi-scale non-sub-sampled shearlet fusion system based on modified central force optimization and contrast enhancement,” Infrared Physics & Technology, vol. 102, pp. 102975, 2019. [Google Scholar]
24. G. N. Raut, P. L. Paikrao and D. S. Chaudhari, “A study of quality assessment techniques for fused images,” IJI-TEE, vol. 2, no. 4, pp. 2278–3075, 2013. [Google Scholar]
 | This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |