Open Access
ARTICLE
An Optimized Implementation of a Novel Nonlinear Filter for Color Image Restoration
Department of Electrical Engineering, College of Engineering, Jouf University, Sakaka, Saudi Arabia
* Corresponding Author: Turki M. Alanazi. Email:
(This article belongs to the Special Issue: Optimization Problems Based on Mathematical Algorithms and Soft Computing)
Intelligent Automation & Soft Computing 2023, 37(2), 1553-1568. https://doi.org/10.32604/iasc.2023.039686
Received 11 February 2023; Accepted 11 April 2023; Issue published 21 June 2023
Abstract
Image processing is becoming more popular because images are being used increasingly in medical diagnosis, biometric monitoring, and character recognition. But these images are frequently contaminated with noise, which can corrupt subsequent image processing stages. Therefore, in this paper, we propose a novel nonlinear filter for removing “salt and pepper” impulsive noise from a complex color image. The new filter is called the Modified Vector Directional Filter (MVDF). The suggested method is based on the traditional Vector Directional Filter (VDF). However, before the candidate pixel is processed by the VDF, the MVDF employs a threshold and the neighboring pixels of the candidate pixel in a 3 × 3 filter window to determine whether it is noise-corrupted or noise-free. Several reference color images corrupted by impulsive noise with intensities ranging from 3% to 20% are used to assess the MVDF’s effectiveness. The results of the experiments show that the MVDF is better than the VDF and the Generalized VDF (GVDF) in terms of the PSNR (Peak Signal-to-Noise Ratio), NCD (Normalized Color Difference), and execution time for the denoised image. In fact, the PSNR is increased by 6.554% and 12.624%, the NCD is decreased by 20.273% and 44.147%, and the execution time is reduced by approximately a factor of 3 for the MVDF relative to the VDF and GVDF, respectively. These results prove the efficiency of the proposed filter. Furthermore, a hardware design is proposed for the MVDF using the High-Level Synthesis (HLS) flow in order to increase its performance. This design, which is implemented on the Xilinx Zynq XCZU9EG Field-Programmable Gate Array (FPGA), allows the restoration of a 256 × 256-pixel image in 2 milliseconds (ms) only.Keywords
Noise frequently degrades image quality and can come from a variety of sources, including electrical disturbances, communication issues, noisy sensors, etc. This noise causes problems in digital image processing. Moreover, denoising and enhancement of images while preserving edge sharpness and image details are essential operations before using them for other image processing techniques.
Nonlinear filters [1] are useful for removing noise from images, as they can effectively preserve the edges and details of the image while smoothing out the noise. They are a type of image filtering technique that involves modifying the pixel values of an image based on their local environment. Some commonly used nonlinear filters include the median filter, the bilateral filter, and the adaptive filter. These filters are widely used in image processing applications such as computer vision, medical imaging, and remote sensing. The Vector Directional Filter (VDF) [2,3] is a type of nonlinear filter that is used to remove noise from color images. Unlike traditional color filters that process each color channel independently, VDF operates on the color vector space, taking into account the spatial correlations between different color channels. This filter employs the angle between the image vectors as an ordering criterion, which allows it to provide an optimal directional estimator. In this case, the vectors of the image with directions atypical in RGB space are removed, and the best estimations of the chromaticity are attained. But the VDF has some limitations, especially when some noisy pixels are not identified or non-noisy pixels are classified as noise. However, in the first scenario, the pixels remain unchanged. For the second scenario, the pixels will be replaced. As a result, these two scenarios may have negative effects, such as blurring fine details or destroying the edges of the reconstructed image.
In this context, to improve the performance of the VDF, a new filter is proposed, which is the Modified VDF (MVDF). Besides, a hardware architecture is suggested for this filter for real-time processing. Nevertheless, to design a hardware architecture, the Low-Level Synthesis (LLS) flow can be used through a hardware description language such as Verilog. But, the LLS needs a high level of skill and much time to design any hardware architecture, especially for complex algorithms [4–6]. For that, in the last decade, a High-Level Synthesis (HLS) flow was established to generate several hardware designs for any algorithm in a few times from a high-level programming language like C or C++ [7–9] and realize the space exploration to select the best design. Thereby, in this work, the HLS flow is used to create an efficient hardware architecture for the MVDF and implement it in the Zynq XCZU9EG Xilinx Field-Programmable Gate Array (FPGA).
The following is the structure of the paper: The related works are illustrated in Section 2. Section 3 gives an overview of the VDF and the theory of the proposed MVDF. In Section 4, several experiments are realized to select the best parameters for the proposed filter, and a comparative study is conducted. A real-time MVDF hardware architecture is designed in Section 5. The paper concludes with Section 6.
A commonly encountered noise in color images is a “salt and pepper” impulsive noise [10]. However, in the literature, several filters are proposed to remove this noise. In fact, for reducing impulse noise from color images, a basic vector filter [11,12] is provided. The vector ordering approach is utilized in these filters. Furthermore, weighted vector filters are suggested in [13–17], where each pixel of the filter window is given a non-negative weight in order to eliminate outliers. In addition, [18,19] provide a hybrid vector filter, which employs several subfilters and produces an output that is a composite of the samples of the input vector. Moreover, fuzzy vector filters are introduced in [20–22]. In this kind of filter, the fuzzy functions created by various distance functions are utilized to determine the weights of a nonlinear adaptive filter. In recent years, popular techniques based on convolutional neural network architecture and deep learning have been suggested to improve the restoration of image quality [23–25].
The most important filters used to preserve the chromaticity of the restored color image and suitable for real-time implementation in the basic vector filter family are the VDF and Generalized VDF [26]. In order for these filters to restore the image, a filter window must be applied to each pixel in the image. This process is carried out without differentiating between a non-noisy pixel and a noisy pixel, which can lead to a poor-quality image. Moreover, these filters require a large amount of time to filter the image and have a high level of computational complexity.
However, in the literature, some solutions are proposed to reduce the computational complexity of these filters. In fact, the authors of [27] suggest a real-time hardware implementation for the VDF. Therefore, the nonlinear function in the VDF is approximated to provide a flexible hardware architecture. But, this approximation leads to a loss in the quality of the restored image. In [28], the authors propose a floating-point implementation of the VDF on the Texas Instruments DSP TMS320C6711. This solution allows filtering a 320 × 320 image in 1342 ms at 150 MHz. Reference [29] introduces some approximations to implement VDF on an Intel Pentium® processor running at 2.66 GHz. These approximations allow a reduction in the processing time but a decrease in the filtered image quality.
In this study, the MVDF is suggested to enhance the quality of the denoised image without increasing the computational complexity relative to the VDF and GVDF filters. In fact, the principle of the MVDF is to detect, through a threshold and neighborhood pixels of the central pixel in the 3 × 3 filter window, whether the selected pixel in the image is corrupted by noise or not before processing it by the VDF. Besides, for real-time execution of the MVDF, the hardware architecture is designed for this filter via the HLS flow.
The VDF allows optimization of the vector direction variation related to the pixel chrominance. Hence, the VDF permits minimizing the angular distance.
Consider x:
The
Assuming that
So, the output of the VDF filter is the vector
The problem with the VDF is that it works with a fixed filter window. This causes a loss in the details of the image and creates a blurred image while filtering the noisy pixels. For that, our purpose is to minimize error in the VDF output and increase its performance by estimating if the selected pixel in the image is corrupted by noise or not before it is processed by the VDF. For that, a threshold (T) and the relationship with neighborhood pixels (RWNP) of the candidate pixel, which is the central pixel
But in the MVDF, there is an extra filter that checks if the central pixel is noisy or not. This filter is based on the computation of the RWNP value. In fact, in the filter window, the minimum order pixels of the VDF output and their closest neighborhoods contain comparable values of the calculated angle relative to another input pixel. So, the RWNP is computed as the distance between the central pixel
So, the output pixel of the MVDF is specified per Eq. (4).
Indeed, from Fig. 1, we can see that for the 3 × 3 filter window nine angular distances are calculated. Then, these distances are arranged in ascending order to determine the f highest similarity input pixels
The performance of our proposed MVDF depends on the f and T parameters. In fact, we have to select the correct values for these parameters to preserve the details of the filtered image and achieve a high visual quality in the restored color image, irrespective of the intensity of the impulsive noise. To realize this study, two standard images are used, which are Lena and Sailboat. These images have a resolution of 256 × 256 pixels and are contaminated by “salt and pepper” impulsive noise of intensities 10% and 20%. This noise is produced by the imnoise function given by the MATLAB tool. The NCD (Normalized Color Difference) and PSNR (Peak Signal-to-Noise Ratio) metrics are used to measure the color image quality objectively. Indeed, the PSNR and NCD metrics are computed through Eqs. (5) and (8), respectively.
where the Mean Squared Error (MSE) for three image components R (Red), G (Green), and B (Blue) is calculated based on Eq. (6)
However, the MSE for each channel in the color image is determined by Eq. (7). j presents the image channel (R, G, and B). X and Y provide the height and the length of the image, respectively. But,
where, the
The MVDF code is written in the programming language C, and it is made with the help of Microsoft Visual Studio Community 2019. The CoreTM i7-1165G7@2.80 GHz Intel processor is used to run the C code for MVDF on the computer. The Timer of the Intel processor is employed to calculate the processing time of our MVDF code. Figs. 2 and 3 illustrate the curves of the PSNR and NCD values in terms of T (from 5 to 90) and f (1, 3, 5, 7, and 9) for the Lena and Sailboat images, respectively. In fact, for each value of f, we have drawn a curve by varying the value of T. Our study was conducted with two intensity levels of impulsive noise, equal to 10% and 20%. Our goal is to study the behavior of the MVDF and select the best values of the f and T parameters in order to provide an excellent restoration of the noisy color image with our proposed MVDF.
It is clear from Figs. 2a and 3a that the MVDF provides the best performance in terms of the PSNR and NCD when f = 5, 7, or 9 and T = 50 for Lena and Sailboat images, which are corrupted by 10% of impulsive noise. On the other hand, if we increase the level of noise to 20% for Lena and Sailboat images, we can see from Figs. 2b and 3b, respectively, that the MVDF provides the best performance when only f = 9 and T = 50. Thus, we can notice from the above findings that when the noise level is increased, it is worth using the nine input pixels (
The performance of our proposed MVDF is compared with that of VDF and GVDF. Indeed, Fig. 5 illustrates the performance comparison in terms of PSNR and NCD metrics for VDF, GVDF, and MVDF. This study is realized for indoor scenes such as Lena (Fig. 5a) and outdoor scenes such as Sailboat (Fig. 5b). These images are contaminated with 3%, 5%, 10%, 15%, and 20% of “salt and pepper” impulsive noise. Nevertheless, we can notice from Fig. 5 that the MVDF provides a better performance compared to the VDF and GVDF in terms of PSNR and NCD metrics, especially for low and medium levels of noise. Besides, we can observe from Fig. 6, which illustrates the execution time in milliseconds (ms) of the VDF, GVDF, and MVDF, that the MVDF reduces the computational complexity relative to the VDF and GVDF. In fact, the MVDF gives a reduction in the execution time of 64.088% and 67.172% relative to the VDF and GVDF, respectively. In light of the above findings, our proposed MVDF is superior to other filters in terms of the quality of the restored image and the reduction of its computational complexity.
To evaluate the efficiency of our proposed MVDF, we extended our study by using more standard test images. By the way, Table 1 presents the performance comparison of the VDF, GVDF, and MVDF for five test images (Lena, Sailboat, Flower, Barbara, and Peppers). This comparison is realized in terms of objective measurement through the PSNR and NCD metrics for impulsive noise intensity between 3% and 20%. Furthermore, Figs. 7 and 8 depict the performance comparison of the subjective measurement of the restored image by the VDF, GVDF, and MVDF for 3% and 10% of the impulsive noise level, respectively. From Table 1, it is worth noting that the MVDF enhances the PSNR by 6.554% and 12.624% and reduces the NCD by 20.273% and 44.147% relative to the VDF and GVDF, respectively. These findings are supported by the subjective evaluations shown in Figs. 7 and 8, which clearly show that the visual quality of the restored image by MVDF is superior to that restored by VDF and GVDF. In fact, we can see that the MVDF preserves the image details and provides a sharper image. But the VDF and GVDF provide a blurred image.
The HLS represents a powerful tool allowing to develop a hardware architecture through high-level languages like C and C++ [30]. In this context, the Vivado HLS is developed by Xilinx to aid engineers in exploring design architecture in the shortest amount of time. In fact, this exploration is realized through the selection of the appropriate Paragma such as PARTITION, PIPELINE, RESOURCE provided by the Vivado HLS environment.
By the way, an optimized hardware architecture for the MVDF is designed through the Vivado HLS 18.1 tool. Fig. 9 depicts the structure of the C code provided as input to this tool. This code reads three image lines to construct the 3 × 3 RGB filter window, which is used to compute the RWNP value. From this value, the MVDF code can keep the central pixel unchanged or determine the filtered pixel through the process presented in Fig. 9. The provided RGB pixel is stored in the internal memory to restore the filtered image.
The MVDF C code is synthesized without adding any Pragma through the Vivado HLS 18.1 tool to be implemented on the Zynq XCZU9EG Xilinx FPGA. Table 2 illustrates the HLS synthesis results of our code. From this table, we can notice that Design 1 uses 8% of Look-up-Table (LUTs), 2% of Flip-Flops (FFs), 7% of BRAM blocks, and 1% of DSP blocks, and needs 1722 ms to restore a 256 × 256 color image at 125 MHz. Therefore, the processing time should be reduced to reach a real-time throughput with the MVDF design. The PARTITION and PIPELINE Pragmas are applied to the C code to process the data in the pipeline. After adding these pragmas, we can observe that the processing time of Design 3 is reduced by 99% relative to Design 1 and that it can restore a 256 × 256 color image in 1 ms at 125 MHz. But, Design 3 uses more FPGA resources, with 88% of LUTs, 84% of FFs, and 89% of DSPs compared to Design 1. Thus, we have employed the RESOURCE Paragma to optimize the FPGA resources. Consequently, we can conclude from Table 2 that Design 4 through the RESOURCE Paragma permits us to reduce the number of LUTs by 44% and FFs by 34% compared to Design 3 and restore a 256 × 256 color image in 2 ms at 125 MHz. However, from these results, it is clear that Design 4 presents high data throughput compared to the CoreTM i7-1165G7@2.80 GHz intel processor, which needs 65 ms to restore a 256 × 256 color image. These results are given for the same image quality between the software and hardware implementations of the MVDF.
In this paper, a new nonlinear filter is proposed for efficient color image restoration. This filter is called MVDF, and it allows to check if a pixel in the image is corrupted or not by noise before changing it to a denoising pixel. For that, the MVDF uses a threshold and the neighborhood pixels of the central pixel in 3 × 3 filter window to make the decision whether to change the central pixel or not. One of the important features of this filter is that, compared to VDF and GVDF, it provides a smooth and clear image without increasing the computational complexity or execution time. Moreover, it can be used for a wide range of applications, like satellite images, character and face recognition, medical imaging, and so on. In fact, the experimental results show that the proposed filter gives better image quality than VDF and GVDF in terms of both objective (PSNR and NCD) and subjective measures, with an average decrease of 65.63% in the execution time. These results are given in particular for color images with lower and moderate “salt and pepper” impulsive noise densities ranging from 3% to 20%. For real-time processing, the hardware architecture is proposed for the MVDF through HLS flow, which allows to restore the image in 2 ms at 125 MHz instead of 65 ms under the CoreTM i7-1165G7@2.80 GHz Intel processor.
Our future work will focus on using deep learning techniques to improve image quality restoration, especially for high levels of “salt and pepper” impulsive noise [31].
Funding Statement: This work was funded by the Deanship of Scientific Research at Jouf University (Kingdom of Saudi Arabia) under Grant No. DSR-2021-02-0393.
Conflicts of Interest: The author declares that he has no conflicts of interest to report regarding the present study.
References
1. B. B. Ahamed, D. Yuvaraj and S. S. Priya, “Image denoising with linear and non-linear filters,” in Proc. ICCIKE, Dubai, United Arab Emirates, pp. 806–810, 2019. [Google Scholar]
2. P. E. Trahanias and A. N. Venetsanopoulos, “Vector directional filters-a new class of multichannel image processing filters,” IEEE Transactions on Image Processing, vol. 2, no. 4, pp. 528–534, 1993. [Google Scholar] [PubMed]
3. R. Lukac, B. Smolka, K. N. Plataniotis and A. N. Venetsanopoulos, “Selection weighted vector directional filters,” Computer Vision and Image Understanding, vol. 94, no. 1, pp. 140–167, 2004. [Google Scholar]
4. A. Ben Atitallah, I. Abid, A. Boudabous and H. Loukil, “A new hardware architecture of the adaptive vector median filter and validation in a hardware/software environment,” International Journal of Circuit Theory and Applications, vol. 49, no. 8, pp. 2329–2347, 2021. [Google Scholar]
5. M. Kthiri, B. Le Gal, P. Kadionik and A. Ben Atitallah, “A very high throughput deblocking filter for H. 264/AVC,” Journal of Signal Processing Systems, vol. 73, no. 2, pp. 189–199, 2013. [Google Scholar]
6. A. Ben Atitallah, M. Kammoun and R. Ben Atitallah, “An optimized FPGA design of inverse quantization and transform for HEVC decoding blocks and validation in an SW/HW environment,” Turkish Journal of Electrical Engineering and Computer Sciences, vol. 28, no. 3, pp. 1656–1672, 2020. [Google Scholar]
7. M. Kammoun, A. Ben Atitallah, K. M. Ali and R. Ben Atitallah, “Case study of an HEVC decoder application using high-level synthesis: Intra prediction, dequantization, and inverse transform blocks,” Journal of Electronic Imaging, vol. 28, no. 3, pp. 033010, 2019. [Google Scholar]
8. M. A. Ben Atitallah, R. Kachouri, A. Ben Atitallah and H. Mnif, “An efficient HW/SW design for text extraction from complex color image,” CMC-Computers, Materials & Continua, vol. 71, no. 3, pp. 5963–5977, 2022. [Google Scholar]
9. A. Ben Atitallah, M. Kammoun, K. M. A. Ali and R. Ben Atitallah, “An FPGA comparative study of high-level and low-level combined designs for HEVC intra, inverse quantization, and IDCT/IDST 2D modules,” International Journal of Circuit Theory and Applications, vol. 48, no. 8, pp. 1274–1290, 2020. [Google Scholar]
10. T. R. Chanu, T. R. Singh and K. M. Singh, “A survey on impulse noise removal from color image,” Turkish Journal of Computer and Mathematics Education, vol. 12, no. 13, pp. 4274–4295, 2021. [Google Scholar]
11. B. Smolka, “Robust sharpening vector median filter,” in Proc. CACS, Taoyuan, Taiwan, pp. 1–6, 2018. [Google Scholar]
12. B. Smolka, “Robustified vector median filter,” in Proc. ICCSE, Vancouver, BC, Canada, pp. 362–367, 2014. [Google Scholar]
13. R. Lukac, “Another generalisation of vector filters,” in Proc. VIPROM, Zadar, Croatia, pp. 273–278, 2002. [Google Scholar]
14. R. Lukac, “Adaptive color image filtering based on center-weighted vector directional filters,” Multidimensional Systems and Signal Processing, vol. 15, no. 2, pp. 169–196, 2004. [Google Scholar]
15. Z. Ma, H. R. Wu and D. Feng, “Partition-based vector filtering technique for suppression of noise in digital color images,” IEEE Transactions on Image Processing, vol. 15, no. 8, pp. 2324–2342, 2006. [Google Scholar] [PubMed]
16. B. Smolka, “Efficient modification of the central weighted vector median filter,” in Pattern Recognition, Berlin, Heidelberg: Springer, 2002. [Google Scholar]
17. B. Smolka, “Rank order weighted vector median filter,” in Proc. IWSSIP, Rio de Janeiro,Brazil, 2010. [Google Scholar]
18. L. Khriji and M. Gabbouj, “Adaptive fuzzy order statistics-rational hybrid filters for color image processing,” Fuzzy Sets and Systems, vol. 128, no. 1, pp. 35–46, 2002. [Google Scholar]
19. I. Abid, A. Boudabous and A. Ben Atitallah, “A new adaptive vector median rational hybrid filter for impulsive noise suppression,” in Proc. STA, Monastir, Tunisia, pp. 417–421, 2015. [Google Scholar]
20. G. Wang, H. Zhu and Y. Wang, “Fuzzy decision filter for color images denoising,” International Journal for Light and Electron Optics, vol. 126, no. 20, pp. 2428–2432, 2015. [Google Scholar]
21. V. Gregori, S. Morillas, B. Roig and A. Sapena, “Fuzzy averaging filter for impulse noise reduction in colour images with a correction step,” Journal of Visual Communication and Image Representation, vol. 55, pp. 518–528, 2018. [Google Scholar]
22. A. Roy, L. Manam and R. H. Laskar, “Region adaptive fuzzy filter: An approach for removal of random-valued impulse noise,” IEEE Transactions on Industrial Electronics, vol. 65, no. 9, pp. 7268–7278, 2018. [Google Scholar]
23. Y. Sato, N. Yamamoto, N. Inagaki, Y. Iesaki, T. Asamoto et al., “Deep learning for bone mineral density and T-score prediction from chest X-rays: A multicenter study,” Biomedicines, vol. 10, no. 9, pp. 2323, 2022. [Google Scholar] [PubMed]
24. A. A. Ramadhan and M. Baykara, “A novel approach to detect COVID-19: Enhanced deep learning models with convolutional neural networks,” Applied Sciences, vol. 12, no. 18, pp. 9325, 2022. [Google Scholar]
25. N. Sharma, L. Saba, N. N. Khanna, M. K. Kalra, J. S. Suri et al., “Segmentation-based classification deep learning model embedded with explainable AI for COVID-19 detection in chest X-ray scans,” Diagnostics, vol. 12, no. 9, pp. 2132, 2022. [Google Scholar] [PubMed]
26. P. E. Trahanias, D. Karakos and A. N. Venetsanopoulos, “Directional processing of color images: Theory and experimental results,” IEEE Transactions on Image Processing, vol. 5, no. 6, pp. 868–880, 1996. [Google Scholar] [PubMed]
27. A. Ben Atitallah, A. Boudabous, L. Khriji and N. Masmoudi, “Reconfigurable architecture of VDF filter for multidimensional data,” International Journal of Circuit Theory and Applications, vol. 41, no. 10, pp. 1047–1058, 2012. [Google Scholar]
28. L. C. Dominguez and V. I. Ponomaryov, “Nonlinear filters for colour imaging implemented by DSP,” in Proc. ICEEE, Mexico City, Mexico, pp. 81–84, 2005. [Google Scholar]
29. M. Emre Celebi, “Real-time implementation of order-statistics-based directional filters,” IET Image Processing, vol. 3, no. 1, pp. 1–9, 2009. [Google Scholar]
30. T. M. Alanazi, A. Ben Atitallah and I. Abid, “An optimized SW/HW AVMF design based on high-level synthesis flow for color images,” CMC-Computers Materials & Continua, vol. 68, no. 3, pp. 2925–2943, 2021. [Google Scholar]
31. C. Tian, L. Fei, W. Zheng, Y. Xu, W. Zuo et al., “Deep learning on image denoising: An overview,” Neural Networks, vol. 131, pp. 251–275, 2020. [Google Scholar] [PubMed]
Cite This Article
This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.