Dynamic Characteristic Testing of Wind Turbine Structure Based on Visual Monitoring Data Fusion

1  Introduction

In the context of the global pursuit of sustainable development goals, many countries are undergoing an energy transition in which renewable energy sources, particularly wind power, are increasingly displacing non-renewable energy sources [1]. As the pursuit of carbon neutrality gains momentum, wind energy is increasingly being adopted and favored worldwide [2,3]. To capture more wind energy within limited costs and reduce electricity costs, there is a need to continually increase the height of components such as rotor blades and nacelles, with safety performance becoming a primary concern. Furthermore, the locations of existing wind farms are progressively expanding into deserts, arid zones, high-seismicity areas, and deep-sea regions, subjecting wind turbine structures to extreme climatic conditions such as low and high temperatures, droughts, and sandstorms [4]. In addition, salinity and humidity in the marine environment can potentially lead to corrosion of wind turbine blades [5]. These environmentally induced damages may lead to reduced power output of wind turbines, and potentially even cause safety incidents, thereby significantly impacting the stable operation and economic benefits of wind farms.

Globally, numerous wind farms were constructed in the 1990s, and since the design life of wind turbines typically spans around two decades, this has necessitated a substantial amount of structural health monitoring tasks [6,7]. In the early stages, wind turbine inspection relied primarily on visual inspection methods, involving direct observation by inspectors or aided by telescopes, but such approaches posed safety risks and were highly susceptible to subjective judgments in the assessment results. Monitoring devices such as thermal imaging, acoustic emission, and ultrasonic waves have also shone brightly in wind turbine structural health monitoring. However, these devices are relatively expensive, require a high level of expertise from inspectors, and are primarily used for factory inspections of wind turbines [8–10]. In the field of civil engineering, common monitoring methodologies involve signal acquisition through sensors, with signal processing techniques employed to monitor structural conditions. Contact sensors, exemplified by accelerometers and displacement sensors, have demonstrated superior performance in numerous large-scale structures [11]. However, contact sensors need to be in contact with the structure, contact sensors are limited by the number of sensors, and dense wiring in the particular configuration of wind turbines becomes a major challenge [12]. Sensor monitoring has the advantage of high stability and accuracy, but further expansion of its application in wind turbine structural health monitoring needs to be considered.

Advancements in computer technology and hardware computing capabilities have led to an increasing utilization of structural health monitoring techniques based on computer vision across various engineering disciplines [13–15]. Vision-based structural health monitoring methods enable signal acquisition of image sequences in an encoded format, featuring non-contact, remote, low-cost, and high-precision capabilities [16]. Similarly, while visual monitoring allows for simultaneous monitoring at multiple points, it is unable to accurately detect displacement signals at desired locations without artificial markers at specific positions, thereby preventing the acquisition of full-field displacement of the entire structure [17,18]. Therefore, the integration of visual monitoring and vibration monitoring data can provide a new approach for traditional structural health monitoring methods.

This paper proposes a method for testing the full-field dynamic characteristics of wind turbine structures, primarily utilizing visual monitoring data in conjunction with vibration monitoring data, and proceeds to conduct experimental validation and comparative analysis. The remainder of this paper is organized as follows: Section 2 introduces the vision-based displacement monitoring method. Section 3 introduces the method of combining visual and vibration data for full-field monitoring. Section 4 conducts testing verification of the method proposed in this article. Section 5 adopts the wind turbine structure for full-field monitoring and dynamic characteristic identification. Section 6 provides a concise summary of the research endeavors presented within this paper.

2  Visual-Based Displacement Monitoring Method

To effectively acquire visual displacement data, this paper employs the Lucas-Kanade Tomasi (LKT) optical flow method for visual processing of structural vibration videos. The overall process is shown in Fig. 1. Firstly, multiple-regions of interest are selected from the wind turbine vibration videos to reduce computational complexity. Subsequently, Shi-Tomasi corner detection and matching are employed, with image pyramids utilized to compensate for excessive motion. Finally, target tracking and displacement calculations are performed through image sequences over time.

Figure 1: LKT optical flow method for identifying the displacement process of wind turbine structures

2.1 Multiple ROI Selection

During visual processing, when confronted with massive image sequence data, one often encounters the issue of computational inefficiency or even computational disruption caused by excessive data volume. To address this challenge, ROI selection technology is introduced to significantly reduce computational burden. The essence of ROI selection lies in accurately identifying the key regions of moving targets for subsequent image processing, thereby excluding irrelevant information. This not only optimizes the utilization of computational resources but also significantly enhances the overall processing efficiency, effectively avoiding unnecessary consumption of computing power.

For wind turbine structural vibration, due to its unique configuration, attempting to perform calculations through a single ROI still results in a significant amount of invalid computational regions. To address this issue, this paper will adopt a multi-ROI selection approach to reduce the visual computational load, specifically by selecting separate ROIs for different blades and the tower of the wind turbine, followed by target tracking. This strategy effectively reduces the visual computation cost while maintaining the accuracy of visual displacement calculation, thereby facilitating its application to the displacement calculation of wind turbine structures. In this paper, multiple ROIs are employed for image preprocessing of different blades and tower sections, thereby avoiding the wastage of substantial computational resources and enhancing the accuracy of visual monitoring.

2.2 Target Tracking Based on LKT Optical Flow Method

The LKT optical flow method constitutes a robust approach for estimating the motion of targets, which is founded upon the concept of optical flow, defined as the instantaneous velocity of pixel movements observed on the imaging plane, emanating from spatially dynamic targets [19]. The LKT optical flow method leverages temporal variations in pixel intensities across image sequences and inter-frame correlations to establish correspondences between successive frames, enabling the computation of motion information for targets across adjacent frames. The LKT optical flow method performs target tracking under three assumptions: brightness constancy, small motion, and spatial coherence. Assuming I (x, y, t) represents the brightness of a pixel (x, y) at time t in an image, if this pixel moves by , between two image frames, the brightness constancy assumption implies:

(1)

By deriving the 1st Taylor expansion using Taylor series and organizing it into a matrix form, the optical flow constraint equation is obtained:

(2)

where and represent the instantaneous velocities of pixels in the x and y directions at time t in the image, respectively.

According to Eq. (2), the temporal gradient of gray-level at a given coordinate position is the product of the spatial gradient of gray-level and the velocity of that position relative to the observer. By incorporating the spatial consistency assumption, which posits that the optical flow values among multiple surrounding points exhibit similarity, the resulting system of equations becomes overdetermined, as there are more equations than unknowns, thereby rendering a direct solution impractical. The Lucas-Kanade method addresses this challenge by utilizing the principle of least squares to derive a solution, enabling the calculation of optical flow values for the pixels.

In practical visual monitoring, a pyramid-style downsampling is introduced on images to compensate for excessive motion. Furthermore, to enhance computational efficiency, this paper employs the Shi-Tomasi corner detection operator for corner detection and matching of targets [20]. Ultimately, precise visual target tracking is achieved by controlling the number of corners, quality factor, and spacing.

2.3 Displacement Calculation

Utilizing the LKT optical flow method, the coordinates of the actual tracking target in each frame of the time series, along with its initial frame coordinates , can be output, enabling the calculation of the wind turbine’s vibrational displacement:

(3)

It should be noted that the displacement calculated by Eq. (3) is based on image coordinates, and the displacement in actual physical coordinates needs to be derived through the scale factor between image and physical coordinates.

3  Full-Field Monitoring Method Based on Visual and Vibration Data

3.1 Scale Factor Acquisition

The displacement data obtained through computer vision techniques are in units of pixels, which do not possess direct physical significance in real-world displacement monitoring. To convert pixel displacement into actual physical displacement, a scale factor must be applied, and this conversion traditionally relies on precise camera calibration or known multi-point structural dimension information [21]. However, calibration conducted on-site for monitoring purposes is inherently constrained by the dimensions of the structure being monitored, thereby presenting substantial challenges and difficulties in practical applications. Specifically, in the context of displacement monitoring of wind turbine structures, accurately determining the distances between monitoring points on the turbines remains a formidable obstacle, significantly impeding both the accuracy and efficiency of the monitoring process. Consequently, the traditional approach to obtaining scale factors is constrained in practical monitoring scenarios, necessitating the integration of additional methods to convert visual pixel coordinates into actual physical coordinates. In recent years, researchers have proposed utilizing the fusion of acceleration and visual data to achieve actual displacement monitoring [22]. Nonetheless, the integration of various filters during data fusion can result in error accumulation, while the inherent computational complexity poses challenges to rapid monitoring. This paper mainly focuses on visual monitoring data, with acceleration data used for scale factor calculation. The specific process is shown in Fig. 2.

Figure 2: Scale factor acquisition process

The precision of acquiring visual displacement is contingent upon the frame rate of the camera. Typically, current cameras possess a frame rate of 50 fps, which is notably inferior to the sampling frequency exhibited by acceleration sensors. If the intention is to derive the scale factor through the integration of acceleration and displacement data, it becomes imperative to initially resample the displacement data. As shown in Fig. 2, the displacement data is initially filtered through a band pass filter to eliminate noise and undesirable frequency components. Subsequently, the original visual displacement data is resampled via interpolation to yield high-sampling displacement data. Concurrently, the acceleration data undergoes similar processing through a band pass filter, and the structural displacement under physical coordinates is derived by applying double integration to the processed acceleration data. Ultimately, the actual scale factor is determined by comparing the visual displacement with the displacement obtained from acceleration integration, utilizing the least squares method. With this scale factor and the visual displacement, the structural displacement under physical coordinates, primarily determined by vision, can be accurately calculated.

The least squares method aims to identify a straight line that optimally fits the processed two-dimensional displacement data, ensuring the best possible alignment with the two sets of data points. The slope of this fitted line represents the scale factor , and its mathematical expression is provided as follows:

(4)

where and are the predicted independent and dependent variables of the sample in the i-th time series, respectively.

Therefore, the displacement in the physical coordinates of the structure can be calculated by the scale factor using Eq. (5):

(5)

Following the previously described processing procedures, the resulting displacement data bears genuine physical relevance and can be promptly employed in subsequent analytical, evaluative, or decision-making endeavors.

3.2 Calculation of Full-Field Displacements of Structure

Contact sensors are inherently constrained to monitoring data at discrete measurement points. In cases where intensive monitoring is deemed necessary, the installation of additional sensors becomes imperative, inevitably leading to an increase in workload. While visual monitoring, in theory, possesses the capability to encompass the entire monitoring field, it encounters challenges in accurately tracking displacement at specific points when confronted with image feature recognition issues, notably image blurring and occlusion of the field of view. Consequently, the development of a method capable of predicting the displacement of unknown points within a structure, based solely on known displacements, holds paramount importance.

Feng et al. [23] updated the parameters of the structure by identifying its stiffness and external excitation forces based on known structural responses. Guided by this idea, this paper achieves accurate prediction of unknown displacement of the structure by comprehensively considering key parameters such as structural vibration equations, boundary condition constraints, and actual damping coefficients. In the shutdown state, the structure of a wind turbine can be modeled as a cantilever beam, with its vibration behavior characterized by the following partial differential equation:

(6)

where is the displacement of the wind turbine at position x and time t; E is the elastic modulus; I is the moment of inertia; is the material density; A is the cross-sectional area.

For the wind turbine structure in a shutdown state, with boundary conditions (fixed at one end) and (free at the other end), the first-order frequency is:

(7)

In practice, vibration systems have damping, and the damping coefficient can be obtained by experiment. For simple models, the damping ratio can be used to describe the damping:

(8)

where c is the damping coefficient; m is the mass; k is the stiffness.

For harmonic motion, the displacement time history can be expressed as:

(9)

where is amplitude; is angular frequency; is phase angle.

For a wind turbine in a shutdown state, due to wave propagation, the displacement away from the fixed end decreases, and thus the propagation velocity of the wave within the wind turbine structure can be expressed as:

(10)

Therefore, the displacement of the wind turbine at the fixed end x can be expressed as:

(11)

where is the attenuation factor.

4  Test Verification

4.1 Test Equipment

To verify the reliability of the proposed method in this paper, a Canon R6 camera was utilized as the visual acquisition device, with video resolution set at 1080 p and frame rate adjusted to 50 fps. The camera adopts a frame rate of 50 fps to ensure the small motion assumption of optical flow method, which can be used for visual monitoring. A laser displacement sensor (LDS) with a sampling frequency of 100 Hz was used for data validation. Two LDSs were set up in this experiment, placed at the top and middle of the blades, respectively. An INV 9812 acceleration sensor was utilized to capture the structural acceleration response, with the sampling frequency set to 1024 Hz. There are a total of 5 accelerometers, of which 3 are evenly distributed on the vertical blades, and the remaining two are located at the tips of the other two blades. This paper uses a simulated shutdown wind turbine model for vibration testing. The wind turbine model is 2.3 m high, with a tower height of 1.5 m and blade length of 0.8 m. All tests were conducted on a scaled-down wind turbine model that simulates shutdown conditions, primarily by applying excitations to the model through initial displacements, initial velocities, and a small-scale shaking table, as shown in Fig. 3.

Figure 3: Test equipment

4.2 Visual Displacement Monitoring Verification

To ensure the feasibility of the proposed method, it is essential to first validate the vision-based approach. In this test, the wind turbine is excited and then allowed to vibrate freely. Visual monitoring is used to monitor the displacement, which is then verified by LDS data. The displacement monitoring data is shown in Fig. 4.

Figure 4: The displacement time history of visual monitoring structures

Fig. 4a shows the application of traditional structural dimension-to-pixel scale factor to vision-based monitoring data. Overall, the discrepancy between the displacement data captured by visual monitoring and LDS is minimal, confirming the feasibility of visual methods in structural displacement monitoring. However, this approach requires precise knowledge of the distances between measurement points, which is often difficult to achieve in practical applications, highlighting the importance of exploring more efficient and accurate methods for scale factor acquisition. As shown in Fig. 4b, although visual monitoring and LDS maintain consistency in monitoring data, the sampling frequency of visual methods is significantly lower than LDS, which makes it difficult to accurately evaluate the errors of visual data in the time domain. Therefore, further research is needed on resampling techniques for visual data.

4.3 Scale Factor Acquisition and Fusion Monitoring Method Validation

Under the premise that visual monitoring is effective, the acquisition of scale factors without resorting to component dimensions necessitates the integration of accelerometer data. Due to the varying sampling frequencies among devices, such as 50 fps for camera, 100 Hz for LDS, and 1024 Hz for accelerometers, the resulting time series of monitored data are inconsistent, hindering the calculation of scale factors and complicating data fusion. Therefore, it is imperative to synchronize the time series of various devices prior to data fusion. In this test, the visual data acquired by the camera were resampled to 1024 Hz to match the time series of the acceleration sensor data, and the acceleration data were double integrated to convert into displacement data.

Fig. 5 shows various signals resulting from the conversion of acceleration to displacement using the method described in this paper. Fig. 5a shows the raw acceleration signal, which has superimposed signals due to various noise interferences present in the laboratory. Without proper filtering, the conversion of this signal to displacement would result in significant errors. In this paper, a 4th order band pass filter in the range of 3~30 Hz is used for noise reduction. Fig. 5b shows the filtered acceleration signal, where the removal of extraneous frequency noise is evident. It is noteworthy that the conversion of acceleration signals to displacement requires a double integration process, which requires the removal of the direct current component from the velocity signal after the first integration to mitigate cumulative errors. This step prevents error accumulation due to constant velocity offsets and improves the accuracy of the displacement calculation, as shown in Fig. 5c. Fig. 5d shows the displacement signal after the second integration of the acceleration, demonstrating that the integrated displacement retains the appropriate trend and can be used as the actual displacement for scale factor calculation.

Figure 5: Various signals for converting acceleration into displacement process

Finally, the proportional factor is calculated by applying the least squares method to the visually measured displacement and the double integrated acceleration displacement. This experiment validates the calculated proportional factor using the top, middle, and bottom sections of the blade, with the results shown in Fig. 6.

Figure 6: Scale factor of wind turbine monitoring points

As shown in Fig. 6, there is a strong correlation between the visually measured displacement data and the acceleration-integrated displacement data. However, as the pixel displacement decreases, the fitting accuracy decreases. The main reason for this is that the method presented in this paper for integrating small oscillations in the acceleration data into displacements introduces inaccuracies in the converted displacements due to noise. The primary reason for this is that acceleration sensors are highly sensitive and are capable of capturing small vibrational noise even when the wind turbine is not experiencing excited vibrations. These vibrational noises, when integrated to determine displacement, inevitably result in inaccuracies. Overall, all data follow an approximately linear distribution, which can be used as evidence of the good fitting effect of the least squares method. To validate the accuracy of the calculated scale factor, this paper uses LDS to verify the visual displacement, taking LDS as the benchmark, and compares it with both the visual data and the fused data. For the sake of brevity in this paper, Fig. 7 shows only the displacement time history data for the top part of the blade.

Figure 7: Comparison of displacement time history after data fusion

Fig. 7 shows the fused data derived from the original visual data after scaling and resampling. As shown in Fig. 7a, the fused data are generally consistent with LDS in the overall structural displacement monitoring, indicating its potential as a displacement monitoring method. Fig. 7b shows a partial displacement time history image, from which the visual monitoring has fewer sampling points and cannot directly calculate errors in the time domain with the LDS-monitored displacement time history. However, after resampling, the fused data in Fig. 7 matches the LDS-monitored data in time series, allowing error analysis.

This paper uses the Root Mean Square Error (RMSE), correlation coefficient ( ), and coefficient of determination ( ) for error analysis. Corresponding equations are [24,25]:

(12)

(13)

(14)

where n is the total number of monitoring, and are vision and LDS data, respectively. and are the average values of two trajectories. and belongs to [0, 1].

Two LDSs were used in this test, so error calculations were performed on the fused data corresponding to the two LDSs. Fig. 8 shows the time domain absolute errors and error distributions of the blade tip. Table 1 shows the error comparisons at two locations.

Figure 8: Error analysis

As shown in Fig. 8, the method proposed in this paper has absolute errors within 1 mm compared to LDS, with most errors concentrated below 0.5 mm, indicating a high level of accuracy. The main reason for the errors above 0.5 mm is the need to keep the laser beam perpendicular to the plane during LDS monitoring. However, due to the angular variations that occur during the vibration of wind turbine blades, deviations in LDS displacement measurements can occur. Therefore, in practical scenarios, absolute displacement measurements obtained by visual monitoring are likely to be more accurate. As shown in Table 1, the comparison of the errors at two monitoring points shows that the RMSE is minimal both at the tip and at the middle of the blade, confirming the accuracy of the proposed method in this paper. Similarly, the value of is infinitely close to 1, indicating an extremely high correlation between the data obtained by the proposed method and LDS monitoring, while the data show that the similarity between the two data sets is almost identical. In summary, the method proposed in this paper exhibits minimal errors compared to LDS monitoring data, and shows extremely high correlation and similarity, making it suitable for displacement monitoring of structures.

4.4 Prediction of Displacement at Unknown Measurement Points

For monitoring points where visual monitoring is difficult or impractical, accurate prediction of unknown structural displacements can theoretically be achieved by utilizing key parameters such as structural vibration equations, boundary condition constraints, and actual damping coefficients. To validate this concept, this paper uses the displacement response at the blade tip to predict the displacement responses at other measurement points along the blade. Due to the complexity of the wind turbine model construction, in this test, the wind turbine model was equivalently replaced with parameters of a cantilever beam with a rectangular cross-section for predicting displacements. To maintain focus and conciseness, only the displacement response at 20 mm below the blade tip was predicted and graphed, using visual monitoring results as a reference. The predicted displacement response is shown in Fig. 9.

Figure 9: Predicting the displacement time history at 20 mm below the blade tip position

As can be seen in Fig. 9, the predicted displacement time histories, both in their entirety and in partial segments, closely match those obtained by visual observation. In addition, the displacements at various points in the time series are in near-perfect agreement with visual observations. Therefore, it is possible to predict the displacement time histories of unknown measurement points within a structure using vibration equations and positional relationships. To examine the relationship between the predicted displacement response at unknown measuring points and the distance from the original monitoring points, Fig. 10 shows the error distributions at distances of 20, 40, 60, and 80 mm from the original monitoring point, and Table 2 lists the corresponding error values.

Figure 10: Error distribution of predicted data at each monitoring point

As shown in Fig. 10, the error in the predicted displacement time histories at the four locations does not exceed 2 mm, with most falling within 0.5 mm, further confirming the feasibility of predicting displacement time histories. Nevertheless, as the distance from the known displacement time history measurement points increases, the variability of errors in the predicted displacements also becomes more pronounced. Similarly, Table 2 confirms this trend, indicating that the errors increase with distance from the known measurement points. The primary reason is that the vibration patterns on the structure change as the distance from known measurement points increases. For instance, when subjected to impact loads, inconsistent vibration patterns may be observed at different locations on the structure at the same time. In this test, the predicted displacement results show satisfactory performance and meet the monitoring requirements. However, in practical large-scale structures, the amplification of errors is inevitable due to the increase in dimensions. Therefore, in actual structural displacement monitoring tasks, visual monitoring serves as the primary method, and this predictive displacement approach is used as a supplement when inaccessible or difficult to monitor measurement points are encountered, ultimately achieving full-field monitoring.

5  Wind Turbine Structural Dynamic Characteristic Testing

5.1 Full-Field Displacement Monitoring of Wind Turbine Structure

In this paper, monocular vision is used for monitoring, focusing solely on the edgewise vibration characteristics of the blades to achieve accurate visual monitoring of wind turbine. The test equipment is identical to that shown in Fig. 3, with the specific layout of the monitoring points shown in Fig. 11. Validation was performed using five acceleration sensors and two LDSs, with the sensor configuration and frequency settings identical to those shown in Fig. 3. Due to the relatively small vibrations in the edgewise direction of the wind turbine compared to its overall dimensions, this paper will primarily focus on the vibration behavior of the blade in the x-direction.

Figure 11: Error distribution of predicted data at each monitoring point

Under the premise of ensuring the accuracy of the method proposed in this paper, dynamic characteristic monitoring of the entire wind turbine structure was conducted. In this test, a vibration table was used for excitation, and the displacement response across the full-field of the wind turbine was monitored. A large number of visual monitoring points were set up in this paper, and for the sake of brevity, only the displacement time histories of three monitoring points on the blades and tower are presented, as shown in Fig. 12. The scale factor was calculated based on the accelerometer at P1 and the visually monitored displacement at P1. Since no tilt angle was used in the camera recording, the actual displacement conversion for all visual displacement responses was performed using the scale factor at P1.

Figure 12: Time history monitoring of full field displacement of wind turbine structure

As shown in Fig. 12, during full-field monitoring of the wind turbine, the displacement time histories of the wind turbine structure show overall consistency, with peak values gradually decreasing as the measurement points move away from a reference position, following the full-field displacement time history pattern. Compared with traditional contact sensors, the vision-based monitoring method allows multiple measurement points to be monitored simultaneously, and for those points beyond visual reach, predicted displacements can be used to perform full-field structural monitoring.

5.2 Vibration Testing of Wind Turbine Structure

Using the vision-based monitoring method in conjunction with an accelerometer to provide the scale factor, the edgewise vibration direction displacement of the wind turbine structure was monitored under the excitation of a shaker table. This test primarily focused on the frequency domain of the entire wind turbine and tower structure, using accelerometers and LDSs to validate the accuracy of the proposed method in the frequency domain. To comprehensively contrast the structural frequency domain information obtained from different monitoring devices, two LDSs were placed at monitoring points P6 and P12 during the tower monitoring. The frequency domain information of the structure measured by three different devices is shown in Fig. 13, with a logarithmic scale adopted for the horizontal axis to better illustrate the comparison.

Figure 13: Comparison of frequency domain information of the overall wind turbine and tower under different monitoring devices

As can be seen from Fig. 13, the three monitoring methods for the blade edgewise vibration direction and tower vibration direction in the overall structure can all effectively identify the first three natural frequencies of the wind turbine blade edgewise vibration, and the overall data distribution is similar to the arrangement of the monitoring points. The main difference in the frequency domain between the overall wind turbine monitoring and the tower monitoring is in the first-order frequency, which is due to the reduction of the structural mass when only the tower without the blades is monitored, which increases the fundamental frequency of the tower. Notably, neither vision-based nor LDS displacement monitoring methods can effectively identify frequencies beyond the third order, while the accelerometer can identify higher order frequencies [26]. Although resampling can be applied to the vision monitoring data to increase the sampling frequency, such linear interpolation methods cannot capture higher frequencies. To compare the stability of the proposed method in the frequency domain, the frequency domain information converted from LDS monitoring is used as a benchmark, and the natural frequencies and errors in the edgewise vibration direction are shown in Table 3.

The data in Table 3 indicate that the errors associated with both vision-based monitoring and LDS detection do not exceed 1%, and that these errors increase with the order of frequency, which is consistent with the monitoring trends. Overall, the frequency accuracy achieved by the method proposed in this paper exceeds 99%, which meets the technical requirements. Furthermore, given the known displacement from vision-based monitoring, structural displacements can be predicted by key parameters such as the structural vibration equation, boundary conditions, and actual damping coefficients, thereby compensating for the limitations of vision-based monitoring caused by factors such as image blur and occlusion.

The vibration modes of the full-field structural displacement time history of the monitored wind turbine were calculated using the Stochastic Subspace Identification (SSI) method and are shown in Fig. 14. Given computational constraints, this paper selected 5 monitoring points on the blades and 6 on the tower, totaling 11 points, deemed adequate to represent the vibration modes of the entire wind turbine structure.

Figure 14: Wind turbine structural mode shape

As shown in Fig. 14, the full-field vibration modes of the wind turbine structure can be effectively calculated using the method proposed in this paper, and the identification of the second-order vibration mode can be achieved by excitation on a small-scale shaking table. Fig. 14 shows the first two vibration modes of the whole wind turbine, where the coupling between the blade and tower vibration modes is observed. However, due to the greater flexibility of the blade, the modal of the blade is more easily excited. In summary, the methodology introduced in this study enables precise remote monitoring of the full-field dynamic characteristics of wind turbine structures, thereby establishing a foundation for the assessment of the condition of large-scale structures, exemplified by wind turbines. Meanwhile, during the tests, it is evident that when the blades vibrate in the edgewise direction, significant vibrations are also observed in the flap-wise direction. Consequently, binocular vision methods can be applied in subsequent research to identify the three-dimensional dynamic characteristics of wind turbines.

6  Conclusions

This paper conducts dynamic characteristic testing of wind turbine structures through the proposed full-field dynamic testing method. Calculate the actual scale factor through visual and acceleration data, and combine it with LKT optical flow method to obtain actual displacement time history data. For monitoring points where visual monitoring is not feasible, structural vibration equations and key parameters such as boundary condition constraints are used to predict the displacements at these locations, ultimately achieving full-field dynamic characteristic monitoring of wind turbine structures. After testing and structural analysis, the following conclusions can be drawn:

(1) Based on the visual displacement time history obtained by the LKT optical flow method, and combining the displacement time history obtained by the double integration of the acceleration data, the scale factor can be calculated by the least squares method, thereby enabling high accuracy calculation of the displacement time history under the physical coordinates of the structure.

(2) Employing the structural vibration equations, boundary condition constraints, and actual damping coefficients as crucial parameters, the displacement time histories at specified monitoring locations can be precisely forecasted given the availability of known displacement time histories, where the prediction accuracy diminishes with increasing distance from the reference point.

(3) The integration of vision-based detection methodologies with displacement prediction techniques facilitates precise full-field displacement time-history monitoring of wind turbine structures, attaining sub-millimeter accuracy in the time domain and exceeding 99% accuracy in the frequency domain.

(4) Considering the unique characteristics of wind turbine structures, a small-scale shaker table was used to perform dynamic characteristic tests on wind turbines under shutdown conditions. The results indicate that the method proposed in this paper can accurately identify the first two mode shapes of the wind turbine structure.

Acceleration data contains high-frequency information. If the high-frequency information collected by acceleration sensors can be fused with visual monitoring data, it may be promising to identify higher-order frequencies of wind turbine structures. In the future, we will focus on the fusion of high-frequency information from acceleration sensors with visual monitoring data, and consider using high-frequency binocular cameras to achieve high-frequency three-dimensional dynamic characteristic identification of the entire wind turbine structure.

Acknowledgement: Thanks to all team members for their work and contributions.

Funding Statement: The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by the National Science Foundation of China (Grant Nos. 52068049 and 51908266), the Science Fund for Distinguished Young Scholars of Gansu Province (No. 21JR7RA267), and Hongliu Outstanding Young Talents Program of Lanzhou University of Technology.

Author Contributions: The authors confirm contribution to the paper as follows: study conception and design: Wenhai Zhao, Wanrun Li; data collection: Wenhai Zhao; analysis and interpretation of results: Wenhai Zhao, Wanrun Li, Ximei Li, Yongfeng Du; draft manuscript preparation: Wenhai Zhao, Wanrun Li, Ximei Li, Shoutu Li, Yongfeng Du. All authors reviewed the results and approved the final version of the manuscript.

Availability of Data and Materials: The data used to support the findings of the study are available from the corresponding author upon request.

Ethics Approval: Not applicable.

Conflicts of Interest: The authors declare no conflicts of interest to report regarding the present study.

References

1. Sun S, Wang T, Chu F. In-situ condition monitoring of wind turbine blades: a critical and systematic review of techniques, challenges, and futures. Renew Sustain Energy Rev. 2022;160:112326. doi:10.1016/j.rser.2022.112326. [Google Scholar] [CrossRef]

2. Hernandez-Estrada E, Lastres-Danguillecourt O, Robles-Ocampo JB, Lopez-Lopez A, Sevilla-Camacho PY, Perez-Sariñana BY, et al. Considerations for the structural analysis and design of wind turbine towers: a review. Renew Sustain Energy Rev. 2021;137:110447. doi:10.1016/j.rser.2020.110447. [Google Scholar] [CrossRef]

3. Oliveira G, Magalhães F, Cunha Á., Caetano E. Continuous dynamic monitoring of an onshore wind turbine. Eng Struct. 2018;164:22–39. doi:10.1016/j.engstruct.2018.02.030. [Google Scholar] [CrossRef]

4. Wang L, Zhang Z. Automatic detection of wind turbine blade surface cracks based on UAV-taken images. IEEE Trans Ind Electron. 2017;64(9):7293–7303. doi:10.1109/TIE.2017.2682037. [Google Scholar] [CrossRef]

5. Ruiz M, Mujica LE, Alférez S, Acho L, Tutivén C, Vidal Y, et al. Wind turbine fault detection and classification by means of image texture analysis. Mech Syst Signal Process. 2018;107:149–67. doi:10.1016/j.ymssp.2017.12.035. [Google Scholar] [CrossRef]

6. Simoncelli M, Zucca M, Ghilardi M. Structural health monitoring of an onshore steel wind turbine. J Civil Struct Health Monit. 2024;14:1423–37. doi:10.1007/s13349-024-00794-w. [Google Scholar] [CrossRef]

7. Vacca G, Vecchi E. UAV photogrammetric surveys for tree height estimation. Drones. 2024;8(3):106. doi:10.3390/drones8030106. [Google Scholar] [CrossRef]

8. Beganovic N, Söffker D. Structural health management utilization for lifetime prognosis and advanced control strategy deployment of wind turbines: an overview and outlook concerning actual methods, tools, and obtained results. Renew Sustain Energy Rev. 2016;64:68–83. doi:10.1016/j.rser.2016.05.083. [Google Scholar] [CrossRef]

9. Amenabar I, Mendikute A, López-Arraiza A, Lizaranzu M, Aurrekoetxea J. Comparison and analysis of non-destructive testing techniques suitable for delamination inspection in wind turbine blades. Compos B. 2011;42(5):1298–305. doi:10.1016/j.compositesb.2011.01.025. [Google Scholar] [CrossRef]

10. Yang B, Sun D. Testing, inspecting and monitoring technologies for wind turbine blades: a survey. Renew Sustain Energy Rev. 2013;21:515–26. [Google Scholar]

11. Ni YQ, Ye XW, Ko JM. Monitoring-based fatigue reliability assessment of steel bridges: analytical model and application. J Struct Eng. 2010;136(12):1563–73. doi:10.1061/(ASCE)ST.1943-541X.0000250. [Google Scholar] [CrossRef]

12. Qiao W, Lu D. A survey on wind turbine condition monitoring and fault diagnosis—Part II: signals and signal processing methods. IEEE Trans Ind Electron. 2015;62(10):6546–57. doi:10.1109/TIE.2015.2422394. [Google Scholar] [CrossRef]

13. Spencer BFJr, Hoskere V, Narazaki Y. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineer. 2019;5(2):199–222. [Google Scholar]

14. Felipe-Sesé L, Díaz FA. Damage methodology approach on a composite panel based on a combination of Fringe Projection and 2D Digital Image Correlation. Mech Syst Signal Process. 2018;101:467–79. doi:10.1016/j.ymssp.2017.09.002. [Google Scholar] [CrossRef]

15. Kuddus MA, Li J, Hao H, Li C, Bi K. Target-free vision-based technique for vibration measurements of structures subjected to out-of-plane movements. Eng Struct. 2019;190:210–22. doi:10.1016/j.engstruct.2019.04.019. [Google Scholar] [CrossRef]

16. Dong CZ, Catbas FN. A review of computer vision-based structural health monitoring at local and global levels. Struct Health Monit. 2021;20(2):692–743. doi:10.1177/1475921720935585. [Google Scholar] [CrossRef]

17. Dong CZ, Bas S, Catbas FN. Investigation of vibration serviceability of a footbridge using computer vision-based methods. Eng Struct. 2020;224:111224. doi:10.1016/j.engstruct.2020.111224. [Google Scholar] [CrossRef]

18. Zhao X, Liu H, Yu Y, Zhu Q, Hu W, Li M, et al. Displacement monitoring technique using a smartphone based on the laser projection-sensing method. Sensor Actuat A-Phys. 2016;246(9):35–47. doi:10.1016/j.sna.2016.05.012. [Google Scholar] [CrossRef]

19. Lucas BD, Kanade T. An iterative image registration technique with an application to stereo vision. In: IJCAI’81: 7th International Joint Conference on Artificial Intelligence, 1981 Aug; Vancouver, BC, Canada. [Google Scholar]

20. Shi J. Good features to track. In: 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1994 Jun 21–23; Seattle, WA, USA. [Google Scholar]

21. Zhao WH, Li WR, Yang MH, Hong N, Du YF. Dynamic characteristics monitoring of wind turbine blades based on improved YOLOv5 deep learning model. Smart Struct Syst. 2023;31(5):469–83. [Google Scholar]

22. Xiu C, Weng Y, Shi W. Vision and vibration data fusion-based structural dynamic displacement measurement with test validation. Sensors. 2023;23(9):4547. doi:10.3390/s23094547. [Google Scholar] [PubMed] [CrossRef]

23. Feng D, Feng MQ. Identification of structural stiffness and excitation forces in time domain using noncontact vision-based displacement measurement. J Sound Vib. 2017;406(5):15–28. doi:10.1016/j.jsv.2017.06.008. [Google Scholar] [CrossRef]

24. Li W, Zhao W, Gu J, Fan B, Du Y. Dynamic characteristics monitoring of large wind turbine blades based on target-free DSST vision algorithm and UAV. Remote Sens. 2022;14(13):3113. doi:10.3390/rs14133113. [Google Scholar] [CrossRef]

25. Khuc T, Catbas FN. Completely contactless structural health monitoring of real-life structures using cameras and computer vision. Struct Control Health Monit. 2017;24(1):e1852. doi:10.1002/stc.1852. [Google Scholar] [CrossRef]

26. Feng D, Feng MQ. Experimental validation of cost-effective vision-based structural health monitoring. Mech Syst Signal Process. 2017;88(12):199–211. doi:10.1016/j.ymssp.2016.11.021. [Google Scholar] [CrossRef]