Intelligent Recognition and Life Prediction of Bridge Fatigue Cracks Based on Multimodal Visual Fusion and Spatiotemporal Convolutional Networks

Bridges, as the core hubs of the transportation infrastructure network, have structural safety that directly relates to regional traffic accessibility and public safety [1, 2]. Under the coupled effects of long-term loads, environmental corrosion, and material aging, fatigue cracks have become the primary cause leading to sudden failure of bridge structures [3, 4]. According to statistics from the International Association for Bridge Maintenance and Safety, approximately 35% of bridge defects worldwide originate from early fatigue crack propagation that is not detected in time. Such defects not only result in maintenance costs reaching up to 40% of construction costs, but may also cause serious casualties [5, 6]. Therefore, achieving early and accurate identification of fatigue cracks and life trend prediction plays an irreplaceable key role in improving the proactiveness of bridge operation and maintenance and reducing life-cycle costs.

The development of bridge crack recognition technology has undergone an important transition from contact-based detection to non-contact visual detection. Among traditional contact-based methods, ultrasonic testing identifies internal cracks through the propagation characteristics of acoustic waves, but its detection range is limited by the coverage area of the probe [7]. Magnetic particle testing is only applicable to ferromagnetic materials and requires strict surface cleanliness [8]. With the rise of machine vision technology, single-modality visual detection methods have been widely studied. RGB image-based methods achieve recognition by extracting texture and edge features of cracks, which has the advantage of convenient data acquisition, but is easily affected by illumination changes and background noise [9]. Depth image-based methods can obtain three-dimensional geometric information of cracks and can effectively distinguish surface depressions from real cracks, but the measurement accuracy of depth sensors limits the recognition capability for micro-cracks [10]. Infrared thermography locates cracks through temperature differences, but it is highly sensitive to ambient temperature [11].

The application of multimodal fusion technology in the field of structural health monitoring has achieved certain progress. Existing fusion strategies can be divided into three categories: early fusion, intermediate fusion, and late fusion [12]. Early fusion integrates multimodal information at the data level and is easily affected by noise differences between modalities. Intermediate fusion performs information interaction at the feature level and needs to solve the problem of mismatched feature dimensions between different modalities. Late fusion combines results at the decision level and cannot fully utilize the complementary information between modalities. Existing studies mostly adopt simple feature concatenation or weighted fusion methods, which fail to fully consider the specific characteristics of each modality, resulting in unsatisfactory fusion performance and making it difficult to meet the detection requirements under complex engineering scenarios [13-15].

Temporal feature extraction is the core step for realizing crack life prediction. Existing studies mostly adopt Recurrent Neural Networks (RNN) models such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) to process temporal data. Such models can effectively capture dependency relationships in the time dimension, but their representation ability for the spatial topology features of cracks is insufficient [15, 16]. Graph Convolutional Networks (GCNs) have unique advantages in processing non-Euclidean data and have been applied to spatial feature extraction of structural damage, but when used alone they cannot explore the temporal patterns of crack propagation [17, 18]. Existing models generally have the problem of insufficient spatiotemporal feature coupling modeling, making it difficult to simultaneously capture the spatial distribution and temporal evolution characteristics of cracks [19, 20].

Comprehensive analysis of existing studies shows that there are three core gaps in the current field of bridge fatigue crack detection. First, the extraction of modality-specific features from multimodal data is insufficient, and the complementary advantages of different modalities are not effectively integrated. Second, the coupling modeling capability of spatiotemporal features is insufficient, making it impossible to simultaneously represent the spatial topology and temporal evolution information of cracks. Third, the feature fusion strategy is overly simple and has limited ability to mine abstract features in hidden layers, resulting in recognition accuracy and robustness of the model that are difficult to meet engineering requirements.

The research objective of this paper is to construct a multimodal intelligent model integrating graph convolution and spatiotemporal convolution, in order to achieve high-precision recognition and life trend prediction of bridge fatigue cracks and provide a new technical means for bridge structural health monitoring. To achieve this objective, the core contributions of this paper are mainly reflected in four aspects. First, a feature transformation unit with graph convolution embedded in a LSTM network is innovatively designed. After modeling crack images as graph structures, spatial topology features are extracted through graph convolution, and temporal dependency relationships are further mined through the LSTM network, thereby achieving collaborative extraction of spatiotemporal features. Second, a dual independent C3D architecture is proposed. According to the feature differences between RGB and depth video modalities, corresponding network parameters and structures are designed respectively, enabling targeted extraction of spatiotemporal features of the two modalities. Third, a feature fusion strategy combining cascade fusion and high-dimensional mapping is constructed. Original feature information of each modality is first preserved through cascade fusion, and then the fused features are mapped to a higher dimension through a fully connected layer to enhance the representation capability of abstract features in hidden layers. Finally, a supervised training mechanism is constructed using a multi-class cross-entropy loss function. Combined with dropout regularization and an early stopping strategy, the recognition robustness of the model under complex conditions such as illumination changes and occlusion is improved.

The remaining chapters of this paper are organized as follows. Chapter 2 systematically elaborates the theoretical foundations of GCN, LSTM networks, C3D, and multimodal fusion, providing theoretical support for model construction. Chapter 3 introduces in detail the overall architecture of the proposed model, including the design details of the three feature extraction branches, the multimodal fusion module, and the training strategy. Chapter 4 quantifies the contribution of core modules through ablation experiments, and verifies the advantages of the “recognition–prediction” framework through comparative experiments and feature fusion effect analysis. Chapter 5 summarizes the core conclusions of the paper and outlines the academic contributions and engineering value of the research.

3.1 Overall architecture of the model

To achieve efficient extraction and fusion of multimodal spatiotemporal features of bridge fatigue cracks, the proposed model adopts an overall topological structure of “three-branch parallel extraction–feature-level convergence fusion–classification decision.” Each module has a clear functional positioning and coordinated connection. Among them, the GCN-LSTM temporal branch takes crack sequence images as input. First, the graph modeling module converts a single-frame image into graph-structured data. Then, a first-order GCN extracts the spatial topology features of cracks, followed by an LSTM network capturing the temporal dependency relationships of the feature sequence, and finally outputs a 512-dimensional feature vector integrating spatiotemporal information. The dual C3D branches process multimodal video data in parallel. The RGB-C3D branch focuses on the texture advantages of RGB video and extracts spatiotemporal features of crack edges and grayscale changes through a 3 × 3 × 3 convolution kernel and five convolution–pooling operations, outputting a 1024-dimensional feature vector. The Depth-C3D branch adapts to the geometric characteristics of depth video, adopts a 5 × 5 × 5 convolution kernel and batch normalization layers to suppress noise and extract spatiotemporal features of three-dimensional morphology, and outputs a 1024-dimensional feature vector. The feature vectors of the three branches are fused cooperatively in the convergence module. First, a cascade operation completely preserves the specificity of each modality and spatiotemporal feature. Then, an 8192-dimensional fully connected layer performs high-dimensional mapping to mine complementary information in hidden layers. Finally, the fused features are input into a Softmax classification layer to output prediction results of four crack states, forming an end-to-end processing process from data input to decision output. Figure 2 shows the topology of the model, intuitively presenting the input data types of each branch and the feature transmission process.

Figure 2. Overall architecture of the multimodal bridge fatigue crack intelligent recognition and life prediction model

3.2 Graph Convolutional Network–Long Short-Term Memory temporal feature extraction branch

The core function of the GCN-LSTM temporal branch is to synchronously extract spatial topology and temporal dependency features from crack sequence images. Its design covers three key stages: feature input preprocessing, construction of the GCN feature transformation unit, and temporal feature output. The parameters and logic of each stage are closely connected. Figure 3 shows the internal logic of the GCN-LSTM module.

In the feature input stage, targeted preprocessing is required to improve data quality and model generalization. The specific parameter design fully adapts to the requirements of subsequent graph modeling and temporal modeling. First, bilinear interpolation is used to uniformly normalize the sequence images to 512 × 512 pixels. This size can completely preserve details such as micro-cracks while accurately matching the subsequent non-overlapping node division of 16 × 16 pixels. Noise suppression adopts a combined strategy of “Gaussian filtering–median filtering.” A Gaussian filter with a size of 5 × 5 is first used to smooth Gaussian noise introduced during image acquisition, and then a median filter with a size of 3 × 3 is used to eliminate the interference of impulse noise on crack edges. Data augmentation is designed according to the continuity characteristics of temporal data, adopting temporally consistent random flipping, brightness adjustment, and small-angle rotation to avoid destroying the temporal evolution pattern of crack propagation. The preprocessed sequence data is input into the branch in the form of temporal windows with a length of 16. The final input form is X_seq∈R^{16 × 512 × 512 × 1}.

Figure 3. Structure diagram of the crack spatiotemporal feature extraction module of the Graph Convolutional Network–Long Short-Term Memory temporal branch

The GCN feature transformation unit is the core that connects spatial and temporal modeling. Its parameter design focuses on the effectiveness of graph structure representation and the coupling adaptability of GCN-LSTM, specifically including graph structure construction and GCN-LSTM coupled computation. Graph structure construction is implemented based on a single-frame preprocessed image. The core parameters are as follows. Nodes are defined as non-overlapping image regions of 16 × 16 pixels. Each frame forms 1024 nodes. The feature vector of each node x∈R⁸ consists of eight statistical features: grayscale mean, grayscale variance, gradient magnitude mean, gradient direction entropy, energy, contrast, correlation, and homogeneity within the region. Thus, a single-frame graph structure G=(V,E) is formed, where E represents the edge set. The construction of the adjacency matrix A∈R^{1024 × 1024} integrates spatial position and feature similarity, where the spatial distance threshold d₀ is set to 48 pixels, and the feature similarity coefficient σ takes the mean value of the standard deviations of all node feature vectors. The specific calculation is shown in Eq. (1):

$A_{i j}=\left[\exp \left(-\left\|x_i-x_j\right\|_2 / \sigma^2\right)\right] \cdot I\left(d_{i j} \leq d_0\right)$                 (1)

In the formula, x_i and x_j are node feature vectors; σ is the feature similarity coefficient; d_ij is the spatial distance between nodes; d₀ is the spatial distance threshold; I( ) is the indicator function.

The coupled computation of GCN and LSTM is realized through parameter dimension matching. GCN adopts a two-layer convolution structure. The input is the node feature matrix of a single-frame graph X_{GCN_in}∈R^{1024 × 8}. After feature aggregation through the normalized adjacency matrix U'=D^−1/2(A+I)D^−1/2, nonlinear transformation is introduced through the ReLU activation function. The convolution weights and bias parameters are respectively set as W₁∈R^{8 × 128}, b₁∈R¹²⁸, and W₂∈R^{128 × 256}, b₂∈R²⁵⁶. The final output feature matrix is X_{GCN_out}∈R^1024×256. The specific calculation is shown in Eq. (2):

$X_{G C N \_ \text {out }}=U^{\prime} \operatorname{ReLU}\left(U^{\prime} X_{G C N \_ \text {in }} W_1+b_1\right) W_2+b_2$                   (2)

In the formula, D is the degree matrix and I is the identity matrix. To adapt to the sequence input format of LSTM, X_{GCN_out} is flattened into a vector of 1 × 262144 according to node order, and then compressed to 256 dimensions through one fully connected layer, obtaining the single-frame spatial feature vector f_spatial∈R²⁵⁶. The spatial features of 16 frames within the temporal window further form a sequence f_spatial-seq∈R^{16 × 256}, which serves as the input sequence of the LSTM network.

Temporal feature output is completed by the LSTM network, whose parameters are optimized according to the temporal evolution characteristics of cracks. The LSTM adopts a two-layer stacked structure to enhance the ability to capture temporal dependencies. The hidden layer dimension is set to 512, the forget gate bias is initialized to 0.1, and the dropout probability is set to 0.2. After the input sequence f_spatial-seq∈R^{16 × 256} is processed by the LSTM network, the hidden state at the last time step is taken as the output feature of the temporal branch f_temporal∈R⁵¹². This vector integrates both the crack spatial topology features extracted by GCN and the temporal dependency information of crack propagation mined by LSTM, providing core temporal feature support for subsequent multimodal fusion.

3.3 Dual convolutional 3D network multimodal feature extraction branch

The core function of the dual C3D multimodal feature extraction branch is to specifically mine modality-specific spatiotemporal features of RGB and depth videos. Through the process of “modality-adaptive parameter design–feature hierarchical extraction–standardized alignment,” it provides high-quality feature input for multimodal fusion. Among them, the RGB-C3D branch focuses on the extraction of appearance features such as texture and edges, while the Depth-C3D branch adapts to the geometric characteristics and noise distribution of depth data. The two branches process data in parallel and share the feature extraction logic but use differentiated parameter configurations.

The RGB-C3D branch adopts the classic architecture of “five convolution–pooling layers + two fully connected layers.” The parameters of each layer are optimized according to crack texture features to ensure the gradual abstraction and dimensional compression of spatiotemporal information. The specific parameter settings are as follows. The first convolution layer adopts a 3 × 3 × 3 convolution kernel with 64 kernels. The stride is set to (1,2,2), and the padding is set to “Same” to maintain the spatial size of the feature map. After ReLU activation, a 3 × 2 × 2 max pooling layer is connected, and the output feature map size is (8,256,256,64). The convolution kernel sizes of layers 2–5 remain 3 × 3 × 3, and the numbers of kernels increase sequentially to 128, 256, 512, and 512. The stride and padding configurations are the same as in the first layer. The corresponding pooling layers all adopt 3 × 2 × 2 max pooling. After five convolution–pooling operations, the output feature map size is compressed to (2,16,16,512). The fully connected layers are designed as a two-level structure of “high-dimensional compression–feature output.” The first fully connected layer receives the flattened feature after pooling with a dimension of 2 × 16 × 16 × 512 = 262144, and the number of neurons is set to 4096. After ReLU activation and dropout, it is input into the second fully connected layer. The number of neurons in the second layer is set to 1024, which directly outputs the RGB modality spatiotemporal feature vector f_RGB∈R¹⁰²⁴, which is consistent with the output dimension of the Depth-C3D branch, ensuring dual-modality feature alignment and providing a dimensional basis for the multi-branch cascade fusion in Section 3.1. The introduction of the ReLU activation function effectively alleviates the gradient disappearance problem, and the design where the spatial dimension stride is larger than the temporal dimension adapts to the characteristic of crack propagation where “spatial morphology mutation is small and temporal evolution is continuous.”

The Depth-C3D branch performs modality-adaptive fine-tuning based on the RGB-C3D architecture. The parameters are optimized mainly for the characteristics of depth video, including single-channel input, narrow grayscale dynamic range, and concentrated noise distribution. First, the single-channel depth video is expanded to three channels in the input layer. At the same time, the input frame size is adjusted to 512 × 512, and the temporal window length is set to 16, which is consistent with the RGB branch. The fine-tuning strategy of the convolution–pooling layers includes three aspects. The number of initial convolution kernels is reduced from 64 to 32 to reduce the amplification effect of high-dimensional noise in shallow layers. The convolution kernel sizes of the first three layers are adjusted to 5 × 5 × 5 to increase the spatial convolution range and improve the ability to capture low-resolution depth cracks. The fourth and fifth layers restore the 3 × 3 × 3 convolution kernels to refine features. Batch normalization layers are added after the first three convolution layers to suppress grayscale offset noise in depth data through mean and variance normalization. The batch size is set to 16 to match the training batch size. The parameters of the fully connected layers are exactly the same as those of the RGB branch, and the output depth modality spatiotemporal feature vector is f_Depth∈R¹⁰²⁴, ensuring alignment of dual-modality feature dimensions.

Multimodal feature normalization is a key step to eliminate dimensional differences between modalities and improve fusion performance. L2 normalization is adopted to uniformly process the output features of the two branches. For the feature vector f_norm∈R¹⁰²⁴, the normalization calculation is shown in Eq. (3). By projecting the feature vector onto the unit hypersphere, the numerical ranges of different modality features remain consistent, avoiding the dominance of RGB texture features in the fusion process due to their higher numerical magnitude. The normalized feature vectors f_RGB-norm∈R¹⁰²⁴ and f_Depth-norm∈R¹⁰²⁴ are directly input into the subsequent multimodal fusion module.

$f_{\text {norm }}=\frac{f}{\|f\|_2}$                    (3)

In the formula, f is the output feature vector of the dual C3D branches; ||f||₂ is the L2 norm of the feature vector.

3.4 Multimodal feature fusion and classification module

The multimodal feature fusion and classification module is the core for the model to achieve decision output. Its design takes “complete preservation of modality specificity–deep mining of hidden-layer complementarity–accurate output of classification results” as the objective. Through a three-level structure of cascade fusion, high-dimensional mapping, and classification output, efficient integration of multi-branch features and accurate identification of crack states are achieved. The selection of this fusion strategy is based on the comparative analysis in Section 2.4. Cascade fusion takes maximizing the preservation of feature specificity of each branch as the core objective. It directly concatenates the output features of the GCN-LSTM temporal branch and the dual C3D modality branches, without performing dimensional compression or weight allocation on the features, thus avoiding the loss of modality information. The high-dimensional mapping layer then maps the 2560-dimensional concatenated features to a higher-dimensional space through an 8192-dimensional fully connected network, using nonlinear transformation to mine hidden complementary information between modalities and compensating for the limitation that cascade fusion only performs simple concatenation.

The cascade fusion strategy takes maximizing the preservation of feature specificity of each branch as the core objective. It directly concatenates the output features of the GCN-LSTM temporal branch and the dual C3D modality branches, without performing dimensional compression or weight allocation on the features, thereby avoiding modality information loss. Combined with the parameter definitions above: the GCN-LSTM branch outputs the temporal feature vector f_temporal∈R⁵¹², which integrates spatial topology and temporal dependency; the RGB-C3D branch outputs the RGB modality feature after five convolution–pooling operations and fully connected layers, and after L2 normalization obtains f_RGB-norm∈R¹⁰²⁴; the Depth-C3D branch outputs the depth modality feature f_Depth-norm∈R¹⁰²⁴ after modality-adaptive fine-tuning and normalization. The cascade operation concatenates the vectors in a fixed order of “temporal feature–RGB feature–depth feature,” forming a concatenated feature f_concat∈R²⁵⁶⁰. This preserves the crack propagation evolution information of the temporal branch, the texture edge information of the RGB branch, and the three-dimensional geometric information of the depth branch, providing a data basis for subsequent complementary information mining.

The high-dimensional mapping layer is designed with the core objective of strengthening the abstract representation capability of features. Its dimensional setting and network structure are optimized based on feature fusion theory and the principle of overfitting control. The dimensional choice is mainly based on two aspects. First, the concatenated feature dimension is 2560, and sufficient parameter capacity is required to mine hidden associations between modalities. Referring to the consensus in the field of feature fusion that “high-dimensional spaces are easier to represent complementary information,” the mapping dimension is set to 8192. Second, “high-dimensional mapping + regularization” is used to balance representation capability and overfitting risk, avoiding parameter redundancy caused by excessively high dimensions. The specific structure is a single hidden-layer fully connected network. The input is concat∈R²⁵⁶⁰, with a weight matrix W_map∈R^{2560 × 8192} and bias b_map∈R⁸¹⁹². A nonlinear transformation is introduced through the ReLU activation function to enhance feature representation capability. Then a dropout layer is connected to randomly deactivate part of the neurons, breaking redundant dependencies between features and improving model generalization. The final output is the high-dimensional fusion feature f_fusion∈R⁸¹⁹². The gradient characteristics of the ReLU activation function are suitable for gradient backpropagation of high-dimensional features, ensuring the effectiveness of parameter updates.

The classification output layer adopts the Softmax function to realize the multi-class mapping of crack states. The target categories are consistent with the task requirements and are divided into four classes: no crack, micro-crack, medium crack, and macro crack. This layer takes the high-dimensional fusion feature f_fusion∈R⁸¹⁹² as input, projects the high-dimensional feature into the category space through the classification weight matrix W_c∈R^{8192 × 4} and bias b_c∈R⁴, and then normalizes it into category probabilities through the Softmax function to output the probability vector y∈R⁴. Among them, y_i represents the probability that the sample belongs to class i. The calculation is shown in Eq. (4):

$y_i=\frac{\exp \left(W_{c, i} f_{\text {fusion }}+b_{c, i}\right)}{\sum_{j=1}^4 \exp \left(W_{c, j} f_{\text {fusion }}+b_{c, j}\right)}$                    (4)

In the formula, W_c,i and b_c,i are the i-th row of the classification weight matrix and the i-th element of the bias vector, respectively. i=1,2,3,4 correspond to the categories of no crack, micro-crack, medium crack, and macro crack. The parameters of the classification layer are initialized using the He normal distribution, which adapts to the gradient characteristics of the ReLU activation function to accelerate model convergence. During training, it works together with the multi-class cross-entropy loss function defined in Section 3.5. Through gradient backpropagation, the parameters of the high-dimensional mapping layer and the classification layer are optimized simultaneously, ensuring classification accuracy and generalization ability.

3.5 Model training strategy

The model training strategy takes “precise loss supervision–efficient convergence optimization–strong generalization guarantee” as the objective. Combined with the characteristics of the four-class crack classification task and the complex structure of multi-branch high-dimensional parameters, a collaborative scheme of “loss function–optimizer–regularization and early stopping” is designed to achieve a balance between convergence efficiency and generalization performance. The parameters of each stage strictly match the model structure described above.

For the four-class task of “no crack, micro-crack, medium crack, macro crack,” the multi-class cross-entropy loss function is adopted to construct the supervision signal. Its core advantage is that it adapts to the class imbalance problem in engineering data where minority classes such as micro-cracks account for a small proportion. Through logarithmic penalty, it strengthens the learning of key category features. The mathematical expression is shown in Eq. (5):

$L=-\left(\frac{1}{N}\right) \sum_{n=1}^N \sum_{m=1}^N y_{n, m} \cdot \log \left(\hat{y}_{n, m}\right)$                     (5)

In the formula, N is the number of samples in a training batch; m=1,2,3,4 correspond to the four labels of no crack, micro-crack, medium crack, and macro crack, respectively; y_n,m is the true label of the n-th sample; ŷ_n,m is the predicted probability output by the classification layer in Section 3.4. This function produces a larger loss value for samples whose predicted probability of the true class is close to 0, thereby specifically strengthening the training weight of difficult samples such as micro-cracks. It directly matches the probability output of the classification layer and provides an accurate supervision signal for gradient backpropagation.

The Adam optimizer is selected to realize efficient updating of model parameters. It combines the advantages of momentum gradient descent and adaptive learning rate, adapting to the complex model structure with multi-branch and high-dimensional parameters in this study. The parameter settings consider both convergence speed and stability: the initial learning rate is set to 1 × 10^-4; the momentum factors are β₁ = 0.9 and β₂ = 0.999; and the numerical stability parameter ε = 1 × 10^-8.

The core advantages of the Adam optimizer are reflected in two aspects. First, for the graph structure parameters of the GCN-LSTM branch and the convolution parameters of the dual C3D branches, it adaptively allocates differentiated learning rates, avoiding update imbalance caused by differences in parameter scale between convolution layers and fully connected layers. Second, through the momentum term, it suppresses loss fluctuation in the later stage of training, ensuring stable convergence of parameter-intensive modules such as the 8192-dimensional high-dimensional mapping layer. During training, a learning rate decay strategy is adopted. When the validation loss does not decrease for five consecutive epochs, the learning rate is reduced to 0.5 times the original value, further improving convergence accuracy.

A triple strategy of “Dropout + L2 regularization + early stopping” is adopted to suppress overfitting, and each measure is precisely matched with the model structure. Dropout layers are deployed as required: the LSTM layer in the GCN-LSTM branch sets the dropout rate to 0.2, and the high-dimensional mapping layer in the multimodal fusion module sets the dropout rate to 0.5. Dropout breaks redundant dependencies between features by randomly deactivating neurons and automatically scales outputs during training to maintain the mean of features. L2 regularization is applied to all trainable parameters with a weight decay coefficient of 1 × 10^-5. By adding the L2 norm term of parameters into the loss function, it suppresses overfitting caused by excessively large parameter magnitudes.

The early stopping mechanism dynamically terminates training based on the performance of the validation set to avoid overfitting on the training set. The F1 score of the validation set is used as the monitoring metric, and the patience value is set to 10. That is, when the validation F1 score does not exceed the historical best value for 10 consecutive epochs, training is terminated and the optimal model parameters are saved. Before training, the dataset is divided into training, validation, and test sets with a ratio of 7:2:1. The validation set uses stratified sampling to ensure that the proportion of each category is consistent with the training set, ensuring the reliability of the monitoring metric.