Robust Automatic Modulation Classification Using Spectrogram Features and a Hybrid CNN-LSTM Network

Automatic Modulation Classification (AMC) is a critical function in modern communication systems that connects the signal detection and demodulation links [1]. It has attracted significant attention in cognitive radios, spectrum monitoring, and military communications. By enabling the automatic identification of modulation formats for incoming signals, AMC helps to improve dynamic spectrum management while making communication systems more flexible in complex and noisy environments [2].

Modulation classification is generally categorized into two main approaches: likelihood-based methods and feature-based techniques [3]. While likelihood-based methods are theoretically optimal, they involve an extensive computation of the likelihood function, which demands prior knowledge of signal characteristics. This complexity renders them impractical for real-time applications. Conversely, feature-based methods extract key signal attributes such as amplitude, phase, and frequency [4], utilizing machine learning or deep learning models for classification. In this regard, deep learning has emerged as a transformative approach due to its capability to autonomously learn high-level representations from raw data, eliminating the need for manually engineered features [5].

The first aspect of AMC is robust performance in low Signal-to-Noise Ratios (SNR), which will be experienced in real communications systems. In these scenarios, it is common that conventional feature-based approaches achieve poor performance, as signal features distortion due to noise challenges their robustness. To improve the feature extraction process, researchers were looking into the application of spectrograms (a time-frequency representation of signals) [6]. They form a more expressive representation of the signal, incorporating both time and frequency information, both of which are significant in decoding modulation types. Spectrogram-based methods, along with robust deep learning architectures, have been reported to have very high classification accuracy, even in adverse noise conditions [7].

The spectrogram provides a two-dimensional view of the time-frequency spectrum of the signal. This is computed using Short-Time Fourier Transform (STFT) [8], which segments the signal into small chunks, calculates its Fourier transform, and plots the magnitude spectrum.

1.1 Advantages of spectrograms for AMC

• Time-frequency representation: Spectrogram provides information over time, as well as frequency, making it advantageous for non-

stationary phenomenon analysis [9].

• Feature Richness: Specific modulation schemes create distinctive images in the spectrogram when applied in a specific environment, which can be captured by deep learning models [10].

• Noise Robustness: Spectrograms will distinguish between signal and noise for more accurate classification.

This work proposes a system for detecting several modulation schemes, commonly used within wireless communication systems based on a hybrid topology of CNN and LSTM networks as depicted in Figure 1. The system uses a spectrogram as its input format, which encodes the time and frequency resolution of the modulated signals. Here, we provide details of a system, its components, operating flow, and strengths.

46b47003-4ccc-400c-bc54-b77388722ba5.png

Figure 1. Hybrid CNN-LSTM architecture for signal classification

3.1 Signal preprocessing

This system receives raw modulated signals as input. Examples of these include BPSK, QPSK, 8-PSK, 16-QAM, 64-QAM, PAM4, GFSK, CPFSK, B-FM, DSB-AM, and SSB-AM modulation schemes. The time-domain input signal is passed through the STFT to transform it in a time-frequency representation (in our case a spectrogram). STFT is a critical instrument for investigating the frequency richness of signals as they progress through time. Basically, it splits the input signal x(t) into overlapping chunks, which are mapped into the frequency domain using the Fourier transform. The signal preprocessing steps are as follows:

• Window Size: The signal is divided into overlapping segments using a window function. We used a 256-point Hamming window for the STFT. This window size was chosen to balance frequency resolution and time resolution, as it provides sufficient detail for the signals used in the study.
• Overlap: The windows overlap by 50%, meaning each window is shifted by half its size. This overlap ensures that we capture the temporal variations in the signal and avoid losing information at the edges of the windows.
• STFT Parameters: is applied with the following parameters:
  • Window Type: Hamming window.
  • Window Size: 256 samples.
  • Overlap: 50% (128 samples).
  • FFT Size: 512 points, which ensures that the frequency bins are adequately resolved.

These preprocessing steps ensure that the spectrograms generated are detailed enough to capture the essential characteristics of the modulated signals while maintaining the computational efficiency required for model training.

Mathematical Formulation of STFT:

$X(t . f)=\int_{-\infty}^{\infty} x(\tau) h(t-\tau) e^{-j 2 \pi f \tau} d \tau$  

where:

• $x(t)$: Input signal (time domain)
• $h(t-\tau)$: The window function applied to each segment.
• $X(t . f)$: The signal in the time-frequency domain, otherwise known as the spectrogram.
• $f$: The frequency variable.

The STFT method splits the signal into small, overlapping windows, performs the Fourier Transform on each, and generates a 2D time-frequency diagram. In this diagram, time is represented on the x-axis, frequency on the y-axis, and the magnitude of the frequency is represented by color or intensity. By using spectrograms, the system effectively captures the key features of modulated signals in a format suitable for further analysis and classification.

3.2 CNNs

The CNN is a precise deep learning architecture used to process image data generally. CNNs have changed the game of computer vision by allowing machines to examine and classify images, identify objects, and even produce new images. Inspired by the human visual system, CNNs excel at identifying spatial hierarchies and complex image patterns.

A CNN consists of a complex arrangement of interconnected layers, each responsible for the extraction and transformation of features from input images. The convolutional layer is the first basic layer, where small filter matrices (kernels) are used to convolve with the input image. This detects local features such as edges, textures, and shapes, producing a feature map indicating important aspects of the image. By combining multiple filters, various features are captured, so the model is more capable of recognizing complex patterns. In CNNs, the equation for convolution is:

$Y_{i.j}=\sum_m \sum_n X_{i+m.j+n} \cdot W_{m.n}+b$     (1)

where:

• $X_{i . j}$ is the input image at position $(i . j)$,
• $W_{m \cdot n}$ is the filter (kernel) at position $(m . n)$,
• $Y_{i.j}$ is the output of the convolution at position $(i . j)$,
• $b$ is the bias term.

The activation function is vital for improving the networks ability to learn complex representations. Their most common activation function is Rectifying Linear Unit (ReLU), which allows non-linearity while retaining important information about the input. ReLU is able to perform much faster, but only sacrifice the lower end of the data without sacrificing anything critical. Where the ReLU activation function is as such:

$\operatorname{ReLU}(x)=\max (0 . x)$    (2)

where:

• If $x$ is positive, the function returns $x$,
• If $x$ is negative, the function returns 0.

After the convolution and activation stages, a pooling layer is applied to refine feature extraction by reducing the spatial dimensions of feature maps. This down-sampling process not only minimizes computational overhead but also enhances the model's robustness against slight variations in input images. The dominant pooling method is max pooling, which retains the most prominent feature by selecting the highest value within a defined region; for Max Pooling, the output is:

$Y_{i.j}=\max \left(X_{i.j} \cdot X_{i+1.j+1}.\ldots.X_{i+k.j+k}\right)$    (3)

where:

• $X_{i.j}$ is the feature map at position $(i . j)$.
• $Y_{i.j}$ is the result after applying max pooling on a region of the feature map.

Table 1 presents the layer configuration of the CNN architecture. Each convolutional layer applies a kernel of size [3 × 3], which serves as a small filter that scans the input feature maps to detect local patterns such as edges and textures. The stride, set to [1 × 1] for convolutional layers, controls how far the kernel moves at each step, ensuring detailed feature extraction. In contrast, the max pooling layers use kernels of size [2 × 2] with a stride of [2 × 2] to downsample the feature maps, reducing spatial dimensions while retaining the most important features.

After extracting the data with 6 convolutional layers, it is then fed through a flattening layer: this turns the multidimensional data into one-dimensional data that can be processed by the LSTM and fully connected layers:

Flatten $(\mathrm{X})=$ Reshape $(\mathrm{X})$     (4)

Table 1. CNN layer configuration for hybrid CNN-LSTM architecture

3.3 LSTM networks

LSTM networks are a specialized form of Recurrent Neural Networks (RNNs) that are designed to manage sequential data, particularly when long-term dependencies need to be captured. These networks excel in addressing the issue of the vanishing gradient, a challenge that causes traditional RNNs to fail at learning patterns over extended sequences. LSTM networks are built with three critical components: the input gate, which controls the information added to the memory; the forget gate, which regulates what data should be removed from the memory; and the output gate, which determines the information to be used as the output at each time step.

The forget gate in an LSTM network is responsible for deciding which parts of the memory (cell state) should be discarded, enabling the network to retain only the most relevant information for future processing.

The sigmoid activation outputs between 0 and 1 values, with 0 indicating full forget and 1 indicating full retention of data. By retaining only relevant historical data, this guarantees that only the information that is truly relevant has influence over the predictions. The equation for the forget gate is:

$f_t=\sigma\left(W_f\left[h_{t-1}.x_t\right]+b_f\right)$    (5)

where:

• $f_t$: Output of forget gate at time step t.
• $\sigma$: a sigmoid activation function, a function producing values between 0 and 1.
• $h_{t-1}$: Hidden state from previous time step.
• $x_t$: is the input at time step t.
• $W_f$: is the weight matrix for the forget gate, and $b_f$ is the bias term.

In flow: The input gate allows new information to flow into the cell state. The first one is an input gate layer that decides which information is going to be updated, along with a candidate value calculated with the tanh function to control the update. This enables the model to strike a balance between past accumulated knowledge and new information gained.

$i_t=\sigma\left(W_i\left[h_{t-1}.x_t\right]+b_i\right)$    (6)

where:

• $i_t$ is the output of the input gate at time step $t$.
• $W_i$ is the weight matrix for the input gate, and b_i is the bias term.

For the candidate memory cell, it is:

$\tilde{C}_t=\tanh \left(W_C\left[h_{t-1}.x_t\right]+b_C\right)$    (7)

where:

• $\tilde{C}_t$ is the candidate memory at time step t,
• tanh Hyperbolic Tangent Activation Function this is similar to the sigmoid function except that its values are between [-1,1].
• $W_C$ is the weight of the candidate memory layer, and $b_C$ is the bias term.

Finally, the output gate decides the information that is passed from the memory cell to the hidden state. It applies a sigmoid activation to decide how much cell state to output. This mechanism enables the model to transfer relevant information in a controlled manner, which significantly stabilizes long-distance connections. Therefore, LSTMs can better model complex correlational sequences more flexibly and better adapt to sequential tasks by estimating the time to remember or forget correlations. By playing the right amount of information to be stored/updated, LSTMs perform a lot better for long-term dependencies, as well as a lot of contextual information, than traditional RNNs.

$o_t=\sigma\left(W_o\left[h_{t-1}.x_t\right]+b_o\right)$    (8)

where:

• $o_t$ is the output of the output gate at time step $t$,
• $W_o$ is the weight matrix for the output gate, and $b_o$ is the bias term.

After that, the cell state collects and updates information that is to be retained or discarded over time. It is updated at each time step using the forget gate and the input gate. The equation for the cell state is:

$C_t=f_t * C_{t-1}+i_t * \tilde{C}_t$    (9)

where:

• $C_t$ is the cell state at time step $t$.
• $C_{t-1}$ is the previous cell state.
• $f_t$ is the output of the forget gate.
• $i_t$ is the output of the input gate.
• $\tilde{C}_t$ is the candidate memory cell.

The hidden state is the output of each step, which is either predicted or forwarded to the next step. It encodes the information of the sequence till the current time step.

Hidden state equation:

$h_t=o_t * \tanh \left(C_t\right)$     (10)

where:

• h_t is the hidden state at time step t,
• o_t is the output of the output gate,
• C_t  is the current cell state.

After LSTM layers, 2 fully connected layers are used to combine the extracted features and learn complex representations for classification:

$z=W \cdot x+b$    (11)

where:

• $W$ is the weight matrix,
• $x$ is the input from the previous layer.
• $b$ is the bias term.

Finally, the output layer uses the Softmax function to convert the raw scores (logits) into probabilities for multi-class classification:

$P(y=i \mid \mathbf{x})=\frac{e^{z_i}}{\sum_j e^{z_j}}$    (12)

where:

• $z_i$ is the raw output (logit) for class i.
• $\sum_j e^{z_j}$ is the sum of the exponentials of all logits to normalize the output to probabilities.

Table 2 shows the LSTM layer configuration of the hybrid CNN-LSTM architecture. The first LSTM layer consists of 128 hidden units with a dropout rate of 0.2 to capture long-range dependencies in the data. The second LSTM layer, with 64 hidden units and the same dropout rate, adds further temporal context while helping to reduce overfitting.

Table 2. LSTM layer configuration for hybrid CNN-LSTM architecture

3.4 Hybrid CNN-LSTM model

This model takes advantage of the strengths of both CNNs and LSTMs. In addition, the use of spectrograms provides a powerful representation by converting signals into the time–frequency domain, allowing the model to extract both spectral and temporal features more efficiently. As illustrated in Figure 2, various modulation types are represented in the spectrograms, which further enhances the model’s ability to distinguish between different signal patterns.

710cdc43-3d1f-48b9-a48b-dda0842a25b8.png

Figure 2. Spectrogram of all modulation types

CNN Layer (6 Layers): Convolutional layers are used in a series to accept the spectrogram as input for extracting the spatial features. These layers identify patterns like edges, textures, and shapes in the spectrogram. Convolutional layers utilize filters to identify local patterns. Pooling layers (like max pooling): reduce the size of the feature maps.

LSTM Layer (2 Layers): Reshape the output of the CNN and input into 2 LSTM layers to learn temporal dependencies. LSTMs are very well adapted to dealing with data from sequences and able to capture long-range dependencies in the signal. The initial LSTM layer handles the input sequential data and captures relevant temporal features. The second LSTM layer adds more details to the time series feature representation and improves the accuracy of the modulation scheme classification. Fully Connected Layer: LSTM output is fed into fully connected layers to project the features onto the classes of modulation. Softmax Activation: The last layer applies a Softmax activation function that results in a probability distribution for the various modulation classes.

3.5 Workflow of the proposed system

Data Collection: Collect a dataset of modulated signals with known modulation types and various SNRs. And the data Generation in MATLAB with 2000 frames for eleven modulation types. Spectrogram Generation: Convert the raw signals to spectrograms via STFT. Data Splitting: Split the dataset into three sets:

• Training Set (70%): Train the CNN-LSTM model.
• Validation Set (15%): Utilized to adjust hyper-parameters and avoid overfitting while training.
• Test Set (15%): Utilized to assess the ultimate performance of the model.