A Novel Variable Forgetting Factor Fast NLMS Algorithm for Time-Varying Systems

Adaptive algorithms represent a crucial class in signal processing [1] and are widely employed to address challenges such as acoustic echo cancellation (AEC) [2, 3], identification systems [4], and speech quality enhancement [5]. Various families of adaptive algorithms exist, each offering unique advantages and suited to specific applications. Among these, the RLS algorithm, a variant of the Kalman filter, is widely regarded as one of the most effective adaptive filters [1, 2]. It is known for its rapid convergence, even with highly correlated inputs. However, its primary drawback lies in its high computational complexity, which is of order $\left.O M^2\right)$ operations per iteration, where $M$ is the length of the adaptive filter. Another category of adaptive algorithms is the stochastic gradient algorithm, which includes the widely used Normalized LMS (NLMS) algorithm and its many variants. These algorithms are especially popular in practical applications due to their ease of implementation and are widely used for their low computational cost, requiring only $O(M)$ operations per iteration. However, a primary limitation of this category is its relatively slow convergence speed, particularly when encountering highly correlated input signals. For this reason, several NLMS variants have been developed to achieve faster convergence while maintaining comparable computational complexity, such as FNLMS algorithm. This algorithm effectively combines the rapid convergence speed of the RLS algorithm with the low computational complexity of the NLMS algorithm, offering a balanced solution that optimizes performance and efficiency [6].

Both RLS and FNLMS algorithms rely on the forgetting factor, a critical parameter that significantly influences key performance metrics, including convergence rate, steady-state MSE, and tracking capability [6-8]. In the conventional implementations of these algorithms, the forgetting factor is typically set between 0 and 1, necessitating trade-offs between various performance criteria. As the forgetting factor approaches unity, the algorithms generally exhibit increased stability and reduced misalignment, but their ability to track time-varying signals diminishes. Conversely, decreasing the forgetting factor enhances tracking capabilities but can lead to decreased stability and increased misalignment. To address these trade-offs, several VFF-RLS algorithms have been proposed [8-16], with particular emphasis on the methods presented in references [8, 13]. This paper introduces a novel VFF design that enhances the ability to track time-varying unknown systems. This approach is integrated with the FNLMS algorithm, resulting in improved convergence speed, reduced steady-state error, and optimized computational complexity. The proposed algorithm demonstrates particular effectiveness in AEC and SI applications. This paper is organized as follows: Section 2 presents the NLMS, RLS, and FNLMS algorithms, along with an overview of their applications in AEC systems. Section 3 provides a concise description of the Practical VFF (PVFF-RLS) algorithm [13] and introduces the proposed NVFF-FNLMS algorithm. Section 4 presents simulation results that validate the performance of the proposed algorithm compared to the previously mentioned algorithms. Finally, Section 5 summarizes our conclusions and findings.

3.1 The practical VFF-RLS algorithm

The practical VFF-RLS algorithm, presented in reference [13] is designed to address system identification problems. As shown in references [8, 13], the choice of the forgetting factor $\lambda$ significantly impacts algorithm performance. A value of $\lambda$ close to 1 negatively influences the algorithm's tracking capability, while a lower λ value enhances tracking but can lead to increased misalignment and steady-state MSE in stationary environments.

The VFF employed in this algorithm is expressed as follows:

$\lambda(n)=\left\{\begin{array}{cc}\lambda_{\max } & \text { if } \zeta(n) \leq \varepsilon \\ \min \left[\frac{\sigma_q(n) \sigma_\eta(n)}{\left|\epsilon+\sigma_e(n)-\sigma_\eta(n)\right|}, \lambda_{\max }\right] & \text { if } \zeta(n)>\varepsilon\end{array}\right.$            (22)

where, $\epsilon$ is a small constant to prevent division by zero, $\lambda_{\text {max}}$ represents the maximum value of the forgetting factor and $\varepsilon$ is a small positive constant. The variable $\zeta(n)$ is called the convergence parameter.

Considering that [8, 13]:

$q(n)=x^T(n) R^{-1}(n-1) \times(n)$            (23)

where, E $\left\{q^2(n)\right\}=\sigma_q^2$, the power estimates are computed using an exponential window:

$\sigma_q^2(n)=\alpha \sigma_q^2(n-1)+(1-\alpha) q^2(n)$             (24)

where, $\alpha$ is a weighting factor, with $\alpha=1-1 /(K M)$ and $K>1$.

In this algorithm, the authors offer a more accurate and practical method to estimate system noise $\eta(\mathrm{n})$, this solution is highly precise, as system noise is typically stationary in practice, and it is calculated in reference [13] through the following steps:

The signal-to-noise ratio (SNR) is described as:

$S N R=\frac{\sigma_y^2}{\sigma_\eta^2}$             (25)

where, E $\left\{y^2(n)\right\}=\sigma_y^2$ is the variance of $y(n)$, in the context of adaptive filtering, we may additionally consider the estimated SNR as [13, 20]:

$S N R_f=\frac{\sigma_{\hat{y}}^2}{\sigma_e^2}$              (26)

where, $\sigma_{\hat{y}}^2=E\left\{\hat{y}^2(n)\right\}$. As the adaptive filter approaches its steady-state condition, it is expected that [13]:

$\hat{y}(n) \approx y(n)$               (27)

And, consequently:

$e(n) \approx \eta(n)$             (28)

Therefore, in this case:

$S N R_f \approx S N R$             (29)

Since $y(n)$ and $\eta(n)$ are uncorrelated, allowing Eq. (1) to be reformulated in terms of variances as:

$\sigma_d^2(n)=\sigma_y^2(n)+\sigma_\eta^2(n)$               (30)

By developing Eq. (29) and considering Eq. (30), the estimate of the system noise power in steady-state can be calculated as follows [13]:

$\sigma_\eta^2(n)=\frac{\sigma_d^2(n) \sigma_e^2(n)}{\sigma_e^2(n)+\sigma_{\hat{y}}^2(n)}$              (31)

where, the power estimates in Eq. (31) are calculated recursively using:

$\sigma_d^2(n)=\alpha \sigma_d^2(n-1)+(1-\alpha) d^2(n)$              (32)

$\sigma_e^2(n)=\alpha \sigma_e^2(n-1)+(1-\alpha) e^2(n)$             (33)

$\sigma_{\hat{y}}^2(n)=\alpha \sigma_{\hat{y}}^2(n-1)+(1-\alpha) \hat{y}^2(n)$              (34)

In terms of energy, Eq. (28) becomes:

$\sigma_e^2(n) \approx \sigma_\eta^2(n)$              (35)

Thus, substituting Eq. (31) into Eq. (35) yields:

$\sigma_d^2(n) \approx \sigma_{\hat{y}}^2(n)+\sigma_e^2(n)$             (36)

From Eq. (36). the convergence parameter $\zeta(n)$ is defined as follows [13]:

$\zeta(n)=\left|\sigma_d^2(n)-\sigma_{\hat{y}}^2(n)-\sigma_e^2(n)\right|$              (37)

This parameter $\zeta(n)$ distinguishes between steady-state and tracking phases, where when the unknown system changes (in tracking scenario), the error become large, so that $\zeta(n)>\varepsilon$, during the steady-state phase, we have $\zeta(n) \leq \varepsilon$ [13]. Algorithm 1 provides the practical VFF-RLS algorithm (PVFF-RLS). A significant drawback of this algorithm is its high computational complexity, which limits its practicality in real-world applications such as AEC. In addition, previous VFF-RLS algorithms have always been tested under abrupt system changes, which are not representative of practical scenarios. As a consequence, subsequent simulation results reveal difficulties in tracking the temporal variations of the unknown system in a non-stationary environment, due to forgetting factor values close to 1. To address these challenges, the next section presents a novel approach with several advantages. It demonstrates superior tracking capabilities compared to the NLMS, RLS, FNLMS, and PVFF-RLS algorithms. Additionally, it outperforms the FNLMS algorithm in terms of convergence speed, final MSE, and misalignment. Significantly, this improved approach achieves these performance enhancements while maintaining lower computational complexity compared to the PVFF-RLS algorithm.

3.2 The proposed VFF FNLMS algorithm

The proposed NVFF design is inspired by the Variable Step-Size (VSS) principle, and has been carefully developed to achieve the key objective of dynamically adjusting the forgetting factor to balance convergence stability and tracking capability. Unlike VSS, where the step size is increased in non-stationary conditions, our NVFF inverts this principle: the forgetting factor is kept close to 1 in stationary conditions to minimize misalignment and set slightly below 1 in non-stationary conditions to enable rapid adaptation. This formulation ensures an optimal trade-off between convergence and tracking and is theoretically supported by a mathematical analysis of $\lambda$ behavior across three scenarios: pre-convergence, stationary, and non-stationary. Moreover, unlike existing VFF-RLS algorithms, which not only suffer from high computational complexity but are also typically tested under abrupt system changes that do not reflect realistic conditions, our evaluation on both artificial and real time-varying systems demonstrated that such approaches struggle to maintain effective tracking. These limitations motivated the development of the NV16FF-FNLMS, which combines low complexity with robust performance in practical scenarios.

This approach is founded upon a fundamental criterion related to the variation in the forgetting factor, defined by the energy ratio between error power and noise power this fundamental criterion justified by the fact that the error significantly varies between stationary and non-stationary systems: it is small in stationary conditions and large in non-stationary conditions. Therefore, based on the error value, the forgetting factor can be dynamically adjusted to optimize the algorithm’s performance. Formally, this criterion is expressed as:

$\lambda(n)=1-\varphi\left|\frac{\sigma_e^2(n)-\sigma_\eta^2(n)}{\sigma_e^2(n)}\right|$              (38)

where, $0<\varphi<1$, serves as a constant that regulates the VFF. In the proposed algorithm, the error signal is estimated using an exponential windowas follow:

$\sigma_e^2(n)=\beta \sigma_e^2(n-1)+(1-\beta) e^2(n)$               (39)

where, $\beta$ is a weighting forgetting factor.

Using the method described in reference [21], the system noise variance can be estimated from $e(n)$ as follows:

$\sigma_n(n)=\sigma_{|e|}(n)$             (40)

where,

$\sigma_{|e|}(n)=\beta \sigma_{|e|}(n-1)+(1-\beta)|e(n)|$                (41)

3.2.1 Derivation of the variable forgetting factor NVFF

In this subsection, we derive the expression for the forgetting factor $\lambda(n)$ in three distinct scenarios preconvergence, stationary, and non-stationary to illustrate how $\lambda(n)$ adapts dynamically in each case.

• Pre-Convergence Case:

At startup, before the filter converges, during the learning phase, we can approximately write:

$\hat{y}(n) \approx 0$             (42)

Substituting Eq. (42) in Eq. (4), we obtain:

$e(n) \approx d(n)$                (43)

Consequently, in terms of energy, Eq. (43) will be:

$\sigma_e^2(n) \approx \sigma_d^2(n)$             (44)

By using Eq. (30) and Eq. (44) in Eq. (38), the $\lambda(n)$ can be expressed as follows:

$\lambda(n)=1-\varphi\left|\frac{\sigma_y^2(n)}{\sigma_y^2(n)+\sigma_\eta^2(n)}\right|$              (45)

$\lambda(n)=1-\varphi\left|\frac{1}{1+S N R^{-1}}\right|<1$               (46)

From Eq. (46), it is evident that $\lambda(n)$ remains strictly less than 1, indicating that $\lambda(n)$ starts at a value smaller than one during the initial phase, whatever of the SNR value.

• Stationary Case

In this case, the forgetting factor by definition should be set very close to 1. This maximizes convergence speed, minimizes the final mean-square error (MSE), and reduces misalignment. Aassuming the filter has sufficiently converged to its true value for Eq. (35) to be satisfied, the VFF in the proposed approach can be expressed as:

$\lambda(n)=1-\varphi\left|\frac{\sigma_\eta^2(n)-\sigma_\eta^2(n)}{\sigma_e^2(n)}\right| \approx 1$              (47)

• Non-Stationary Case

The forgetting factor reaches its maximum value in the stationary case. However, if the echo system changes over time, disturbances may affect the echo signal, represented as:

$y(n)+y_{n s}(n)$            (48)

where, $y_{n s}(n)$ is a disturbance resulting from temporal variations of the unknown system, and $y(n)$ stays near to $\hat{y}(n)$. Consequently, the desired signal $d(n)$ in Eq. (1) becomes:

$d(n)=y(n)+y_{n s}(n)+\eta(\mathrm{n})$              (49)

Assuming $y(n), y_{n s}(n)$ and $\eta(\mathrm{n})$ are independent quantities, the energy of $d(n)$ can be approximated as:

$\sigma_d^2(n)=\sigma_y^2(n)+\sigma_{y_{n s}}^2(n)+\sigma_\eta^2(n)$              (50)

Furthermore, by equating Eq. (36) and Eq. (50) and applying the assumption $\sigma_y^2(n) \approx \sigma_{\hat{y}}^2(n)$, the error power can be represented through an alternative expression:

$\sigma_e^2(n) \approx \sigma_{y_{n s}}^2(n)+\sigma_\eta^2(n)$             (51)

As a result, substituting Eq. (51) into Eq. (38) yields:

$\lambda(n) \approx 1-\varphi\left|\frac{\sigma_{y_{n s}}^2(n)}{\sigma_{y_{n s}}^2(n)+\sigma_n^2(n)}\right|$              (52)

In Eq. (52), it is important to observe that the term $\left|\frac{\sigma_{y_{n s}}^2(n)}{\sigma_{y_{n s}}^2(n)+\sigma_\eta^2(n)}\right|$ always lies between 0 and 1, as the denominator is greater than the numerator, ensuring:

$0<\left|\frac{\sigma_{y_{n s}}^2(n)}{\sigma_{y_{n s}}^2(n)+\sigma_\eta^2(n)}\right|<1$             (53)

As a result, we have:

$0<1-\varphi\left|\frac{\sigma_{y_{n s}}^2(n)}{\sigma_{y_{n s}}^2(n)+\sigma_\eta^2(n)}\right|<1$                (54)

Thus, in the non-stationary case we conclude that:

$0<\lambda(n)<1$               (55)

Finally, the VFF expression for the proposed NVFF-FNLMS is given by:

$\lambda(n)=\left\{\begin{array}{cc}\lambda_{\max } & \text { if } \zeta(n) \leq \varepsilon \\ 1-\varphi\left|\frac{\sigma_e^2(n)-\sigma_\eta^2(n)}{\sigma_e^2(n)+\delta_0}\right| & \text { if } \zeta(n)>\varepsilon\end{array}\right.$              (56)

where, $\delta_0$ small constant is added to avoid division by zero. The recursive estimators of $\sigma_d^2(n)$ and $\sigma_{\hat{y}}^2(n)$ are estimated as follows:

$\sigma_d^2(n)=\beta \sigma_d^2(n-1)+(1-\beta) d^2(n)$                (57)

$\sigma_{\hat{y}}^2(n)=\beta \sigma_{\hat{y}}^2(n-1)+(1-\beta) \hat{y}^2(n)$                 (58)

In this NVFF-FNLMS algorithm, the choice of the parameter $\varphi$ is critical to ensure effective performance, as it governs the behavior of the forgetting factor $\lambda(n)$, which must always remain strictly less than 1 . Theoretically, $\varphi$ must be smaller than 1, a condition derived from the analysis of Eq. (46) under the assumption that the SNR is always positive. After mathematical manipulation, Eq. (46) yields the following condition:

$0<\varphi<1+\mathrm{SNR}^{-1}$             (59)

Since $\mathrm{SNR}^{-1}$ is typically much smaller than unity, for example, at $\mathrm{SNR}=50 \mathrm{~dB}$ we have $\mathrm{SNR}^{-1}=0.00001$, it can be approximated that $1+\mathrm{SNR}^{-1} \approx 1$. Consequently, the simplified condition is obtained as follows:

$\varphi<1$            (60)

Furthermore, for noise estimation method $\sigma_\eta(n)=\sigma_{|e|}(n)$ in (40) it is important to note that this approximation is a simplified assumption. In the non-stationary case, the error signal can be expressed as $e(n)=\eta(n)+y_{n s}(n)$. Consequently, the error power is approximated by $\sigma_e^2(n) \approx \sigma_{y_{n s}}^2(n)+\sigma_\eta^2(n)$, This decomposition shows that the error is not purely due to noise, but also contains a component related to system variations. From a noise estimation viewpoint, this assumption introduces a drawback, since part of the error variance is wrongly interpreted as noise, leading to an overestimation of $\sigma_\eta^2(n)$. However, the presence of $y_{n s}(n)$ in the error signal can also be regarded as beneficial for tracking, because it implicitly conveys information about system variations and enables the VFF to adapt more effectively in non-stationary environments. Therefore, while this estimation method is reliable in stationary conditions, its limitation in non-stationary cases should be acknowledged, as it simultaneously degrades the accuracy of noise estimation but enhances the tracking capability of the algorithm.

Finally, the theoretical analysis reveals that the proposed NVFF aligns with the general principle of a VFF. When the real system exhibits stationary behavior, the proposed NVFF approaches a value of one, enabling the algorithm to achieve a lower steady-state error. In contrast, under non-stationary conditions, the denominator in Eq. (52) increases, resulting in a decrease in the forgetting factor $\lambda(n)$ to lower values, which enhances the algorithm’s ability to track time variations of the system.

3.2.2 Stability analysis of the proposed NVFF-FNLMS

To establish the stability condition of the newly proposed NVFF-FNLMS algorithm, which is a variant of the FNLMS algorithm incorporating a VFF technique, we refer to the stability and convergence condition of the adaptation step-size $\mu_{F N L M S}$ derived in reference [6]. This condition is itself related to the adaptation step-size $\mu_{N L M S}$ of the classical NLMS algorithm, obtained using approximate mean-square analysis as described by Slock [22]. The analysis focuses on how the FNLMS algorithm influences the adaptation gain compared to the NLMS algorithm. For this purpose, we assume that the input signal is a white Gaussian stationary process. Under this assumption, we can approximate $e_p(n) \approx x(n)$, and the variance of the prediction error $\alpha(n)$ for the proposed algorithm is evaluated as follows:

$\alpha(n)=\lambda(n) \alpha(n-1)+x^2(n)$               (61)

The Eq. (61) can be expanded iteratively as:

$\alpha(n)=\lambda(n)\left[\lambda(n-1) \alpha(n-2)+x^2(n-1)\right]+x^2(n)$              (62)

$\alpha(n)=\lambda(n) \lambda(n-1) \alpha(n-2)+\lambda(n) x^2(n-1)+x^2(n)$             (63)

$\begin{gathered}\alpha(n)=\alpha(0) \prod_{i=1}^n \lambda(i)  +\sum_{k=0}^{n-1}\left[x^2(n-k) \prod_{j=0}^{k-1} \lambda(n-j)\right]\end{gathered}$              (64)

where, we adopt the convention that the empty product is equal to 1. If we now consider the expected value under the assumption that $x(n)$ is a stationary white Gaussian process with variance $\sigma_x^2$, we obtain:

$\begin{aligned} & E\{\alpha(n)\}=E\{\alpha(0)\} \prod_{i=1}^n E\{\lambda(i)\} & +\sigma_x^2 \sum_{k=0}^{n-1} \prod_{j=0}^{k-1} E\{\lambda(n-j)\}\end{aligned}$               (65)

Based on the stationary case, in the steady state we observe that $\lambda(n)$ is almost constant and close to 1. Therefore, $\lambda(n)$ tends toward a stationary value, denoted as $\lambda_{\text {sta}}$ (the stationarycase) very close to 1 (slightly less than unity for analysis purposes). Under this assumption, the expression simplifies to:

$\begin{gathered}E\{\alpha(n)\}=\alpha(0) \lambda_{\mathrm{sta}}{ }^n+\sigma_x^2 \sum_{k=0}^{n-1} \lambda_{\mathrm{sta}}{ }^k \\ =\alpha(0) \lambda_{\mathrm{sta}}{ }^n+\sigma_x^2 \frac{1-\lambda_{\mathrm{sta}}}{1-\lambda_{\mathrm{sta}}}\end{gathered}$              (66)

Thus, for $n \rightarrow \infty$, the mean variance of the prediction error signal converges to:

$\alpha(n)=\frac{\sigma_x^2}{1-\lambda_{\mathrm{sta}}}$             (67)

Now we also adjust the other equations of $\tilde{\boldsymbol{c}}(n)$ and $\gamma(n)$ by their asymptotic values:

$\tilde{\mathbf{c}}(n)=\frac{\boldsymbol{x}(n)}{\frac{\lambda_{\text {sta }}}{1-\lambda_{\text {sta }}} \sigma_x^2+c_0}$             (68)

$\gamma(n)=\frac{1}{1-\tilde{\mathbf{c}}(n) \boldsymbol{x}(n)} \approx \frac{1}{1+\frac{M \sigma_x^2}{\frac{\lambda_{\mathrm{sta}} \sigma_x^2}{1-\lambda_{\mathrm{sta}}}+c_0}}$               (69)

The adaptation gain of the NLMS algorithm, under the stability condition $0<\mu_{N L M S}<2$, is approximately given by [6, 17]:

$\tilde{\mathbf{c}}_{N L M S}(n) \approx-\frac{\bar{\mu}_{N L M S}}{M \sigma_x^2} \boldsymbol{x}(n)$            (70)

By applying the approximations in Eqs. (67)-(69), the adaptation gain of the proposed NVFF-FNLMS algorithm can be expressed as follows:

$\begin{gathered}\tilde{\mathbf{c}}(n) \approx-\frac{\mu_{N L M S}}{M \sigma_x^2\left(1+\frac{\lambda_{\mathrm{sta}}}{M\left(1-\lambda_{\mathrm{sta}}\right)}+\frac{c_0}{M \sigma_x^2}\right)} \boldsymbol{x}(n) \\ \approx-\frac{\mu_{N L M S}}{M \sigma_x^2\left(1+\frac{\lambda_{\mathrm{sta}}}{M\left(1-\lambda_{\mathrm{sta}}\right)}+\frac{c_0}{M \sigma_x^2}\right)} \boldsymbol{x}(n)\end{gathered}$          (71)

Finally, by comparing Eqs. (70) and (71), the condition stabiliti of the proposed algorithm can be stated as:

$0<\mu_{\mathrm{NVF}}=\frac{\mu_{N L M S}}{1+\frac{\lambda_{\text {stat }}}{\left(1-\lambda_{\text {stat }}\right) M}+\frac{c_0}{M \sigma_x^2}}<2$             (72)

Based on the stability condition of the NVFF formulation, we find that this inequality is always satisfied for $0<\mu_{N L M S}<2$. Moreover, from the derivation of $\lambda(n)$ in the pre-convergence, stationary, and non-stationary cases, we observe that $0< \lambda(n)<1$. Therefore, we conclude that the proposed algorithm is stable even when the forgetting factor varies dynamically.

Algorithm 2 gives an overview of the proposed algorithm.

In this paper, we introduced an enhanced FNLMS algorithm incorporating a novel VFF, termed the NVFF-FNLMS algorithm, specifically designed for AEC and System Identification (SI) applications. The proposed algorithm was comprehensively compared with the original NLMS, FNLMS, RLS, and PVFF-RLS algorithms, as well as the VFF-FNLMS and VSS-FNLMS algorithms, which also aim to improve tracking performance in time-varying systems. Performance evaluations demonstrate that the proposed NVFF-FNLMS algorithm outperforms classical FNLMS and NLMS algorithms in terms of tracking ability, steady-state MSE, convergence speed, and misalignment for both stationary and non-stationary signals and systems. Compared to previous VFF-FNLMS algorithms, the proposed NVFF-FNLMS exhibits superior tracking performance, achieving lower misalignment and a reduced final MSE.

The NVFF-FNLMS algorithm also demonstrates superior adaptability to both real and artificial time-varying systems compared to VFF-RLS algorithms, making it more suitable for non-stationary environments. Despite its performance gains, the NVFF-FNLMS algorithm remains computationally efficient compared to the more complex PVFF-RLS algorithm, making it a practical choice for real-time applications such as AEC.

While the proposed algorithm demonstrates significant improvements in convergence speed, tracking capability, and final MSE within the tested scenarios, certain limitations remain. Its applicability is constrained under conditions that have not yet been fully explored. In particular, the theoretical performance limits for tracking time-varying systems are not yet completely established, due to the lack of predictive models for varying rates of system change and distortions. Moreover, the NVFF-FNLMS algorithm may face limitations in scenarios involving two simultaneous systems (e.g., a time-varying system under double-talk conditions) or in environments characterized by strong impulsive noise.

Future work will aim to extend the theoretical analysis of tracking limits, enhance robustness under severe noise conditions, and develop hardware implementations (e.g., FPGA) to provide a more accurate assessment of real-time performance and facilitate integration into large-scale, real-world signal processing systems.

In conclusion, the NVFF-FNLMS algorithm proposed in this paper represents a practical, efficient, and effective solution for real-time applications such as AEC.