Design of New Hybrid Controller Mixing TD3 with NNFZ and SMC for Mobile Robot Trajectory Tracking

The trajectory tracking problem for nonholonomic wheeled mobile robots (WMRs) represents a fundamental challenge in modern robotics, with critical applications in autonomous vehicles, warehouse automation, service robotics and precision agriculture. These systems operate under nonholonomic motion constraints, characterized by the inability to move instantaneously in arbitrary directions, which significantly complicates the control-design process. This challenge is further exacerbated by nonlinear dynamics, model uncertainties, and external disturbances, including irregular terrain conditions, payload variation, and sensor noise [1-3].

Over the past few decades, numerous control strategies have been proposed to address these challenges. Classical methods, such as proportional-integral-derivative (PID) controllers and kinematic model-based approaches [4, 5], provide simplicity and computational efficiency; however, their performance deteriorates in uncertain or dynamic environments. Nonlinear model-based techniques, including backstepping and feedback linearization [5, 6], improve robustness but depend on accurate system identification. Recently, intelligent control approaches have been introduced. Reinforcement Learning (RL), particularly the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm [7, 8], has demonstrated strong adaptability to complex dynamics, although it suffers from sample inefficiency and lacks formal stability guarantees [9, 10]. Neuro-Fuzzy Inference Systems (NFIS) [11, 12] combine the learning capability of neural networks with the interpretability of fuzzy logic, enhancing robustness to nonlinearities and uncertainties, but may exhibit slow adaptation in highly dynamic environments. Sliding Mode Control (SMC) [13-15] is renowned for its robustness and invariance to matched uncertainties; however, its implementation often suffers from high-frequency chattering. Hybrid approaches have also emerged, such as fuzzy-SMC [16] and RL-enhanced classical controllers [17], which show improved accuracy and better sim-to-real transfer. However, to the best of our knowledge, the three-way integration of TD3, NFIS, and SMC remains largely investigated.

This study addresses this gap by proposing a novel hierarchical hybrid control architecture that synergistically integrates the TD3, NFIS, and SMC for WMR trajectory tracking. The architecture operates across three hierarchical levels: TD3 provides high-level adaptive decision-making and long-term strategy optimization, NFIS enables real-time parameter adaptation to handle system uncertainties, and SMC ensures robust low-level control execution with guaranteed stability.

The main contributions of this study are as follows:

• Development of a novel three-layer hierarchical control architecture that seamlessly integrates TD3 for adaptive high-level decision-making, NFIS for online parameter tuning, and SMC for robust low-level control execution
• Formulation of a comprehensive TD3 reward function that effectively encodes multiple performance objectives including tracking accuracy, control smoothness, energy efficiency, and robustness metrics
• Rigorous Lyapunov-based stability analysis providing theoretical guarantees for asymptotic convergence under bounded disturbances and parameter uncertainties
• Extensive simulation validation demonstrating 30% improvement in RMSE, 42% reduction in IAE, and superior robustness compared to standalone controllers across diverse trajectory patterns
• A comprehensive robustness analysis under multiple disturbance scenarios, including external forces, parameter uncertainties, and measurement noise, showed consistent performance advantages.

The remainder of this paper is organized as follows: Section 2 formulates the trajectory tracking problem for nonholonomic wheeled mobile robots and presents the system models. Section 3 details the proposed hierarchical hybrid control architecture that integrates TD3, NFIS, and SMC. Section 4 provides a theoretical stability and convergence analysis of the control scheme. Section 5 describes the simulation environment and experimental setup used for the validation. Section 6 reports and discusses the performance results, including trajectory tracking accuracy, robustness under disturbances, and comparative evaluations against the baseline controllers. Finally, Section 7 concludes the paper and outlines the future research directions.

3.1 Hybrid control architecture overview

The proposed hybrid control architecture integrates three complementary control paradigms in a hierarchical structure [7, 8, 10, 13] to address the complex requirements of robotic path tracking in the presence of uncertainty and disturbance.

As illustrated in Figure 2, the architecture leverages a synergistic combination of learning- and model-based control strategies using a three-layer framework. The hierarchical architecture consists of (1) a high-level Twin Delayed Deep Deterministic Policy Gradient (TD3) agent [7], for strategic decision-making and adaptive parameter optimization, (2) a mid-level Neural Network Fuzzy (NNFZ) [10, 12] controller for nonlinear compensation and uncertainty handling, and (3) a low-level Sliding Mode Controller (SMC) [13-15] for robust tracking and disturbance rejection. This multilayer approach addresses the limitations of individual control methods while exploiting their respective strengths through intelligent coordination of the control methods.

2.png

Figure 2. Hybrid control architecture overview

3.2 Proposed hybrid control architecture

The integration of these control paradigms enables the system to achieve robust performance across varying operational conditions, from high-precision tracking in stable environments to aggressive maneuvering under significant disturbances [8, 16, 17, 18]. The TD3 agent provides long-term strategic planning and online adaptation capabilities, the NNFZ controller handles nonlinear system dynamics and model uncertainties, and the SMC ensures robust stability and finite-time convergence.

3.2.1 High-level: TD3 agent design

State and Action Spaces

The TD3 agent operates with a comprehensive state representation that captures the instantaneous and historical tracking information. Building upon recent advances in deep reinforcement learning for robotics, the state vector is formulated as

$\mathbf{s}=\left[e_1, e_2, e_3, \dot{e}_1, \dot{e}_2, \dot{e}_3, \int e_1 d t, \int e_2 d t, \int e_3 d t, v_d, \omega_d, \dot{v}_d, \dot{\omega}_d\right]^T \in \mathbb{R}^{13}$     (13)

where, $e_i$ represents the position and orientation errors, $\dot{e}_i$ denotes the error derivatives for damping, $\int e_i d t$ provides integral terms for steady-state error elimination, and $v_d, \omega_d$ with their derivatives capture the reference trajectory dynamics.

The action space consists of adaptive control parameters that are dynamically optimized as follows.

$a=\left[K_{p 1}, K_{p 2}, K_{p 3}, K_{d 1}, K_{d 2}, K_{d 3}, \lambda_1, \lambda_2, \eta_1, \eta_2\right]^T \in \mathbb{R}^{10}$     (14)

where, $K_{p i}$ and $K_{d i}$ are the proportional and derivative gains for the PD controller components, $\lambda_i$ are the sliding surface parameters that determine the convergence rate; and $\eta_i$ are the switching gains that balance robustness and chattering. These parameters were bounded within physically meaningful ranges to ensure system stability and actuator feasibility.

Reward Function Design

The reward function was carefully designed to encode multiple, often conflicting, objectives inherent to robotic control. Following the principles of multi-objective optimization in reinforcement learning, the reward function is formulated as

$\begin{gathered}R=-\alpha_1\|e\|^2-\alpha_2\|\dot{e}\|^2-\alpha_3 \int\|e\|^2 d t-\alpha_4\|u\|^2-\alpha_5 \\ \|\dot{u}\|^2+\alpha_6 R_{\text {safety}}+\alpha_7 R_{\text {efficiency}}\end{gathered}$     (15)

The safety component encourages operation within safe velocity bounds as follows:

$R_{\text {safety }}=\exp \left(-\beta_1\left(|v|-v_{\text {safe }}\right)^2\right) \cdot \exp \left(-\beta_2\left(|\omega|-\omega_{\text {safe }}\right)^2\right)$     (16)

The efficiency term penalizes the excessive energy consumption.

$R_{\text {efficiency}}=-\gamma_1 \int\left(v^2+\omega^2\right) d t$     (17)

The weighting coefficients $\alpha_i, \beta_i, \gamma_i$ are determined through systematic hyperparameter optimization using Bayesian optimization techniques to balance the tracking accuracy, control smoothness, safety constraints, and energy efficiency.

TD3 Algorithm Implementation

The TD3 Algorithm 1 addresses the overestimation bias inherent in traditional actor-critic methods through the use of twin critic networks and delayed policy updates.

The algorithm employs dual critic networks $Q_1\left(\boldsymbol{s}, \boldsymbol{a} ; \theta_{Q_1}\right)$ and $Q_2\left(\boldsymbol{s}, \boldsymbol{a} ; \theta_{Q_2}\right)$ with parameters $\theta_{Q_1}$ and $\theta_{Q_2}$, and an actor network $\pi\left(s ; \theta_\pi\right)$ with parameters $\theta_\pi$. Target networks with parameters $\theta^{\prime}_{Q_1}, \theta^{\prime}_{Q_2}$, and $\theta^{\prime}_\pi$ are maintained to ensure training. The algorithm incorporates several key innovations: target policy smoothing through noise injection to reduce the variance in value estimates, delayed policy updates every $d$ steps to reduce the per-update error, and clipped double Q-learning to mitigate the over-estimation bias. The exploration noise $\epsilon$ is gradually annealed during training to transition from exploration to exploitation as follows:

Algorithm 1: TD3-Based Parameter Adaptation

3.2.2 Mid-level: Neural network fuzzy controller

NNFZ Architecture

The Neural Network Fuzzy (NNFZ) [10, 12] controller combines the universal approximation capabilities of neural networks with the interpretability and robustness of fuzzy logic systems [13, 14]. The controller employs a five-layer architecture that systematically transforms crisp error inputs into control outputs using fuzzy reasoning processes.

Layer 1 (Input Layer): Receives normalized error signals $e=\left[e_1, e_2, e_3\right]^T$ representing position and orientation tracking errors.

Layer 2 (Fuzzification): Computes membership degrees using Gaussian membership functions with adaptive parameters:

$\mu_{A_{i j}}\left(x_i\right)=\exp \left(-\frac{\left(x_i-c_{i j}\right)^2}{2 \sigma_{i j}^2}\right)$     (18)

where, $c_{i j}$ and $\sigma_{i j}$ represent the center and width of the $j$th membership function for the $i$th input, respectively. These parameters are adaptively tuned during online learning to capture the nonlinear system dynamics.

Layer 3 (Rule Layer): Implements fuzzy rules using T-norm operations (product inference).

$w_j=\prod_{i=1}^3 \mu_{A_{i j}}\left(x_i\right)$     (19)

This layer encodes the expert knowledge of the control strategy using IF-THEN rules, with each node representing the firing strength of a particular rule.

Layer 4 (Normalization): Normalizes firing strengths to ensure numerical stability:

$\bar{w}_j=\frac{w_j}{\sum_{k=1}^N w_k}$     (20)

Normalization ensures that the contributions of all rules sum to unity, providing a probabilistic interpretation of rule activation.

Layer 5 (Defuzzification): Computes control outputs using Takagi-Sugeno-Kang (TSK) consequent functions:

$\mathbf{u}_{N N F Z}=\sum_{j=1}^N \bar{w}_j\left(p_{j 0}+p_{j 1} e_1+p_{j 2} e_2+p_{j 3} e_3\right)$     (21)

where, $p_{j i}$ is the consequent parameter that defines the linear relationship between the inputs and outputs for each rule.

Online Learning Algorithm

The NNFZ parameters are updated online using a hybrid learning algorithm that combines gradient descent for premise parameters and recursive least squares for consequent parameters. The gradient descent update for the premise parameters is as follows:

$\theta_{i j}(k+1)=\theta_{i j}(k)-\eta(k) \frac{\partial E}{\partial \theta_{i j}}$     (22)

where, the error function is defined as

$E=\frac{1}{2}\left\|\mathbf{y}_d-\mathbf{y}\right\|^2$     (23)

The learning rate was adaptively adjusted to ensure convergence as follows:

$\eta(k)=\frac{\eta_0}{1+\beta \sqrt{k}}$     (24)

where, $\eta_0$ is the initial learning rate, $\beta$ is the decay factor, and $k$ is the iteration index. This adaptive scheme balances fast initial learning and convergence stability.

3.2.3 Low-level: Sliding mode controller

Sliding Surface Design

The sliding mode controller [13] provides robust tracking performance by designing an appropriate sliding surface that ensures finite-time convergence. The sliding surface is defined as:

$s=\left[s_1, s_2\right]^T=\left[\dot{e}_1+\lambda_1 e_1, \dot{e}_2+\lambda_2 e_2\right]^T$      (25)

where, $\lambda_i>0$ is a design parameter that determines the convergence rate on the sliding surface. The choice of linear sliding surfaces ensures computational efficiency while maintaining a robust performance.

Control Law

The SMC control [14, 15] law comprises equivalent and switching components to ensure both sliding surface attractiveness and system robustness.

$u_{S M C}=u_{e q}+u_{s w}$     (26)

The equivalent control maintains the system trajectory on the sliding surface once it is reached as follows:

$u_{e q}=\left[v_d \cos \left(e_3\right)+\lambda_1 e_1, \omega_d+\lambda_2 e_2\right]^T$     (27)

This component is derived from the condition $\dot{\boldsymbol{s}}=0$ and represents the nominal control effort required in the absence of uncertainty. The switching control ensures finite-time convergence to the sliding surface as follows:

$u_{s w}=-\left[\eta_1 \operatorname{sign}\left(s_1\right), \eta_2 \operatorname{sign}\left(s_2\right)\right]^T$     (28)

where, $\eta_i>0$ are switching gains that must be chosen to be larger than the upper bound of uncertainties to guarantee robustness.

To mitigate the chattering phenomenon inherent in traditional SMC, a boundary layer approach is employed:

$\boldsymbol{u}_{s w}=-\left[\eta_1 \operatorname{sat}\left(s_1 / \phi_1\right), \eta_2 \operatorname{sat}\left(s_2 / \phi_2\right)\right]^T$     (29)

where, $\operatorname{sat}(\cdot)$ is the saturation function defined as

$\operatorname{sat}(x)= \begin{cases}\operatorname{sign}(x) & \text { if }|x|>1 \\ x & \text { if }|x| \leq 1\end{cases}$     (30)

where, $\phi_i$ defines the boundary layer thickness, providing a trade-off between tracking accuracy and control smoothness.

3.2.4 Integration mechanism

The integration of the three control layers is achieved through an intelligent weighted combination scheme that dynamically adjusts the contribution of each controller based on the system state and performance metrics [19-26]. The final control signal is generated as follows:

$\boldsymbol{u}=w_1 \boldsymbol{u}_{T D 3}+w_2 \boldsymbol{u}_{N N E 7}+w_3 \boldsymbol{u}_{S M C}$     (31)

The weights are computed using a SoftMax function to ensure smooth transitions and numerical stability.

$w_i=\frac{\exp \left(\alpha_i\right)}{\sum_{j=1}^3 \exp \left(\alpha_j\right)}$     (32)

where, $\alpha_i$ is the confidence score determined by the TD3 agent based on the current system performance, uncertainty levels, and operational context. This adaptive weighting mechanism allows the system to seamlessly transition between different control modes, leveraging TD3’s learning capability during the exploration phases, NNFZ’s approximation power for nonlinear dynamics, and SMC’s robustness during disturbances. The coordination between the control layers follows a hierarchical decision-making process. The TD3 agent at the highest level monitored the overall system performance and adjusted the parameters and weights of the lower-level controllers. The NNFZ controller provides smooth control actions for nominal operation, whereas the SMC intervenes when a robust performance is required owing to significant disturbances or model uncertainties.

3.2.5 Complete implementation algorithm

The complete hybrid control Algorithm 2 integrates all three control layers with online learning and adaptation capabilities. Algorithm presents the detailed implementation procedure for both the training and execution phases.

Algorithm 2: Complete Hybrid TD3–NNFZ–SMC Control

5.1 Simulation environment

The proposed hybrid controller was implemented in Python 3.12.3 using PyTorch 1.10 for the TD3 implementation, and simulations were conducted on a model of the TurtleBot3 Waffle Pi robot [27-30]. The simulation environment was developed using the following specifications (Table 1).

Table 1. Robot parameters

5.2 Controller parameters

Here outlines the essential configuration settings for the Twin Delayed Deep Deterministic Policy Gradient (TD3) controller. The provided Table 2 offers a clear and comprehensive overview of the primary and essential parameters, such as the learning rates for both the actor and critic networks, the discount factor, and other vital configurations. Altogether, these specific parameters are fundamentally essential for the overall performance and successful optimization process of the controller itself.

Table 2. Controller configuration parameters

5.3 Computational analysis

Total cycle meets real-time requirements (80-100 Hz control loops) on standard CPU without GPU acceleration, demonstrating practical feasibility (Table 3).

Table 3. Controller configuration parameters

5.3.1 Test trajectories

Two reference trajectories were designed to evaluate the controller performance.

Circular Trajectory:

$\begin{aligned} & x_d(t)=2 \cos (0.2 t) \\ & y_d(t)=2 \sin (0.2 t) \\ & \theta_d(t)=0.2 t+\pi / 2\end{aligned}$     (41)

Figure-Eight Trajectory:

$\begin{aligned} & x_d(t)=2 \sin (0.2 t) \\ & y_d(t)=\sin (0.4 t) \\ & \theta_d(t)=\arctan \left(\frac{\dot{y}_d}{\dot{x}_d}\right)\end{aligned}$     (42)

5.4 Disturbance scenarios

To evaluate robustness, four disturbance scenarios were implemented.

1. External Forces: Random impulse forces ($\pm$ 5N) applied every 2-3 seconds
2. Parameter Uncertainty: $\pm$ 20% variation in mass and inertia parameters
3. Measurement Noise: Gaussian white noise ($\sigma$ = 0.05 m for position, $\sigma$ = 0.1 rad for orientation)
4. Combined: All disturbances applied simultaneously

This study introduces an innovative hierarchical hybrid control framework that effectively combines the Twin Delayed Deep Deterministic Policy Gradient (TD3), Neural Network Fuzzy (NNFZ) control, and Sliding Mode Control (SMC) for tracking the trajectory of mobile robots. This approach overcomes the inherent limitations of each control method while capitalizing on their complementary benefits. The main contributions and findings include:

•Architectural Innovation: The three-tier hierarchical design facilitates the seamless integration of learning-based adaptation (TD3), nonlinear compensation (NNFZ), and robust control (SMC), resulting in superior performance compared to any single controller.

•Performance Improvements: Experimental results show a 30% reduction in RMSE, a 42% enhancement in IAE, and a 20% faster convergence rate compared to individual controllers across various trajectory patterns.

•Robustness Enhancement: The hybrid controller consistently performed well under different disturbance conditions, with a worst-case performance drop of only 34.9% compared to 83.5% for TD3 alone, indicating greater resilience to uncertainties and external disturbances.

•Theoretical Guarantees: A thorough Lyapunov-based stability analysis offers formal assurances of asymptotic convergence under bounded disturbances, ensuring safe application in real-world scenarios.

The proposed hybrid control framework marks a significant step forward in mobile robot trajectory tracking, providing a balanced solution that combines adaptability, robustness, and high performance. Its modular architecture allows customization to meet specific application needs, making it applicable to a broad range of robotic systems beyond differential drive platforms.

Future research directions include the following: (1) hardware implementation and real-world testing on physical robot platforms, (2) expansion to multi-robot coordination and formation control scenarios, (3) incorporation of vision-based feedback for improved environmental awareness, (4) development of online learning mechanisms for continuous adaptation in dynamic environments, and (5) exploration of transfer learning techniques to reduce the training time for new robot configurations.