Reinforcement Learning-Based Control Strategy for BLDC Motors

Brushless DC (BLDC) motors run on a direct voltage source. They are becoming more popular and widely used. This is because they are constantly being improved. New materials make them stronger and more reliable [1]. They also use less energy, which makes them more efficient. In addition, the electronic parts that control them are getting better. These parts now use digital circuits instead of brushes, which were used in traditional DC motors [2]. These motors serve as a replacement or alternative solution to conventional motors, as they offer a very good power-to-weight ratio and a higher capacity to operate across wider speed ranges [3]. The weaknesses of brushed motors, which BLDC motors successfully address, include lower efficiency, mechanical wear at the commutation level, the need for regular inspection and maintenance, and the requirement for expensive control systems. Due to their favorable electrical and mechanical characteristics, BLDC motors are frequently used in high-precision control systems such as in automotive, aerospace, medical applications, measuring equipment, industrial automation, and electromechanical actuation systems. Numerous control techniques have been developed to maximize the performance and robustness of BLDC motor drive systems [4].

A large number of control models, such as the classical PID controller, can be optimized using online or offline optimization algorithms, or through methods such as fuzzy logic or artificial neural networks (ANN) to determine their parameters [5]. The non-adaptive fuzzy logic controller (FLC) [6], the adaptive fuzzy logic controller, ANFIS [7], sliding mode control, as well as model predictive control (MPC) [8], have also been developed for the speed control of brushless DC motors. The majority of industrial applications still rely on classical PID controllers due to their simplicity and robustness. However, classical PID controllers are generally not effective when the processes involved are higher-order systems, time-delay systems, nonlinear systems, complex or imprecise systems, lacking accurate mathematical models, or subject to uncertainties [9].

Today, methods such as neural networks, random forests, and adaptive neuro-fuzzy inference systems (ANFIS) are used to estimate or dynamically adjust the parameters of PID controllers, to model complex systems, or to predict changes in the behavior of electric motors [10].

Machine learning has revolutionized the field of electric motor control by introducing intelligent approaches capable of adapting to dynamic and nonlinear environments [11]. Unlike traditional methods based on strict mathematical models, machine learning enables the extraction of control laws from measured data, even in the presence of uncertainties, disturbances, or variations in system parameters [12, 13].

More recently, reinforcement learning (RL) has attracted significant interest in the control of electric motors, particularly in electric vehicles, drones, and autonomous robots. This approach is based on the interaction between an agent and its environment, learning to optimize its actions by maximizing a cumulative reward, which is particularly well-suited for speed or torque control. Algorithms such as Q-learning [14], Deep Q-Networks (DQN) [15], and Proximal Policy Optimization (PPO) enable the implementation of adaptive and optimal control strategies without the need for an accurate model of the system [16, 17].

Given the limitations of conventional control methods particularly their reliance on accurate models and lack of robustness in the face of uncertainty Reinforcement Learning is attracting growing interest as a model free, data driven control strategy. Unlike traditional approaches, RL enables the design of optimal control policies through interaction with the environment, learning from system states, performed actions, and received rewards [18]. This approach allows RL agents to identify effective strategies without requiring complex mathematical models, making it especially suitable for systems that are difficult to model. Among its main advantages is its dynamic adaptability, which ensures good performance despite variations in temperature, wear, or external disturbances. RL also stands out for its ability to operate without an explicit model, in contrast to PI or MPC controllers that require precise motor modeling [19]. Finally, it excels at handling the nonlinearities inherent in electric motors and power electronics, by learning complex relationships directly from data. These features make RL a promising approach for intelligent motor control in real world, uncertain environments [20].

Highly precise speed control of electric motors, especially brushless motors, is a major challenge in most industrial and automotive applications, due to the high demands placed on performance, stability and robustness. Conventional control techniques such as PID controllers or fuzzy logic sometimes demonstrate their limits, especially under non-linear, uncertain or disturbed conditions. In this context, reinforcement learning is an interesting alternative. It enables the development of intelligent controllers capable of learning and conforming to system dynamics in real time, without the need for a precise model. This is the motivation behind this article, which aims to:

• Exploit the application of artificial intelligence methods in electric motor control systems, in particular BLDC motors;
• Software design of an intelligent RL-based controller for high-performance, precise control of BLDC motors;
• Overcome the difficulties of standard methods by applying a more flexible, higher-performance approach to BLDC engine control;
• Provide a strong solution to the uncertainties and non-linearities present in real-action scenarios.

This article, which focuses on the speed control of a brushless motor using reinforcement learning, is structured to cover all the essential aspects of this innovative approach. It begins with an introduction that presents the general context, the main control techniques developed in the literature, as well as the motivation behind this study. The second part is devoted to the mathematical modeling of the BLDC motor, a crucial step to accurately represent the system to be controlled. The third part introduces the reinforcement learning algorithm used, detailing the role of the agent and the training process implemented. The fourth part is dedicated to the presentation and analysis of the obtained results, allowing for an evaluation of the effectiveness of the proposed method. Finally, a conclusion summarizes the main contributions of the study and suggests future research directions to improve and extend this approach.

This section is structured into two subsections. The first presents the fundamental principles of reinforcement learning, detailing its key components and main algorithms, as well as the rationale for algorithm choice, with a particular focus on the algorithm used in this article to design and train the agent dedicated to brushless motor control. The second subsection is devoted to an in-depth analysis of the agent training results, with the aim of evaluating its performance and effectiveness in motor control.

3.1 The fundamental principles of reinforcement learning

Reinforcement learning is a machine learning technique in which an agent learns to perform optimal actions by interacting with an environment [24]. Unlike supervised learning, which requires labeled datasets, reinforcement learning operates through a trial-and-error process, where the agent receives feedback in the form of rewards based on the actions it takes [25]. This makes RL particularly well-suited for problems where it is difficult to define explicit rules or provide comprehensive training data.

Reinforcement learning is based on the components illustrated in Figure 2. These components are defined on a rigorous mathematical foundation, and each plays a vital role in the overall RL process, contributing to decision-making and the optimization of the agent’s performance.

2.png

Figure 2. Diagram of the learning process with the various functions

Agent: Acts as a central learning and decision-making entity. It interacts with the environment by observing it, performing appropriate actions and adjusting its behavior according to the rewards received based on the quality of its decisions [26].

Environnement: Describes the external system with which the agent interacts. It provides the agent with observations about its current state, reacts to actions by evolving towards new states, and assigns rewards according to the effects of these actions [27].

States: State, Denoted s, describes the configuration of the environment at a given moment and provides essential information that the agent uses to guide its decisions. In the context of motor control, a state can represent physical quantities such as motor speed, position or current [28].

Actions: Actions, a, represent the decisions or interventions the agent can make to influence the environment. For example, in the case of a motor, this might involve adjusting the applied voltage or switching the inverter control signals. The agent's main objective is to choose the most effective actions to obtain favorable results [29].

Policy: The policy, noted π, defines the agent's decision-making strategy by establishing a correspondence between the observed states and the actions to be taken. It governs the agent's behavior in each situation. A policy can be deterministic, associating a single action with each state, or stochastic, selecting actions according to a probability distribution [30].

Reward: The reward, r, is a digital signal sent by the environment to the agent after each action, indicating the level of relevance or quality of the result obtained. It can be positive, negative or zero, and plays a central role in learning by guiding the agent towards the desired control objectives. For example, a high reward may be assigned when the target speed is reached, while exceeding current limits may result in a penalty. A reward function is often defined to minimize speed error while respecting system constraints [31].

Algorithm: Reinforcement learning algorithms are widely used in motor control, each offering specific advantages depending on its learning approach. Value-based methods, such as Deep Q-Networks (DQN), estimate the value of actions to guide policy selection and have proven effective in high-dimensional control tasks. Policy-based methods, such as Policy Gradient (PG), directly optimize the control policy without relying on a value function, allowing flexible adaptation to complex environments. Actor-critic methods combine both approaches for improved performance [32]. Among these, the Deep Deterministic Policy Gradient (DDPG) algorithm is particularly well-suited for continuous control in motor applications. Building on DDPG, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm introduces significant improvements by reducing overestimation bias in the critic network, enabling more stable and reliable learning. TD3 stands out as a powerful and efficient solution for precise control of brushless motors in continuous action spaces.

In this study, the choice of the Twin Delayed Deep Deterministic Policy Gradient algorithm for controlling the speed of a BLDC motor is justified by the fact that TD3 represents an enhanced version of the Deep Deterministic Policy Gradient. DDPG relies on an actor–critic architecture, where the actor $\mu\left(s \mid \theta^\mu\right)$ generates a deterministic action for a given state, and the critic $Q\left(s, a \mid \theta^Q\right)$ estimates the value of that action. The critic is updated by minimizing the following loss function:

$L\left(\theta^Q\right)=\mathbb{E}_{\left(s, a, r, s^{\prime}\right) \sim \mathcal{D}}\left[\left(Q\left(s, a \mid \theta^Q\right)-\left(r+\gamma Q\left(s^{\prime}, \mu\left(s^{\prime} \mid \theta^{\mu^{\prime}}\right) \mid \theta^{Q^{\prime}}\right)\right)\right)^2\right]$     (9)

However, DDPG generally suffers from instability issues due to Q-value overestimation, poorly controlled noisy exploration, and policy update oscillations. To overcome these limitations, TD3 introduces three major enhancements:

• Clipped Double Q-learning: Two Q-value estimators $Q_1$ and $Q_2$ are used, and the target value is computed using their minimum to reduce overestimation:

$y=r+\gamma \min _{i=1,2} Q_i\left(s^{\prime}, a^{\prime}\right)$ where $a^{\prime}=\mu\left(s^{\prime} \mid \theta^{\mu^{\prime}}\right)+\epsilon, \epsilon \sim \mathcal{N}\left(0, \sigma^2\right)$     (10)

• Target Policy Smoothing: Gaussian noise is added to the target action to make the policy more robust to minor variations:

$a^{\prime}=\mu\left(s^{\prime}\right)+\operatorname{clip}(\epsilon,-c, c)$ with $\epsilon \sim \mathcal{N}\left(0, \sigma^2\right)$     (11)

This prevents the agent from exploiting irrelevant sharp changes in the value function 3. Delayed Policy Updates: The policy network $\mu$, along with the target networks $Q^{\prime}$ and $\mu^{\prime}$, are updated less frequently (every $d$ steps), which helps reduce variance and improve learning stability. Thanks to these enhancements, the TD3 algorithm is employed in this study to enhance the robustness of the BLDC motor speed control in the presence of load torque disturbances. Experimental results demonstrate that TD3 achieves superior performance in terms of Integral Square Error (ISE), reduced overshoot, and improved robustness compared to DDPG. Moreover, TD3 generates a smoother control effort, thereby minimizing mechanical and thermal stress on the motor and its power electronics. Consequently, the selection of TD3 for this application is strongly supported by its ability to learn more stable, accurate, and generalizable control policies in dynamic and noisy environments such as those encountered in BLDC motor systems.

The Table 1 shows the algorithm defining a reward function used by a reinforcement learning (RL) based controller for speed control of a BLDC motor. The algorithm is initiated by estimating the absolute error between speed reference and actual motor speed. If this error is greater than or equal to 1, a quadratic penalty is applied in the form of a negative reward $-e^2$, to discourage large errors. On the other hand, for errors less than 1, a non-linear transformation is applied, assigning the error a negative value of its square root, which reinforces learning even in the case of small deviations. If the error is strictly negative (a rare but covered theoretical case), it is simply reduced to zero. This reward mechanism ensures fast, accurate tracking of the speed setpoint, while guaranteeing learning stability and robustness to disturbances. By dynamically decoupling from the error amplitude, this function enables the RL agent to adjust the control signal efficiently to minimize speed deviation in non-linear and uncertain environments.

Table 1. An algorithm for the process of learning

3.2 Training agent and results analysis

The Figure 3 illustrates the evolution of the reward per episode during training of a Twin Delayed Deep Deterministic Policy Gradient agent for speed control of a BLDC motor. The x-axis represents the episode number, while the y-axis shows the reward obtained, reflecting the control performance. The curves show both rewards per episode (thin, irregular lines) and rolling averages (thick lines), highlighting the agent's progressive improvement over time. Initially, rewards are low, reflecting an inefficient control strategy. However, with learning, rewards increase steadily, indicating that the agent is progressively optimizing motor speed control. Convergence towards a higher reward reflects a stabilization of the agent's strategy, and thus a more efficient and stable speed control.

3.png

Figure 3. The evolution of reward per episode during training of a TD3 agent

4.png

Figure 4. The table of training progress

Figure 4 shows a summary of the training of an rlTD3Agent agent applied to the speed control of a BLDC motor via a TD3-based reinforcement algorithm. Training took place over 100 episodes and lasted about 7 minutes, ending normally after reaching the maximum number of episodes. Rewards indicated control performance: the final episode reward was -2541.72, and the average reward over all episodes was -3046.06, showing progressive improvement. The Q0 value for the episode (-2216.58) reflects the estimated quality of the policy in the initial state. These results show that the agent has learned a relatively effective, albeit perfectible, control strategy for stabilizing engine speed while minimizing error.

This section aims to present and analyze the results obtained from the simulation carried out using MATLAB Simulink for the speed control of a brushless motor, by using one of the machine learning tools, which is reinforcement learning. The analysis of the results is conducted through two scenarios to thoroughly evaluate the validity of the controller used. The first focuses on the tracking performance of the imposed speed reference, and the second on robustness, by varying the load torque to observe the system's response. The parameters used in this simulation are presented in the Table 2.

Table 2. Parameters for the simulation [33]

5.1 First scenario: Tracking performances

5.1.1 Performance tracking under no-load conditions

The purpose of this part of the first scenario is to study the motor's response at no load, without applying any load torque. Figure 6 shows the motor response with a reference variation from 80 rad/s to 100 rad/s. For the first startup interval, the motor speed follows the 80 rad/s reference after just 1.5 seconds, and for the second interval, it follows the 100 rad/s reference after just 1.2 seconds. As for the error, it vanishes, as shown in Figure 7. Figure 8 illustrates the controller’s output voltage during the speed variations, clearly showing the interaction of the agent, which forms the basis of the controller, with the speed changes. Due to the proportionality between the brushless motor speed and the voltage applied to the motor, the controller generates a voltage of 125 V for a speed of 80 rad/s and 156 V for a speed of 100 rad/s. These results in this section are obtained in the absence of load torque, as shown in Figure 9.

6.png

Figure 6. Motor speed variation during change in reference speed without load condition

7.png

Figure 7. Speed tracking error with null load torque

8.png

Figure 8. Controller output voltage during speed reference variations without load torque

9.png

Figure 9. Load torque applied to the motor

5.1.2 Performance tracking under load conditions

In order to further evaluate the performance, this second part of the first scenario applies a constant load torque to the motor, T = 2 Nm. As already mentioned in the first part, Figure 10 shows the motor's response to a variation in the reference speed from 80 rad/s to 100 rad/s, but this time with a load torque equal to 2 Nm. At startup, the motor follows the reference after 1.9 seconds, and for the second variation, the response follows the reference after 1.5 seconds. Similarly, when decreasing the speed from 100 rad/s to 80 rad/s, the results are the same, as shown in Figure 11. Regarding the error between the two speeds, it vanishes, as shown in Figure 12. Figure 13 shows the controller output applied to the brushless motor. The controller generates a voltage of 174.5 V to reach the reference speed of 80 rad/s, and 205.7 V to reach the speed of 100 rad/s. These results are obtained under the application of a constant load torque of 2 Nm, as shown in Figure 14. To more rigorously evaluate the speed tracking performance, a more complex reference speed scenario was adopted, as illustrated in Figure 15. The resulting curve demonstrates accurate and fast speed tracking, even in the presence of abrupt changes in the reference, with a response time ranging from 0.5 s to 1.8 s.

10.png

Figure 10. Motor speed variation during change in reference speed with load condition

11.png

Figure 11. Motor speed variation when decreasing reference speed with load torque

12.png

Figure 12. Speed tracking error with load condition

13.png

Figure 13. Controller output voltage during speed reference variations with load torque

14.png

Figure 14. Load condition applied to the motor

15.png

Figure 15. Load condition applied to the motor

5.2 Second scenario: Robustness performances

Regarding the robustness of the controller against external disturbances such as load torque variations, two tests were carried out to properly evaluate its robustness. The first test involves varying the torque in an increasing manner, from 2 Nm to 4 Nm. Figure 16 shows that the motor speed closely follows the desired value of 100 rad/s, with a downward deviation of about 20% from 100 rad/s, which is expected during a torque increase. However, the speed returns to its reference value after only 1.8 seconds, which sufficiently demonstrates the effectiveness of the controller in handling torque variations. Figure 17 shows that the speed error is eliminated. The voltage generated by the controller is illustrated in Figure 18, where it can be seen that the motor supply voltage increases from approximately 205.7V to 255.5V as the load torque varies from 2 Nm to 4 Nm. These measurements were taken under a variable load torque increasing from 2 Nm to 4 Nm, as shown in Figure 19.

16.png

Figure 16. Variations in motor speed as load torque increases

17.png

Figure 17. Speed error during load torque variation and growth

18.png

Figure 18. Controller output voltage as load torque increases

19.png

Figure 19. Increasing load condition applied to the motor

Regarding the second robustness test, it consists of varying the load torque in a decreasing manner, from 4 Nm to 2 Nm, in order to observe the response of the controller-motor system. Figure 20 shows that the motor speed perfectly follows the desired value of 100 rad/s, with an overshoot of about 20% relative to 100 rad/s, which is logical when the load torque decreases. However, the motor returns to its reference speed after only 1.6 seconds, which sufficiently demonstrates the effectiveness of this controller in handling torque variations. Figure 21 shows that the speed error cancels out. The voltage generated by the controller is illustrated in Figure 22, where it can be seen that the motor supply voltage drops from approximately 255.5 V to 205.7 V as the load torque decreases from 4 Nm to 2 Nm. These measurements were taken under a decreasing variable load torque from 4 Nm to 2 Nm, as shown in Figure 23. These two robustness tests demonstrate the reliability and the advantages of using reinforcement learning-based agents employing the Twin Delayed Deep Deterministic Policy Gradient algorithm in the regulation and control of brushless motors. To further evaluate the robustness of the TD3-based reinforcement learning controller, Figure 24 illustrates its resilience against sudden and rapid load torque disturbances ranging between 0 N.m, 1 N.m, and 2 N.m showed in the Figure 25. The controller maintains a high-performance response with ripple limited between 20% and 30%, and a response time ranging from 0.5 s to 1.8 s.

20.png

Figure 20. Variations in motor speed as load torque decreases

21.png

Figure 21. Speed error when varying and decreasing load torque

22.png

Figure 22. Controller output voltage as load torque decreases

23.png

Figure 23. Decreasing load condition applied to the motor

24.png

Figure 24. Speed response to load torque variations

As part of the evaluation and enhancement of the approach studied in this article, which focuses on speed regulation of a BLDC motor using a Reinforcement Learning-based controller (TD3 algorithm), this section presents a comparative study between the RL controller and two conventional controllers: PID and Fuzzy Logic. The comparison is based on four key criteria: robustness to disturbances, real-time adaptation capability, design complexity, and maintenance requirements. It is worth noting that the performance of the RL-based controller, as demonstrated in this work, could be further improved with a more powerful PC and CPU, since the approach requires a significant training time. The results of this comparative analysis are summarized in the following Table 3.

25.png

Figure 25. The variation in load torque applied to the motor

Table 3. Comparison table between the three controllers