A New Approach Based on Fuzzy-Q-Learning Algorithm to Control 3 Level T-Type Voltage Source Converter

Conventional tuning methods are based on adequate modeling of the system to be tuned and analytical processing using the transfer function or equations of state. Unfortunately, these are not always available. These control techniques have proven their effectiveness in many industrial regulation problems. Advanced control methods (Adaptive Controller, Predictive Control, Robust Control ...) meet the requirements of a number of highly non-linear systems. It is in this same deal that the fuzzy modeling and control methods are positioned [1]. The majority of complex industrial systems such as the voltage source converter are difficult to control automatically. This difficulty arises from: - their non-linearity, - the variation of their parameters, - the quality of the measurable variables. These difficulties have led to the advent and development of new techniques such as fuzzy control, which is particularly interesting when there is no precise mathematical model of the process to be controlled or when the latter presents strong nonlinearities or imprecisions [2]. In many applications of VSC controller, the results obtained with a fuzzy controller are better than those obtained with a conventional control algorithm. In particular, the fuzzy control methodology appears useful when the processes are very complex to analyze by conventional techniques. Several works in the field of VSC control have shown that a fuzzy logic regulator is more robust than a conventional regulator.

Despite this performance of fuzzy logic, it remains limited in the case of dynamic systems, a change of a parameter or an external disturbance can affect the functioning of this system, and so, even the technique of fuzzy logic requires an improvement. In this context, H.R. Berenji proposes a reinforcement of this technique; it is the Fuzzy Q-learning approach [3, 4]. The principal of this last one, which make the fuzzy controller in dynamic mode, will be detailed in the next section

By investigating in the quality of this approach, and using Direct Power Control (DPC) [5, 6], in this work, we analyzed the effect of this combination in helping a VSC to work properly.

In the same objective, not just the technic of control it performed but the structure of the VSC also has been changed, for that, a 3 level T-type VSC converter represent the studied system [7].

This paper is organized as follows: in Section 2, three level T-type VSC topology and its modeling are presented. The Proposed fuzzy Q learning direct power control (FQL DPC) is developed in Section 3. The application of the proposed control approach with 3 level DPC technic on the studied VSC is presented in Section 4. Sections 5 describe the simulations results and their discussions and finally conclusions are mentioned in the last section.

3.1 Fuzzy-Q-Learning

In the Fuzzy-Q-Learning (FQL) algorithm, the membership functions of the input variables are defined by natural knowledge extraction. As for the number of rules, due to the determined input domain structure of the SIF, it is automatically determined by the number of linguistic terms of each variable. The modifiable characteristics of the apprentice therefore consist only of the parameters concerning the conclusions [12, 13].

The principle of the learning algorithm, Fuzzy-Q-Learning, consists in choosing for each rule a conclusion among a set of conclusions available for this rule (called limited vocabulary). It then implements a process of competition between actions. The management of this process is conventionally carried out by associating with each action a quality depending on the state. This quality then determines the probability of choosing the action, which must be discreet. To also allow the management of continuous actions, we propose to no longer consider the quality of actions according to states but according to rules. To do this, the actions corresponding to each rule have a quality. From this local quality, the algorithm chooses, for each activated rule (i.e., those describing the current state), an action among the set of discrete actions available in these rules. The final action (discrete or continuous) is then the result of the inference obtained with the various locally elected actions.

The matrix q makes it possible not only to implement local policies to rules, but also to represent the evaluation function of the global t-optimal policy.

Figure 4 schematizes the structure of the apprentice proposed for the FQL algorithm in the case of discrete actions and continuous actions.

4.jpg

Figure 4. FQL in the case of discrete actions and continuous actions

3.2 FQL architecture for the optimization of conclusions

The architecture of the proposed FQL algorithm for the optimization of conclusions is shown in Figure 5.

The premise part of the SIF is fixed (all the states), the parameters of the conclusions, i.e., all the actions are available thanks to a priori knowledge of the system to be controlled, (27 actions for one 3-level inverter). The qualities of the actions are evaluated by the evaluation function Q [14, 15].

Consider a Takagi-Sugeno-type SIF, with initialization of identical conclusions for each fuzzy rule. A rule has the following form:

$\begin{aligned}

&R_{i}: \text { if } S_{1} \text { is } L_{1}^{i} \& \ldots \ldots . \& S_{N_{I}} \text { is } L_{N_{I}}^{i} \text { then } Y \text { is } a_{1} \text { with } q_{1}^{i}\\

&\begin{array}{l}

\text { or } a_{2} \text { with } q_{2}^{i} \\

\text { or } a_{3} \text { with } q_{3}^{i} \\

\quad \vdots \\

\text { or } a_{k} \text { with } q_{k}

\end{array}

\end{aligned}$     (9)

5.jpg

Figure 5. Architecture of the proposed FQL algorithm

And $i=1 \ldots N$

With,

- R_i the its basic rule;

- Y the output variable;

- $S=\left(S_{1}, \ldots \ldots \ldots, S_{N_{I}}\right)^{T}$ input variables whose domain of definition is denoted S;

- $L_{j}^{i}$ a linguistic term of the fuzzy variable used in the rule and whose membership function is noted $\mu_{L_{j}^{i}}$;

- {a1, a2, a3, a4, a5, …, ak} is the set of actions to be applied, it is either predefined from knowledge of the system to be controlled, or initialized randomly. It remains invariant during the learning process.

At the end of the learning process, the elected action Ae for each fuzzy rule is the action that has maximum quality.

At each time step. The SIF calculates the output action and the corresponding quality. This action is applied on the system which returns a reinforcement signal r. From this feedback signal, the quality value updated using the time difference method. An exploration / exploitation policy is implemented at the level of the FQL algorithm only at the start, thereafter it will be ensured by itself.

3.3 Procedure for executing the FQL algorithm

We describe the execution of the FQL algorithm over a time step. It can be divided into six main stages. Let t + 1 be the current time step, the apprentice applied the action (discrete or continuous) elected at the previous time step, and in return received the primary reinforcement due to the transition from state to state. He then perceives this state through the truth values of his rules and performs the following six steps:

(1). Calculation of the t-optimal evaluation function of the current state:

$Q_{t}^{*}\left(S_{t+1}\right)=\sum_{R_{i} \in A_{t}}\left(\operatorname{Max}_{U \in U}\left(q_{t}^{i}(U)\right) \alpha_{R_{i}}\left(S_{t+1}\right)\right.$     (10)

(2). Calculation of the TD error:

$\tilde{\varepsilon}_{t+1}=r_{t+1}+\gamma Q_{t}^{*}\left(S_{t+1}\right)-Q_{t}\left(S_{t}, U_{t}\right)$     (11)

(3). Update of the matrix:

$q_{t+1}^{i}=q_{t}^{i}+\beta \tilde{\varepsilon}_{t+1} e_{t}^{i^{T}}, \forall R_{i}$     (12)

(4). Election of the action applied on the state

• Discrete actions: inference of the overall quality of each discrete action from their local qualities:

$q_{t+1}^{S_{t+1}}(U)=q_{t+1}(U) \cdot \varphi_{t+1}^{T}, \forall U \in U,$     (13)

And election of one of these actions according to the exploration / exploitation policy:

$E_{U}\left(q_{t+1}^{S_{t+1}}\right)=A r g M a x_{U \in U}\left(q_{t+1}^{S_{t+1}}(U)+\eta^{S_{t+1}}(U)+\rho^{S_{t+1}}(U)\right)$     (14)

• Continuous actions: for each activated rule, election of a local action and inference of the continuous action to be applied:

$U_{t+1}\left(S_{t+1}\right)=\sum_{R_{i} \in A_{t+1}} E_{U}\left(q_{t+1}^{i}\right) \cdot \alpha_{R_{i}}\left(S_{t+1}\right), \forall U \in U$     (15)

where, election is defined by:

$E_{U}\left(q_{t+1}^{i}\right)=\operatorname{ArgMax}_{U \in U}\left(q_{t+1}^{i}(U)+\eta^{i}(U)+\rho^{i}(U)\right)$     (16)

5. Update of eligibility traces:

$e_{t+1}^{i}\left(U^{i}\right)=\left\{\begin{array}{l}

\gamma \lambda \cdot e_{t}^{i}\left(U^{i}\right)+\phi_{t+1}^{i},\left(U^{i}=U_{t+1}^{i}\right) \\

\gamma \lambda \cdot e_{t}^{i}\left(U^{i}\right), \quad \text { sinon }

\end{array}\right.$     (17)

6. New calculation of the evaluation function corresponding to the elected actions and the current state with the newly updated parameters:

$Q_{t+1}\left(S_{t+1}, U_{t+1}\right)=\sum_{R_{i} \in A_{t+1}} q_{t+1}^{i}\left(U_{t+1}^{i}\right) \alpha_{R_{i}}\left(S_{t+1}\right)$     (18)

This value will be used for the computation of the error TD at the next time step.

For the studied system represented in Figure 6 above, the application of the proposed technic is illustrated by Figure 8 to 15 that represent the obtained results after simulation.

Note that the simulation was done using Matlab/SimPowerSystem environment and a fault was occurred at 0.7s and 1.5s designed by a change in desired reference of active power and at t=2.2s for reactive power.

First of all, Figures 8 and 9 show respectively, the active and reactive power in the AC side.

As showing in this result, the such active and reactive desired power mentioned by its references (red line), are perfectly obtained and they follow the references values, even when a change in refences happens (load variation for example), the VSC based on the FQL-DPC approach ensure this change in fraction of seconds and without any losses of response.

8.jpg

Figure 8. Active power

9.jpg

Figure 9. Reactive power

This response can be confirmed by evaluate the current and the voltage of the phases, that represented in Figure 10.

Taking phase (a) for the evaluation, we can see clearly that the VSC save it voltage in the desired reference, where the current change rapidly without losing it nature of signal, which is properly sinusoidal.

For better evaluation of the behaviour of the current, Figures 11, 12 and 13 shown the different current shift in the circuit.

From Figure 12, we can see the variation of current called by the load follow by a variation in the value of the current from -5A to 5A between 0.7s and 1.5s, in this period the behaviour of the current is slightly changed but it saves its sinusoidal shape (Figure 11).

Also, the current in the middle point which name as I_o showing in Figure 13 has variate with any change which confirm the reaction and the operation of the VSC base on Q learning DPC technique under any condition of working.

10.jpg

Figure 10. Voltage and current of phase a AC side

11.jpg

Figure 11. Three phase AC current

12.jpg

Figure 12. Load Current

13.jpg

Figure 13. Current zero in middle point

14.png

Figure 14. DC voltage of the DC side

15.png

Figure 15. DC voltage of each capacity

Finally, in order to evaluate the characteristics of the control, we base on the response of the voltage on the DC side and the two capacitors builds the DC part of the VSC.

Figures 14 and 15 shown the response of the DC side, from these two figures, we can see the performance of the command adapted to control of the VSC.

At the transient period (reference variation), we can note exceedances of 11% with a response time of 1e^-3s with a final value exactly the same (700V).