Optimization of Deep Reservoir Computing with Binary Genetic Algorithm for Multi-Time Horizon Forecasting of Power Consumption

Energy is part of everyday life. Its role is extremely crucial; it is the driving force behind economic activity and personal and social development. On the one hand, due to the growth of human society with the increase of energy efficiency and the use of developed artificial intelligence techniques, on the other hand, the growth of population and economy leads to high energy consumption, which produces a major challenge [1]. Today, the world is facing a major double challenge in the field of energy. The first challenge is the lack of safe and adequate energy sources and the second is the large-scale renewable energy grid-connected power generation may have adverse effects on power grids [2]. Energy sectors around the world are under huge pressure to assure a stable and reliable energy supply. Depending on the data of BP Statistical Review of World Energy, in 2018, global primary energy consumption knows the biggest growth since 2013 by 2.8% [3].

The global transition to a smart grid is justified by the need to satisfy the ever-increasing electricity consumption and to guarantee the sustainable and secure supply of the power system [4]. Smart Grid is based essentially on integration of renewable energy sources [5]. In addition, many countries have opted for a renewable energy development policy due to rising oil prices and global climate change [6]. In these cases, forecasting of consumption power and renewable energy power production becomes more critical than ever before, in managing smart grid systems, which satisfies future sustainable needs. Under these circumstances, there are many methods for power forecasting, it can be classified to four main approaches: Mathematical approaches, machine learning approaches, deep learning approaches and hybrid approaches [7]. Mathematical technique has been applied in the past to predict power time series [8]. This can be classified into two categories: persistence model and statistical method. Unfortunately, it usually produces forecasts with poor accuracy and does not work well with non-linear data as well [8]. Because of these limitations, machine learning introduces better accuracy and performance, such as the SVM (Support Vector Machine) [9], ANN (Artificial Neural Network) [10], ELM (Extreme Learning Machine) [11].

In recent years, The ANN methods are developed, under the name of deep learning [12], which are considered complex machine learning models with many layers, hyper-parameters and algorithms for making a high accuracy and reality. Generally, deep learning used for three problems, regression (prediction), classification, and clustering [13]. As a result, it has many models used depend a nature of problems. RNN (Recurrent neuron network) is a famous one for power forecasting as a time series data [14, 15], because of his ability to memorize information from the previous states [14]. Recently RNN has developed in many forms as LSTM (long short-term memory), GRU (Gate recurrent units), and ESN (Echo state network) form to deal with problems such as vanishing gradient, exploding gradient, longtime training, and optimizing the flexibility of data training [16-18]. Both the consumption and production power are generally non-linear, volatile and uncertain. In this field, the Echo state Network (ESN) is the best due to his ability of modeling such kind of time series data [18]. ESN's hidden layer is replaced by a dynamic reservoir composed of several weakly connected neurons, that should make information processing easier [18]. In addition, during the learning process, only the read weights are trained on the basis of linear regression (LR) algorithms [18]. As a result, ESN has a fast convergence speed and can attain the overall optimal solution. In recent years, the introduction of many layers frameworks in the RNN has attracted much interest [15]. Recently, the Echo state network has been developed to have many architectures with additional layers and hyper-parameters under the name of DeepESN (Deep Echo State Network) [19]. It plays a very powerful learning role and in solving difficult time-series problems [19]. However, this type of deep learning architecture is typically developed by experts through trial and error, which takes a long time to obtain precise results [19]. Graphics processing unit (GPU) performance has increased significantly in recent years, which has led to the widespread use of reinforcement learning and evolutionary algorithms to make it possible to automate the process of finding the best model structure [20, 21]. Therefore, the use of a hybrid model combined between DeepESN and some optimization model can avoid the trial and error procedure and eventually obtain a strong increase in model performance. There are several methods to optimize forecasting models, generally, they can be classified into two big categories: Statistical methods, and Natural-inspired algorithms.

Starting with statistical methods, this kind of optimization models have a large bound of utilization in the field of energy forecasting, it is used to optimize the fitness function or loss function by finding the best hyperparameter or parameter values of the prediction model. The following lines show some works that used optimization statistical methods. Grid search (GS) was applied in [22] by Raviv et al., it is one of the basic methods that depend on finding the best values of hyperparameters, based on his performance metrics GS shift the values into the optimal point that represent optimal solution. By the way, in one of our articles [23] we used GS to optimize a hybrid model that constitutes from GRU and PCA methods to predict Global horizontal irradiations, choosing learning rate, bash size, dropout as optimized hyperparameters and MSE as loss function, so that gives a great result in term of efficiency and performance. Especially for neuron network parameters, Gradient Descent (GD) considered a conventional optimization algorithm that used to find the best weight values by moving to a minimum local point of fitness function. It has been reported in the literature that GD outperforms GS in the area of finding the best neuron network (Weight) parameters. It can be found countless papers that use GD, among them [24] written by Amarasinghe et al. Some studies as [25] and [26] use cross validation, which is applied mainly in the preprocessing of the data for prediction models. Generally, this method is applied in parallel to hyperparameters optimization, by splitting the datasets into one for training the model and different set to test optimized hyperparameters that make, to achieve the best one on a dataset. This method has many types including: K-fold cross-validation, leave p out cross-validation, hold-cross validation [27]. The community of researchers in the filed developing optimization methods applied other form of statistical and probabilistic methods in their works, which are named the family of Bayesian optimization. It is based on probabilistic equation to achieve the best objective function. Among the works that use these methods we found in Refs. [28, 29]. In this term, these papers find that Bayesian methods outperforms GS and CV methods. Nevertheless has a limitation, represented in that the covariance function parameters the must be tuned, which make the competition more complex [30]. There are also others statistical optimization approaches that are used in filed of energy forecasting as Quasi Newton Method which used by conzaleg et al. to optimize the weight in ARMAX model for time series forecasting [31]. Dynamic Integrated Forecast System Dicast applied by Sulaiman et al. [32]. Levenberg Marquardt used in [33] to optimize Self-recurrent wavelet NN (SRWNN) for short term load forecasting. Excavated association rules used in [34] for RBFNN hyperparameters optimization. Non-homogeneous poison process (NHPP) used by Yue et al. [35] and Altering Direction of Multiplier method used by Yu et al. [36].

In the field of energy forecasting, the natural-inspired algorithms have a big attraction by researchers and developers, e.g. for achieving a good efficient energy management, the paper [37] targets energy generation forecasting from renewable sources by using ANN model optimized by differential evolution (DE), which can deal with multiple hyperparameters of forecasting model. DE can be considered as a technique for the global optimization of nonlinear and non-differentiable continuous space functions or objective function. In literature, genetic algorithm (GA) used in many works related to prediction models situation. E.g. in [38], which used it to optimize hyper-parameters and parameters of Deep learning model with stacked auto-encoders aiming to forecast wind power. Moreover, GA was used in one of our works [39], with other form which is Binary GA (BGA), this version shows great result in term of architecture hyperparameters (i.e. number of units and number of layers) optimization of deep learning models (i.e. Simple RNN, LSTM, GRU). GA can be found also in another form, which named micro genetic algorithms (MGA), among the papers that used this method, we found [40], so that it was applied as an optimizer of weight of relevant vector machines, to predict electricity price. One of the most famous optimization algorithms is told Particle swarm optimization (PSO) that is used to look for the maximum or minimum of a function defined on a multidimensional vector space, which represent hyperparameters bound values. Raza et al. in [41] used both the gradient learning techniques and PSO to optimize the parameters of neuron networks as Feed forward neuron network (FFNN), back propagation neuron network (BPNN) to predict electricity load demand and he found that the gradient learning has a poor convergence performance, leading to inefficient model training. On the other hand, PSO address the optimization with height performance leading to obtain the best forecasting model. There are also many other natural inspired algorithms as Cuckoo Search Algorithms (CSA) using by Xiao et al. to optimize the coefficient weight of NN models [42], Modified firefly algorithm (MFA) which was used in many works, among them [43], which hybridize SVR with MFA to achieve better short-term load forecasting. Quasi oppositional artificial Bee colony optimization (QOABCO), which has a great impact to the forecasting model for electricity price and load, in the published work of Shayeghi et al. [44]. Novel shark search algorithm (NSSA) which is can be found in work [45], that represent an improved Elman NN by using (NSSA) to predict short-term load electricity. The family of meta-heuristic algorithms also applied in some works, as can be found in paper of Artich of Chou and Ngo [46], which used a hybrid version of ARMA and SVR model tuned using one of meta-heuristic methods. This family of methods can be seen also in [47], so that the authors here proposed an LSTM model aiming to forecast electricity price and demand, some hyperparameters of this model optimized using jaya optimization algorithm which is considered as one of the meta heuristic optimization family. Final natural inspired algorithm optimization method in our review is the Improved Environment Adaption Algorithm, which is applied by Singh et al. in his work [48], his application targets to tune the weights and the hyperparameters of ANN model, as a result giving to the model more precision and capability to overcome the over and under fitting issues. All of these methods can give relatively a pretty good result, by achieving the optimal values of certain parameters and hyperparmeters of Deep learning model. However, in the field of deep learning architecture optimization PSO, GA and DE are the most used. To summarize this related works, it is found that the PSO is the best one in term of accuracy compared with others but is suffering from low speed of computing. In addition, it is found that the GA achieves good fitness function value compared with DE. The BGA has significant optimization in this field and better in term of computing processing speed than simple GA.

In general, the hyperparameters can be divided into two categories training hyperparameters and architecture hyperparmeters [49]. The training hyperparameters are the variables which deal with how deep learning is trained (e.g. learning rate and bash size in RNN) or (e.g. Leaking rate and spectral radius in ESN). While the architecture hyperparmeters indicate the structure of network of deep learning. The BGA will be used in this work to select the best architecture of DeepESN by finding the adequate number of deep ESN units (neurons) and hidden layers, as result improving performance of DeepESN. In this study, a presentation of theoretical basis of ESN and DeepESN and there Architectural and learning hyperparameters will be presented, including different advantage of DeepESN compared with ESN in term of minimizing forecasting error and their ability to deal with long time series data as consumption power. This work also will show the capability of BGA to find best values of architectures hyperparameters. So that will be compared with other optimization algorithm as simple GA, PSO and DE. In addition, this paper gives a comparison of different hybrid deep learning models i.e. RNN-BGA, LSTM-BGA, GRU-BGA, ESN-BGA with DeepESN-BGA. The comparison of results will be based on MSE, RMSE and MAE as evaluation metrics to get the best model in term of accuracy, and it will base on computation time to compare the processing duration of different models.

In general, the motivations of this work are based in first on the optimisation of the architectural part of the DeepESN using binary genetic algorithms that can improve the prediction of the energy consumption by the deep learning mode. Testing the proposed model in the prediction of different cases of short-term time horizons. Achieving the best time computation with high accuracy and performance in different short horizons. Finally, finding the best way to deal with architectural hyperparameters of DeepESN, by testing different optimization models. The contribution of this study can be identified as follows:

•   To the best of our knowledge, there is no work has optimized the architected hyperparameters of DeepESN by using our proposed model can improve the prediction operation of deep learning.

•   The prediction in this work has been used for different time horizon (1 min, 5min, 10 min, 20 min) aiming to evaluate the model in different case of short-term prediction.

•   A comparison between different deep learning models and DeepESN-BGA, in term of accuracy and time computation, deepESN has been shown great outperformance in this field.

•   A comparison between hybrid DeepESN-BGA and other hybrid models i.e. DeepESN-GA, DeepESN-PSO, DeepESN-DE. This comparison is the first one for consumption power forecasting. BGA has been given optimal values of architecture hyperparameters (i.e. number of units of each layer, number of layers).

This paper is organized as follows: Section 2 general description of the data used for forecasting. Sections 3 for descriptions of GA and BGA and their proprieties. Sections 4 describe the basic ESN. Section 5 for proposed Deep ESN and their advantage respectively. Section 6 introduces the experimental models used and his application. Section 7 shows the representation of the results and the discussions. Finally, Section 8 for conclusions and summarizes the directions for future research.

3.1 Genetic algorithm

Nature has always been a great source of inspiration for all humankind. Genetic Algorithms (GA) are search algorithms based on the concepts of natural selection and genetics. GAs are a subset of a much larger branch of computing known as evolutionary computing. It is frequently used to solve optimization problems in research, specifically, in the task of parameters of machine learning and deep learning [51]. The GA process is divided into two parts:

• Identify the appropriate chromosomal encoding to be used as solutions or parameters to be tested.

• Determine the fitness function to be used to test the solutions obtained from the GA to determine if they are suitable to be used as next-generation solutions.

Genetic algorithms are carried out in different steps as shown in Figure 2 above. Firstly, it is important to initialize a population, which is a group of individuals who can solve the problem in question. An individual is characterized by a set of factors called genes and when these genes are linked together, they are called a chromosome and a group of chromosomes makes a population. After selection of the population, test the solutions got from GA whether it is appropriate to be used as a next generation solution or not by using fitness functions. In this study, Fitness Function is Mean Square Error (MSE).

The operation of mating the individuals makes a new solution from the old solutions, this operation named crossover, allowing the algorithms to have many solutions and to select the best of them. After this, mutation step, that gives to the solutions more variety, which can support the algorithms to be unpredictable and to be not blocked in an unending loop of selections or crossovers.

3.2 Binary genetic algorithm

Binary GA, is another version of GA, this version steal has the same algorithm method. The difference here in representation of individuals which are the solution of fitness function, so that it be initialized as binary number, to present a binary chromosomes of gens of 1 or 0, then BGA try to assemble these chromosomes to create a population of binary number. The fitness function is represented in function of chromosome. After selection of the population, test the solutions got from BGA whether it is appropriate to be used as a next generation solution or not by using fitness functions. The description of each step of the mechanism of BGA is presented as follow:

2.png

Figure 2. Flowchart of genetic algorithm

Firstly, it is important to define the fitness function which is minimized, the variables which represent in this work the architecture hyperparameters of the deep learning model, and all parameters related with BGA as number of bites per gen, population size, mutation rate, selection process, mating process, and convergence condition. Secondly, it must initialize population, according to the number of hyperparameters and the population size. Thirdly, the population, including the chromosome is decoded to produce a solution space of variables. Then, the fitness function of each chromosome is calculated and listed, aiming to minimize them. Additionally, the operation of natural selection is operated in the process, the BGA here create what is named offspring in each generation by making the selection of linked chromosomes, this operation based on selecting a set of fittest chromosomes from which the parent will be chosen, and all relative chromosomes are thrown and be substituted by produced offspring. Once the natural selection is finished, the mating step start. Generally, three approaches can be used to perform the mating: single point crossover, double point crossover and uniform crossover. In the first one, a random selecting of a crossover point between the first and last bit of the chromosomes by swapping the bites. In the second one, here the exchange of the chromosomes bites is existing between two crossover points. In the third one, the gens here are chosen arbitrary, based on one of the relative genes of the parent chromosomes. After the production of offspring, the mutation happens, that enable the BGA to give populations new information, in the case of less efficiency of population in term of cost, this population will be rejected during the next iteration. In this term, the mutation rate indicates the number of bits is mutated in the population. Finally, the Algorithm verifies whether all the given convergence condition is met. If they are met, the operation stops by achieving the most likely variables that can minimize the fitness function. If it is not, the iteration of several generations is activated by an algorithm, which repeats all of his process, including decoding, cost computation, mating and mutation.

5.1 Architecture and algorithm of DeepESN

4.png

Figure 4. DeepESN architecture

DeepESN model is a new extended model of ESN, the most important character of DeepESN is the stacked hierarchy of reservoirs it could be has many reservoir layers as shown in Figure 4. The input of the first reservoir layer in each step j is equal to the outside input directly, while the previous layer output fed another reservoir layer. In order to make it more convenient, reservoir layers have identical number of neurons $N_r$. Moreover, $N_l$ indicate the total number of reservoir layers, and the $X^{\left[N_l\right]}(j)$ gives the states of the reservoir layer at time j, the reservoir states of the first layer and those layers at levels higher than 1 are updated with Eq. (7) and (8) respectively:

$X_{(j)}^{[1]}=F\left(w_{i n} \times U(j)+\widetilde{w}^{(1)} \times X^{[1]}(j-1)\right)$           (7)

$X_{(j)}^{[l]}=F\left(w^{(l)} \times X^{[l-1]}(j)+\widetilde{w}^{(l)} \times X^{[l]}(j-1)\right)$         (8)

$F=\left[F^{(1)}, F^{(2)} \ldots F^{\left(N_l\right)}\right]$           (9)

$F^{(l)}\left(l=1,2,3 \ldots, N_l\right)$          (10)

With $l=1,2,3 \ldots . N_l$, illustrate the evolution of reservoir states at layer l, assuming that F represents the function of activation for reservoir states in each layer (Eq. 9 and 10). DeepESN have weights metric as ESN model, in addition to $w^{(l)}$ matrix, which describe the relation between $N_l \times N_l$ represent the internal recurrent weights for layer l, and $\widetilde{w}^{(l)}$ the inter-layer connection weights from layer 1 to layer $N_l$, accordingly.

In ESN, each of the $w_{i n}$ and $w_{b a c k}$ is created at random according to a uniform distribution from the range [0, 1] and maintained throughout all phases of training, the same things for $w_{i n}$ and $w^{(l)}$ of DeepESN model. The w of ESN, and $\widetilde{w}^{(l)}$ of DeepESN are a scattered matrix initiated randomly, and they are scaled by using the relation as $\frac{\rho \times w}{\left|\gamma_{\max} \quad \right|}$ or $\frac{\rho^{(l)} \times \widetilde{w}^{(l)}}{\left|\gamma^{(l)}{ }_{\max } \quad \right|}$. With $\left|\gamma_{\max }\right|$ or $\left|\gamma^{(l)} \max\right|$ represent the spectral radius which is means the highest absolute intrinsic value of w, and ρ or $\rho^{(l)}$ is a hyperparametres of setting the spectral radius [56], with $l=1,2,3 \ldots ., N_l$. In order to enable the current internal states of the reservoir are simply associated with the input history over a few iterations, the spectral radius ρ has to be less than 1 [19, 55]. Within the learning procedure, only the $w_{\text {out }}$ has to be trained based on the LR algorithms, which allows for highly successful learning [56].

5.png

Figure 5. Flowchart of DeepESN method

As ESN model hyperparameters, the DeepESN has LR, connectivity rate α, spectral radius ρ, and instead of N, it has the reservoir scale $N_r$ in each reservoir. In addition, of these hyperparametres, the number of stacked reservoir layers $N_l$ is very important in DeepESN. For the calculation of output, Figure 3 illustrates how the state-output connected. The global stat is showed in Eq. (11), then it updates in Eq. (13). The relation between reservoirs and output layer are weighted by using $w_{\text {out }}$ of $N_o *\left(N_r * N_l\right)$. By using this connection state-output method, the playback component can attribute various weights to the dynamics supplied by each layer [57]. The output function is applied for collecting the complete state I of $\left(J-J_0+1\right) \times\left(N_r * N_l\right)$ and the target vector T of $\left(J-J_0+1\right) * N_o$, with the $J_0$ is the washout time step, based on condition of $J_0 \leq J$, after this compute the weights of the output based on this Eq. (12). Figure 5, represent the different steps of deepESN methods functionality.

$X(j)=\left[X_{(j)}^{(1)}, X_{(j)}^{(2)}, \ldots, X_{(j)}^{\left(N_l\right)}\right]$          (11)

$w_{\text {out }}=\left(I^{-1} * T\right)^t$          (12)

$Y(j)=w_{\text {out }} * X(j)$          (13)

5.2 Advantages of DeepESN

DeepESN have three advantages in comparison with simple ESN, among them:

• DeepESN has more wealthy layer states, which gives him specific characteristics of a dynamic system elaborated in different reservoirs. This feature approved in the result of an experience in [58]. Some other works in the field of dynamical system theory illustrate that the dynamics of reservoirs state change in relation with their contractility [59].
• DeepESN has a very powerful short-term memory capacity (STMC). Considering that the STMC is applied for knowing the ability to recall the last input based on present states [58]. According to some work as [60], the DeepESN is better than simple ESN in STMC, in case of the same number of recurrent neurons.
• DeepESN has a great calculating ability, due to the removed of connections from the input layers to the reservoir layers higher than one and the connection between the grater and lower reservoir layers. That allows to minimize the number of non-zero recurring links [61, 62]. In addition, In the case of the overall number of recurrent neurons is $N_r * N_l$, the updating of each state for DeepESN are $L\left(N_l * N_r^2\right)$, but for ESN are $L\left(N_l^2 * N_r^2\right)$  [63]. As a result, clearly, the loss of DeepESN is less in comparison to ESN.

This work aims to apply the advantage of DeepESN to deal with energy consumption prediction problems.

In term of optimization algorithms comparison, it can be seen that the values of hyperparameters are selected optimally to find the minimum fitness function by using BGA. It outperforms some comparative methods (GA and DE) based on their accuracies as it can be illustrated in Figures 11, 12, 13, 14 and Table 5. DeepESN-PSO has achieve good results especially in long term forecasting. However, PSO takes a long time in processing to select the best hyperparameters values.

Figure 15 and 16 shows the time processing of each prediction model. As can be seen in Figure 15, the fastest model in terms of computation time is the ESN, while the slowest is the GRU. The proposed model, even when coupled with a BGA, is still better than GRU, LSTM and RNN. In Figure 16, hybridization with BGA outperforms that with other optimization models. Due to his binary characteristic BGA that lead to achieve best solution rapidly compared with others. These results indicate that the proposed model statistically has very interesting results, which make it a powerful method in the field of short-term power prediction.

7.png

Figure 7. The comparison between prediction methods and true values for 1 min time horizon

8.png

Figure 8. The comparison between prediction methods and true values for 5 min time horizon

9.png

Figure 9. The comparison between prediction methods and true values for 20 min time horizon

10.png

Figure 10. The comparison between prediction methods and true values for 30 min time horizon

11.png

Figure 11. The comparison between optimized Deep-ESNs and true values for 1 min time horizon

12.png

Figure 12. The comparison between optimized Deep-ESNs and true values for 5 min time horizon