New Meta-Heuristic Computer-Oriented Algorithms to Solve Unconstrained Optimization Problems

The difficulty is in tailoring optimization algorithms to the real-world applications that they find in several scientific and technological domains [1]. As a result, certain algorithms are only useful for solving complicated issues, which supports the idea that it's important to constantly adopt new techniques for improvement [1]. Meta-heuristic algorithms (MAs) are a class of optimization algorithms that are used to handle difficult issues that are beyond the scope of conventional techniques. These algorithms are used to search a wide search area in order to find the global optimum of a problem. They are inspired by natural processes such as evolution, swarm behavior, and genetics [2, 3]. Darwin's theory of evolution served as the basis for the development of the genetic algorithm (GA) recently [4]. GA and the differential evolution (DE) algorithm both employ the same operators, namely crossover and mutation. But DE uses a different approach [5]. The collective foraging activities of fish and bird species serve as an inspiration for particle swarm optimization (PSO) [6]. The ABC algorithm is a computer method that mimics honeybees' information-sharing capacities and foraging strategies [7]. Utilizing the laws of motion and gravity, the gravity search algorithm (GSA) is a computer optimization method [8]. Numerous sophisticated algorithms, as demonstrated in references [9-14], base their depiction on the actions of real-world occurrences and biological entities.

CGAs are classified as effective methods for reaching minimization points for optimization problems with the least possible iterations, but the solutions are considered local. Therefore, the process of linking these algorithms with MAs gives them greater efficiency in finding minimizations of the same problems globally as in references [15-20].

The difference between conjugate gradient methods and mayfly optimization algorithm lies in how they approach and optimize solutions to mathematical optimization problems, especially in terms of the mathematical mechanism and theoretical basis behind each. Let's review the main differences:

Conjugate gradient methods are analytic algorithms that rely on gradients to optimize the objective function, and are commonly used in unconstrained optimization problems, especially when the objective function is a quadratic or linear function with derivative functions. They work by calculating gradient directions at each step, but instead of following the gradient directly, they use a technique that makes the direction at each step orthogonal (conjugate) to the previous directions, which makes them avoid excessive recompilation. As common applications of these algorithms, they are usually used in convex function optimization problems, such as least squares problems in machine learning, as well as in large vector optimization in machine learning. Their features are very effective in large problems where Hessian matrices are computationally expensive. They almost guarantee a local solution in quadratic function problems. The mayfly optimization algorithm is an evolutionary algorithm inspired by the natural behavior of mayflies in nature. It is a type of swarm intelligence algorithm, which uses the concept of attraction between individuals to update locations and minimize the value of the objective function. The algorithm's working mechanism simulates the behavior of mayflies in terms of mating and movement, where groups of females and males form a "swarm", and each individual move towards other individuals according to rules based on their fitness, trying to minimize or optimize the value of the objective function. Applications of these algorithms are used in optimizing complex nonlinear functions, and in problems that require a broader exploration of the solution space, such as engineering system design, and multidimensional optimizations. Its features are suitable for highly complex and discontinuous optimization problems, and it can exit local solutions due to the mating and exploration mechanism.

The paper's structure is as follows: in Section 2, we shall provide a different formulation of the CGAs. In Section 3, under specific assumptions, we examine the theoretical components and determine the global convergence properties of the Fletcher and Reeves conjugate gradient algorithm (FR)-CGA. A detailed description of the original mayfly algorithm is provided in Section 4. To improve both algorithms' performance, we combined the FR-CGA and mayfly algorithms in Section 5. The sixth section of the research concentrates on the digital side. Seven unconstrained functions are treated with a unique combination algorithm mayfly optimization algorithm and conjugate gradient algorithm (MOA-CG), and the results are compared with the mayfly method.

When comparing the theoretical convergence properties of a particular conjugate gradient method, it depends on the properties (convergence rate, ensuring convergence towards the optimal solution (Global vs Local Convergence), reliability and stability, computational efficiency, scalability). These properties will be available if the functions are less complex and also if certain conditions are available to achieve them, such as the global convergence of the algorithm. In this section, we will show the advantages of the proposed algorithm from the theoretical side, relying on some necessary and important conditions such as the basic assumption.

Assumption 3.1

The existence of the value $x_1$, which is defined as the restricted level set $S=\left\{x: f(x) \leq f\left(x_1\right)\right\}$ indicates the existence of a number (B>0) s. t.

$\|x\| \leq B, \forall x \in S$           (8)

If $g$ is Lipschitz continuous and $f$ is continuously differentiable in certain neighborhoods $N$ of $S$, then there is a constant $\mathrm{L} \geq 0$ s. t.

$\| g(x)-g\left(x_k\right)| | \leq \mathrm{L}| | \mathrm{x}-\mathrm{x}_k| |, \forall \mathrm{x}, \mathrm{x}_k \in N$        (9)

Theorem 3.2

If we assume that two new algorithms, Eqs. (5) and (6), produce $x_{k+1}$ and $d_{k+1}$, and we use PWP (3) to yield $\alpha_k$, then the direction holds such that s. t.

$d_{k+1}^T g_{k+1} \leq-c\left\|g_{k+1}\right\|^2, \forall k \geq 1$           (10)

Proof: Multiply both sides of Eq. (5) from the right side by $g_{K+1}$, we have:

$\begin{gathered}d_{k+1}^T g_{k+1}=-\left\|g_{k+1}\right\|^2+\vartheta_k \beta_k^{F R} s_k^T g_{k+1} \\ d_{k+1}^T g_{k+1}=-\left\|g_{k+1}\right\|^2+\left(1-\mu_k\right) \frac{d_k^T y_k}{\left\|d_k\right\|^2\left\|y_k\right\|^2} \beta_k^{F R} s_k^T g_{k+1}\end{gathered}$

where, $s_k^T g_{k+1}=s_k^T y_k+s_k^T g_k>s_k^T y_k$, then

$\begin{gathered}d_{k+1}^T g_{k+1} \leq-\left\|g_{k+1}\right\|^2+\left(1-\mu_k\right) \frac{d_k^T y_k}{\left\|d_k\right\|^2\left\|y_k\right\|^2} \beta_k^{F R} s_k^T y_k \\ d_{k+1}^T g_{k+1} \leq-\left\|g_{k+1}\right\|^2+\left(1-\mu_k\right) \frac{\alpha_k\left(d_k^T y_k\right)^2}{\left\|d_k\right\|^2\left\|y_k\right\|^2} \frac{\left\|g_{k+1}\right\|^2}{\left\|g_k\right\|^2} \\ d_{k+1}^T g_{k+1} \leq-\left\|g_{k+1}\right\|^2+\left(1-\mu_k\right) \alpha_k \frac{\left\|g_{k+1}\right\|^2}{\left\|g_k\right\|^2}\end{gathered}$

Upon simplification, a satisfactory descent is obtained for this algorithm in the following manner:

$d_{k+1}^T g_{k+1} \leq-c\left\|g_{k+1}\right\|^2$

where, $c=\left[1+\left(1-\mu_k\right) \frac{\alpha_k}{\left\|g_k\right\|^2}\right]>0$.

Theorem 3.3

Assuming that Assumption (3.1) A is satisfied, let us consider any CGA (2)-(4), where $d_{k+1}$ satisfies condition (10) and $\alpha_k$ is determined using Eq. (3).

$\sum_{k \geq 1} \frac{1}{\left\|d_k\right\|^2}<+\infty$        (11)

Then, we have

$\lim _{k \rightarrow \infty} \inf \left\|g_k\right\|=0$         (12)

In this discourse, we shall proceed to expound upon the theory of global convergence by drawing upon the conditions posited in preceding proof theories.

Theorem 3.4

Assuming that $\left|\mu_k\right| \leq 1$, there exists a positive constant $(i=1,2)$ for each $k$ that is less than or equal to zero. These constants are designated as $\gamma_1$ and $\gamma_2$, respectively, with $\gamma_1 \leq\left\|g_k\right\| \leq \gamma_2$. Then, the PWP search determines the updated CGA and $\alpha_k$, where $g_k$ is equal to zero for a given k or $\lim _{k \rightarrow \infty} \operatorname{in} f\left\|g_k\right\|=0$.

Proof: Due to the fulfilment of the descent state, we have $\left\|d_k\right\| \neq 0$. This observation is made in conjunction with the Lipchitz condition.

$\begin{gathered}\left\|y_k\right\|=\left\|g_{k+1}-g_k\right\| \leq L\left\|s_k\right\| \\ \left\|d_{k+1}=-g_{k+1}+\vartheta_k \beta_k^{F R} s_k\right\| \\ \left\|d_{k+1}\right\| \leq\left\|g_{k+1}\right\|+\left|\vartheta_k\left\|\beta_k^{F R} \mid\right\| s_k \|\right.\end{gathered}$

where,

$\begin{gathered}\left|\vartheta_k\right|=\left|\frac{d_k^T y_k}{\left\|d_k\right\|^2\left\|y_k\right\|^2}-\mu_k \frac{d_k^T y_k}{\left\|d_k\right\|^2\left\|y_k\right\|^2}\right| =\frac{2 \alpha_k}{\left\|s_k\right\|\left\|y_k\right\|} \leq 2 \alpha_k L\end{gathered}$

Given the assumption that D known and $\left|\beta_k^{F R}\right| \leq \frac{\gamma_2^2}{\gamma_1{ }^2} \equiv E$, the updated search direction can be expressed:

$\left\|d_{k+1}\right\| \leq \gamma_2+2 \alpha_k L E$

This implies $0<\sum_{k=1}^{\infty} \frac{\left(g_k^T d_k\right)^2}{\left\|d_k\right\|^2}<\infty$ when $\sum_{k=1}^{\infty} \frac{\left\|g_k\right\|^4}{\left\|d_k\right\|^2}<$ $\frac{1}{c^2} \frac{\left(g_k^T d_k\right)^2}{\left\|d_k\right\|^2}<\infty$

Within this segment of the paper, we present a novel algorithm (MOA-CG) that relies on the conjugate gradient algorithm put forth by Eqs. (5) and (6). We have demonstrated that this algorithm has adequate descent and convergence to the minimum point with the algorithm (MOA) in order to identify optimal solutions for the optimization functions. Combining the two algorithms gives us their advantages together, i.e., finding analytical solutions using algorithmic methods inspired by nature that are close to ideal within complex solution spaces.

The suggested algorithm is available for viewing in the paragraph that follows:

5.1 MOA-CG algorithm

Step 1: Initialized mayfly algorithm parameter (nPop= 20, Max-Iter=2000, g=0.8, a1=1.0, a2=1.5, a3=1.5, β=2, d= 5, f1=1).

Give initial CG parameter: variable $x_0 \in R^n, \delta \in$ $[0,0.5]$ and $\sigma \in[\delta, 1]$. Let $\mathrm{k}=0, d_0=-g_0$.

Step 2: The best value of step to algorithm results: set $\alpha_k$ from (PWP) in formula (3) and compute the $\vartheta_k$ from Eq. (6).

Step 3: Evaluate the parameters used by the new search direction (4)-(5).

Step 4: Find the new point happened $x_{k+1}$ as formulas (2)-(4). 

Step 5: Compute population based on f(x) and find the global best (gbest).

Step 6: If the maximum iteration has not been achieved, compute the following:

• The Eqs. (17)-(18) can be utilized to determine the speed and position of individual female mayflies.

• The Eqs. (13)-(15) can be utilized to determine the speed and position of individual male mayflies.

• Arrange the mayflies into categories and assign them a ranking based on the function f(x).

• Produce male and female progeny, denoted as Eqs. (19)-(20) respectively.

• The process of offspring development involves the random division of offspring into male and female individuals.

• Replace the least desirable elderly persons with the most exemplary fresh ones.

• The pbest and gbest values are updated.

Step 7: If the sum of duplicate values has reached its maximum, terminate the process and display the optimal value without any adjustments in the variable k. Proceed to Step (2).

Based on the data presented in the preceding section, we draw the conclusion that the new algorithm, MOA-CG, performs better than the traditional algorithms, MOA, when it comes to finding minimization points for known testing functions (i.e., in terms of the number of iterations and the time it takes to reach the optimal point). Furthermore, the traditional proposed CGA converged more strongly than when the FR parameter was used; however, when algorithms with comprehensive behavior were introduced to reach the minimization of functions, its efficiency increased in convergence and decreased the number of iterations. Traditional methods (such as conjugate gradient) provide local convergence, high computational speed and efficiency on specific problems such as quadratic or linear functions. Modern algorithms inspired by nature provide high flexibility in exploring global solutions and scalability, but they require more computations and may be less stable in terms of theoretical convergence. Therefore, the choice of algorithm often depends on the nature of the specific problem, whether it is convex, linear, or non-convex and complex. It is feasible to combine new conjugate gradient techniques with the mayfly method in future study. The conjugate gradient algorithm suggested in the paper can also be used with other clever algorithms.