A Hybrid Algorithm based on Invasive Weed Optimization and Particle Swarm Optimization for Global Optimization

In this paper, an effective combination of two Metaheuristic algorithms, namely Invasive Weed Optimization and the Particle Swarm Optimization, has been proposed. This hybridization called as HIWOPSO, consists of two main phases of Invasive Weed Optimization (IWO) and Particle Swarm Optimization (PSO). Invasive weed optimization is the natureinspired algorithm which is inspired by colonial behavior of weeds. Particle Swarm Optimization is a swarm base Algorithm that uses the swarm intelligence to guide the solution to the goal. IWO algorithm is the algorithm which is not benefit from swarm intelligence and PSO converges to the local optimums quickly. In order to benefit from swarm intelligence and avoidance from trapping in local solutions, new hybrid algorithm IWO and PSO has been proposed. To obtain the required results, the experiment on a set of benchmark functions was performed and compared with other algorithms. The findings based on the nonparametric tests and statistical analysis showed that HIWOPSO is a more preferable and effective method in solving the highdimensional functions. Keywords—Invasive weed optimization; Particle Swarm Optimization; Global optimization; Hybrid algorithm


INTRODUCTION
One of the affective methods in finding the best solution in numerical problems is the Optimization technique.In optimization, only a few solutions are considered the best which are called as the goal.Classical optimization techniques have some deficiencies on solving the complex optimization problems.These deficiencies are primarily interdependent on their inherent search systems.These classical optimization methods are strongly under effects of choosing proper objectives, constraints functions and type of variables.They also do not grant a universal result approach that can be used to solve problems where various type of variables, objective and constraint functions, are used [1].For covering these deficiencies ,a new method with the name of Metaheuristic was designed, which is mainly originated from artificial intelligence research that developed by researchers [2].A Metaheuristic is an algorithm designed for solving the various types of hard optimization problems without having to fully accommodate to each problem.The Greek word meta indicates that these methods are higher-level heuristics.The primary features of Metaheurisitc methods are as follows: they are nature-inspired (meaning that they have originated from nature physics, behavior and etc); stochastic components are one of the inseparable parts of these methods (involving random variables); they aren't gradient base method and don't use them; at the beginning of program, they have several parameters which needs to adjusts properly.Metaheuristic algorithms combine various intelligent procedures and guide basic heuristic methods [3].These algorithms are inspired from different things such as natural phenomena, natural selections and social behaviors and applied in solving the optimization problems.Examples of the recently metaheurtistc algorithms are Vortex search [4], WOA (whale optimization algorithm) [5], MBA (mine blast algorithm) [6], WCA(water cycle algorithm) [7], and SFS (stochastic fractal search) [8].
The PSO [9] is the other Metaheuristic algorithm which has been utilized in the optimization of many problems.This algorithm uses the strategy of birds and folks in migration for finding better solutions.Individuals in the PSO are called as particles and each particle has velocity in the searching space.Particles are distributed randomly in the searching space and positions of the particles are changed based on the velocity which has been calculated.These particles tend to move toward the best positions which causes to seek a better position and find the best.One of the deficiencies that can be specified for PSO is that it often falls to the local minimum quickly, missing better opportunities when facing multimodal functions [10].
IWO is a nature inspired algorithm which is getting much more attention because it shows efficient exploration and dissimilarity properties [11] and took an exceptional place for solving continuous optimization problems.In formal IWO, the seeds are uniformly spread on the search space [11].After all, if the searching space is too large, its not efficient to use this type of distribution for the searching.Also, initialization of the parameters is so important task in IWO and trapping in local solution is a probable event.Hence ,it was found that the efficiency of an IWO algorithm to achieve success goal in problems relies too much on its initial parameters and these parameters should be wisely selected based on the problem to be solved.To overcome these problems, a novel hybrid IWO and PSO algorithm is proposed and implemented for solving continuous optimization problems.Hybridization of IWO with other algorithms has been investigated in many studies.MICA-IWO [12]is a new type of hybrid method which www.ijacsa.thesai.orgcombines two well-known Metaheuristic approaches: IWO and imperialist competitive algorithm (ICA) [13].ICA has outperformed many of the already existing stochastic and direct search global optimization techniques.The hybrid MICA-IWO method has been used for handling optimal reactive power dispatch (ORPD) problem.Other recently hybrid method based on IWO is the hybrid IWO/WDO [14] algorithm.In IWO/WDO, wind driven optimization (WDO) [15] algorithm has been combined with IWO algorithm for nulling pattern synthesis of uniformly spaced linear and nonuniform circular array antenna.
In this paper, we will combine PSO and IWO global optimization algorithms, and propose the novel hybrid algorithm based on these algorithms which are jointly called as HIWOPSO.As PSO has Swarm intelligence, this could provide more variant population for IWO which could help in finding better places by using the previous experiences.To avoidance of trapping in local solutions and searching the large area greatly, the mutation function has been used which applied for checking other places and give chance for them.Proposed method is tested on the benchmark functions and compared with other famous algorithms.The statistical analysis Friedman test [16] is performed on the results and compared with other algorithms results.The Convergence diagram and normality diagrams for the statistical analysis is also presented.
In the real world, many problems have been proposed and optimization problems are one of them [17].The optimization problems are single or multi-objective.The multi-objective is the problem with more than one objective function (m>c1) and single objective is a problem with one objective function (m=1).The main goal in this procedure is to seek the global minimum or maximum.The function may have more than one minimum or maximum which is called as the local, but only one of them is the global maximum or minimum.The point x* is the global minimum if f(x*)_f(x) for all the x in the searching space S. Optimization problem may consist of one or more mathematical functions which need to be optimized.The general form of the optimization problem is indicated in Eq. (1).MinimizeF (f1(x);…; fm(x)), x = (x1,….,xn)€ S. ( Where n is the decision variables, m is the number of objectives, x is decision vector and S is searching space.If the problem has one objective function (m=1), then it should be indicated as Eq.(2).minimizef(x), x = (x,…,xn) € S. ( The rest of the paper is organized as follows: Section 2 illustrates the PSO and IWO algorithms, and section 3 discusses the HIWOPSO algorithm, its parameters and boundary control.Section 4 presents 26 benchmark test functions applied for the experiments.Finally, the last section presents the concluding remarks.

II. RELATED WORKS
The PSO is the one of the Metaheuristic algorithms which is originated from the nature.This algorithm was introduced by Kenndy and Eberahart in 1995 [9].The PSO is originated from the birds and folks migration behavior, living in small and large numbers of groups.The birds use a method for finding food and migration, which has been used in this algorithm.In this method, only the birds know their distance from food, but they don't know the location of the food thus, following the other neighboring birds is the best way for surviving.
The PSO consist of elements with the name of particles which is a probable solution in the searching space.The main steps in the PSO algorithm are as follow: first, particles are distributed randomly in the searching area and PSO starts the process with these particles.In this searching process, particles only follow the one which is nearer to the goal and has better fitness value.Each particle has a velocity which is represented by Vi and calculated by Eq. ( 3) in the Ddimensional searching space.Particles are under the effect of personal (   ) and swarm experiences (   ) and the position is updated by Eq. ( 4).
In Eq. ( 3) and ( 4), Xi represents the ith particle of the population, c1 and c2 are the learning coefficients, r1 and r2 are random values between [0 1], ! is the inertia weight, and Vi is the ith member of particles velocity.   and    are the personal best and generation best.
Invasive weed optimization was first introduced by Mehrabian and Locus [11] and it is One of the population based optimization algorithms which is originated from colonial behavior of weeds.The IWO algorithm is a very simple and yet efficient algorithm in finding optimum solution of the objective function, which is implemented based on the natural and basic features of weeds in a colony such as reproduction, growth and competition to survive.In comparison to other algorithms, IWO is simpler and has adequate ability and convergence rate to the global optimum solution of the objective function.Some of the major features of this algorithm which specifies it from other methods are reproduction, space distribution and exclusive competition.
In order to simulate behavior of a weed, we have the following algorithm, according to [11].
Step 1: Initial population production: a population of N0 seeds is randomly distributed in an n dimensional space.
Step 2: Reproduction: each seed grows and turns into a mature plant and then, begins seed production for newer generation.The amount of seeds produced by a plant increases linearly between two possible values of minimum (Smin) and maximum (Smax) possible amounts of produced seeds.The amount of produced seeds for the ith plant in every repeat is dependent to its goal value (Fi), its best (Fbest) and worst (Fworst) goal values in that repeat and is calculated with the following equation: Step 3: Distribution space the randomness and assimilation of the algorithm are related to this stage.The produced seeds are distributed in the d dimensional search space with normal distribution which has zero mean and different variance of (N(0;_t)d).In this state, the seeds will be near the breeder plant.Although, the standard deviation decreases from initial amount (_initial) to final amount (_final) in each repeat, in the simulations, non-linear variation of standard deviation causes satisfactory results which are illustrated below: In Eqs.(13)(15), T represents the maximum number of repeats related to (t) and is the non-linear modulation factor.In this status, the positions of seeds (Sj) for the ith plant (wi) are calculated as follows: =   + N(0, δ  )  , 1 <= j≤ numseed(i) (7) Step 4: Exclusive competition: by several repeats, the number of plants produced by rapid reproduction reaches its maximum value (Wmax), in this situation, every plant is permitted to produce seeds by in accordance with reproduction method.The seeds are authorized to spread in search space with correspondence to distribution space method, when the seeds find their position; they form a colony alongside their parent plants.Then members with less propriety are deleted in order to number of members reach its maximum allowed value.In this method, the parent plants combine with their children and the plants with most propriety from the group are preserved and allowed for replacement.This crowd control mechanism will be imposed on next generations until reaching the final period.Step 5: If the criterion satisfied end otherwise, return to Step 2.

III. PROPOSED MODEL
The swarm intelligence is some kind of artificial intelligence which has been established based on group behaviors in decentralized and self-organizing systems.This systems usually included population of the simple agents that interact locally with each other and their environment.Some samples of this system that can be mentioned are ants groups, birds flock, fishes flock, bacterize bulk and animals herd.In order to use of swarm intelligence, it needs to use the behavior of these systems in the proposed method.So PSO algorithm has been using with IWO algorithm to give the behavior of swarm intelligence to agents of IWO algorithm and use it in guiding the solution of the problems to the goal.As previously mentioned, in IWO algorithm, each father (weed) produce some child (seed) that these Childs distribute around the father with a kind of normal distribution.In proposed hybrid method, this distribution could be based on the some kind of normal distribution or guided through the goal by using the swarm intelligence and previously experience of father.Using the previously experiences is like this, if each weed (wi) could give good result, it stores the result as the experience in the experience memory and these experiences inherited to this weed Childs and used in guiding the solutions of the problem like the PSO algorithm.Each child in proposed method (HIWOPSO) inherited two things from their father and that is the velocity (Vi) and experience (wp).The main process of proposed method is like this: first, the population with size npop0 of weeds (wi) are distributed randomly around the searching space and their position is initialized randomly.The cost of each weed is computed from the position which calculated previously and at the Start, the value of velocity is zero (Vi = 0).first, The experience of each weed is the value which previously initialized to the position and cost for each weed.After producing population with size of npop0, algorithm enters to the main iteration loop which at start have a section for computing the number of producing child for each weed by Eq. ( 5) and also the velocity for each weed is computed like the following equation: Where wi is ith weed in population and wp is personal experience and wbest is the global best , other variables are described in PSO algorithm and This equation is same as the PSO velocity update equation.For each of the wi enters to a loop and one solution is created by the seed distribution formula in IWO (newsol) Eq. ( 9) and another solution is created by PSO (newsol2) Eq (10).
Newsol =   +   (10) New solutions newsol and newsol2 are compared with each other and if newsol2 better than newsol then newsol value will be replaced by the newsol2 value.The newsol is compared with wi experience and if newsol better than wp it will set as wi experience and it also if it is better than global best, the global best will be updated with newsol.After using the PSO and IWO for creating new solutions, its time to reduce the probability of trapping in local solutions.
For decreasing this probable event , using mutation function could be help full after producing new solution.The mutation function applied to the new solution (newsol) and the output (mutatedsol) will be compared with newsol.The mutatedsol is compared with newsol and if the mutation creates better solution then the value of newsol will be replaced by the mutatedsol value and newsol inserted to the population.The mutation function described briefly in next subsection.After inserting the newsol to the population, it must be control the size of population not to exceed from maximum size which was initialized at the start (npop).
The mutation function which has been using in proposed algorithm, uses random selection of dimension and replaces that dimension values with the random values that has been obtained from function domain.For example if the input for the mutation function is the following input=[0.70.5 0.6 0.8] with boundary [L U]=[0.4 0.9] and if the mutation function selects random dimensions 2 and 4 then that dimensions values will be changed with random real value from domain [0.4 0.9] and output=[0.7 0.45 0.6 0.65].Random selection of dimension could be any order of input dimension size for example, if d=30 then random dimension could be f{1 5 18 20} , f1{ 2 3 4 5 6 7 19} and etc...The number of dimension that selected for replacement is determine randomly in each execution of mutation function.
The pseudo code for proposed method described as follows: HIWOPSO algorithm is consists of 2 main parts and these parts are executed in consecutive form in each cycle which are showed at the flow chart Fig. 1.In each part process is performed on weeds of population with size of pop-size and weed position is d dimensional vectors.The main operator for each part effects on time complexity.The sections that alter time complexity are: PSO update equation, IWO position update equation.We can explain complexity analysis of HIWOPSO in worse case and prove the fastest execution of these two algorithms combination like this: As the HIWOPSO algorithms are executed in consecutive form, then the time complexity for IWO is O(nd) and for PSO is also O(nd), therefore the HIWOPSO algorithm is run in O(nd) complexity because of the consecutive form of execution in IWO and PSO algorithms.For analyzing convergence of the Metaheuristic algorithm, Markov chain Monte Carlo method is the one of the preferable methods for this task [18].
Most Metaheuristic algorithms can be sighted in the framework of Markov chain from statically viewpoint.Now if look at the proposed hybrid method closely using the framework of Markov chain Monte Carlo, each algorithm in HIWOPSO essentially forms a Markov chain and the appropriate better solutions which created in each iteration, replace with previous one.Convergence analyzing based on the Markov chain for algorithm HIWOPSO is performed as follows: Definition 1. Assume that the best weed is shown by  * := {  * ∈ : ( * ) = min (()⃓  ∈  )} where X is probable solution and f is fitness function.The number of best weeds in weeds population is shown by (): ⃓ N ∩  * ⃓ .Definition 2. Algorithm convergence with probability 1 to the best if this condition is true : , where g indicates generation number and N0 is random initial population.
Theorem 1. HIWOPSO algorithm converges to its globally best solution with probability 1.
Parameter adjustment is a non-negligible task which is required to be performed properly in order to get a better result in solving various problems.Besides, the parameter adjustment also is necessary for controlling the boundary whenever the algorithm finds a new solution [19].The (HIWOPSO) needs boundary control for a weed wi, because its position is required to be in the searching space, which is a boundary between [L U], (L is the lower bound and U is the upper bound of the searching space).The method that controls the boundary is as follows: p = Max(X,L); q = Min(p, U) (11) Where Min and Max are the functions that select the minimum and maximum among the input pairs, X is the input weed position and q is the output which has been controlled in the boundary range [L U].Since proposed algorithm is combination of two algorithms IWO and PSO then it is consisted of these two algorithms parameters.HIWOPSO parameters with initial settings are mentioned in Table 1.

IV. DISCUSSION AND EVALUATION
In this paper, the proposed algorithm has been evaluated with a set of benchmark functions which are a subgroup of unimodal and multimodal functions.These functions have various dimensions such as 2, 4, 10 and 30.Tables 3 highlights a multimodal test functions and Table 2 shows unimdal test functions.UNIMODAL TEST FUNCTIONS (D: DIMENSIONS) Multimodal test functions have many local minimums, and therefore, they are hard to be solved simply because of the trapping in local solutions.The (HIWOPSO) has been evaluated by these functions and search the global minimum for them.In Table 4, the results for algorithms GA [20], DE [21], PSO, BA [22] and IWO are compared with HIWOPSO.Conditions for experiment and parameter settings for these algorithms is explained in reference [23].The experiment has been performed on the computer with following features: CPU 2.1 GHZ, Ram 8 GB and Matlab 2016 running on computer with windows 7. The NFE= 500,000 (number of function evaluation) was set as Stopping criteria and the values minimum than 1E-12, presented as 0 same as other methods.The mean value and Std Dev (standard deviation) have been calculated from 30 independent runs.In Table 4, the HIWOPSO found the minimum results for most of the functions with best standard deviations.The results of Friedman non-parameteric test [16] are also presented in this table and HIWOPSO could rank the best with value 2.75.The low p-value indicates that the results are remarkably different with each other and figure 2 depicts the results for this test with a bar diagram.For analyzing the procedure of convergence in (HIWOPSO), figure 3   of Kolmogorov-Smirnova and Shapiro-Wilk [16] has been performed for the four functions of F6, F7, F12 and F20 which are hard to be solved, and here, standard deviation is not zero for them.Table 5 presents the results for this test and the pvalue, df (degree of freedom), and the statistics for this test are also presented.Accordingly, the p-value which is higher than the significant _ =0.05 is considered a normal distribution and the lower than that value is supposed to be an abnormal distribution.Based on the p-value and the test of normality of Kolmogorov-Smirnova, the function of F7 result is normal and other functions have the abnormal distribution.Figure 4 shows the normal and abnormal distributions, the histogram, QQ-plot and Box-plot for the two functions, F7 and F12.In this figure, F7 is a normal distribution and F12 is considered abnormal.As can be inferred, in normal distributions, the V.
CONCLUSION AND FUTURE WORKS The IWO is, arguably, one of the most efficient nature inspired Metaheuristic algorithms, which has outperformed most of the algorithms in solving the various optimizing numerical problems.Furthermore, one of the practical Metaheuristic algorithms which have been used most widely in the optimization is the PSO.The PSO algorithm is a simple and practical algorithm which is more amenable to combination with others.In the current study, the PSO and IWO algorithms are combined in order to design a new hybrid method for seeking the global solution.The proposed hybrid algorithm begins to search by creating solutions by IWO and PSO algorithms and used the better one as the input to the mutation function.Mutation output is compared with the input and better one selected as the new seed.The experiment based on benchmark functions and non-parametric ranking showed that the proposed hybrid is more dominant and competent than other famous algorithms.Moreover, based on the results of the test of normality and convergence, this proposed hybrid algorithm mostly had abnormal distributions for results and converged to the optimum solutions in minimum CPU Time.

Fig. 2 .
Fig. 2. Bar diagram for non-parametric Friedman test results for Functions F1-F26 minimum CPU Time and faster against the other algorithms IWO and PSO.Also, the test of normality

TABLE IV .
HIWOPSO COMPARISON WITH GA, DE, PSO, BA, AND IWO (UNIMODAL FUNCTION SET), BOLD VALUES REPRESENT