Adaptive Group Organization Cooperative Evolutionary Algorithm for Tsk-type Neural Fuzzy Networks Design

—This paper proposes a novel adaptive group organization cooperative evolutionary algorithm (AGOCEA) for TSK-type neural fuzzy networks design. The proposed AGOCEA uses group-based cooperative evolutionary algorithm and self-organizing technique to automatically design neural fuzzy networks. The group-based evolutionary divided populations to several groups and each group can evolve itself. In the proposed self-organizing technique, it can automatically determine the parameters of the neural fuzzy networks, and therefore some critical parameters have no need to be assigned in advance. The simulation results are shown the better performance of the proposed algorithm than the other learning algorithms.


I. INTRODUCTION
In the field of artificial intelligence, neural fuzzy network [1][2][3] refers to combinations of neural networks and fuzzy logic.Because the advantages of neural fuzzy networks are powerful computation ability and human-like reasoning ability from neural networks and fuzzy systems, respectively, it has good performance for solving complex nonlinear problems.The neural fuzzy networks can perform the nonlinear mapping once the system parameters are trained based on a sequence of input and desired response pairs, and it does not require mathematical descriptions of system.Therefore, the determination of parameters is a critical issue for neural fuzzy networks.
The backpropagation (BP) [4,5] is a common method and widely used for training neural fuzzy networks.It is well known that BP is an approximate steepest descent algorithm.The steepest descent algorithm is the simplest, and the minimization method.The advantage of steepest decent algorithm is very simple, requiring calculation only of the gradient, the disadvantage of steepest descent algorithm is that training time is generally longer than other algorithms; based on initial weight values, it is often to find the local optimal solution but not global optimal solution.Besides, BP training performance depends on the initial values of the system parameters, and for different network topologies one has to derive new mathematical expressions for each network layer.
Considering the aforementioned disadvantages one may be faced with suboptimal performance even for a suitable neural fuzzy network topology.Hence, the capability of training parameters and finding the global solution while optimizing the overall structure are important.The evolutionary methods using for training the fuzzy model has become a popular research field [6][7][8][9][10][11][12][13][14][15][16][17][18][19][20] because evolutionary methods simultaneously evaluate many points in the search space and are more likely to converge toward the global solution.The evolutionary fuzzy model is a learning process to generate a fuzzy model automatically by incorporating evolutionary learning procedures.
Recently, several improved evolutionary algorithms have been proposed [16][17][18][19][20][21][22].In [16], Bandyopadhyay et al. used the variable-length genetic algorithm (VGA) that allows the differentia of lengths of chromosomes in a population.Carse et.al. [17] used the genetic algorithm to evolve fuzzy rule based controllers.In [18], Chen proposed an efficient immune symbiotic evolution learning algorithm for compensatory neuro-fuzzy controller.In [19], Lin presented a novel selfconstructing evolutionary algorithm to design a TSK-type fuzzy model.In [20], the group-based symbiotic evolution (GSE) was proposed to solve the issue of the traditional genetic algorithm that all the fuzzy rules were encoded into one chromosome.In [21], Lin proposed a hybrid evolutionary learning algorithm to combine the compact genetic algorithm and the modified variable-length genetic algorithm to perform structure/parameter learning to construct a network dynamically.Hsu [22] proposed a multi-groups cooperationbased symbiotic evolution (MGCSE) to train a TSK-type neural fuzzy network (TNFN).Their results showed that MGCSE can obtain better performance and convergence than symbiotic evolution.Although MGCSE being a good approach for training a TNFN, there is no systematic way to determine suitable groups for selecting chromosomes.
Although the above evolution learning algorithms [16][17][18][19][20][21][22] can improve the traditional genetic algorithms, these algorithms may encounter one or more of the following issues: 1) all fuzzy rules represent one chromosome; 2) the random group selection of fuzzy rules; 3) the numbers of fuzzy rules and group numbers have to be assigned in advance.
Recently, the data mining approach has been widespread used in several fields [23][24][25][26][27][28][29][30].The data mining can be regarded as a new way of data analysis.One goal of data mining is to find association rules among frequent item sets in transactions.In [23], the authors proposed a mining method of ascertain www.ijarai.thesai.orglarge item sets to find association rules in transactions.Han et al. [24] proposed the frequent pattern growth (FP-Growth) to mine frequent patterns without candidate generation.Wu et al. [30] proposed a data mining method based on the genetic algorithm that efficiently improve the Traditional genetic algorithm by using analysis support and confidence parameters.
In order to solve aforementioned problems, this paper proposes an adaptive group organization cooperative evolutionary algorithm (AGOCEA) for designing a TSK-type neural fuzzy network.The AGOCEA adopts the group symbiotic evolution (GSE).Each population in the GSE is divided to several groups and each group represents a set of the chromosome that belongs to one single fuzzy rule.To solve the problem of random group selection, a data mining based group selection method was used to select the better groups.The adaptive group organization was used to solve the some parameters have to be assigned in advance.This paper is organized as follows.In the Section II, a brief description of TSK-type neural fuzzy network is introduced.The proposed AGOCEA is described in the Section III.In the Section IV, the simulation results are presented.The conclusions are summarized in the Section V.

II. THE CONCEPT OF THE TSK-TYPE NEURAL FUZZY
NETWORKS A Takagi-Sugeno-Kang (TSK) type neural fuzzy network (TNFN) [1] employs different implication and aggregation methods from the standard Mamdani fuzzy model [3].Instead of using fuzzy sets the conclusion part of a rule, is a linear combination of the crisp inputs.The fuzzy rule of TSK-type neural fuzzy network is shown as following equation: ) and x 2 is A 2j (m 2j ,  2j ) and … and x n is A nj (m nj ,  nj ) THEN y=w 0j +w 1j x 1 +…+w nj x n . ( where n is the number of the input dimensions and j is the number of the fuzzy rules.
The structure of the TSK-type neural fuzzy network is shown in Fig. 1.It is a five-layer network structure.The functions of the nodes in each layer are described as follows: Layer 1 (Input Node): No function is performed in this layer.The node only transmits input values to layer 2. That is Layer 2 (Membership Function Node): Nodes in this layer correspond to one linguistic label of the input variables in layer 1; that is, the membership value specifying the degree to which an input value belongs to a fuzzy set is calculated in this layer.For an external input x i , the following Gaussian membership function is used: where m ij and σ ij are, respectively, the center and the width of the Gaussian membership function of the jth term of the ith input variable x i .Layer 3 (Rule Node): The output of each node in this layer is determined by the fuzzy AND operation.Here, the product operation is utilized to determine the firing strength of each rule.
The function of each rule is Layer 4 (Consequent Node): Nodes in this layer are called consequent nodes.
The input to a node in layer 4 is the output delivered from layer 3, and the other inputs are the input variables from layer 1 as depicted in Fig. 1.
where the summation is over all the inputs and where w ij are the corresponding parameters of the consequent part.
Layer 5 (Output Node): Each node in this layer corresponds to one output variable.The node integrates all the actions recommended by layers 3 and 4 and acts as a defuzzifier with ) where M k is the number of fuzzy rule.The learning process of the AGOCEA involves ten operators: initialization, two phase self organization, data mining based group selection, fitness assignment, reproduction, crossover, mutation, calculation of group similarity, splitting process, and lumping process.The whole learning process is described step-by-step as follows:

A. Initialization
Before the AGOCEA is designed, individuals forming several initial groups should be generated.The initial groups of the AGOCEA are generated randomly within a fixed range.

B. Two phase self organization (TPSO)
After every group is initialized, the AGOCEA adopts pervious research which was TPSO [29,31] to decide the suitable selection times of each number of rules (in this paper the number of rules lie between [M max , M min ]); that is, it determines the selection times of M k groups which form a TNFN with k rules.
After the TPSO, the selection times of the suitable number of rules in a TNFN will increase, and the selection times of the unsuitable number of rules will decrease.The details of the TPSO are listed as follows: Step 0.
Initialize the probability vectors: Step 1. Update the probability vectors according to the following equations: ThresholdFitnessvalue is a predefined threshold value.
Step 2. Determine the selection times according to the probability vectors as follows: where Selection_Times represents total selection times in each generation, Total_Velocy is summation of the probability vectors is the selection times of M k groups that use to select k chromosomes for constructing a TNFN.
After step 2, the selection times of every numbers of rules in a TNFN are obtained.Then the where TPSOTimes is a predefined value; Best_Fitness g represents the best fitness value of the best combination of chromosomes in gth generation; Best_Fitness represents the best fitness value of the best combination of chromosomes in current generations.Eqs. ( 19)- (21) represents that if the fitness is not changed for a sufficient number of generations, the suitable M k groups may fall in the local optima solution.

C. Data mining based group selection (DMGS)
After the TPSO step, the selection times of each rule number in a TNFN is decided.The AGOCEA then performs selection step.The selection step in the AGOCEA can be divided by selection of groups and chromosomes.In the selection of groups, this paper uses the DMGS to improve the random selection.In the DMGS, the groups are selected according to the frequent patterns found by FP-Growth.In the proposed DMGS, the FP-Growth finds the frequent groups from a transaction (in this paper a transaction means a set of the M k group indexes that perform well).After the frequent group indexes have been found, the DMGS selects the M k groups indexes according to the frequent group indexes.To avoid the frequently-occurring groups from falling in the local optimal solution, the DMGS uses three actions to select M k groups.The three actions defined in this paper are normal, search, and explore.The detail of the DMGS is shown as follows: Step 0. The transactions are building as follow equation: TNFNRuleSet i represents ith group index in M k group indexes that are selected to form a TNFN with M k rules.For example, as shown in Table I, the first transaction of the transaction set means the 3 rules TNFN that select from 1st group, 4th group, and 8th group has a well performance.

TABLE I.
TRANSACTIONS IN A FP-GROWTH.

Transaction index Group indexes
Step 1.Normal action: After building up the transactions, the DMGS selects group according to different action types.If the action type is normal action, the DMGS selects the group as following equation: Accumulator NormalTimes where Accumulator is defined in Eq. ( 20); GroupIndex[i] represents selected ith group index of the M k group indexes and P size represents there are P size groups in a population in the AGOCEA.
Step 2. Find the frequent groups: If the action is searching or exploring action, the DMGS uses the FP-Growth [24] to find frequent group indexes in transactions.The frequent group indexes are found according to the predefined Minimum_Support.The Minimum_Support means the minimum fraction of transactions that contain an item set.The FP-Growth algorithm can be viewed as two parts: construction of the FP-tree and FP-growth.The sample transactions shown in Table II   Step 3.

Select the group indexes according to different actions:
After obtaining the frequent item sets, the DMGS selected group indexes according to different actions that describe as follows: In the searching action, the group indexes are selected from the frequent item as follow equations: FrequentItemSet q Random FrequentPool where SearchingTimes is a predefined value that judge to perform searching action; FrequentPool represents the sets of frequent item set that obtain from FP-Growth; FrequentPoolNum presents the total number of sets in FrequentPool and FrequentItemSet[i] presents a frequent item set that select from FrequentPool randomly.In Eq. ( 24), if M k greater than the size of FrequentItemSet[i], the remaining groups are selected by Eq. ( 23).
In the exploring action, the group indexes are selected according to the frequent item as follow equations: where where ExploringTimes is a predefined value that judge to perform exploring action.
Step 4. After selecting M k group indexes, the k chromosomes are selected from M k group as follows: [ ] , where [1, ] 1, 2, , where N c represents the number of chromosomes in each group; ChromosomeIndex[i] represents the index of a chromosome that select from ith group.
The illustration of the DMGS is shown in Fig. 5 with simple descriptions as follows: suppose the TPSO determines that 4 rules are expected, and 3 out of 7 groups, group 2, 3 and 6, are deemed as frequent groups.If the current action type of the DMGS is normal action, then 4 random groups will be selected to form a TFS.If the search action is taken, then frequent group 2, 3 and 6 will be selected.The remaining one group will be draw randomly from group 1, 4, 5 and 7.If the explore action is taken, then the 4 non-frequent group 1, 4, 5 and 7 will be selected in case of the problem of local optimum.

D. Fitness assignment
The fitness value of a rule (an individual) is calculated by concatenating this individual with elites of other groups selected by DMGS.The details for assigning the fitness value are described as follows: G , the M k groups selected by the DMGS; G j ．p i denotes the ith individual of the jth group; y j refers to the elite individual of the jth group.Then the fitness of the individual G j ．p i can be computed as follows: www.ijarai.thesai.org

E. Reproduction
To perform reproduction, elite-based reproduction strategy (ERS) [22] is adopted in this study.In ERS, every chromosome in the best combination of Mk groups must be kept by performing reproduction step.In the remaining chromosomes in each group, the roulette-wheel selection method [32] is adopted for proceeding with the reproduction process.
Then the well-performed chromosomes in the top half of each group [14] proceed to the next generation.The other half is generated by performing crossover and mutation operations on chromosomes in the top half of the parent individuals.

F. Crossover
In this step, a two-point crossover strategy [32] is adopted.Once the crossover points are selected, exchanging the site's values between the selected sites of individual parents can create new individuals.These individuals are offspring which inherent the parents' merits.

G. Mutation
The utility of the mutation step can provide some new information to every group at the site of an individual by randomly altering the allele of a gene.
Thus mutation can lead to search new space which can avoid falling into the local minimal solution.In the mutation step, uniform mutation [33] is adopted, and the mutated gene is drawn randomly from the domain of the corresponding variable.

H. Calculation of group similarty
In order to achieve self adaptive group organization, it must determine the group similarity first.This paper involves the three measurements to determine the group similarity: 1. group centers, 2. group distance, and 3. group standard deviation.
Group centers: Group standard deviation: where y j is the center of the jth group, m j is the total number of jth group, ij l x is ith chromosome in the jth group, d ij is the Euclidean distance between the ith group and jth group,   i j  is the ith bit standard deviation in the jth group,   0 i j  is the largest standard deviation of jth group.
After calculating above three measurements, it will be used to adjust the group organization of the neural fuzzy network by following two processes: splitting process and lumping process.

I. Splitting process
, where 0  is standard deviation splitting threshold, it means the chromosomes in the jth group are very dissimilar, so the AGOCEA will call Splitting process.
The Splitting process will divide jth group into two groups, which are j+ group and j-group, by following step: the top 50% (fitness value) chromosomes in the jth group will put into j+ group, the other 50% chromosomes in jth group will put into j-group.The other 50% in the j+ group and j-group will generate randomly.After Splitting process, the dissimilar group will be separated into different groups, and total number of group will increase.

Lumping process
If 0 ij dd  , where d 0 is lumping threshold, it means the chromosomes are very similar between ith group and jth group, so the AGOCEA will call Lumping process.The Lumping process will merge ith group and jth group into a new group.The new group consists of the top 50% chromosomes from ith group and the top 50% chromosomes from jth group.After Lumping process, the similar groups will be merged into a new group, and total number of group will decrease.

IV. SIMULATION RESULTS
The example used for identification of nonlinear dynamic system given by Narendra and Parthasarathy [34] is described as following difference equation: The output of above equation depends nonlinearly on both its past value and the input, but the effects of the input and output values are not additive.The training input patterns are random value in the interval [-2, 2].To determine the performance of the algorithms, this example adopts the root mean square error (RMSE).The definition of the RMSE is: where   ŷk is desired output, y(k) is model output, and N is number of data.
In order to determine performance of the difference learning algorithm, this example is compared AGOCEA with HESP [35], ESP [36], MCGSE [22], SANE [37], and GA [6].All algorithms were learned for 500 generations and repeated for 50 trails.The initial parameters of the AGOCEA are given in Table IV.In these figures, the symbol "o" represents the desired output of the nonlinear dynamic system, and the symbol "*" represents the output of all algorithms.It can be seen from the Fig. 5-10 that the model output of AGOCEA has more accuracy than the other comparing learning algorithms.www.ijarai.thesai.org      Figure 12 (a)-(f) show the identification error between the desired output and all algorithms' output.As shown in Fig. 12 (a)-(f), the AGOCEA illustrated the smaller error than other algorithms.Figure 13 provides the learning curve of the various learning algorithms, it can be seen from the learning curve that the AGOCEA converge faster and better than the other learning algorithms.Table V shows the results obtained from a RMSE analysis of the various learning algorithms.There was a significant difference between the proposed AGOCEA and the other learning algorithms.No matter which performance index is, the proposed AGOCEA has the better performance than the other learning algorithms.V. CONCLUSIONS In this paper, the AGOCEA is proposed for designing TSKtype neural fuzzy network.The proposed AGOCEA not only determine the suitable number of fuzzy rules and group number but also efficiently tune the free parameters in the TNFN.The AGOCEA adopts the GSE that each population is divided to several groups and each group represents only one fuzzy rule.In order to solve the problem of random group selection, a data mining based group selection method was used to select the better groups.Furthermore, the adaptive group organization was proposed to solve the some parameters have to be assigned in advance.The simulation results show that the AGOCEA trained TNFN is superior to other methods.Although the proposed AGOCEA can obtain better results in comparison with the other learning algorithms, it still has a limitation.The important limitation lies in the fact that the proposed AGOCEA emphasize the network parameter learning and the group structure organization, it is a two level learning structure.While the problems become more complex, there is possible that increase the levels of learning structure.

Fig. 3 .
Fig. 3.The structure of the chromosome in the AGOCEA.

Fig. 4 .
Fig. 4. The structure of the chromosome in the AGOCEA.

Step 4 .
used to select k chromosomes form M k groups to construct a TNFN.In the proposed TPSO, for avoiding suitable M k groups may fall in the local optima solution, the TPSO proposed two different actions to update the k M V .Decide the deferent action according to the following equations: If Accumulator  TPSOTimes (19) Then do Step1, Step2, and Step 3, If Best_Fitness g = Best_Fitness (20) Then Accumulator = Accumulator + 1, If Accumulator > TPSOTimes (21) Then do Step 0 and Accumulator = 0,

Fitness
represents the fitness value of TNFN with M k rules, TransactionNum is the total number of transactions Transaction j [i] represents the ith item in the jth transaction, [] k M

Figure 6 -
11 show the output of all algorithms for the input signal 

Fig. 7 .
Fig. 7.Figure 4.2 Identification Results Of The Desired Output And The HESP.

Fig. 9 .
Fig. 9. Identification Results Of The Desired Output And The MCGSE.

Fig. 10 .
Fig. 10.Identification Results Of The Desired Output And The SANE.

Fig. 11 .
Fig. 11.dentification Results Of The Desired Output And The GA.
are taken as examples.Minimum_Support = 3 is considered in this example.Frequent group indexes generated by FP-growth shown in Table III are then thrown into the pool that's named FrequentPool.www.ijarai.thesai.org

TABLE II .
SAMPLE TRANSACTIONS.

TABLE IV .
INITIAL PARAMETERS OF THE AGOCEA.
Fig. 6.Identification Results Of The Desired Output And The AGOCEA.

TABLE V .
RMSE COMPARISON OF VARIOUS LEARNING ALGORITHMS.