ABCVS : An Artificial Bee Colony for Generating Variable T-Way Test Sets

To achieve acceptable quality and performance of any software product, it is crucial to assess various software components in the application. There exist various softwaretesting techniques such as combinatorial testing and covering array. However, problems such as t-way combinatorial explosion is still challenging in any combinatorial testing strategy, as it takes into consideration the entire combinations of input variables. Therefore, to overcome this problem, several optimizations and metaheuristic strategies have been suggested. One of the most effective optimization algorithms based techniques is the Artificial Bee Colony (ABC) algorithm. This paper presents t-way generation strategy for both a uniform and variable strength test suite by applying the ABC strategy (ABCVS) to reduce the size of the test suite and to subsequently enhance the test suite generation interaction. To assess both the effectiveness and performance of the presented ABCVS, several experiments were conducted applying various sets of benchmarks. The results revealed that the proposed ABCVS outweigh the existing based strategies and demonstrated wider interaction between components as opposed to AI-search based and computational based strategies. The results also revealed higher prospect of ABCVS in the aspect of its effectiveness and performance as observed in the majority of case studies. Keywords—T-way testing; variable-strength interaction; combinatorial testing; covering array; test suite generation; artificial bee colony algorithm


I. INTRODUCTION
For investigate acceptable quality and performance for any software product, it is important to assess the various software components in the application.Software testing is afforded with many techniques and tools, divided into two major categories; Black box and White box testing.The inner components of the system under test (SUT) in performing white-box testing, are considered during the generation of the test case, while the input variables and how they interact as a significant function in black-box testing, occurs during test suite production [1].Due to the complexity of the most software systems, therefore, all-inclusive testing taking the entire configurations and interactions into consideration is not feasible due to computational constraints as well as the need for sampling strategies [2].From several of the techniques available for black box software, both combinatorial testing (CT) in addition to covering array (CA) are suitable approaches used for testing purposes.In fact, some studies of t-way interaction testing have indicated this form of testing to be effective in identifying nearly all of the flaws in a typical software system [3].Notably, various uniform and variable strength t-way techniques are documented throughout the literature in this domain.
However, combinatorial explosion in a combinatorial testing strategy such as a t-way is a problem.Moreover, while it is not feasible to generate the entire combination of variables in this case, it is necessary that a CA of minimum size be generated.Generally, the minimum size of a CA is an unspecified priori.Therefore, it is advisable to generate as much CA as possible with several test cases within an acceptable amount of time.Various strategies have been found in the literature to produce the test suite size with uniform and variable strength interaction.Accordingly, to assess such a strategy, three features are required, namely; 1) the array size of the test suites that denote the effectiveness, 2) speed of the strategy, which refers to performance, and 3) portability of the strategy to support adequate high interaction strength.Different approaches have been adopted using numerous strategies, such as; Artificial Intelligence (AI) based and Pure Computationalbased approaches [4].These strategies are based on pure computational-based approaches which are characterized by their high performance but having worst efficiency, for instance; In-Parameter-Order-General (IPOG-D) [5] and Intelligent Test Case Handler (ITCH) [6].While ITCH generates test suites of small array sizes, the IPOG D strategy, on the other hand, has a fast-paced approach.Previous studies have suggested that when a small interaction strength is considered (i.e.t ≤ 3), it can cover most flaws in a typical software system [7][8][9][10][11][12].Subsequent research studies although, have proposed the need to support the higher strength interaction, particularly for complex systems [2,4,[13][14][15].
The appropriate solution to address the problems associated with the computation of the minimal test suite is by employing a search-based software engineering (SBSE) technique [16][17][18][19].By drawing inspiration from the SBSE technique, various strategies have already been suggested towards the problem of uniform and variable strength interaction (e.g.Simulated Annealing (SA) [9-11, 20, 21], Genetic Algorithm (GA) [12,[21][22][23][24][25][26][27], Practice Swarm Optimization (PSO) [7,[28][29][30], Ant Colony Algorithm (ACA) [8,12], Harmony Search Algorithm (HS) [4,31,32], Cuckoo Search (CS) [3,33,34], and the Bat Algorithm (BA) [35][36][37][38][39]).Even though relevant strategies have been found to be useful based on AI, they are noted in the literature as being slow due to costly computations [3,4,7].Another challenge encountered for some AI-based strategies that are considered to be slow is support for small interaction strengths (i.e.t ≤ 4) for generating test suites.Therefore, in this paper, the ABC algorithm is proposed as an efficient and www.ijacsa.thesai.orgacceptable strategy, known as the Artificial Bee Colony Strategy (ABCVS) continuous to our previous research [40,41] for uniform and variable strength interaction t-way minimal test suite production.It is anticipated that ABCVS will address this problem.In fact, experimental results have revealed that the proposed ABCVS is able to support higher interaction strengths up to t = 6 compared to other Artificial Intelligence (AI)-based strategies.Furthermore, ABCVS it can compete against these other strategies in the majority of the case studies examined in the literature regarding efficiency (test suite generation) and performance (speed) and against other AI-based strategies having higher interaction strengths.The remainder of this paper is structured accordingly.Section II presents the background to this study, (i.e.CA and MCA concepts) which is followed by Section III which surveys "state of the art" testing strategies in this area.Section IV provides an overview of an ABC which is then followed by Section V describing the proposed strategy, consisting of two parts: (1) construction of the covering matrix, and (2) the proposed ABCVS.Section VI illustrates the tuning of the ABCVS parameters.Section VII evaluates the ABCVS by conducting several benchmark experiments in terms of effciency and performance alongside with statistical analysis evaluation by using Wilcoxon signed-rank test in Section VIII.Section IX discusses the advantages, limitations and threats to the validity of the approach, which followed by Section X providing overall conclusions to the study and presenting recommendations for future work.

II. COVERING ARRAY WITH PROBLEM DEFINITION MODEL
The strategy of a t-way testing is one of the most minimization criteria used regarding the number of test cases list.The t-way testing is a combinatorial approach, and "t" is indicates the interaction strength [12], and one of the input parameters.For example, assume that the (SUT) behavior is represented by user input then the external and internal events or the modes of the system parameters are controlled by separate parameters (P).In this case, each separate value Vi has a parameter, Xi.These Vi are selected from a fixed set of values, consisting of value members.Each p-tuple…(X1, X2, …,Xp) shows a test case where Xi ∈ Vi and 1 ≤ i ≤ P [29].Using a practical software example, let us consider a web configurable software system.This system is consists of fivecomponents, namely; a device, a processor, the operating system (OS), browser, and a display.Fig. 1 illustrates their relationships.This system has employed as a simple illustration of the main idea in t-way testing regarding variable strength, and uniform strength.Each component of this system is known as a separate parameter, and each parameter has one or more values.Table I displays the parameters of the system and their values.In this example, the number of parameters is five, consisting of 3-parameters with 2-values and 2-parameters with 2-values.
To produce different software settings of the system, the software should be tested to identify any existing defects.In this case, a test case can be used to determine particular settings of the software.The total number of test cases are called the test suite, where the test suite must identify existing defects.For instance, assuming device 1 (Phone) is not able to use OS 2 (Windows).Therefore, the system will fail because the operating system (OS) is Windows based and the device parameter is the phone.Therefore, to detect faults, using 2-way schema (Phone,-, Windows,-,-) at least one occurrence time must be covered by the generated test suite.
To obtain "perfect" software that performs with no defects, it is important to test all software components to cover all possible parameters.In this example, 72 test cases (i.e. 2 × 2 × 3 × 3 × 2) are required to cover all cases.Nowadays, softwaretesting costs are rising due to the evolution and complexity of the software and an increasing the number of parameters and software levels.Since the failure of a limited number of parameters is mainly caused by the interaction of the parameters [12], it is not efficient to produce a comprehensive test case.Therefore, to solve this problem, a particular technique should be used such as a t-way testing approach as an alternative to a more comprehensive test, where 2 ≤ t ≤ p.In addition, it is worth noting that the "t" value denotes that the final test suite consists of all possible t-way combinations as an alternative to the p-way.However, it cannot identify the interactions that are responsible for the software failure caused if the "t" value is low.
All possible interaction combinations of the parameters are covered by the combinatorial test, called the t-way CA for the test that generates the test suite.In addition, the "t" value defines the covering depth as well as the strength, and it is considered an important factor that must be determined by the tester.Overall, mathematically both the t-way approach and CA have a direct relationship.Therefore, to define the t-way approach, the following concept is used where CA is an array size N × p, where N contains test cases, consisting of two features: Each column i contains some members of set Vi, |Vi| = Vi.The rows of each sub-array of size N × t, cover all combinations |Vk 1 | × |Vk 2 | ×…×|Vk t | of t at least once [29]; and if v 1 = v 2 =…== v p =v, then all possible cases are obtained as given in Eq. ( 1) The CA is denoted by CA (N; t, p, v 1 , v 2 , …..,vp), and if v 1 = v 2 =…= v p = v, then it is denoted by CA (N; t, p, v) or CA (N; t, v p ) [29].In some research references, if vi is different, it is called a Mixed Covering Array (MCA) [38,[42][43][44].
The interactions between the parameters are important given that most software is unstable, due to some of these interactions having less impact.Some others interactions are more likely related to system failure.In this case, CA along www.ijacsa.thesai.orgwith the variable strength is used effectively to determine the various interactions.Various covering strengths exist in this structure between the various sets of parameters.The VS Covering Array (VSCA) is denoted by VSCA (N; t, p, v, {C}) where {C}, is consists of one or more CA (t′, p′, v) and p' is a subset of p.A web configurable software system example is next shown to understand VSCA.In the web configurable software assumes that all parameter combinations require 2way and a device, processor and OS that requires 3-way interaction.In this case, the configuration system is indicated as VSCA (N; 2, 2 3 3 2 , CA (3, 2 2 3 1 )).  3 3 2 , CA (3, 2 2 3 1 )).For this reason, using VSCA not only covers all 2-way interactions of complete parameters, but also covers all 3-way interactions of the specified parameters.Throughout Over the last few decades, there have been many studies in the literature that grouped into four main sets to solve the CA problem [45]: (A) mathematic methods, (B) greedy algorithms, (C) heuristic search algorithms, and (D) random methods.

A. Mathematic Methods
The mathematician researcher uses mathematic methods derived from the extensions of mathematical functions like the Orthogonal Array (OA).To generate CA for optimal mathematics, it is only available for specific configurations (i.e. the parameters and values are required to be uniform) [12].Notably, there are several strategies designed based on this approach such as TConfig [46] and Combinatorial Test Service (CTS) [47] strategies for the generalisation of an OA.However, a major problem exists for the OA (i.e. as the solution is only available for small configurations because of the restriction).

B. Greedy Algorithms
Most greedy algorithm (GA) solutions initially begin to generate all possible combinations (i.e.solutions) based on the input variables; then, generate one test at a time until all uncovered combinations are covered.A greedy algorithm is different in this case compared to the other strategies for generating test cases.In order to produce a test case using greedy algorithms [3], there are two main approaches; (1) Onetest-at-a-time (one-line-at-time) and ( 2) One parameter-at-atime.
The AETG was the first type of strategy using the OTAT approach to generate the test suite.AETG attempts to select a number of test case candidates in every single cycle covering most of the interactions and adds the selected test case to the final test suite in a "greedy" fashion.Another greedy strategy was used for many years, where mAETG [9] and mAETG-sat [57] were extended to support the constraints.PICT [48] helps in the generation of all interactions and randomly chooses the required test cases by using the OTAT approach.Because of its random attitude, this strategy tends to offer a non-optimal test size.In addition, PICT has been able to generate test suite sizes up to t > 6.
As a variant of the AETG, TVG [52] produced the best results based on the three algorithms, namely; Random, Plusone, and T-reduced sets.However, given the limited literature available, the details relating to the use of each algorithm remains indistinct.Nevertheless, based on our experience relative to the implementation of TVG, T-reduce usually generates the most optimal outcome, unlike the other related algorithms.In fact, TVG supported the interaction strength (i.e., t < 10) to produce the test suite.
Another strategy showing good results regarding efficiency for most configuration systems, as well as having good speed using the OTAT approach, is the Jenny strategy [56].This strategy initially produces the test suite in order to cover oneway interaction only and was then further developed to extend the test suite to cover every 2-way interaction.This process, pending all t-way interactions, tends to be covered.The major competitor of Jenny is the Intelligent Test Case Handler, known as ITCH [6] which depends entirely on an exhaustive search algorithm for the producers of the interaction test suite.However, it is considered to be one of the slowest strategies compared to the other computational strategies regarding its efficiency to produce good results for some configuration systems.CTE_XL [54,55] is another strategy based on the Classification-Tree Method (CTM) that utilises the OTAT approach called Classification-Tree Editor eXtended Logics (CTE_XL).The fundamental idea behind this strategy is based on several features that produce test suites by receiving the input data which is then divided into the two subsets being entirely different, and then gathering these subsets to produce the test case which is then is added to the final test suite.However, the CTE-XL strategy underpins low interaction strength for producing the test suites (i.e., t ≤ 3).The most reliable strategy in the computational technique that uses three algorithms for producing the test suite is GTWay [2].In this strategy, the utilised algorithms are the t-way pair generation, the parser algorithm, as well as the backtracking algorithm.In the first instance, the parser algorithm helps to ensure that the system configuration parameters are guided in order to be utilised by the algorithm of the t-way pair generation.Then, the algorithm of the t-way pair generation helps in the generation of the required t-way parameter interaction based on a symbolic representation that is defined based on the parser algorithm.Finally, the backtracking algorithm needs to iteratively go through the t-way pair sets to add up the values of the t-way parameter's interactions for the generation of a complete test t-way suite.Although, addressing the strength of the high interaction in this instance is necessary (i.e.t ≤ 12).
Compared to the OTAT approach, the OPAT approach is where the algorithm begins initially by choosing two parameters.Then, the test suite is horizontally extended by adding the rows one-by-one until the interaction parameters have been covered, starting with a vertical extension to choose more parameters, of which the process continues until the end.One of the strategies that utilised the OPAT approach is IPO.Other strategies based on the IPO include; IPOG [58], IPOG-D [5], IPOG-F [59], IPO-s [60], and ParaOrder [61].However, ParaOrder and IPOG strategies can support a variable strengthcovering array.
In addition, SA, GA and TS are "grand" strategies that support and only produce a test suite in CA.The results of the SA strategy have indicated that it is stronger compared to both, GA and ACA.Furthermore, both SA [10] and ACA [8] algorithms were extended to support the variable strength interaction up to (i.e.3-way), where ACA is called, Ant Colony System (ACS).Searching the large space using SA to find the best test case utilised the Binary Search Algorithm (BSA).Meanwhile, ACA tried to find the best path (i.e.where every path represents a test case) until selecting the best path.In this case, the best path was selected based on the calculated weight.In GA, it begins randomly by selecting the test case after initialising the population, where each test case represented one chromosome.Then, through a fitness function, the chromosome"s weight is calculated, after changing the functions (mutation and crossover) and at the end of the process, the best test case (i.e.best chromosome) is added to the final test suite.All these procedures continue until the required coverage is met.Another strategy that supports CA in by generating the test suites based on the PSO algorithm and fuzzy techniques called Fuzzy Self-Adaptive PSO (FSAPSO) [29].FSAPSO is better compared to CS and DPSO regarding efficiency, but the performance is weak due to the Fuzzy computations.However, in this case, the strategy supported the interaction strength up to t = 4 to produce the final test suite.
In addition to SA and ACA [12], other strategies also are based on the GA algorithm with less modification in the structure which produces a good result regarding efficiency.These strategies are PWiesGen [25], PWiesGenPM [22], tuned Genetic (GS) [62] and Genetic Algorithm for Pairwise Test Sets (GAPTS) [26].Notably, PWiesGenPM is better compared to GAPTS and GS is stronger than PWiesGenPM, GAPTS and PWiesGen.In order to address the variable strength problem, PWiesGen was extended [23] [to overcome this problem] with interaction strength (i.e.t = 6).This strategy is called PWiesGen-VSCA.
According to the literature, the first HS or AI that supports the interaction strength (i.e.t = 6) is the PSTG [7] strategy based on the PSO algorithm.Further, this strategy shows good performance, but weak efficiency for the strength less than t = 3 against the available strategies found in the literature.Calculating the weight to select the best test case to be stored needs big data by using a new approach.Accordingly, this strategy was extended to support the variable strength called VS-PSTG.Another strategy like PSTG that produces the test suite up to (i.e.t = 6) is the CS strategy [3], which is precisely like PSTG but is faster by using the utilised Cuckoo algorithm to reduce the search space.
One of the most efficient strategies used to produce strong results and also supports strengths (i.e.t = 15) is HSS [4] which is based on the Harmony Search Algorithm.Either this is where the algorithm imitates the musicians behavior, which endeavors to compose enthralling music from random sampling or from improvisations (that is, adjusting a tune from their memory).In this strategy, it adds the test case in each iteration to the test suite and then goes on until the required coverage is met.Another efficient strategy used to produce a strong result, but having less support compared to HSS regarding strengths (i.e.t = 6), is HHH [63].This strategy is characterised by using High-Level Hyper-Heuristic (HHH) as the first strategy [based on literature], where it uses four algorithms (i.e.meta-heuristic algorithm) instead of one.The four algorithms are; Teaching-Learning based Optimization (TLBO), PSO, Global Neighborhood Algorithm (GNA) and the Cuckoo Search algorithm.This strategy employs Tabu Search as the high-level search, then selects one of the four algorithms in each step, and depends highly on three explicit operators, on the basis of improvement, diversification, and intensification to adaptively choose the best meta-heuristic at any point in time to produce test cases.
The test suite is also produced using the PSO family [28] according to the following strategies; DPSO, DMS-PSO, APSO, TVAC, CLPSO and CPSO.As any AI algorithm has disadvantages and advantages due to randomisation, PSO as one of the algorithms also has inherent weaknesses regarding efficiency, which is the velocity function.To address this problem, DPSO was used to produce a good result by generating the best solution.This strategy supports the interaction strength of more than 6, but the published result [28] is up to 4.

D. Random Methods
This method is an ad hoc approach where a random test case is selected by utilising input distribution [45].Section 3.2 mentioned that TVG utilised three methods to test case generation.One of these methods is a random method.

IV. OVERVIEW OF ARTIFICIAL BEE COLONY
The ABC algorithm is designed to trigger the honeybee colony foraging manner.A quintessential honeybee swarm comprised of three basic constituents e.g.source of food / employed foragers/failed recruits (onlookers and scout bees) [64].The employed bees linked to a specific food source.They transmit important information such as (location, navigation and the profitability) of the food source and convey the data with the rest of the standby bees at the hive.The onlooker bees are responsible for food source discovery utilizing the information provided by employed bees.The scout bees assigned to random hunt the new food source.It is presumed that, the employed bees who are lacking of food source transformed into scout bees and begin a new search for the food source.It is inferred that the number of food source equates to the number of employed bees in the colony.In conclusion, the solution to the optimization problem is represented by the food source stand; meanwhile, the quality www.ijacsa.thesai.org(fitness) of the mentioned solution coincides with the quantity of food source [65].ABC initiated a random population distribution of SN solutions (food source positions) in the search space, where SN signify the size of employed or onlooker bees.Each solution X i (i = 1, 2, …, SN) will essentially be a D-dimensional vector provided that the number of optimization parameters to be D. All solutions generated at this stage can be obtained from Eq. (2).
Here, X min and X max are respectively the lower and upper boundary parameters for solution xi in dimension j (j = 1, 2 … D), and rand [0,1] is a scaling factor representing a random number between [0,1].The D-dimensional solutions (food source positions) generated in the initialization step (C = 0) are subject to repeated cycles (C = 1, 2 …, MCN), until a termination criterion is satisfied.One ABC cycle required both implementation of global and local probabilistic search/selection.Every cycle demands various task performance by various bee types as illustrated in the flowchart (Fig. 2).The operations are initially independent and can be justified in distinct manner as below for clearer vision of the ABC methodology.
1) Employed bee step: After the employed bees convey the information to onlooker bees post evaluation of their sources fitness (solutions).Each employed bee agitated the old solution (X ij ) in its memory to create a potential solution using the Eq (3) below: Here, K{1, 2, …, D} and j{1, 2, …, SN} (k s i) are randomly chosen indexes, and rand [-1,+1] is a random number between [1,1], which works as a scaling factor.Evidently, the perturbation on solution is inversely proportional to optimum solution approached in the search space.The employed bee will also evaluate the fitness of the new (perturbed) solution and according to greedy-selection scheme, supposing the fitness value is better, it will replace the old one from employed bee memory.
2) Onlooker bee step: The duty of onlooker bee it to search for food source (solution), in reference to the association of probability value and food source Pi.The calculation of Pi is as the expression below in Eq. 4: The fitness value of certain solution is signified by fit and solution number is referred to the subscript index.By comparing Pi against a randomly picked number ranging between [0, 1], the probabilistic selection is applied.The selection will be only be accepted if the random number generated is less than or equal to Pi.The approval of probabilistic selection will determine the authorization of an onlooker bee assignment to a given solution.Normally, the calculation of fitness value of solutions in problems minimization is carried as the following in Eq. 5: The objective function for solution i is signified by fi.Assuming the selected food source matches with Pi probability, with Eq. ( 3) the onlooker bee will select a better food source (solution) in the area of the previous one in her memory.Suppose the fitness value of the solution is better, the onlooker bee will auto update the latest solution in her mind, disregarding the old one, which is akin to employed bees.

3) Scout bee step:
The scout bees are assigned to search random food source in order to discover better solution for the global optimization problem.The scout bees are different from employed/onlooker bees as they are not committed to old solution in order to create trial solution.They obtained their samples from a broad set of D dimensional vectors, in condition it is still within the search space zone.In ABC, the solution will be disregard if the (non-global) solution cannot be improvised post a pre-evaluated cycle"s number.This will affect the assigned employed bee and transformed it to scout bee with limited scout type behavior.The limit also known as the value of this pre-evaluated cycle numbers is a crucial control factor of this algorithm.The limit is expressed as below.

Limit = c.n e .D
Where, Ne signify the number of unemployed bees, while c is constant coefficient with a recommended value of 0.5 or 1.ABC application minimum requirement is one scout bee implementation.Scout-type operations hypothetical searches in the completely D-dimensional space provide exceptional effectiveness to the ABC method in searching the best global solution.Scout bees are independent when it comes to global optimum solution discovery in comparison to other bee types.www.ijacsa.thesai.orgBoth (employed/onlooker) concurrently check on their local candidate solutions for the global best.Thus, it is impossible for ABC to be trapped in local optima [64].

V. PROPOSED STRATEGY
In this study, to fulfil the optimal test suite, a new strategy is proposed.The proposed strategy uses ABC algorithm as the backbone algorithm to generate the test suites.Generally, the ABCVS strategy undergoes three phases during the construction of the uniform and variable t-way test suite generation using; (1) ABCVS input analysis algorithm, (2) ABCVS interaction generation algorithm and (3) generating the test suite using ABC algorithm as shown in Fig. 3.In Section 5.1, it discusses how to set up the input for the next phase and describes the generating of the interaction tuples by using the interaction generation algorithm.The ABCVS test suite construction is described in Section 5.2.

A. Input Analysis & Interaction Generation Element Algorithm
This section illustrates how the strategy receiving the CA, MCA and VSCA (inputs) and generation of the interaction parameter (P) combinations, compute and store the interaction element for each of the parameters to be used later for the test case coverage evaluation; based on the values (v) and the specified interaction strengths (t).To clarify the input analysis and the interaction generation elements, Fig. 4 shows a small VSCA configuration as an example illustrated by using a flowchart.
The value interaction elements of each parameter are then constructed based on the parameter interactions.As shown in the previous example (N; 2, 2 3 3 1 , {CA (3, 2 3 )}), the main configuration has four parameters, where the first three parameters have two values (0 and 1), and the fourth parameter has three values (0, 1, and 2) with interaction strength t m = 2.The sub-configuration has three parameters, each having two values (0 and 1) with interaction strength t s = 3.As illustrated in Fig. 4, the generation approach for the main configuration (t m = 2) has six possible combinations of the interaction's parameters.For example, the possibility of the interaction elements for combinations (0 B2 0 B4) is 2 ×3 for both the second and fourth parameters, and the possibility of the interaction elements for combinations (B1 0 B3 0) is 2 ×2 between the first and third parameters.In order to check the availability of the parameters, the algorithm scans the binary digit of the combination, where each value of the other parameters is included in the interaction element.Regards to the rest of the values (i.e., replaced with "X") indicates "don"t care" is the excluded value in the interaction element.The interaction elements of the sub-configuration were produced similarly to that as depicted in Fig. 4. Generating the interaction elements for each test case continues until creating the final test suite.This method is illustrated in the next section.

B. The Test Suite Generation Algorithm
According to Fig. 3 that showed the responsible algorithm for the parameter combination generator, and the interaction elements combination generator based on the interaction strength, this will generate the possible interaction elements.In this strategy, a unique answer is produced in each step of the algorithm, which has no effect on the next test case selection.Selecting a test case is the most challenging problem in test suite generation, to achieve the test objective with a minimum number of test cases.In ABCVS, each procedure in this algorithm generates the best solution, which does not affect the next test case selection.For example, at every stage, the first of each bee (employed and onlooker) is randomly valued by the scout bee to the number of food source (which represents a test case) within a particular area.Following the valuation, the quality of food source is called the fitness value (coverage or weight) which is calculated based on the fitness function, and the results are presented to the employed bees.The employed bees optimise the solutions (food source position), calculate their qualities again, and present the results to the onlooker bees.The onlooker bees calculate the probability of selecting the solutions.Then, these values are normalised, and the food www.ijacsa.thesai.orgsource with higher quality (fitness) are greedily selected, and the proposed solutions are optimised again by the random values and the values of neighbors.The quality of the proposed solutions based on the fitness function is again specified.The onlooker bees investigate the solutions and select the best solution for storing.If it is an obsolete solution, the scout bee generates a new one.Otherwise, if the algorithm is finished, the final solution is registered as the output of the first stage, if not the algorithm should repeat.For instance, in Fig. 5, the test case is selected based on the maximum fitness (coverage or weight).The test case (1000) has the maximum weight of coverage, which is seven.Therefore, this test case (1000) adds to the final test suite by the test suite generation algorithm.Regarding the test case (1001) that does not reach the maximum coverage of fitness, the algorithm updates the population (food source) search space randomly to obtain the best fitness.

VI. TUNING PARAMETER SETTING OF ABCVS
There are three important control parameters in the proposed ABCVS strategy.These control parameters are the colony size number (employed bee and onlooker bee (population size)) equal to the number of food source, the value of Limit and the maximum cycle number (MCN).Growing the population size improves the outcome regarding both the generation time and the array size.In ABCVS, generating the final test suite plays a critical role based on the employed and onlooker phase.It is worth mentioning that these two phases are different in determining the value of configurations.In other words, both of t, p and v affect the number of iterations for the employed and onlooker phase.
An experiment were conducted on CA (N; 2, 5 7 ) with a variable number of the bee to determine this value.As shown in Fig. 6 conducted, the best value of this configuration for the number of bees is 1 (i.e.food source = population /2) when the number of cycles ( 70) is a test suite of size 38 test cases, and when the number of bees is 2, the number of cycles (100) is 37. Increasing the value of bees up to 3 and 4, when the number of cycles is (100) and ( 90), respectively, the algorithm achieves a test case of size 36 test cases.In these experiments, the Limit is kept constant = 100.Therefore, the value increasing does not affect the size, only increasing in the generation time.In Fig. 6, different colours that depend on the number of cycles show the size of the test cases.Fig. 7 illustrates the effects of the colony size (population size) on the array size in the same configuration CA (N; 2, 5 7 ), and shows the best average test suite size for CA (N; 2, 5 7 ).The population of size is variable from 1 to 10, and 20 to 100, when the number of cycles also is variable from 10 to 100 and Limit = 100 is considered.Regarding the test suite array size analysis, Meta-heuristic is a non-deterministic algorithm based on randomisation.Therefore, this analysis has two values.First, the best value as illustrated in Fig. 6, and the second value is the average of five independent runs.The best average is shown in Fig. 7 for the configuration CA (N; 2, 5 7 ), where the best value for this configuration is 36 test cases.
The analysis, in this case, is aimed at determining the suitable value for the colony size (population size) that depends on changing the values of p, t and v of the configuration.Each configuration then determines the suitable experiments.Therefore, the best average value is selected when the population size is 50, and the maximum number of cycles is 80.Note that this value is related to the configurations in the tables of this paper.

VII. EVALUATION
In this section, there are two parts in the evaluation of the test suite generation strategies; (1) the efficiency evaluation (i.e.test suite size) and (2) the performance evaluation (i.e. the time of test suite generation) [62].The test suite size and the interaction strength are not affected by the operating system and hardware but depend on the steps of the algorithm's running.Different strategies are selected in different tables based on the criterion; Tables IV-VIII are divided into two categories.In the first category, the available strategies that can be implemented in our system are included regarding efficiency; ABCVS is compared with these strategies.The second category contains unavailable strategies as discussed regarding efficiency and performance by other researchers.The comparison of these strategies has been conducted based on efficiency and performance.Therefore, it implements some of the AI strategies since many are unavailable in the same conditions of the proposed strategy (i.e.hardware and software).Regarding the Tables IV-VIII of configurations, the mentioned strategy's values are available, and used the same value as that published in the papers; otherwise, implementing the strategies on the system is used to obtain the result.
The characteristics of the hardware system for conducting the experiments of the proposed strategy consisted of a Windows 7 (OS) desktop computer with 3.40 GHz Xeon (R) CPU E3 and 8GB RAM.The Java language JDK 1.8.was used to code and implement the ABCVS.In this paper, the best outcome in the tables is shown in bold and in dark cells.Also, "NA" (Not Available) means not publishing, and "NS" (Not Supported) means does not support the corresponding configuration.

A. Evaluation of the Efficiency
In Table IV, the array size evaluation is divided into five parts; the interaction strength 2 ≤ t ≤ 6 and parameter 3 ≤ p ≤12 and value and V = 3 for configuration CA (N; t, p, v).The outcome of IPOD-D and IPOG are adopted from the Advanced Combinatorial Testing System (ACTS) [66].These strategies do not normally generate good results as well as for PICT, TConfig and Jenny.Whereas, IPOG-D, IPOG and TConfig support generation of the interaction strength up to t = 6.As shown in Table VII, other AI-based strategies are studied.The results of CS and PSTG are unavailable and are obtained from the papers.CS and PSTG generate similar results.The strongest strategy in terms of producing the best result is DPSO, where it showed results for t ≤ 4 obtained from [24], for each configuration of 30 separate runs.Regarding the weak performance of DPSO (i.e.speed), it is impossible to implement each configuration of 30 runs because in CA (2512; 6, 3 12 ) 23,620 s are required, thus will take more than 7 days.Other results are available from [67].The result for t > 4 is obtained with 3 runs only.CS another AI-based strategy has generated a good result.In this case, the obtained result for t > 4 is adopted in 3 runs only because of the low performance of this strategy.To be fair, the ABCVS strategy is adopted in 5 runs for t > 4, and 20 runs for the value t ≤ 4.
The values in Tables V, VI and VII, are adopted from the corresponding paper [7].Regarding the configuration system VSCA (N; 2, 3 15 , {C}) in Table V, where the PICT and WHITCH strategy produces undesirable results, these strategies have the capability to generate a test suite to t = 6.However, WHITCH produced better results.Another computational-based strategy is ParaOrder that had a better result compared to PICT and WHITCH up to t = 3 but does not support in generating a test suite for t ≥ 4. As shown in this table, AI-based strategies are considered the strongest strategies.The strongest strategies that generate the best results are SA and ACS when the strength is 3.However, these strategies are not able to produce a test suite of more than 3. Regarding the strategies such as PwiseGen-VSCA, ABCVS, GS and PSTG, these are better than the other strategies in terms of support strength of more than 3 and less than 6.IPOG and GS have the support strength more than 6.
Table VI shows the configuration VSCA (N; 3, 3 15 , {C}), where PSTG, GS and ABCVS strategies generate good results for strength up to t = 6, but IPOG and GS produce a better test suite for strength more than 6.However, ACS and SA are not able to generate a result due to not having support strength of more than 3. Also, the results for Density and ParaOrder are not available.www.ijacsa.thesai.orgRegarding the configuration system VSCA (N, 2, 4 3 5 3 6 2 , {C}) in Table VII, the results of PwiseGen-VSCA and GS strategies are not available as well as for the ParaOrder and Density strategies for most subsets ({C}), but not all.Furthermore, WHITCH and PICT have support strength up to 6, but these strategies do not generate a good result.ABCVS, TVG and PSTG strategies have the capability to generate better results for strength t = 6, and ACS and SA produced a better result for strength t = 3.The last evaluation in terms of time generation, the configuration CA (N; 3, 7, v) for 2 ≤ v ≤ 10 is used in Table X.In this evaluation will test the impact of increasing the values on time generation.IPOG and IPOG-D are the fastest strategies in terms of the performance and DPSO is strongest on terms of array size.ABCVS generates close results to DPSO and GS.

VIII. STATISTICALLY EVALUATION
In order to evaluate the strategy regarding array size (i.e.effectiveness), a statistical method is another way to assess the significance of strategy.In this case, the Wilcoxon signed-rank test is used between ABCVS and each strategy in the experimental tables 95% confidence level (i.e.α =0.05).The reason for adopting this method is that the Wilcoxon signedrank test takes into account the difference between two sets.To study the difference of the two sets, this test is ideal.In other words, this test is measured from a subject group and can be rated.
Due to the multiple comparisons, we need to control the error rate.Bonferroni-Holm correction was adopted to adjusting α value (i.e. in other words, based on Holm"s sequentially rejective step down procedure [26]) depending on the first stored p-value (Asymp.Sig. (

IX. DISCUSSION
AI-based strategies are characterised by producing a strong result, although, not without having a complex structure and repetition of which the complex structure of the strategy is inversely correlated with the strategy"s strength.For instance, ACO, GA and SA can produce a test suite with t <= 3.While CS and PSTG reduce complexity by changing the structure and raising the performance speed to support strengths up to t = 6.Notably, GS supports strengths up to t = 15.www.ijacsa.thesai.orgIPOG, PICT and Jenny are computational based strategies having the capability to produce a test suite for t > 6 like AIbased strategies.However, computational strategies do not produce good results but instead, do have good performance.Therefore, it is impossible to obtain a strategy that can support high interaction strength with perfect performance and efficiency.The ABCVS indicates that its efficiency regarding generating a test suite with t = 6 competes well compared to the other existing strategies.Further, ABCVS is better as compared to computational ones and can compete with AIbased strategies regarding performance and efficiency.
Although, ABCVS like any other strategy have limitations; for example, its contribution towards supporting variable strengths interaction.The ABCVS efficiency for strengths (i.e.t ≤ 3) is slightly lower compared to others.Given the limited literature in this field, many of these strategies are not available publicly and therefore, in this study, the configurations reported and available in publications have been used.Therefore, further experiments need to be conducted in order to obtain a precise evaluation of the proposed strategy.This paper proposes an efficient strategy called ABCVS, based on the ABC algorithm for both uniform and variable CAs.Supporting variable strength is the main contribution of ABCVS.In addition to the supporting variable strength, ABCVS can generate a test suite up to t = 6 and can produce a good result with suitable performance.To study the impact of parameters like population size or a number of cycles, different experiments have been conducted.The suitable tuning parameters of the proposed ABCVS strategy results in better ABCVS, regarding higher interaction, performance and efficiency.Furthermore, different experiments have been conducted on different configurations to compare ABCVS with existing strategies, where ABCVS shows it can compete against the other strategies regarding both efficiency and performance.As part of our future work, we want to study other metaheuristic approaches to hybrid these with ABC to increase efficiency.This hybridisation should be performed in a way that does not decrease performance and can increase the support for test suite generation for t > 6.

Fig. 5 .
Fig. 5.An Illustration Example for Selection Test Case based on the Maximum Coverage.

Fig. 7 .
Fig. 7.The Best Average Test Suite Size with Variant Number of Bee for CA (N; 2, 5 7 ).
Table III displays the complete test suite to cover this configuration.Three more test cases as shown in Table III, are added to Table II in order to construct VSCA (12; 2, 2

TABLE IV .
A TEST SUITE SIZE FOR CA (N; T, 3 P ) WITH 2 ≤ T ≤ 11 AND 3 ≤ P ≤ 12

TABLE X .
TEST SUITE SIZE AND TIME FOR CA (N; 3, V7) WITH 2 ≤ V ≤ 6 Evaluation of the Performance In TableVIII, the configuration CA (N; t,37) for 2 ≤t ≤6 is used in terms of array size and time.TConfig strategy speed relies on t, where by increasing the t time will increase exponentially.IPOG-D and IPOG are fastest computationalbased strategies, where the test suite generation time almost near to zero that indicates the high performance.ABCVS and DPSO performance are very slowly.PICT, Jenny and GS are faster than ABCVS, but these strategies are not able to generate the final test suite in a less than day.Therefore, the proposed strategy it's better in terms of performance than PICT, Jenny and TConfig.The result of the proposed strategy shows the generation time of a test suite does not really rely on increasing t.
TableIXdisplays the results in terms of array size and time of the strategies.The configuration CA (N; 3, 3 P ) is used to evaluate the variable of p on time generation.The growth of p has shown an impact on the proposed strategy in terms of time generation, but has less impact on TConfig performance.
2-tailed)) in scaling in ascending order.Therefore, α Holm is adjusted based on: Note: Where M indicates the overall number of paired comparison and i indicates the test number.The test is executed using a software tool called SPSS, where if the value is less than α Holm of the Asymp.Sig.(2tailed), it indicates a significant difference between the two sets.There are four values to evaluate ABCVS; Ranks ABCVS>, ABCVS<, and ABCVS= are used.In other words, the results of the proposed strategy are greater, smaller or equal to the other existing strategies.Two values have a Statistical Test part; Asymp.Sig.(2-tailed) and Z.The value of Asymp.Sig.(2-tailed)indicates the significant difference between the two sets and that the value does not exceed α Holm.Regarding the value Z, it is out of the scope of this paper (i.e.not important).The strategies with N/A and N/S results are considered incomplete and ignored samples, as there is no available result for the specified test configuration.Tables XI-XIV present a statistical test for the Tables IV-VII and TableXIpresents the result of TableIVfor the Wilcoxon test.ABCVS shows there is a significant difference with other strategies in column Asymp.Sig.(2-tailed), except for CS, which has a significant difference from ABCVS.

Table
XII presents the test results of TableV.IPOG, ParaOrder, Density, TVG, GS, SA, ACS and PwiseGenVSCA produce a test suite for strength less than 3 for t > 3 and the results are considered "missing".For this reason, the IPOG, ParaOrder, Density, TVG, PSTG and GS results are better than ABCVS.Whereas, ABCVS performs better than WHITCH, PICT, SA, ACS and PwiseGen-VSCA in this table.

Table
XIII presents the test results ofTable VI, where ABCVS excelled compared to WHITCH and GS strategies.Table XIV presents the test results of Table VII.TVG and PSTG are shown to have a significant difference compared to ABCVS.However, ABCVS excelled with the WHITCH, and PICT strategies.

TABLE XI .
WILCOXON SIGNED RANK SUM TEST FOR TABLE IV.

TABLE XII .
TWILCOXON SIGNED RANK SUM TEST FOR TABLE V.

TABLE XIII .
WILCOXON SIGNED RANK SUM TEST FOR TABLE VI.

TABLE XIV .
WILCOXON SIGNED RANK SUM TEST FOR TABLE VII.