A New Feature Filtering Approach by Integrating IG and T-Test Evaluation Metrics for Text Classification

High dimensionality is one of the main issues associated with text classification, such as selecting the most discrepant features subset for classifier's effective utilization is a difficult task. This significant preprocessing stage of selecting the relevant features is often called feature selection or feature filtering. Eliminating the non-relevant and noise features from the original feature set will drastically reduce the size of the feature set and the time complexity of the classification models and also improve or maintain their performance. Most of the existing filtering method produced a subset with relatively high number of features without much significant impact on running time, or produced subset with lesser number of features but results in performance degradation. In this paper, we proposed a new bi-strategy filtering approach that integrates Information Gain with t-test that selects a subset of informative features by considering both the score and ranking of respective features. Our approach considers the results' disparity produced by the benchmark metrics used in order to maximized and lessen their advantage and disadvantage. The approach set a new threshold parameter by computing V-score of the features with minimum scores present in both the two subsets and further refined the selected features. Hence, it reduces the size of the features subset without losing much informative features. Experiment results conducted on three different text datasets have shown that the proposed method is able to select features that are highly discrepant and at the same time achieves a significant improvement in terms of classification accuracy and F-score at the cost of a minimum running time.

@article{Ado2021,
  title     = {A New Feature Filtering Approach by Integrating IG and T-Test Evaluation Metrics for Text Classification},
  journal   = {International Journal of Advanced Computer Science and Applications},
  volume    = {12},
  number    = {6},
  year      = {2021},
  publisher = {The Science and Information Organization},
  author    = {Abubakar Ado and Mustafa Mat Deris and Noor Azah Samsudin and Aliyu Ahmed},
  doi       = {10.14569/IJACSA.2021.0120657},
  url       = {https://doi.org/10.14569/IJACSA.2021.0120657}
}