Feature Selection on Imbalanced Domains: A Stability-Based Analysis

A large body of literature has shown the beneficial impact of feature selection on the efficiency, interpretability, and generalization ability of machine learning models. Most of the existing studies, however, focus on the effectiveness of feature selection algorithms in identifying small subsets of predictive features, often neglecting the stability of the selection process, i.e., its robustness with respect to sample variation, which can be crucial for the actual exploitation of the results. In particular, little research has so far investigated the stability of feature selection methods in class-imbalanced domains, where some classes are underrepresented and any perturbation in the set of training records can strongly affect the final selection outcome. This work aims to investigate this important issue by studying the stability of different selection algorithms across high-dimensional datasets that present different levels of class imbalance. To this end, a methodological pipeline is discussed which allows a joint evaluation of the selection outcome both in terms of stability and final predictive performance. Although not exhaustive, our experiments provide very useful insight into which methods can be more stable on imbalanced data while still ensuring good generalization results.