The Role of Diversity in the Design of Multiple Classifier Systems

Multiple Classifiers Systems (MCS) perform information fusion of classification decisions at different levels overcoming limitations of traditional approaches based on single classifiers. We address one of the main open issues about the use of Diversity in Multiple Classifier Systems: the effectiveness of the explicit use of diversity measures for creation of classifier ensembles. So far, diversity measures have been mostly used for ensemble pruning, namely, for selecting a subset of classifiers out of an original, larger ensemble. Here we focus on pruning techniques based on forward selection, since they allow a direct comparison with the simple estimation of accuracy of classifier ensemble. We empirically carry out this comparison for several diversity measures and benchmark data sets, using bagging as the ensemble construction technique, and majority voting as the fusion rule. Our results provide further and more direct evidence to previous observations against the effectiveness of the use of diversity measures for ensemble pruning, but also show that, combined with ensemble accuracy estimated on a validation set, diversity can have a regularization effect when the validation set size is small. Whereas several existing pruning methods use some combination of individual classifiers accuracy and diversity, it is still unclear whether such an evaluation function is better than the bare estimate of ensemble accuracy. We empirically investigate this issue by comparing two evaluation functions in the context of ensemble pruning: the estimate of ensemble accuracy, and its linear combination with several well-known diversity measures. This can also be viewed as using diversity as a regularizer, as suggested by some authors. To this aim we use a pruning method based on forward selection, since it allows a direct comparison between different evaluation functions. Experiments on thirty-seven benchmark data sets, four diversity measures and three base classifiers provide evidence that using diversity measures for ensemble pruning can be advantageous over using only ensemble accuracy, and that diversity measures can act as regularizers in this context. Focusing on ensemble creation technique well-known as Bagging, the computational power and demand of Neural Networks (NNs) approved in both researches or in applications. The weight connections of the NNs holds the real ability for the NNs model to efficient performance. We aim to analyze the weight connections of the trained ensemble of NNs, as well as investigating their statistical parametric distributions, we present a framework to estimate the best-fit statistical distribution from a list of well-known statistical parametric distributions. This work is the first attempt in the state-of-art to explore and analyze the weights of a trained ensemble of 1000 neural networks. Consequently we aim in our future work to employ the outcomes to withdraw the weight connections value from approximated best-fit distribution instead of training the ensemble of NN classifiers from scratch.