An Empirical Study of Performance Metrics for Classifier Evaluation in Machine Learning

A variety of classifiers for solving classification problems is available from
the domain of machine learning. Commonly used classifiers include support vector
machines, decision trees and neural networks. These classifiers can be configured
by modifying internal parameters. The large number of available classifiers and
the different configuration possibilities result in a large number of combinatiorrs of
classifier and configuration settings, leaving the practitioner with the problem of
evaluating the performance of different classifiers. This problem can be solved by
using performance metrics. However, the large number of available metrics causes
difficulty in deciding which metrics to use and when comparing classifiers on the
basis of multiple metrics. This paper uses the statistical method of factor analysis
in order to investigate the relationships between several performance metrics and
introduces the concept of relative performance which has the potential to case the
process of comparing several classifiers. The relative performance metric is also
used to evaluate different support vector machine classifiers and to determine if the
default settings in the Weka data mining tool are reasonable.