On the Design of Bayes Consistent Loss Functions for Classification

Google Tech Talk July 30, 2010 ABSTRACT Presented by Hamed Masnadi-Shirazi. The machine learning problem of classifier design is studied from the perspective of probability elicitation, in statistics. This shows that the standard approach of proceeding from the specification of a loss, to the minimization of conditional risk is overly restrictive. It is shown that a better alternative is to start from the specification of a functional form for the minimum conditional risk, and derive the loss function. This has various consequences of practical interest, such as showing that 1) the widely adopted practice of relying on convex loss functions is unnecessary, and 2) many new losses can be derived for classification problems. These points are illustrated by the derivation of novel losses which are not convex, but do not compromise the computational tractability of classifier design, and are robust to the contamination of data with outliers. It is argued that such robustness requires loss functions that penalize both large positive and negative margins. Also, the connection between risk minimization and probability elicitation is extended to the cost sensitive setting in a manner that guarantees consistency with the cost-sensitive Bayes risk, and associated Bayes decision rule and a new procedure for learning cost- sensitive classifiers is proposed. Hamed Masnadi-Shirazi is a PhD student in the Statistical Visual Computing Lab at the University of California, San Diego. He received his B.S. in Electrical Eng. from Shiraz University, Iran and University of Texas at Arlington in 2003. His research interests are in computer vision, statistical signal processing and machine learning. At present, he is working on the theory of Bayes consistent loss functions for classifier design, with applications to robust classification, cost sensitive learning, real time face and object detection and brain EEG surprise signal detection.