TABLE OF CONTENTS Pierre Geurts
Department of Electrical Engineering and Computer Science, University of Li ge, Belgium. Postdoctoral Researcher, F.N.R.S., Belgiu m
| Abstract | In this chapter, the important concepts of bias and variance are introduced. After an intuitive introduction to the bias/variance tradeoff, we discuss the bias/variance decompositions of the mean square error (in the context of regression problems) and of the mean misclassification error (in the context of classification problems). Then, we carry out a small empirical study providing some insight about how the parameters of a learning algorithm influence bias and variance. |
| Keywords: | bias, variance, supervised learning, overfitting |
The general problem of supervised learning is often formulated as an optimization problem. An error measure is defined that evaluates the quality of a model and the goal of learning is to find, in a family of models (the hypothesis space), a model that minimizes this error estimated on the learning sample (or dataset) S. So, at first sight, if no good enough model is found in this family, it should be sufficient to extend the family or to exchange it for a more powerful one in terms of model flexibility. However, we are often interested in a model that generalizes well on unseen data rather than on a model that perfectly predicts the output for the learning sample cases. And, unfortunately, in practice, good results on the learning set do not necessarily imply good generalization performance on unseen data, especially if the "size" of the hypothesis space is large...
