In practice, it is insufficient merely to identify the values of the parameters that minimize the sum of squared errors. We need also to consider the accuracy of our estimates. Here, we address this topic with Bayesian statistics, which describes how our uncertainty in the parameter values is changed by doing the experiments. Let us consider single-response regression of a model

We have a particular set of N measured responses y ? ^N, and wish to estimate the unknown parameter vector ? ? ^P and the statistical properties of the random error ? ? . While the error may not be truly stochastic, we assume that it has the properties of a random variable, since presumably we have no practical way of predicting the error value in any single experiment.

As statistics is based upon probability theory, it is helpful to cite again two possible means of defining probabilities. One way to think about probability the frequentist approach is based upon the relative occurrences of events in many repetitive trials. Let us say that the probability of observing an event E in an independent random trial is p( E). The frequentist way of defining the value of the probability of observing E, 0 ? p( E) ? 1, is to say that if we perform a large number T of such trials, with the observed number of occurrences...