Eliciting and Analyzing Expert Judgment: A Practical Guide

Many methods can help characterize and handle uncertainties, even in the restrictive environment of expert judgment applications. This chapter examines some of the easier ways. This book's concept and definition of uncertainty cover a very broad sense, including the four basic sources of uncertainty that prevail throughout the data and the analysis.
This chapter suggests the following steps to control, or at least understand, this total uncertainty characterization. First, the section Obtaining Uncertainty Measures discusses which uncertainty measures the answer data require. Second, the section Modeling Uncertainties discusses how to model uncertainties in the data. This chapter concludes in Comparison of the Methods with a comparison of various methods for handling uncertainties. Chapter 18 discusses the relationship between uncertainty and the inference process.
Different kinds of uncertainties become important in any sampling or experimental data-gathering process. Some uncertainties can be controlled and, therefore, reduced to an acceptable noise level of influence simply by taking larger samples or by doing careful experimental design and measurements. Other uncertainties cannot be controlled or reduced by any practical means. We must learn to accept these uncertainties. The most we can do is understand their importance and effects.
Uncertainties can stem from different sources, definitions, sampling errors, nonsampling errors (e.g., missing data), and scientific or modeling techniques (Stoto, 1988). All four sources create uncertainties in expert judgment applications. Careful elicitation can help reduce the uncertainty by following the procedures in Part II. Large sample sizes can reduce sampling error...