Eliciting and Analyzing Expert Judgment: A Practical Guide

This chapter consists of three major sections. Section one, Aggregation Scheme Selection, presents several recommended and widely accepted aggregation estimators as well as methods used to form aggregation distributions. Section two, Application Environments, covers aggregation using the methods in the first section in various problem settings and environments that involve multiple experts, decision makers, and analysts. Section three presents the similarities between solving the aggregation problem and solving the problem of characterizing uncertainties (chapter 17).
Combining the experts' answers into either a single value (estimator) or a single distribution of values can prove a most difficult analytical problem. This aggregation is a mathematical aggregation. A variety of techniques can perform mathematical aggregation; however, many of these techniques impose restrictions on the data, the experts, the analyst, and on the interpretations of results. The techniques presented in this chapter for both estimators and distributions reflect the general philosophy of the book: from the elicitation side, the aggregation should not require the experts to be force-fitted into unknown or uncomfortable modes of providing data; from the analysis side, the aggregation should not be so complex that an analyst requires a doctorate in mathematics to understand and use it. To achieve both goals, this chapter presents only some of all the available techniques. However, most of the general types of estimators are given in some form. For a complete report on all the different types of mathematical aggregation schemes, see a survey paper by Genest and Zidek (1986).