Eliciting and Analyzing Expert Judgment: A Practical Guide

The purpose of eliciting and analyzing expert judgment is to use all the available information in cases where data does not exist or where data is sparse. The analyst uses the experts' information to draw conclusions. This process is called making inferences. This chapter describes two types of inferences: general and statistical. This chapter discusses the inferences of both types that are possible, accounting for the design features and analysis methods used in the problem. This chapter also presents some comments about inference relating to expert judgment applications.
Statistical inferences are based on statistical principles of sampling. For example, analysts use results from a sample that is representative of a larger population to make statements regarding that population. In expert judgment applications, however, such extended inference is usually not possible. For one reason, the information from the experts is not a random sample (or representative sample) of the true state of knowledge. In most cases, the analyst may not even be able to form a random sample of the experts for the elicitation. Therefore, making inferences from expert information about the true state of the universe is not recommended.
In most expert judgment applications, the experts' knowledge represents the state of the existing or available knowledge. In that sense, general inferences can be made as follows: the results from the experts' information can be used to draw conclusions about the existing or available knowledge base which may or may not represent the true state of...