7.2 The Message Understanding Conferences

Much NLP research was funded by U.S. government agencies in the 1970s and 1980s, but these agencies became frustrated at the difficulty of evaluating competing approaches, when researchers chose their own issues, processing methods, evaluation methods, and data. The solution was to establish a methodology of competitive evaluation, where sponsored researchers and others would agree to develop systems to process the same data, and formalize their analysis results in a standard notation. A series of seven "Message Understanding Conferences" (MUC) was held, with the last in 1998 ^[6] The names of the various tasks identified in MUC evaluations, and the methods used for evaluation in MUC, have become widely adopted and adapted outside MUC. Familiarity with IE, as defined in MUC, is advantageous to aid understanding of IE carried out under other auspices (see Chapter 9).

MUC also had a strong influence on the design of IE systems. Systems are usually modular, with a module robustly doing one level of analysis. Hobbs ^{[7] [8]} identifies the components of a generic IE system: tokenizer, sentence splitter, tagger, morphological analyzer, chunker, gazetteer, NE module, discourse module, template extractor, and template combiner. Chapter 2 goes into further detail on this aspect.