Data cleansing is the process of finding errors in data and either automatically or manually correcting the errors. A large part of the cleansing process involves the identification and elimination of duplicate records; a large part of this process is easy, because exact duplicates are easy to find in a database using simple queries or in a flat file by sorting and streaming the data based on a specific key. The difficult part of duplicates elimination is finding those nonexact duplicates for example, pairs of records where there are subtle differences in the matching key. Data cleansing, which we discuss in Chapter 9, focuses mostly on the following.

• Parsing, which is the process of identifying tokens within a data instance and looking for recognizable patterns. The parsing process segregates each word, attempts to determine the relationship between the word and previously defined token sets, and then forms patterns from sequences of tokens. When a pattern is matched, there is a predefined transformation applied to the original field value to extract its individual components, which are then reported to the driver applications.

• Standardization, which transforms data into a standard form. Standardization, a prelude to the record consolidation process, is used to extract entity information (e.g., person, company, telephone number, location) and to assign some semantic value for subsequent manipulation. Standardization will incorporate information reduction transformations during a consolidation or summarization application.

• Abbreviation expansion, which transforms abbreviations into their full form. There are different kinds of abbreviation. One...