Overview

When the quality of your data is suspect, how can you trust the results of any analysis based on that flawed data? In the business intelligence/data warehouse user community, there is a growing confusion as to the difference between data cleansing and data quality. Although many data cleansing products can help in applying data edits to name and address data or help in transforming data during an extract/transform/load (ETL) process, there is usually no persistence in this cleansing. Each time a data warehouse is populated or updated, the same corrections are applied to the same data.

In reality, improved data quality is the result of a business improvement process that looks to identify and eliminate the root causes of bad data. A critical component of improving data quality is being able to distinguish between "good" (i.e., valid) data and "bad" (i.e., invalid) data. But because data values appear in many contexts, formats, and frameworks, this simple concept devolves into extremely complicated notions as to what constitutes validity. This is because the validity of a data value must be defined within the context in which that data value appears. For example, there may be many ways in which customers refer to a corporation, but there is only one legal name under which the corporation is officially registered. In most contexts any of the corporate aliases may be sufficient, whereas in other contexts only the legal name is valid.

This chapter centers on the importance of defining data...