TABLE OF CONTENTS Jonathan I. Maletic
Kent State University
Andrian Marcus
Wayne State University
| Abstract | This chapter analyzes the problem of data cleansing and the identification of potential errors in data sets. The differing views of data cleansing are surveyed and reviewed and a brief overview of existing data cleansing tools is given. A general framework of the data cleansing process is presented as well as a set of general methods that can be used to address the problem. The applicable methods include statistical outlier detection, pattern matching, clustering, and Data Mining techniques. The experimental results of applying these methods to a real world data set are also given. Finally, research directions necessary to further address the data cleansing problem are discussed. |
| Keywords: | Data Cleansing, Data Cleaning, Data Mining, Ordinal Rules, Data Quality, Error Detection, Ordinal Association Rules |
The quality of a large real world data set depends on a number of issues (Wang et al., 1995; Wang et al., 1996), but the source of the data is the crucial factor. Data entry and acquisition is inherently prone to errors, both simple and complex. Much effort can be allocated to this front-end process with respect to reduction in entry error but the fact often remains that errors in a...
