Finally we are going to talk about data. Most of the book so far has basically centered on planning and infrastructure what goes into the project before you actually start. At this point, it is time to get our hands dirty and I really mean it! No matter what any external consultant tells you about the status of what is in your data, there is no excuse not to settle down for a good hard look at your data sets just to see whether they really display the characteristics you think they do. This process, a large part of which can be automated, is referred to as data profiling.

The goal of profiling data is to discover metadata when it is not available and to validate metadata when it is available. Data profiling is a process of analyzing raw data for the purpose of characterizing the information embedded within a data set. Data profiling incorporates column analysis, data type determination, and intercolumn association discovery. The result is a constructive process of information inference to prepare a data set for later integration. This chapter discusses these issues and describes the data profiling process.