Chapter List

Part Overview

• Avoiding Pitfalls

• Overcoming Obstacles

• Managing Projects to Success

Overview

Data mining initiatives sometimes fail. As with any system development project, there are many factors that can contribute to failure. Some of these factors are beyond the control of the data mining analysts (budget cuts for example), but most are the result of mistakes in methodology or technique.

The term "failure" itself requires definition. For most development projects, "failure" means that the effort did not lead to the creation of a system that met the specification. This definition doesn't really fit data mining projects, which tend to be more "exploratory" than constructive. "Failure" for a data mining project usually means one of the following two outcomes:

1. Not finding relevant patterns that are actually present in the data and could be generalized

2. Finding (and exploiting) patterns that do not generalize, are not relevant, or are not actually present in the data

Both of these failures are bad, but the second is generally worse. It could initiate the implementation of a system that, as will be discovered after much time and expense, can't possibly work.

The circumstances leading to data mining failure are ubiquitous. Some are inherent to data mining itself, so there are no simple tricks or data mining tools that will completely insulate a careless analyst from them. The application of a sound data mining process in conjunction...