Overview

If we were to assess the size of the data sets that are common in any major industry, we would not be surprised at how much data is being created, collected, and stored, even before considering aggregating this data into any sort of data warehouse. It would not be unusual to see large customer databases, accompanied by transaction data sets (e.g., orders, call detail records [CDRs], insurance policies) that are one or two orders of magnitude larger. For example, it would not be unreasonable to expect a telecommunications company to log millions, if not billions, of CDRs each day; a data set containing a year's worth of CDRs can easily exceed a terabyte of data. Add in party reference data, customer service detail records, order transactions, service orders, billing, shipping, etc., and we have the makings of a huge data integration and analysis problem.

When we combine all that information into an analytical framework, we are presented with two major barriers to the timely capture, analysis, and exploitation of information. The first is that the sheer amount of information available overwhelms any individual's ability to understand it, and the second is that the time to complete the amount of processing needed to digest that information is likely to exceed the window of opportunity for exploiting any results.

A successful business intelligence (BI) strategy encompasses more than just the desired analytical functionality. It must also incorporate expectations about the timeliness of the applications. Luckily, there have been significant innovations in...