A buzzword in the ’90s, one rarely hears mention of data mining these days. Was it a victim of its own hype or has it just been rebranded?

By Gail Balfour

Why would a store put disposable baby diapers next to six packs of beer? The question comes from an oft-quoted parable of data mining at its finest. The mid-’90s legend goes something like this: a retail chain looked at its sales data and discovered young fathers often purchase both beer and diapers on a Friday night, supposedly because they stop off to pick up diapers and happen to see the beer.

The decision: place the unlikely duo side by side on the shelves and drive sales. The result: sales of beer increased.

The story has countless versions and no one quite knows how much of it actually happened, but it perfectly illustrates the shiny promise of data mining: using sophisticated tools to comb through existing data, hidden customer buying habits would float to the surface, netting increased sales.

And yet, nobody seems to be talking about data mining these days. So what happened?

It’s about the analysis
“The concept is great. And vendors love it because it is such great marketing speak. But when you unpack what data mining is and try to implement it, it’s actually fairly complicated, which is why we don’t really see data mining in the business tech headlines anymore,” said George Goodall, senior research analyst with London, Ont.-based Info-Tech Research Group.

According to Goodall, the biggest mistake people make about data mining is thinking it is a technology process. It isn’t. Successful implementations rely on a clear business strategy, knowing what questions to ask and having someone trained to analyze the data.

When he looks at a data mining success story, the first question Goodall likes to ask is not what tool was used but who performed the analysis. Unfortunately, few companies invest in the calibre of knowledge worker required, he said. “The actual implementation is very, very difficult. There are many great tools to do data mining, but they are not crystal balls. What it really comes down to is the individual analyst,” he said. “And not every enterprise is willing to hire a creative Ph.D. in mathematics or physics to do that kind of work.”

Some sectors, however, were willing to do exactly that. Casinos in Las Vegas have had “stunning success” with data mining, Goodall said. They are able to profile customers and their behaviours, offer package deals, provide incentives and drive customer spending and margins.

“The trick there is that the technology is only one part—they have people putting a lot of effort on analysis. The good analysts are also very creative, in the sense that they have the drive to find oddities or phenomena and track down the sources. “That’s a rare combination, and certainly something that a software vendor can’t deliver.”

Loren Hicks, a Toronto-based senior management consultant, agrees with Goodall. “You are still going to have to ask [the software] definable questions. You need a culture in which you can look at numbers dispassionately. It’s very difficult to do well.”

He said a successful implementation requires a team that’s both creative and mathematical, a challenging combination because “marketing people act about 180 degrees away from how statisticians behave.”

So choose strategy
Unfortunately, it’s common for enterprises to look at data mining as a technology initiative, when in reality it should be seen almost exclusively as a business decision, Goodall said. “So it gets punted over to the CIO or the VP of technology to shepherd the process. But without defined metrics, without the strategy, it becomes very difficult to use the technology to any benefit at all.”

Therein lies the trouble, and one big reason data mining tools have gotten a bad rap.

Anne Milley, director of analytics with business-intelligence vendor SAS Institute in Carey, N.C., admits even the best data mining tool on the market still requires a thorough corporate strategy behind it.

“If a company wants to start doing more analysis and get more value from their data, but they don’t have a clearly articulated problem, then it’s very hard to get started,” she said. “People have a guided notion that they can buy a black box that will solve all their problems. But thinking is still required.”

Milley said the term “data mining” itself can be quite multidisciplinary, which may lead to some confusion as to how it is defined. “Data mining got remarketed a little bit, and to some extent, over hyped,” she said. “Really, it draws most heavily on plain garden-variety predictive modelling and dealing with noisy, messy data.”

Data mining by any other name
Far from an art in decline, however, Milley insists data mining is actually being done more than ever but under the name “business intelligence.” The market has been expanded by tools that make that critical analysis process simpler: increased usability, better data visualization and faster computation.

“More people than ever can make sense of data. I think the uptake continues and the market for analytics still has solid growth.”

Milley agreed with Goodall that implementing a data mining project should be a business decision first, but cautioned that gaining the support of executives can be challenging. “It is often a slow adoption, especially when companies are not treating their data and analytics as assets,” she said. “But when you’ve got senior management’s buy-in, you can do more with the data and be strategic, and that sets the stage for enterprise success.”

One way to prove the value of data mining is by starting small, or with a pilot project. Some companies may only need a simple tool such as an Excel spreadsheet at first, she said. “But to really wallow in your data, and understand factors that are most influencing [customer behaviours], you will need something a little more powerful.”

Seeing trees in the forest
Starting with a pilot project was just what CIBC did 12 years ago when first implementing data mining tools from SAS, said Daymond Ling, director of modelling and analytics at the bank in Toronto. It wanted to merge silos of information between the banking, credit and mortgage divisions—a large undertaking for a pilot—and a lot was riding on its success.

“We didn’t know what we would get out of it. If it turned out to be a dud, the whole project team would be disbanded. That was the risk we took,” Ling said.

But when the company started implementing the project, something unexpected happened. “We realized we were talking to the same customer over and over (across our different divisions). By focusing on the customers rather than the products, we began solving problems we were never able to solve before,” he said. “Here was this window into the customers’ lives, something you could only get by looking at the customer.”

Ling began to detect patterns of behaviour, from student loans to first jobs, to credit cards and then mortgages. “I don’t want to sound corny, but it’s kind of like being with the customer along his whole life cycle. I find that fascinating. This is not just data, these are real people.”

Hicks said it is common, when data mining is done correctly, to come up with information that no one on the team expected. That’s the whole point.

“Consumer behaviour will surprise you every time. You are looking for information that you don’t know exists, like what colour pyjamas a customer bought 15 years ago. With enough colour data to spot a buying pattern, we know if you lead, follow or don’t care about fashion trends. Then, as we do promotions, we either send you the new arrivals, the closeouts or nothing.

“But it’s still very easy to draw the wrong conclusions from data,” he said. “Never throw any data away. Storage is cheap these days.” So it sounds like data mining, in the right hands, still holds promise after all—especially if the right strategy and team is in place. But what about the tools? Many companies already have all the software they need, Goodall said.

“Take a hard look at the systems you have in place right now. Often I talk to customers who have been wooed by a [data-mining software provider] and they kind of get led down the garden path, and at the end of the day are really not getting any more benefit, other than a Web-based user interface, than what they had before. “The first step is for companies to exploit their existing databases.

IT managers can take an image of a database, normalize the image and then generate a set of standard reports. These reports can then be further analyzed using the basic capability of a tool like Excel. Microsoft Reporting Services is another starting point for enterprises with SQL Server. These approaches obviously lack the full capability of mature BI platforms but they are often all mid-sized enterprises require.

Definitions

Data mining: “The science of extracting useful information from large data sets or databases,” according to Principles of Data Mining. In business terms, that means finding hidden sales trends that, when leveraged, will increase revenue.

Data warehouse: The main repository of corporate data on which both data mining and BI processes are based. Creating a warehouse often involves scrubbing and rationalizing data to make it useful for BI processes.

Business intelligence: A related concept, BI takes a wider look at a company’s business, encompassing such factors as competitive sales data, industry trends, economic and geographical indicators, etc.