Advanced Data Mining

Data Mining (or knowledge discovery in databases, KDD) is the process of automatically searching large volumes of data for patterns using tools such as classification, association rule mining, clustering, neural networks etc.

Data Mining is especially used in case of large amounts of data, e. g. for social network and shopping basket analysis, and Decision Trees.

What is the difference compared to Statistics and Econometrics? Data Mining includes many methods from the two other fields and extends them with new algorithms. In addition, Data Mining generates completely new methods of data analysis, not possible with statistical means alone.

Applications

Given the large amount of methods, algorithms and heuristics, which figure under the name of Data Mining, the applications are very numerous and diverse. Some examples from decision support for various industries, such as financial services, insurance, retail chains, the telecoms sectors, are:

• Customer segmentation to find out what customer profiles are in your customer base
• Customer scoring for credit assessment before underwriting
•  Model customer response to various direct marketing efforts and chose the best type of contact (call, e-mail, letter, mobile text message) for each customer
• Cross selling analysis and potential for new cross sales
• Churn prediction via cancellation analysis
• Credit card fraud detection
• Shopping basket analysis
• Social network analytics (community effects, Web 2.0 analytics)
• And many other application, which are not accessible to statistical methods alone …

• Assess the probability of college dropouts based on application records
• Assess whether public monetary incentives for academic institutions to admit students from poor backgrounds are working or if they need fine tuning
• Spot trends in citizen choices to adjust future policies, e. g. for anticipate the need for improved infrastructure, when commuting is increasing
• Predict which of the citizens applying for unemployment benefits are in danger of becoming long-term unemployed and offer targeted support to those individuals only
• Mine tax files for fraud detection
• And many others …

Example 1: Shopping basket analysis

One example of value added analytics only possible with methods from Data Mining is Shopping Basket Analysis: Shopping items, such as wine and pasta, which are bought more often together (so they can be found in a shopping baskets together in the same basket) have stronger "links" between each other. To increase sales, these items should be placed in nearby shelves in a super market. Based on sales slip data, Data Mining algorithms can discover all the links between products and product categories in order to help optimize the shelf layout to achieve a maximum of cross-selling in unplanned purchases by customers.

Click the image above to see a live shopping basket link analysis.

Example 2: Neural networks for financial modeling

Artificial Neural Networks (ANN for short) are imitations of human brains. They are a form of computer system with many nodes connected and interacting with each other. By exposing them to real data, they discover the most complex relationships between variables – automatically. These results and in turn used to build better prediction models, e. g. to predict stock markets. See the image below to find out how ANN compares to traditional prediction methods.

Advantages of Data Mining

As mentioned above, Data Mining is used, when Statistics or Econometrics fail. There are a number of reasons why this could be the case, e. g.:

• Large, even massive amounts of data,
• A data structure, which is too complex to be able to develop a parsimonious set of hypotheses as required for most statistical methods (e. g. regression analysis) to work properly,
• Complex and changing relationships between variables in different parts of the data space.

Data Mining at Analyx

At Analyx, we are applying the Data Mining methods to large and complex data sets. In addition we develop tailor made models for client problems in decision support in business as well as policy (see applications above).

Our experienced team of data miners and business consultants can also advice clients on how to use Data Mining methods to get new insights and discover new information in existing data sets.

© 2006 Analyx sp. z o.o. sp. k.

Back to homepage