Business Value in an Ocean of Data: Data Mining from a User Perspective电子书

Table of Contents

Foreword

Recommendations

1. Data Mining in General

1.1. Why Data Mining?

1.2. Technology for Uncovering Hidden Information

1.3. Pitfalls and Difficulties

1.3.1. Popular Beliefs About Data Mining

1.3.2. Analytical Booby Traps

Ideal Types

Siren Voices of the Mining Tool

Non-usable Rules

Overfitting

1.4. Data Mining and Business

Business Understanding

Expertise

1.4.1. Tools

1.4.2. Business Value

1.4.2. Defining the Business Objective

1.5. Data Mining and Statistics

1.6. Data Mining and Conventional Data Analytics

1.7. Data Mining and Manual Analysis

1.8. Business Value in an Ocean of Data

1.9. The Structure of This Book

1.10. The Authors of This Book

2. Business Intelligence and Data Warehouses

2.1. Business Intelligence (BI)

2.1.1. Positioning of Data Mining Within the BICC

2.1.2. The Future of BI

2.2. Data Warehouses. From Source Systems to Data Mining — Data Warehouses and Data Marts

2.2.1. OLTP

2.2.2. Data Warehouse – The Basics

2.2.3. Subject Orientation

2.2.4. Integration

2.2.5. Storing Data from Several Periods

2.2.6. Non-volatility

2.2.7. Accessibility

2.2.8. Transformation

2.2.9. Management-driven

2.2.10. Sources for a Data Warehouse

2.2.11. How Does Data Reach the Data Warehouse?

2.2.12. Normalization, Denormalization and Redundancy

2.2.13. Data Warehouse – Central Layer

2.3. Metadata

2.3.1. Metadata

2.3.2. Metadata Storage

2.3.3. Requirements for a Metadata Maintenance Tool

2.4. Exploitation

2.4.1. Data Marts

2.4.2. Analytical Data Layers

2.5. OLAP

2.6. Other Exploitation Technologies

3. Lessons from Data Warehousing Projects

3.1.Establishing a Corporate Data Warehouse

3.1.1. The Goals of Implementing a Data Warehouse

3.1.1.1. Long-term Objectives of Data Warehouse Implementation

3.1.1.2. Short-Term Objectives of Implementing the Data Warehouse

3.2.1. Definition of Successful Data Warehouse Implementation

3.1.3. Risks

3.1.4. Reasons for Data Warehouse Project Failures

3.2. The Conditions Necessary for a Successful Implementation

3.2.1. Establishing a BI Strategy As the First Step of the Implementation

3.2.2. Ongoing Support from Senior Management, Strong Sponsor

3.2.3. Iterative Implementation

3.2.4. Implementation As a Common Task for Business Units and IT

3.2.4.1. Users Have a Key Role During Implementation

3.2.4.2. Selecting Key Users

3.2.4.3. Documentation

3.2.4.4. Applications

3.2.5. Unified Concepts, Data Quality

3.2.6. Continuous Management of User Expectations, Publication of Results

3.2.7. Establishing a BICC, a Team of Professionals, IT Operatives and a User Support Team

4. The Methodology of Data Mining Projects

4.1. Data Mining Projects

4.2. Business-Oriented View

4.3. Methodologies

4.3.1. CRISP-DM Methodology

Business Understanding

Data Understanding

Data Preparation

Modelling

Evaluation

Deployment

4.3.2. The Time Requirements of Project Steps

4.4. Success Factors for Data Mining

4.4.1. Asking the Right Question

4.4.2. Giving the Right Answer

4.4.3. Feedback

4.4.4. Consultation

5. A Typical Data Mining Problem: Predictive Modelling

5.1. Predicting the Future

5.1.1. Predictive Modelling

5.1.2. A Predictive Modelling Example

5.2. The Predictive Modelling Process

5.3. The Purpose of Predictive Modelling

5.4. Scorecard

5.5. Making Predictions Based on the Past

5.6. Setting up an Analytical Environment, Collecting Data and Definitions

5.7. Examining and Understanding the Data

5.7.1. Exploration of the Data

5.7.2. Examining Relationships

5.8. Data Modification, Preparing Data for Modelling, Modelling

5.8.1. Machine Learning

5.8.2. New Dimensions, New Distributions

5.8.3. Binning for More Accurate Model Fitting

5.8.4. Binning Techniques

5.9. Building and Applying the Model

5.9.1. Variable Selection

5.9.2. The Predictive Model and Its Evaluation

5.10. Applying the Final Model

6. Segmentation

6.1. The Purpose of Segmentation

6.2. Customer Segmentation

6.3. Elements of Customer Segmentation

6.3.1. Master Data

6.3.2. Customer Behaviour

6.4. Preparation of Data

6.5. Segmentation Base Table

6.6. Segmentation and Profiling

6.7. Segment Creation Methods

6.7.1. Clustering

6.7.2. The Disadvantages of Clustering

6.7.3. Cube-Based Segmentation

6.7.4. Segmentation Supported by Decision Trees

6.7.5. Anticipated Segmentation

6.8. Operating the Segmentation

7. Behaviour Prediction (Early Warning)

7.1. Another Task for Predictive Data Mining

7.1.1. Behaviour, and How It Is Influenced

7.1.2. Who? — The Client

7.1.3. What? — The Forms of Behaviour

7.1.4. When? — The Timing

7.1.5. Why? — Finding the Reasons

7.2. Exploring Behaviour

7.2.1. What Is Behaviour?

7.2.2. An Example of Churn Analysis

7.3. Time Series: The Foundation of Behaviour Modelling

7.3.1. Time Series

7.3.2. Seasonal Effects

7.3.3. Campaign Effects

7.3.4. Time Series Matching

7.3.5. Stability Analysis

7.4. Behaviour Modelling: The Practice

7.4.1. Analytical Base Table

7.4.2. Modelling

7.4.3. Model Assessment

8. Campaign Optimization

8.1. Why Should Marketing Campaigns Be Optimized?

8.2. The Business Goal of Campaigns

8.3. The Main Question for Campaign Optimization

8.4. Planning Campaigns

8.5. Campaign Interactions

8.6. The Utility Score

8.6.1. Income

8.6.2. Risk

8.6.3. Channel Preference

8.6.4. Client Value

8.6.5. Product Affinity

8.6.6. Churn Probability

8.6.7. Campaign History

8.6.8. Campaign Sensitivity

8.6.9. The Utility Scorecard

8.7. Excessive Communication and Exclusion

8.8. Constraints

8.8.1. Quotas

8.8.2. Channels

8.9. Allocation of Offers

8.10. The Optimal Allocation

8.10.1. Definition of the Problem

8.10.2. The Mathematical Background to Optimal Allocation

8.10.3. Problems with Optimal Allocation

8.10.4. The Sub-optimal Algorithm

8.10.5. Dates and Channels

8.10.6. Handling Exclusions and Constraints

8.10.7. An Example

8.10.8. Why Is Using the ‘Greedy Algorithm’ Not Optimal for Utility Scores?

8.10.9. The Sub-optimal Algorithm in Detail

8.10.10. Preliminary Exclusion

8.10.11. Allocation

8.10.12. Reports

8.10.12.1. Statistics

8.10.12.2. Channel Usage

8.10.12.3. Comparing Excluded and Allocated Offers

8.11. Summary

9. The Implications of Data Quality

9.1. The Business Implications of Data Quality

9.2. Data Quality Issues

9.2.1. Data Quality Requirements

9.2.2. Reference Points

9.2.3. Risks

9.3. Improving Data Quality

9.3.1. Processing Data Errors

9.3.2. Data Exploration

9.3.3. Data Profiles

9.3.4. Data Standards

9.3.5. The Life Cycle of Data Errors

9.3.6. Background Processing

9.3.7. Online Solutions

9.4. Methodologies and Algorithms for Data Cleaning

9.4.1. Cleaning Data

9.4.2. Error Characteristics: Types of Data Cleaning Errors

9.4.3. Data Flaws and Psychology

9.4.3.1. Normalisation

9.4.3.2. Pair Matching

9.4.3.3. Group Evaluation

9.4.4. Manual Verification

9.4.5. Master Records

9.5. Back-Testing

9.6. An Example of Cleaning of Master Data: A Register of Prisoners of World War II

9.6.1. Contraction

9.6.2. Normalisation

9.6.3. Classification of Pairs

10. Networks for Business Purposes

10.1. Network Perspective

10.2. About Networks

10.2.1. What Is a Network?

10.2.2. Network Types

10.2.2.1. Connectivity

10.2.2.2. Type of Nodes and Edges

10.2.2.3. The Orientation of Edges

10.2.2.4. Network Topology

10.2.2.5. Edge Weight

10.3. Business Use of Networks

10.3.1. Networks and BI

10.3.2. Applications

10.3.2.1. Sales, Cross-selling

10.3.2.2. Risk and Fraud Analysis

10.3.2.3. Churn Analysis

10.3.2.4. Other Applications

10.4. Defining Networks

10.4.1. Defining Nodes

10.4.1.1. Identification

10.4.1.2. Typifying

10.4.1.3. Attributes

10.4.2. Edge Definition

10.4.2.1. Identification

10.4.2.2. Typifying

10.4.2.3. Attributes

10.5. Preparing Network Data

10.5.1. Challenges of Tabular Data Storage

10.5.2. External Data Sources

10.5.3. Data Quality Matters Within Networks

10.5.3.1. Identification of Nodes

10.5.3.2. Identification of Edges

10.5.3.3. Matching Internal and External Data

10.5.4. Sampling Network Data

10.6. Visualisation

10.6.1. Visual Features of Edges and Nodes

10.6.2. Features of Good Visualisation

10.6.3. Functions Required of a Visualisation Tool

10.6.4. Animating Changes in the Network

10.7. Analysis

10.7.1. Basic Analytics: Indicators and Metrics

10.7.1.1. Environmental Statistics

10.7.1.2. Key Actors and Groups

10.7.2. Advanced Analytics

10.7.2.1. Path Search

10.7.2.2. Structures

10.7.3. Network Analysis Tool

10.8. Network Simulation

10.9. Storing Network Data

10.9.1. Storage in a Relational Database

10.9.2. Storing in Memory

10.9.3. Graph Databases

11. Outlook

11.1. Industrial and Commercial Applications

11.1.1. The Steel Converter and Empirical Knowledge

11.1.2. Data and Databases for Hot Rolling

11.2. Agricultural Applications

11.2.1. Satellite-Based Crop Estimation

11.2.1.1. A Method for Detailed Crop Yield Estimations

11.2.1.2. A Method for Robust Crop Yield Estimations

11.2.2. Pest Prediction

11.2.3. Wine and Bayesian Networks

11.2.4. Cows and Numbers

11.2.5. Agricultural Scenarios

11.2.6. Possibilities for Optimization

11.3. Service and Trade

11.3.1. Applications of Pattern and Sequence Finding

11.3.2. Applications in Logistics

11.3.3. Product Recommendation

11.4. Other Areas of Interest

11.4.1. Counterterrorism

11.4.2. Healthcare

11.4.3. Data Mining Competitions

Notes

Bibliography

Data Mining

Data Warehouses

Data Warehouse Projects

Predictive Modelling

Campaign Optimization

Data Quality

Networks

Outlook

Index

A

B

C

D

E

F

G

H

I

L

M

N

O

P

R

S

T

U, V, W

Introductions