Chapter 33. Matching (Data Quality)

Table of Contents

Introduction
Method description
Blocking indexes
Attribute similarity evaluations
Record classification
Usage
Features
Data requirements
Model building and testing
Model Application
Examples
References

Introduction

Data Quality is one of the problems to deal with when it comes to data mining. It includes among others data standardization and records deduplication.

Data standardization means transforming the given textual data, dates or numbers from various formats into one chosen format, e.g. "Warszawa" and "Wa-wa" (city name) or "Al. Jerozolimskie" and "JEROZOLIMSKIE" (street name) should be replaced with one uniform value. Another example could be replacing different notations of a bank's name with one chosen notation.

Data deduplication means identifying and removing duplicates from a given dataset in order to avoid statistical inaccuracy or a trivial error of sending more than one copy of a message to one person. It is common practice to standardize the dataset prior to deduplication.

The above tasks can be carried out with a combination of various algorithms which are part of the Matching module.