Table of Contents
Data Quality is one of the problems to deal with when it comes to data mining. It includes among others data standardization and records deduplication.
Data standardization means transforming the given textual data, dates or numbers from various formats into one chosen format, e.g. "Warszawa" and "Wa-wa" (city name) or "Al. Jerozolimskie" and "JEROZOLIMSKIE" (street name) should be replaced with one uniform value. Another example could be replacing different notations of a bank's name with one chosen notation.
Data deduplication means identifying and removing duplicates from a given dataset in order to avoid statistical inaccuracy or a trivial error of sending more than one copy of a message to one person. It is common practice to standardize the dataset prior to deduplication.
The above tasks can be carried out with a combination of various algorithms which are part of the Matching module.