The most important features available in the Matching Module are:

Record matching can be processed on both numerical and categorical data.

Extra conditions are present when setting the set identifier (id) attribute: it must have unique values, must not containt any missing values and has to be the only column of that type.

For every attribute one can specifty how AdvancedMiner should deal with missing values: ignore nulls or assume a constant similarity/distance value ("Null Similarity")

As described in the AdvancedMiner in practice chapter, model building consist of:

Choosing Weighted Matching Function Settings entails using Weighted Matching Algorithm Settings by default and Probability Matching Function Settings entails Probability Matching Algorithm SettingsS. Different Attribute Usage Sets are used for both types of function settings as well.

Function Settings

In the figure below depicting the function settings window only one parameter (maximum output size) has a direct impact on the results. Both the Weighted Matching Function Settings and Probability Matching Function Settings have the same properties window.

Figure 33.1. Matching: Function Settings

Table 33.1. Matching: Function Settings

Name	Description	Possible values	Default value
Maximum Ouput Size	the maximum size of the output set given as a percentage of the input set's size; zero means no limit; when this limit is exceeded the algorithm terminates and a warning is printed; this setting assures that a huge data set will not be accidentally created when other settings - especially blocking and cut-offs - are inappropriate	positive real numbers (e.g. 100 means 100%)	400

Wieghted Matching Algorithm Settings

Th table below describes the Algorithm Settings which are common to both Weighted and Probability Matching algorithms.

Figure 33.2. Matching: Algorithm Settings

Table 33.2. Matching: Algorithm Settings

Name	Description	Possible values	Default value
Case sensitive matching/grouping	if enabled then all the matching/grouping comparisons will be case sensitive	boolean	true
CutOff Value	pairs with a score above that value will be written to output, the rest will be suppressed	positive real numbers	1.0
Intervals Count	Used to create a score histogram for viewing in Freq; setting to zero causes creating the histogram only for values greater than CutOff	positive integer numbers	100
Ignore NULL in groups	If set to true algorithm will ignore every pair with at least one NULL blocking attribute	boolean	true
Square similarity	[EXPERIMENTAL FEATURE] If set to true the square value of the similarity for every pair of compared attributes is taken	boolean	false

The Weighted Matching Algorithm Settings contains one additional property.

Table 33.3. Data Quality: Weighted Matching Algorithm Settings

Name	Description	Possible values	Default value
Normalize score	If set to true the score values are normalized to range <0,1>	boolean	false

If the Probability Matching Algorithm Settings is chosen some other properties may be set, which are described in the table below.

Figure 33.3. Data Quality: Probability Matching Algorithm Settings

Table 33.4. Data Quality: Probability Matching Algorithm Settings

Name	Description	Possible values	Default value
Use Values Frequency	if set to false (not set) then the u probability value is estimated as 1/N where N is the number of different values for the given attribute; if set to true the u probability is based on the frequency of values for the given attribute	boolean	true

Attributes usage sets

The table below describes the attribute usage set common to both Weighted and Probability matching algorithms

Table 33.5. Data Quality: Attributes Usage Set

Name	Description	Possible values	Default value
Name	attribute name	not editable	attribute's name
Usage	how should the given attribute be used	id, match, groupBetween, block, inactive	match
Group	not used
Algorithm	the matching algorithm if usage is set to match	qgram2, qgram3, jaro, strict, levenshtein, zip, custom	jaro
Null Similarity	When a value is compared with a missing value how should it be treated? This field's value is returned as the comparison result.	positive real number from the range <0,1>	0.5
Min Similarity	When two records are compared using on of the attributes the comparison result must be greater or equal to this minimal similarity value; in other case the processed pair will be classified as a non-match	positive real number	0
Custom Matching	code for the custom matching procedure.	text: Java code	empty
Use Custom Preprocessing	Is there any preprocessing to be done on that attribute values?	boolean	false
Custom Preprocessing	The implementation of the custom preprocessing algorithm	text: Java code	empty

Figure 33.4. Data Quality: Attributes Usage Set (Weighted Matching)

Table 33.6. Data Quality: Attributes Usage Set (Weighted Matching)

Name	Description	Possible values	Default value
Weight	How important is the given attribute; the comparison result (similarity value) will be multiplied by that weight	positive real number	1.0

Figure 33.5. Data Quality: Attributes Usage Set (Probability Matching)

Table 33.7. Data Quality: Attributes Usage Set (Probability Matching)

Name	Description	Possible values	Default value
M Prob	the probability that when the values agree the two compared records are really a match; it is the measure of how trustworthy the data is	a real number from the range (0,1)	0.95

Usage

The possible values for that field are:

• id - use as the unique identifier; there can be only one field with that type of usage set; the attribute's values must be unique and there can't be any missing values
• match - use this attribute as a matching field; user should also choose the comparison method (Algorithm column)
• block - divide the data set into blocks grouping by this attribute
• groupBetween - divide the data set into blocks and compare the elements between different blocks; it can be set only for one attribute and additional setup has to be done using methods available only from a Gython script
• inactive - don't use this attribute it any way

Algorithm

Possible values for that field are:

• jaro - the jaro metric
• qgram2 - the 2-gram metric
• qgram3 - the 3-gram metric
• strict - strict (char by char) string comparison
• levenshtein - the Levenshtein metric
• zip - the length of the compressed representation of a given string (for details see here and here)
• custom - an attributes comparator is written in Java by the user; at this time it is available only via Gython scripts and not in GUI

The result of the matching process is a table with a number of columns among which the three ones given below are always present:

from specialized.dataquality import *
                

matching_results(buildTaskName, sourceData, sourceDataId=None, cutOff = 0.0, dbType='SQLITE', silent=0)