Sun Match Engine applications use the Sun Match Engine specifically for standardization and probabilistic weighting, while the master index application determines survivorship. This process relies on the logic specified in the configuration files of the master index project and of the Sun Match Engine.
The following topics provide information about how the Sun Match Engine works with master index applications to standardize data and formulate matching weights.
Searching and Matching in Sun Match Engine Applications (Repository)
Standardization and Matching Process in Master Index Applications (Repository)
When a new record is passed to the master index database, the master index application selects a subset of possible matches from the database. The master index application then uses the Sun Match Engine matching algorithm to assign a matching probability weight for each record in this subset (known as the candidate selection pool). To create the candidate selection pool, the master index application makes a series of query passes of the existing data, searching for matches on specific combinations of data. These combinations are defined by the blocking query, which is defined in the Candidate Select file and specified in the Threshold file.
Matching is performed on the fields included in the match string defined in the Match Field file. Each field is assigned a matching weight. The weights for each field are summed to determine the matching probability weight for the entire record (known as the composite weight). Before matching on some fields, such as the first name, the index might standardize the field based on information in the standardization files. You can customize how each field is weighted by modifying the match configuration file.
The standardization and matching processes use logic that is defined by a combination of Sun Match Engine configuration files and master index configuration files. During the standardization and match processes, the following occurs.
The Sun Match Engine receives an incoming record.
The Sun Match Engine standardizes the fields specified for parsing, normalization, and phonetic encoding. These fields are defined in the StandardizationConfig section of the Match Field file and the rules for standardization are defined in the Sun Match Engine standardization configuration files.
The master index application queries the database for a candidate selection pool (records that are possible matches) using the blocking query specified in the Threshold file. If the blocking query uses standardized or phonetic fields, the criteria values are obtained from the database.
For each possible match, the master index application creates a match string (based on the match columns in the Match Field file) and sends the string to the Sun Match Engine.
The Sun Match Engine checks the incoming record against each possible match, producing a matching weight for each. Matching is performed using the weighting rules defined in the match configuration file.
The data string that is passed to the Sun Match Engine for match processing is called the match string and is defined in the MatchingConfig section of the Match Field file. The Sun Match Engine configuration files, the blocking query, and the matching configuration are closely linked in the search and matching processes. The blocking query defines the select statements for creating the candidate selection pool during the matching process. The matching configuration defines the match string that is passed to the Sun Match Engine from the records in the candidate selection pool. Finally, the Sun Match Engine configuration files define how the match string is processed.
The Sun Match Engine configuration files are dependent upon the match string, and it is very important when you modify the match string to ensure that the match type you specify corresponds to the correct row in the match configuration file (matchConfigFile.cfg). For example, if you are using person matching and add “MaritalStatus” as a match field, you need to specify a match type for the MaritalStatus field that is listed in the first column of the match configuration file. You must also make sure that the matching logic defined in the corresponding row of the match configuration file is defined appropriately for matching on the MaritalStatus field.
The Sun Match Engine breaks down fields into various components. For example, it breaks addresses into floor number, street number, street name, street direction, and so on. Some of these components are similar and are typically stored in the same field in the database. In the default configuration, for example, when the standardization engine finds a house number, rural route number, or PO box number, the value is stored in the HouseNumber database field. You can customize this as needed, as long as any field you specify to store a component is also included in the object structure defined for the master index application.
The Sun Match Engine uses field identifiers to determine how to process fields that are defined for normalization or parsing. The IDs are defined internally in the match engine and are referenced in the Match Field file. The field IDs you specify for each field in the Match Field file determine how that field is processed by the standardization engine. The field IDs for person names determine how each name is normalized. The field IDs for business names specify which business type key file to use for standardization. The field IDs for addresses determine which database fields store each field component and how each component is standardized.
Table 3 lists each field component generated by the Sun Match Engine along with their corresponding field IDs. You can only specify the predefined field IDs that are listed in this table.
Table 3 Standardization Field Identifiers
Indicators are used in the Match Field file to reference the type of matching and standardization to perform on each field. You must specify one of these indicators, called match types and standardization types, for the fields you define for standardization or matching. The match types correspond to the match types listed in the first column of the match configuration file (matchConfigFile.cfg). The standardization types are defined internally in the match engine. The Sun Match Engine uses these types to determine how to process each field.
Table 4 lists the default standardization types; Table 5 lists the default match types. You can modify the match type names but not the standardization type names. For more information about match and standardization types, see Master Index Match Types and Field Names (Repository) in Understanding Sun Master Index Processing (Repository). Note that the match types you can specify in the Match Field file (listed in Table 5) are not the same values you specify for the Match Type field drop-down list in the wizard when you create the master index application.
Table 4 Standardization Types
This indicator ... |
processes this data type ... |
---|---|
Address | |
PersonName |
Pre-parsed name fields (including any first, middle, last, or alias names). |
BusinessName |
The standardization types listed above correspond to the three categories of match types listed below. You can also specify miscellaneous match types, which do not correspond to any standardization types.
Table 5 Match Types
This indicator ... |
processes this data type ... |
---|---|
The parsed name field of a business name. |
|
The parsed organization type field of a business name. |
|
The parsed association type field of a business name. |
|
The parsed alias type field of a business name. |
|
The parsed industry sector field of a business name. |
|
The parsed industry type field of a business name. |
|
The parsed URL field of a business name. |
|
The parsed street name field of a street address. |
|
The parsed house number field of a street address. |
|
The parsed street direction field of a street address. |
|
The parsed street type field of a street address. |
|
A first name field, including middle name, alias first name, and alias middle name fields. |
|
A last name field, including alias last name fields. |
|
The day, month, and year of a date field. |
|
The month and year of a date field. |
|
The hour, day, month, and year of a date field. |
|
The minute, hour, day, month, and year of a date field. |
|
The seconds, minute, hour, day, month, and year of a date field. |
|
A generic string field. |
|
A numeric field. |
|
A field containing integers. |
|
A field containing real numbers. |
|
A field containing a social security number. |
|
A field containing a single character. |
|
Any field on which you want the Sun Match Engine to use prorated weights. |
|
Any field you want the Sun Match Engine to match character for character. |
The Sun Match Engine configuration files are designed to perform very specific functions in the standardization and match processes. These files should only be modified by personnel with an understanding of the Sun Match Engine and an understanding of the data integrity requirements of your organization. Modifications to both the master index configuration files and the Sun Match Engine configuration files should be made while the master index application is in the preproduction stages. Modifying the files after master index application has moved into production might cause variances in matching weights and data processing.
The most common modifications to the Sun Match Engine configuration files are generally in the match configuration file, where you can fine-tune the weighting process. This file defines probabilities used by the algorithm to determine a matching probability weight for each match field. You can use the match comparison functions provided by the Sun Match Engine to fine-tune the matching logic in this file. Another common modification is inserting additional names or terms into category files, such as the first name category file (personFirstName*.dat).
Depending on your data requirements, you might need to modify additional standardization files. Some of the patterns files (most notably the address patterns files) are very complex and should only be modified by personnel who thoroughly understand the defined patterns and tokens. If you modify standardization files, make sure you modify them for each national domain specified in the Match Field file.