Oracle Java CAPS Match Engine and the Oracle Java CAPS Match Engine

Implementing the Oracle Java CAPS Match Engine with a master index application requires some customization to the Match Field file in the master index project. You can also customize the Oracle Java CAPS Match Engine configuration files to better suit your data standardization and matching requirements.

The following topics provide information about the required customization and how the Match Field file corresponds to the configuration files.

• Master Index Components and the Oracle Java CAPS Match Engine

• Configuring the Master Index Matching Service (Repository)

Master Index Components and the Oracle Java CAPS Match Engine

Oracle Java CAPS Match Engine applications use the Oracle Java CAPS Match Engine specifically for standardization and probabilistic weighting, while the master index application determines survivorship. This process relies on the logic specified in the configuration files of the master index project and of the Oracle Java CAPS Match Engine.

The following topics provide information about how the Oracle Java CAPS Match Engine works with master index applications to standardize data and formulate matching weights.

• Searching and Matching in Oracle Java CAPS Match Engine Applications (Repository)

• Standardization and Matching Process in Master Index Applications (Repository)

• The Master Index Match String (Repository)

• Oracle Java CAPS Match Engine Field Identifiers

• Oracle Java CAPS Match Engine Match and Standardization Types

• Oracle Java CAPS Match Engine Configuration File Modifications

Searching and Matching in Oracle Java CAPS Match Engine Applications (Repository)

When a new record is passed to the master index database, the master index application selects a subset of possible matches from the database. The master index application then uses the Oracle Java CAPS Match Engine matching algorithm to assign a matching probability weight for each record in this subset (known as the candidate selection pool). To create the candidate selection pool, the master index application makes a series of query passes of the existing data, searching for matches on specific combinations of data. These combinations are defined by the blocking query, which is defined in the Candidate Select file and specified in the Threshold file.

Matching is performed on the fields included in the match string defined in the Match Field file. Each field is assigned a matching weight. The weights for each field are summed to determine the matching probability weight for the entire record (known as the composite weight). Before matching on some fields, such as the first name, the index might standardize the field based on information in the standardization files. You can customize how each field is weighted by modifying the match configuration file.

Standardization and Matching Process in Master Index Applications (Repository)

The standardization and matching processes use logic that is defined by a combination of Oracle Java CAPS Match Engine configuration files and master index configuration files. During the standardization and match processes, the following occurs.

1. The Oracle Java CAPS Match Engine receives an incoming record.

2. The Oracle Java CAPS Match Engine standardizes the fields specified for parsing, normalization, and phonetic encoding. These fields are defined in the StandardizationConfig section of the Match Field file and the rules for standardization are defined in the Oracle Java CAPS Match Engine standardization configuration files.

3. The master index application queries the database for a candidate selection pool (records that are possible matches) using the blocking query specified in the Threshold file. If the blocking query uses standardized or phonetic fields, the criteria values are obtained from the database.

4. For each possible match, the master index application creates a match string (based on the match columns in the Match Field file) and sends the string to the Oracle Java CAPS Match Engine.

5. The Oracle Java CAPS Match Engine checks the incoming record against each possible match, producing a matching weight for each. Matching is performed using the weighting rules defined in the match configuration file.

The Master Index Match String (Repository)

The data string that is passed to the Oracle Java CAPS Match Engine for match processing is called the match string and is defined in the MatchingConfig section of the Match Field file. The Oracle Java CAPS Match Engine configuration files, the blocking query, and the matching configuration are closely linked in the search and matching processes. The blocking query defines the select statements for creating the candidate selection pool during the matching process. The matching configuration defines the match string that is passed to the Oracle Java CAPS Match Engine from the records in the candidate selection pool. Finally, the Oracle Java CAPS Match Engine configuration files define how the match string is processed.

The Oracle Java CAPS Match Engine configuration files are dependent upon the match string, and it is very important when you modify the match string to ensure that the match type you specify corresponds to the correct row in the match configuration file (matchConfigFile.cfg). For example, if you are using person matching and add “MaritalStatus” as a match field, you need to specify a match type for the MaritalStatus field that is listed in the first column of the match configuration file. You must also make sure that the matching logic defined in the corresponding row of the match configuration file is defined appropriately for matching on the MaritalStatus field.

Oracle Java CAPS Match Engine Field Identifiers

The Oracle Java CAPS Match Engine breaks down fields into various components. For example, it breaks addresses into floor number, street number, street name, street direction, and so on. Some of these components are similar and are typically stored in the same field in the database. In the default configuration, for example, when the standardization engine finds a house number, rural route number, or PO box number, the value is stored in the HouseNumber database field. You can customize this as needed, as long as any field you specify to store a component is also included in the object structure defined for the master index application.

The Oracle Java CAPS Match Engine uses field identifiers to determine how to process fields that are defined for normalization or parsing. The IDs are defined internally in the match engine and are referenced in the Match Field file. The field IDs you specify for each field in the Match Field file determine how that field is processed by the standardization engine. The field IDs for person names determine how each name is normalized. The field IDs for business names specify which business type key file to use for standardization. The field IDs for addresses determine which database fields store each field component and how each component is standardized.

Table 3 lists each field component generated by the Oracle Java CAPS Match Engine along with their corresponding field IDs. You can only specify the predefined field IDs that are listed in this table.

Table 3 Standardization Field Identifiers

Oracle Java CAPS Match Engine Match and Standardization Types

Indicators are used in the Match Field file to reference the type of matching and standardization to perform on each field. You must specify one of these indicators, called match types and standardization types, for the fields you define for standardization or matching. The match types correspond to the match types listed in the first column of the match configuration file (matchConfigFile.cfg). The standardization types are defined internally in the match engine. The Oracle Java CAPS Match Engine uses these types to determine how to process each field.

Table 4 lists the default standardization types; Table 5 lists the default match types. You can modify the match type names but not the standardization type names. For more information about match and standardization types, see Master Index Match Types and Field Names (Repository) in Understanding Oracle Java CAPS Master Index Processing (Repository). Note that the match types you can specify in the Match Field file (listed in Table 5) are not the same values you specify for the Match Type field drop-down list in the wizard when you create the master index application.

Table 4 Standardization Types

The standardization types listed above correspond to the three categories of match types listed below. You can also specify miscellaneous match types, which do not correspond to any standardization types.

Table 5 Match Types

Oracle Java CAPS Match Engine Configuration File Modifications

The Oracle Java CAPS Match Engine configuration files are designed to perform very specific functions in the standardization and match processes. These files should only be modified by personnel with an understanding of the Oracle Java CAPS Match Engine and an understanding of the data integrity requirements of your organization. Modifications to both the master index configuration files and the Oracle Java CAPS Match Engine configuration files should be made while the master index application is in the preproduction stages. Modifying the files after master index application has moved into production might cause variances in matching weights and data processing.

The most common modifications to the Oracle Java CAPS Match Engine configuration files are generally in the match configuration file, where you can fine-tune the weighting process. This file defines probabilities used by the algorithm to determine a matching probability weight for each match field. You can use the match comparison functions provided by the Oracle Java CAPS Match Engine to fine-tune the matching logic in this file. Another common modification is inserting additional names or terms into category files, such as the first name category file (personFirstName*.dat).

Depending on your data requirements, you might need to modify additional standardization files. Some of the patterns files (most notably the address patterns files) are very complex and should only be modified by personnel who thoroughly understand the defined patterns and tokens. If you modify standardization files, make sure you modify them for each national domain specified in the Match Field file.

Configuring the Master Index Matching Service (Repository)

To configure a Oracle Java CAPS Match Engine application for specific data types and for the Oracle Java CAPS Match Engine, you must customize the Matching Service by modifying the Match Field file in the master index project. Configuring the matching service consists of the following four tasks.

• Master Index Standardization Configuration (Repository)

• Master Index Match String Configuration (Repository)

• Match and Standardization Engine Configuration

• Master Index Phonetic Encoder Configuration (Repository)

Master Index Standardization Configuration (Repository)

The StandardizationConfig section of the Match Field file determines which fields are normalized, parsed, or phonetically encoded and defines the nationality of the data being processed. The standardization section includes the following structures.

• Normalization Structures

• Standardization Structures (Parsing and Normalization)

• Phonetic Encoding Structures

The StandardizationConfig section defines fields that will be normalized, fields that will be parsed and normalized, and fields that will be phonetically encoded. The standardization types you specify in this section correspond to the match configuration file; the field IDs you can specify are listed in Table 3.

Normalization Structures

The normalization structure defines fields that are already parsed, but need to be normalized. It also tells the Oracle Java CAPS Match Engine where to place the normalized data in the object structure. Matching on any of these fields is determined by the match string and the logic is defined in the match configuration file.

Of the three data types processed by the Oracle Java CAPS Match Engine, only the person name data type is expected to provide information in fields that are already parsed; that is, the first, last, and middle names appear in separate fields, as do the suffix, title, and so on. The person standardization files define logic for normalizing person name fields. By default, only the names you specify for matching in the wizard are defined for normalization. You can define normalization for additional name fields, such as maiden name, spouse’s name, and so on. For each normalization structure, you must specify the national domains for the data you are processing.

Defining New Fields for Normalization

The fields you define for normalization in the Match Field file can include any name fields. If you define normalization for fields that are not currently defined for normalization in the Match Field file, make the following additional changes.

1. In the Match Field file, define the normalization structure, using the appropriate standardization type (PersonName), domain selector, and field IDs (FirstName, MiddeName, or LastName).

2. Add the new fields that will store the normalized field value to the appropriate objects in the Object Definition file.

3. If any of the normalized fields are to be used for blocking, modify the Candidate Select file by adding the new fields to the blocking query.

4. Regenerate the master index application in NetBeans to include the new fields in the database creation script, the outbound Object Type Definition (OTD), and the method OTD.

5. To specify that the new normalized fields be used for matching, do the following:

   1. Determine the match type or the match comparison function you want to use to match the normalized data, and modify the match configuration file (matchConfigFile.cfg) if needed.

   2. Add the new normalized field to the match-columns element of the MatchingConfig section of the Match Field file, making sure to use the appropriate match type from the match configuration file.

Standardization Structures (Parsing and Normalization)

The fields that must be parsed, and possibly normalized, are defined in a standardization structure in the StandardizationConfig section of the Match Field file. The standardization structure tells the Oracle Java CAPS Match Engine where to place the standardized information extracted from the parsed fields. The target fields you specify for standardization facilitate searching by the parsed values. Matching on any of these fields is determined by the match string and the logic is defined in the match configuration file.

The Oracle Java CAPS Match Engine expects business names and street address information in free-form text fields that must be parsed and normalized prior to matching. The logic for parsing and normalizing street address information is contained in the address standardization files; the logic for parsing and normalizing business names is contained in the business standardization files. You can customize the standardization of these data types by modifying the appropriate patterns file. For each standardization structure, you must specify the national domains for the data being processed.

Defining New Fields for Standardization

The fields you define for standardization in the Match Field file can include any street address or business name field. Perform the following steps if you need to define one of these field types for standardization.

1. If necessary, modify the patterns file for the type of data you are standardizing.

   You can define new input and output patterns or modify existing ones.

2. Define the standardization structure, using the appropriate standardization type (BusinessName or Address), domain selector, and field IDs (described in Table 3).

3. Add the new fields that will store the parsed or normalized data to the appropriate objects in the Object Definition file.

4. If any of the parsed or normalized fields are to be used for blocking, modify the Candidate Select file by adding the new fields to the blocking query.

5. Regenerate the master index application in NetBeans to include the new fields in the database creation script, the outbound Object Type Definition (OTD), and the method OTD.

6. To specify that the new standardized fields be used for matching, do the following:

   1. Determine the match type or the match comparison function you want to use to match the parsed data, and modify the match configuration file (matchConfigFile.cfg) if needed.

   2. Add the new standardized field to the match-columns element of the MatchingConfig section of the Match Field file, making sure to use the appropriate match type from the match configuration file.

Phonetic Encoding Structures

The fields to be phonetically encoded are defined in a phonetic encoding structure in the StandardizationConfig section ofthe Match Field file. The phonetic encoding structure tells the Oracle Java CAPS Match Engine where to place the phonetic data created from the fields that are encoded. You can define any field in the object structure for phonetic encoding.

Defining New Fields for Phonetic Encoding

The fields you define for phonetic encoding in the Match Field file can include any field.

1. Determine the type of phonetic encoder to use to convert the field.

   You can use any of the encoders described in Table 7.

2. Define the phonetic encoding structure, using the appropriate encoders.

3. Add the new fields that will store the phonetic values to the appropriate objects in the Object Definition file.

4. If any the phonetic fields are to be used for blocking, modify the Candidate Select file by adding the new fields to the blocking query.

5. Regenerate the master index application in NetBeans to include the new fields in the database creation script, the outbound OTD, and the method OTD.

Master Index Match String Configuration (Repository)

The MatchingConfig section of the Match Field file determines which fields are passed to the Oracle Java CAPS Match Engine for matching (the match string). If you are matching on fields parsed from a free-form text field, define each individual parsed field you want to use for matching. The default fields listed in the MatchingConfig section depend on the fields you specified for matching in the wizard (for Oracle Java CAPS Master Patient Index, the default fields are FirstName, LastName, DOB, Gender, and SSN).

The match types you can use for each field in this section are defined in the first column of the match configuration file. Make sure the match type you specify has the correct matching logic defined in the match configuration file.

Match and Standardization Engine Configuration

The MEFAConfig section of the Match Field file defines which standardization and match engines will be used by the master index application. By default, the master index application is already configured to use the Oracle Java CAPS Match Engine for matching and standardization. For more information, see Understanding Oracle Java CAPS Master Index Configuration Options (Repository).

Table 6 lists the elements in the Match Field file that define the match and standardization engine, along with the appropriate values for the Oracle Java CAPS Match Engine.

Table 6 Oracle Java CAPS Match Engine Standardization and Match Classes

Master Index Phonetic Encoder Configuration (Repository)

The Oracle Java CAPS Match Engine supports several phonetic encoders, which are defined in the PhoneticEncodersConfig section of the Match Field file. Any encoders specified in the phonetic encoding structures (see Phonetic Encoding Structures) must also be defined in the PhoneticEncodersConfig section. The classes for the encoders are listed in Table 7.

• Soundex - This algorithm is an industry standard for phonetically encoding first names.

• French Soundex - This algorithm is based on the Soundex algorithm, but is customized for French characters and names.

• Refined Soundex - This algorithm is similar to the Soundex algorithm, but is optimized for spell checking.

• NYSIIS - This algorithm is an industry standard for phonetically encoding last names.

• Metaphone - This algorithm is similar to the Soundex algorithm, but is better at identifying words that sound similar. This encoder is limited to encoding a single word in ASCII format containing only characters in the A - Z range. No punctuation or numbers can be in the input string.

• Double Metaphone - This algorithm is an improvement on the Metaphone algorithm, at times returning two encodings for a word that could have multiple pronunciations.

Table 7 Phonetic Encoder Classes for the Oracle Java CAPS Match Engine