4 Setting Match Field Variations and Agreement/Disagreement

System-dependent Matching

It is possible to associate a different set of match columns with different systems. For example, if System A has reliable SSN data but System B does not have reliable SSN data, execute matching with a different set of match columns that do not have SSN when the data comes from System B.

System-dependent matching matches different configurations for distinct system sources; that is, two different system sources with the same match field type (for example, FirstName) will have access to different matching configuration information. This means that the FirstName/LastName comparators for System A will have different agreement /disagreement weights when compared with those for System B.

Conditional Matching

Conditional criteria with a set of match columns are capable of determining which set of columns will be used for matching. This is especially useful when the nature of a person-identity parameter defines the match columns that are to be used. For example, if an incoming Person ID is a driver's license, "driverLicense" is used as the match field; however, if an incoming Person ID is a medical record, "medicalID" is used as the match field, and so on.

Frequency-based Matching

Frequency-based algorithms at the field-level provide frequency-based matching that result in frequency-based weights. This means that matches on less-frequent data must obtain a higher match weight than more frequent data (that is, names with higher frequencies must evaluate to lower match scores). This feature is important when we deal with an odd distribution of data. For example, if the field is a person's first name, then we may have to deal with data with very high frequency (for example, "John") and very low frequency (for example, "Alrik") in a place like California. This feature helps readjust the associated weights for the first name so that occurrences of "John" have less impact than "Alrik" in the overall weight. The customer and/or implementer must be able to set a step-down of the associated weight (as a percentage) and also manage the lists of field values to be included in a frequency based deployment. As this feature is pluggable, you can replace the default weight adjustments with plug-in classes.

Alias Matching and Field Swapping

Swapping or substituting names during matching, such as first and second names (for example, Karl Robert Els could also be known as Robert Els) so that it is possible to be match on Karl Els and Robert Els)

Swapping of first and last names (for example, James Dean) sometimes needs to be done as the first and last names have been wrongly entered into computer systems (for example, FirstName=Dean, LastName= James).

Matching of a first name must be done with occurrences in aliases as well. This means that OHMPI matches FirstName=First Name (for example, James) and FirstName=Alias (for example, Jim).

Cap for Agreement Matching

There is a maximum cap on agreement weights for a certain group of match fields. For example, if the match fields are DOB, SSN, FirstName, [address, Phone#], the group [address, Phone#] fields are given individual maximum agreement scores of 10 and 10. However, since the maximum cap for this group [address, Phone#] is also set to 10, if any one of these fields matches, the group is considered a good match candidate. In a cap group, the fields are associated similar to an 'OR' operator, and they are typically used to reduce false positives. For example, we can get false positives because a street name [address] and a home phone number [Phone#] would both match for someone else living at the same address. Since it is probable that this person would also have the same last name, we would be generating a weight that is too high for this particular situation. Only one of the weights [street name or home phone number] should be used, or the weight should be capped as the higher of the two values. This is a common problem when there are multiples of related fields used for matching. The matching process assumes that each match field is independent of the other and can break down when they are not, such as using a home phone number and street name for matching purposes.

Waterfall Matching

Waterfall Matching includes MatchSet, Conditional Matching, and System-based Matching.

Understanding the XML Elements

The Design-time configurations are stored in an XML file called match-ext.xml. The match-ext.xml file has three main elements:

• <matchSet> allows for multiples of MatchSet and properties of matchSet

• <frequencyBasedFields> defines fields whose weights are adjusted based upon the frequency of certain strings

• <fieldsSubstitution> contains a multiple set of fields whose data is swapped for matching & blocking

Sample XML File

Below is an example of the match-ext.xml file. See XML File Explanation for details.

<MatchExtConfiguration>      <processUnconditionalMatchSets>false</processUnconditionalMatchSets>      <matchSet ID="1">
          <scoreMultiplier>.9</scoreMultiplier>    <matchColumns>     <matchColumn>      <columnName>Enterprise.SystemSBR.Person.FirstName_Std</columnName>      <matchType>FirstName</matchType>      <agreementWeight>10</agreementWeight>      <disagreementWeight>-10</disagreementWeight>     </matchColumn>    <matchColumn>      <columnName>Enterprise.SystemSBR.Person.LastName_Std</columnName>      <matchType>LastName</matchType>    </matchColumn>    </matchColumns>      <conditions>       <fieldCondition>        <field>Enterprise.SystemSBR.Person.ID</field>        <value>DL</value>       </fieldCondition>       <systems>        <system>HospitalA</System>        <system>HospitalB</System>       </systems>      </conditions>      <childMatchSet capAgreement="10" capDisagreement="-10">      <matchColumns>      <matchColumn>
       <columnName>Enterprise.SystemSBR.Person.Phone.PhoneNum</columnName>
       <columnName>Enterprise.SystemSBR.Person.Phone.PhoneNum</columnName>
       <matchType>Phone</matchType>        <agreementWeight>10</agreementWeight>        <disagreementWeight>-10</disagreementWeight>      </matchColumn>      <matchColumn>        <columnName>Enterprise.SystemSBR.Person.Address.Staddress</columnName>        <matchType>StAddress</matchType>      </matchColumn>      </matchColumns>      </childMatchSet>      </matchSet>   <fieldSubstitution>    <matching>     <substituteField>      <targetField>
         Enterprise.SystemSBR.Person.FirstName
      </targetField>
      <sourceField>
         Enterprise.SystemSBR.Person.LastName
      </sourceField>      <sourceField>
         Enterprise.SystemSBR.Person.Alias.FirstName
      </sourceField>     </substituteField>    </matching>    <blocking>     <substituteField>      <targetField>Enterprise.SystemSBR.Person.FirstName_phon</targetField>      <sourceField>Person.LastName</sourceField>      <phoneticEncodertype>Soundex</phoneticEncodertype>     </substituteField>    </blocking>   </fieldSubstitution>

 <frequencyBasedFields>   <field>    <fieldName>Enterprise.SystemSBR.Person.FirstName_Std</fieldName>   </field>   <alternateMatchColumn>      <columnName>Enterprise.SystemSBR.Person.FirstName_Std</columnName>      <matchType>LastName</matchType>   </alternateMatchColumn>   </field>   <maxPercentWeightVariation>50</maxPercentWeightVariation>   <frequencyWeightReducerPlugin>
      customPackage.CustomFreuqencyReducer
   </frequencyWeightReducerPlugin>
 </frequencyBasedFields>
</MatchExtConfiguration>

XML File Explanation

This section explains the match extensions in the match-ext.xml file. When setting up matching, simply comment out the match extensions you do not want to use for matching in the match-ext.xml file.

<MatchExtConfiguration>

/** There can be multiple match sets. A matchSet has a unique String ID. Each match set is evaluated independently. The match score evaluated from all the match sets for a given tuple is the one having the highest match score. A matchSet can contain child-matchSets, but a child matchSet should be unconditional.

*/<matchSet ID="1">
  <scoreMultiplier>.9</scoreMultiplier>/** A scoreMultiplier, is multiplied with the match score computed for that set of match fields, to determine total score. */   <matchColumns>    <matchColumn>       <columnName>Enterprise.SystemSBR.Person.FirstName_Std</columnName>       <matchType>FirstName</matchType>/** agreementWeight & disagreementWeight are optional fields. The default value is chosen from matchconfig.cfg */       <agreementWeight>10</agreementWeight>       <disagreementWeight>-10</disagreementWeight>     </matchColumn>     <matchColumn>       <columnName>Enterprise.SystemSBR.Person.LastName_Std</columnName>       <matchType>LastName</matchType>     </matchColumn>   </matchColumns>/* 

The given match set has a condition associated with it. A given pair of tuples is compared using a particular matchSet only if Conditions for this matchSet is evaluated to true. Conditions can be composed of many individual fieldCondition instances. Each fieldCondition has an implicit "AND" operator with other fieldCondition instances. A fieldCondition is true only if the field at run time has a value that is specified in a value element in the fieldCondition. The set of fieldConditions also has an implicit "AND" with systems. However, the set of systems within the systems element has an implicit "OR" operator. So this particular matchSet is run through the match engine only if the given Person object at run time has ID="DL" and belongs to either system HospitalA or HospitalB.

The conditions element can be empty, which implies that it is an unconditional matchSet. Similarly, the systems element can be empty, which means this "conditions" is true for all systems.

*/
<conditions>  <fieldCondition>    <field>Enterprise.SystemSBR.Person.ID</field>    <value>DL</value>  </fieldCondition>  <systems>    <system>HospitalA</System>    <system>HospitalB</System>  </systems></conditions>/** childMatchSet contains set of match columns. Total cap for aggregate of agreement and disagreement weights can be specified in childMatchSet attributes*/<childMatchSet capAgreement="10" capDisagreement="-10">  <matchColumns>     <matchColumn>       <columnName>Enterprise.SystemSBR.Person.Phone.PhoneNum</columnName>       <matchType>Phone</matchType>       <agreementWeight>10</agreementWeight>       <disagreementWeight>-10</disagreementWeight>     </matchColumn>     <matchColumn>       <columnName>Enterprise.SystemSBR.Person.Address.Staddress</columnName>       <matchType>StAddress</matchType>     </matchColumn>  </matchColumns></childMatchSet></matchSet>/** 

A directive to the Match Enhancer to use values from sourceField as a value for targetField during the matching of targetField. The values for fields specified in sourceField will be used in addition to the value for targetField at runtime matching. For example, If FirstName = John, lastName = Smith, and Alias.FirstName = Jack, then the first name comparisons would include the values ['John','Smith','Jack']. These can be used both at matching and for blocking.

Additionally, if PhoneticEncoder-type is specified, then that encoding is executed on the Blocking.sourceField value, before using it as additional value for targetField blocking.

*/<fieldSubstitution>  <matching>    <substituteField>      <targetField> Enterprise.SystemSBR.Person.FirstName</targetField>       <sourceField weightMultiplier=.9>
         Enterprise.SystemSBR.Person.LastName
       </sourceField>       <sourceField>
         Enterprise.SystemSBR.Person.Alias.FirstName
       </sourceField>     </substituteField>   </matching>   <blocking>     <substituteField>       <targetField>
         Enterprise.SystemSBR.Person.FirstName_ph
       </targetField>       <sourceField>
         Enterprise.SystemSBR.Person.lastName
       </sourceField>       <PhoneticEncoder-type>S oundex</PhoneticEncoder-type>     </substituteField>   </blocking> </fieldSubstitution> /**

Frequency based scoring is applied to set of <field>. A field includes elements:

• Fieldname is a fieldname whose score is reduced based upon the frequency of the given string that requires matching.

• maxPercentWeightVariation specifies the maximum percentage in the score that can be reduced, regardless of any frequency-based agreement weight reduction rules.

• FrequencyWeightReducerPlugin specifies a frequency reducer that can evaluate how much agreementWeight of a given field should be reduced based upon the frequency percentage of a given field value.

*/     <frequencyBasedFields>       <field>         <fieldName>
           Enterprise.SystemSBR.Person.FirstName_Std
         </fieldName>         <alternateMatchColumn>           <columnName>
             Enterprise.SystemSBR.Person.FirstName_Std
           </columnName>           <matchType>LastName</matchType>         </alternateMatchColumn>       <field>       <maxPercentWeightVariation>50</maxPercentWeightVariation>       <frequencyWeightReducerPlugin>
         customPackage.CustomFreuqencyReducer
       </frequencyWeightReducerPlugin>     </frequencyBasedFields>
   </MatchExtConfiguration>

Use the XML Editor to define the different MatchSets and FrequencyBasedFields.

Frequency Weight Reducer Plugin Interface

package com.sun.mdm.index.matching.matchingext;import java.util.Map;import java.util.HashMap;import java.util.List;import java.util.Set;import com.sun.mdm.index.matching.matchingext.generated.FrequencyBasedFields;import com.sun.mdm.index.matching.matchingext.generated.Field;import com.sun.mdm.index.util.ConnectionUtil;import java.sql.Connection;import java.sql.Statement;import java.sql.ResultSet;import java.sql.SQLException;/** * Computes reduction factor for a corresponding field and value based on its input frequency. This reduction factor * is used to reduce the agreement weight for such field value. String with higher frequencies  * should return a lower reduction factor. OHMPI frequency based normalizer has a reduction factor algorithm based on frequency * of a name value. So this plugin should be written only if the default agreement reduction provided by OHMPI needs to be changed. * A (reduction factor returned by this method)*(agreement Weight of a field), gives the new agreement weight for the field. *  *  * @author  sdua * @version $Revision: 1.1 $ */public interface FrequencyWeightReducer {        /**                    returns agreement weight reduction factor for a String that participates in matching process.    @param field field name for which reduction factor should be computed    @param value value for the field name. value typically would mean person's name.    @param percentFrequency percentage frequency of this field value relative to all other values.    The implementation class for this interface needs to be specified in match-ext.xml.   */   double computeReductionFactor(String field, String value, double percentageFrequency);}

Default Behavior of Frequency-based Reduction in Agreement Weights

The reduction in agreement weight due to high frequency of a word follows a linear function but which has not one adjustment rate (slope) but gives different weight adjustment rates at different frequency ranges. Below are the ranges of frequencies, for which there are corresponding ranges of agreement adjustment weights. So the lower and higher word percentFrequency corresponds to lower and higher range of agreement multiplication factor. Any weight reduction within a range of percentFrequency is proportionally reduced.

if (percentFrequency > 10%) new agreementWt = 20% of original agreementWt

else if (percentFrequency is 5% - 10%) new agreementWt = 20% - 30% of original agreeementWt

else if (percentFrequency is 1% - 5%) new agreementWt = 30% - 50% of original agreeementWt

else if (percentFrequency is .1% - 1%) new agreementWt = 50% - 70% of original agreeementWt

else if (percentFrequency is .01% - .1%) new agreementWt = 70% - 90% of original agreeementWt

Setting Up the match-ext.xml to Perform Matching

Basically all you need to do to set up matching is edit the match-ext.xml file. To do this:

1. Start NetBeans and open a project.

2. Select the Open Project icon in the NetBeans toolbar.

3. In the Open Project window select the project you want to open and click Open Project.

4. In the Projects pane (on the left side of NetBeans) and under Configuration (which is open), right-click match-ext.xml and select Edit.

   The XML Editor opens.

5. Edit the match-ext.xml file, changing the values based upon your matching needs and comment out matching that you do not want to use.

6. Save the file.

7. Deploy your project

8. Open your project in the MIDM or in a web services client.