Oracle® Healthcare Master Person Index Match Engine Reference Release 3.0 E62304-01 |
|
|
PDF · Mobi · ePub |
The OHMPI Match Engine compares records using probabilistic matching to determine the proximity of the records. This comparison of records is configured at design time specific to a custom application, and is based on a set of fields and their agreement and disagreement weights.
OHMPI 2.0 introduces a host of innovative matching options, collectively referred to as power match options, to the OHMPI Match Engine, including the capability to create variations in sets of match fields and agreement/disagreement weights. The bottom-line benefit is availability of sophisticated match options that results in higher levels of granular matching and higher levels of accuracy in match results. Value substitution can be used to reduce data entry errors that cause last name and first name swapping in records. Ability to configure multiple MatchSets based on various conditions enables customers to configure powerful OHMPI solutions catering to customers' requirements for varying levels of data fields and sources of data.
This chapter includes the following sections:
OHMPI provides a pluggable interface MatcherAPI, which is a matching layer that is built on top of the Match Engine.
Note:
These features will work only if you have not modified default MatcherAPI implementation- SbmeMatcherAdapter
, that is specified in the <matcher-api> tag of the mefa.xml
file.Any previous project will work seamlessly with this release.
It is possible to associate a different set of match columns with different systems. For example, if System A has reliable SSN data but System B does not have reliable SSN data, execute matching with a different set of match columns that do not have SSN when the data comes from System B.
System-dependent matching matches different configurations for distinct system sources; that is, two different system sources with the same match field type (for example, FirstName) will have access to different matching configuration information. This means that the FirstName/LastName comparators for System A will have different agreement /disagreement weights when compared with those for System B.
Conditional criteria with a set of match columns are capable of determining which set of columns will be used for matching. This is especially useful when the nature of a person-identity parameter defines the match columns that are to be used. For example, if an incoming Person ID is a drivers license, "driverLicense" would be used as the match field; however, if an incoming Person ID is a medical record, "medicalID" would be used as the match field, and so on.
Frequency-based algorithms at the field-level provide frequency-based matching that result in frequency-based weights. This means that matches on less-frequent data must obtain a higher match weight than more frequent data (that is, names with higher frequencies must evaluate to lower match scores). This feature is important when we deal with an odd distribution of data. For example, if the field is a person's first name, then we may have to deal with data with very high frequency (for example, "John") and very low frequency (for example, "Alrik") in a place like California. This feature helps readjust the associated weights for the first name so that occurrences of "John" have less impact than "Alrik" in the overall weight. The customer and/or implementer must be able to set a step-down of the associated weight (as a percentage) and also manage the lists of field values to be included in a frequency based deployment. As this feature is pluggable, you can replace the default weight adjustments with plug-in classes.
Note:
Frequency-based matching involves calculating frequencies with representative large data that might use one of the profiler functions.Swapping or substituting names during matching, such as first and second names (for example, Karl Robert Els could also be known as Robert Els) so that it is possible to be match on Karl Els and Robert Els)
Swapping of first and last names (for example, James Dean) sometimes needs to be done as the first and last names have been wrongly entered into computer systems (for example, FirstName=Dean, LastName= James).
Matching of a first name must be done with occurrences in aliases as well. This means that OHMPI matches FirstName=First Name (for example, James) and FirstName=Alias (for example, Jim).
There is a maximum cap on agreement weights for a certain group of match fields. For example, if the match fields are DOB, SSN, FirstName, [address, Phone#], the group [address, Phone#] fields are given individual maximum agreement scores of 10 and 10. However, since the maximum cap for this group [address, Phone#] is also set to 10, if any one of these fields matches, the group is considered a good match candidate. In a cap group, the fields are associated similar to an 'OR' operator, and they are typically used to reduce false positives. For example, we can get false positives because a street name [address] and a home phone number [Phone#] would both match for someone else living at the same address. Since it is probable that this person would also have the same last name, we would be generating a weight that is too high for this particular situation. Only one of the weights [street name or home phone number] should be used, or the weight should be capped as the higher of the two values. This is a common problem when there are multiples of related fields used for matching. The matching process assumes that each match field is independent of the other and can break down when they are not, such as using a home phone number and street name for matching purposes.
Waterfall Matching includes MatchSet, Conditional Matching, and System-based Matching.
MatchSet defines a set of match fields along with optional agreement and disagreement weights. It also contains information about conditional matching based on systems and/or individual fields.
There are two kinds or pools of MatchSets
Conditional MatchSets: These MatchSets are associated with conditions that are based on a system associated with input data or conditions that are based on record field values. A given conditional MatchSet qualifies given input data only if its conditions satisfy input data.
Unconditional MatchSets: These MatchSets do not have any condition associated with them.
The matching system processes all conditional MatchSets first and selects the MatchSets whose conditions satisfy the input data. Afterwards the system proceeds to do actual matching of input data with the selected set of MatchSets. Each "MatchSet" matching evaluation with input data, will yield a score that is based on probabilistic matching of input data with match fields and agreement and disagreement fields specified in that MatchSet.
The mefa-ext.xml
file supports a tag called ProcessUnconditionalMatchSets which has two values: true and false.
ProcessUnconditionalMatchSets - true: After processing conditional match sets, the system proceeds to all unconditional match sets. The overall match score is the maximum of the scores evaluated from all these match sets.
ProcessUnconditionalMatchSets - false: After processing conditional MatchSets, the system processes an unconditional MatchSets pool only if none of MatchSets in the conditional MatchSets pool satisfy the input data condition. In other words, if one or more Conditional MatchSets satisfy input data condition, then the unconditional MatchSets are not processed.
Table 4-1 MatchSet, Conditional Matching, and System-based Matching Examples
MatchSet name | MatchSet ID | Description | Input 1 SO with System A | Input 2 SO with System B | Input 3 SO with System C | Input 4 SO with System E |
---|---|---|---|---|---|---|
MS – System A or B + Gender = M |
1 |
System A or B and Gender condition |
X |
X |
- |
- |
MS – System A + ColorOfHair = W |
2 |
System A and ColorOfHair condition |
X |
- |
- |
- |
MS – System C |
3 |
System C no field condition |
- |
- |
X |
- |
MS – System C or D |
4 |
System C or D no field level condition |
- |
- |
X |
- |
MS Default (added by system based on |
5 |
Default contains no system or field level conditions |
- |
- |
- |
X |
MatchSet Gender =F |
6 |
No system with condition |
- |
- |
- |
X |
MatchSet CoH = black |
7 |
No system with condition |
- |
- |
- |
X |
MatchSet condition = some other |
8 |
No system with condition |
- |
- |
- |
X |
In Table 4-1, above, X represents the matching will be performed based on weights defined in corresponding MatchSet in column 1. For example, if the system object comes in from System A, the matching engine will go through MS ID 1 and 2 and do the waterfall logic to get the maximum weight based on two sets of comparison. Similarly, if input comes from system B the only MatchSet that satisfies the system match is "MS – System A or B" ID 1 which is MS – System A or B so the matching is done based on configuration in matchset 1 and all other matchsets will be ignored including default and 6, 7, and 8. Thus there is no waterfall matching invoked.
However if input with system E comes in, notice there is no matchset defined for system E, the extension matching will go through all matchsets that have no system attached. In this case the matching will be performed based on 5,6,7,8 matchset and waterfall logic will kick in to retrieve the maximum weight out of 4 results.
The Design-time configurations are stored in an XML file called match-ext.xml
. The match-ext.xml
file has three main sections:
<matchSet>
allows for multiples of MatchSet and properties of matchSet to support conditional and System-dependent Matching
<frequencyBasedFields>
defines fields whose weights are adjusted based upon the frequency of certain strings
<fieldsSubstitution>
contains a multiple set of fields whose data is swapped for matching & blocking
The first five sections below provide information you need to be aware of, as well as an example of the XML file you will edit to set up your MatchSets. The sixth section provides a simplistic procedure on where and how to set up the match-ext.xml
file for the matching you require.
The Design-time configurations are stored in an XML file called match-ext.xml
. The match-ext.xml
file has three main elements:
<matchSet>
allows for multiples of MatchSet and properties of matchSet
<frequencyBasedFields>
defines fields whose weights are adjusted based upon the frequency of certain strings
<fieldsSubstitution>
contains a multiple set of fields whose data is swapped for matching & blocking
The MatchSet is set of matching fields and also consists of their agreement and disagreement weights. The core enhancement in the features set is using some of Match fields with their agreement/disagreement weights to match pairs of data, which results in different match score. The highest match score from all sets of match sets is taken to determine if there is an assumed match, duplicate match, or non-match.
Note:
The match threshold and the duplicate thresholds are fixed irrespective of the type of MatchSet that is used.Each MatchSet can be associated with:
A set of systems
A set of conditions based on [field = value]; each such condition has an AND operator between themselves
scoreMultiplier
When there are many MatchSets to match against, some MatchSets can be more reliable than others to find assumed or potential thresholds. This factor is multiplied with the score computed by MatchEngine to normalize it with respect to the application Match threshold and Duplicate threshold. So if scoreMultiplier is .8, and total score computed for a MatchSet is 50, then it is normalized to 40.
child MatchSet
The MatchSet can contain a child MatchSet. The child MatchSet can be used for capping agreement/disagreement weights for group of match fields. This represents the 'OR' functionality within a match set.
FrequencyBasedFields consists of:
set of fields whose max agreement weights will be normalized to a lower weight depending upon known frequency of field data in the system.
maxPercentWeightVariation
maxPercentWeightVariation is the percentage of maximum variation from the agreement weight. If it is 50%, and the agreement weight for a field is 20, then regardless of how the agreement weight was reduced via frequency based rules, the max agreementWeight reduction would be 20*.5 = 10.
Note:
This is not same score computed by the Match Engine, which depends upon a match comparison of two field values.Computation of weight reduction
A plugin interface is provided that can override default rules on how much agreement weight should be reduced with the high frequency of a word.
Note:
Based upon the frequency of a field value, it only affects the maximum agreement weight. The disagreement weight is not affected. For example, If a name has an agreement weight of +10, the disagreement weight = -10. However, when high frequency words such as "John" and "John" are matched, the total agreement weight is lowered to perhaps +5. If one name is "George" and other name is "John" they do not match, but their disagreement score would still be -10.Every name in FrequencyBasedFields has its absolute frequency recorded in a frequency table (see Table 4-2). You populate this table off-line using a custom process that is outside the scope of OHMPI.
FrequencyTable map in memory: During startup time of an OHMPI server (for example WebLogic), the frequency table is loaded in memory and their frequency percentages for each name are computed. If a name is not in memory it has low frequency and is given maximum agreement weight.
fieldsSubstitution consists of:
Value Substitution
The feature provides functionality to support swapping of the fields during the matching process. The swapping functionality helps to resolve cases in which certain field values such as Last Name is erroneously entered instead of First Name. Field substitution will allow the match to be performed based on swapped field values in addition to the normal matching process based on MatchSet.
Examples would be:
<fieldSubstitution> <matching> <substituteField> <targetField>Enterprise.SystemSBR.Person.LastName</targetField> <sourceField>Enterprise.SystemSBR.Person.FirstName</sourceField> </substituteField> </matching> <blocking> <substituteField> <targetField>Enterprise.SystemSBR.Person.LastName</targetField> <sourceField>Person.FirstName</sourceField> </substituteField> </blocking> </fieldSubstitution>
The absence of the <fieldSubstitution> element from the match-ext.xml
file is treated as if the Value Substitution feature has been turned off.
Description of Various Elements:
matching defines the various elements needed to configure the matching part of value substitution
targetField defines the target field whose value will be substituted by one or more sourceField
sourceField defines the source field that will contain input data and also be queried to get the substitute value
Alias Matching
The feature enhances existing matching by enabling matching done based on previously used names or aliases.
It is a function to match a First Name against FirstName and Aliases. The value substitution construct described above could be used to achieve this functionality. The target field would describe the actual field that needs to be matched against source fields such as Aliases and so on.
Below is an example of the match-ext.xml
file. See XML File Explanation for details.
<MatchExtConfiguration> <processUnconditionalMatchSets>false</processUnconditionalMatchSets> <matchSet ID="1"> <scoreMultiplier>.9</scoreMultiplier> <matchColumns> <matchColumn> <columnName>Enterprise.SystemSBR.Person.FirstName_Std</columnName> <matchType>FirstName</matchType> <agreementWeight>10</agreementWeight> <disagreementWeight>-10</disagreementWeight> </matchColumn> <matchColumn> <columnName>Enterprise.SystemSBR.Person.LastName_Std</columnName> <matchType>LastName</matchType> </matchColumn> </matchColumns> <conditions> <fieldCondition> <field>Enterprise.SystemSBR.Person.ID</field> <value>DL</value> </fieldCondition> <systems> <system>HospitalA</System> <system>HospitalB</System> </systems> </conditions> <childMatchSet capAgreement="10" capDisagreement="-10"> <matchColumns> <matchColumn> <columnName>Enterprise.SystemSBR.Person.Phone.PhoneNum</columnName> <columnName>Enterprise.SystemSBR.Person.Phone.PhoneNum</columnName> <matchType>Phone</matchType> <agreementWeight>10</agreementWeight> <disagreementWeight>-10</disagreementWeight> </matchColumn> <matchColumn> <columnName>Enterprise.SystemSBR.Person.Address.Staddress</columnName> <matchType>StAddress</matchType> </matchColumn> </matchColumns> </childMatchSet> </matchSet> <fieldSubstitution> <matching> <substituteField> <targetField> Enterprise.SystemSBR.Person.FirstName </targetField> <sourceField> Enterprise.SystemSBR.Person.LastName </sourceField> <sourceField> Enterprise.SystemSBR.Person.Alias.FirstName </sourceField> </substituteField> </matching> <blocking> <substituteField> <targetField>Enterprise.SystemSBR.Person.FirstName_phon</targetField> <sourceField>Person.LastName</sourceField> <phoneticEncodertype>Soundex</phoneticEncodertype> </substituteField> </blocking> </fieldSubstitution> <frequencyBasedFields> <field> <fieldName>Enterprise.SystemSBR.Person.FirstName_Std</fieldName> </field> <alternateMatchColumn> <columnName>Enterprise.SystemSBR.Person.FirstName_Std</columnName> <matchType>LastName</matchType> </alternateMatchColumn> </field> <maxPercentWeightVariation>50</maxPercentWeightVariation> <frequencyWeightReducerPlugin> customPackage.CustomFreuqencyReducer </frequencyWeightReducerPlugin> </frequencyBasedFields> </MatchExtConfiguration>
This section explains the match extensions in the match-ext.xml
file. When setting up matching, simply comment out the match extensions you do not want to use for matching in the match-ext.xml
file.
Note:
The samplematch-ext.xml
file that is delivered with OHMPI is commented out. To use this file you must remove these comments at the beginning (<!--
) and end (-->
) of the sample file. Also, you need to modify the object names and field names to your project object model and configure it based upon your business needs.<MatchExtConfiguration>
/** There can be multiple match sets. A matchSet has a unique String ID. Each match set is evaluated independently. The match score evaluated from all the match sets for a given tuple is the one having the highest match score. A matchSet can contain child-matchSets, but a child matchSet should be unconditional.
*/<matchSet ID="1"> <scoreMultiplier>.9</scoreMultiplier>/** A scoreMultiplier, is multiplied with the match score computed for that set of match fields, to determine total score. */ <matchColumns> <matchColumn> <columnName>Enterprise.SystemSBR.Person.FirstName_Std</columnName> <matchType>FirstName</matchType>/** agreementWeight & disagreementWeight are optional fields. The default value is chosen from matchconfig.cfg */ <agreementWeight>10</agreementWeight> <disagreementWeight>-10</disagreementWeight> </matchColumn> <matchColumn> <columnName>Enterprise.SystemSBR.Person.LastName_Std</columnName> <matchType>LastName</matchType> </matchColumn> </matchColumns>/*
The given match set has a condition associated with it. A given pair of tuples is compared using a particular matchSet only if Conditions for this matchSet is evaluated to true. Conditions can be composed of many individual fieldCondition instances. Each fieldCondition has an implicit "AND" operator with other fieldCondition instances. A fieldCondition is true only if the field at run time has a value that is specified in a value element in the fieldCondition. The set of fieldConditions also has an implicit "AND" with systems. However, the set of systems within the systems element has an implicit "OR" operator. So this particular matchSet is run through the match engine only if the given Person object at run time has ID="DL" and belongs to either system HospitalA or HospitalB.
The conditions element can be empty, which implies that it is an unconditional matchSet. Similarly, the systems element can be empty, which means this "conditions" is true for all systems.
*/ <conditions> <fieldCondition> <field>Enterprise.SystemSBR.Person.ID</field> <value>DL</value> </fieldCondition> <systems> <system>HospitalA</System> <system>HospitalB</System> </systems></conditions>/** childMatchSet contains set of match columns. Total cap for aggregate of agreement and disagreement weights can be specified in childMatchSet attributes*/<childMatchSet capAgreement="10" capDisagreement="-10"> <matchColumns> <matchColumn> <columnName>Enterprise.SystemSBR.Person.Phone.PhoneNum</columnName> <matchType>Phone</matchType> <agreementWeight>10</agreementWeight> <disagreementWeight>-10</disagreementWeight> </matchColumn> <matchColumn> <columnName>Enterprise.SystemSBR.Person.Address.Staddress</columnName> <matchType>StAddress</matchType> </matchColumn> </matchColumns></childMatchSet></matchSet>/**
A directive to the Match Enhancer to use values from sourceField as a value for targetField during the matching of targetField. The values for fields specified in sourceField will be used in addition to the value for targetField at runtime matching. For example, If FirstName = John, lastName = Smith, and Alias.FirstName = Jack, then the first name comparisons would include the values ['John','Smith','Jack']. These can be used both at matching and for blocking.
Additionally, if PhoneticEncoder-type is specified, then that encoding is executed on the Blocking.sourceField value, before using it as additional value for targetField blocking.
*/<fieldSubstitution> <matching> <substituteField> <targetField> Enterprise.SystemSBR.Person.FirstName</targetField> <sourceField weightMultiplier=.9> Enterprise.SystemSBR.Person.LastName </sourceField> <sourceField> Enterprise.SystemSBR.Person.Alias.FirstName </sourceField> </substituteField> </matching> <blocking> <substituteField> <targetField> Enterprise.SystemSBR.Person.FirstName_ph </targetField> <sourceField> Enterprise.SystemSBR.Person.lastName </sourceField> <PhoneticEncoder-type>S oundex</PhoneticEncoder-type> </substituteField> </blocking> </fieldSubstitution> /**
Frequency based scoring is applied to set of <field>
. A field includes elements:
Fieldname is a fieldname whose score is reduced based upon the frequency of the given string that requires matching.
maxPercentWeightVariation specifies the maximum percentage in the score that can be reduced, regardless of any frequency-based agreement weight reduction rules.
FrequencyWeightReducerPlugin specifies a frequency reducer that can evaluate how much agreementWeight of a given field should be reduced based upon the frequency percentage of a given field value.
*/ <frequencyBasedFields> <field> <fieldName> Enterprise.SystemSBR.Person.FirstName_Std </fieldName> <alternateMatchColumn> <columnName> Enterprise.SystemSBR.Person.FirstName_Std </columnName> <matchType>LastName</matchType> </alternateMatchColumn> <field> <maxPercentWeightVariation>50</maxPercentWeightVariation> <frequencyWeightReducerPlugin> customPackage.CustomFreuqencyReducer </frequencyWeightReducerPlugin> </frequencyBasedFields> </MatchExtConfiguration>
Use the XML Editor to define the different MatchSets and FrequencyBasedFields.
package com.sun.mdm.index.matching.matchingext;import java.util.Map;import java.util.HashMap;import java.util.List;import java.util.Set;import com.sun.mdm.index.matching.matchingext.generated.FrequencyBasedFields;import com.sun.mdm.index.matching.matchingext.generated.Field;import com.sun.mdm.index.util.ConnectionUtil;import java.sql.Connection;import java.sql.Statement;import java.sql.ResultSet;import java.sql.SQLException;/** * Computes reduction factor for a corresponding field and value based on its input frequency. This reduction factor * is used to reduce the agreement weight for such field value. String with higher frequencies * should return a lower reduction factor. OHMPI frequency based normalizer has a reduction factor algorithm based on frequency * of a name value. So this plugin should be written only if the default agreement reduction provided by OHMPI needs to be changed. * A (reduction factor returned by this method)*(agreement Weight of a field), gives the new agreement weight for the field. * * * @author sdua * @version $Revision: 1.1 $ */public interface FrequencyWeightReducer { /** returns agreement weight reduction factor for a String that participates in matching process. @param field field name for which reduction factor should be computed @param value value for the field name. value typically would mean person's name. @param percentFrequency percentage frequency of this field value relative to all other values. The implementation class for this interface needs to be specified in match-ext.xml. */ double computeReductionFactor(String field, String value, double percentageFrequency);}
The reduction in agreement weight due to high frequency of a word follows a linear function but which has not one adjustment rate (slope) but gives different weight adjustment rates at different frequency ranges. Below are the ranges of frequencies, for which there are corresponding ranges of agreement adjustment weights. So the lower and higher word percentFrequency corresponds to lower and higher range of agreement multiplication factor. Any weight reduction within a range of percentFrequency is proportionally reduced.
if (percentFrequency > 10%) new agreementWt = 20% of original agreementWt
else if (percentFrequency is 5% - 10%) new agreementWt = 20% - 30% of original agreeementWt
else if (percentFrequency is 1% - 5%) new agreementWt = 30% - 50% of original agreeementWt
else if (percentFrequency is .1% - 1%) new agreementWt = 50% - 70% of original agreeementWt
else if (percentFrequency is .01% - .1%) new agreementWt = 70% - 90% of original agreeementWt
Basically all you need to do to set up matching is edit the match-ext.xml
file. To do this:
Start NetBeans and open a project.
Select the Open Project icon in the NetBeans toolbar.
In the Open Project window select the project you want to open and click Open Project.
In the Projects pane (on the left side of NetBeans) and under Configuration (which is open), right-click match-ext.xml and select Edit.
The XML Editor opens.
Edit the match-ext.xml
file, changing the values based upon your matching needs and comment out matching that you do not want to use.
Save the file.
Deploy your project
Open your project in the MIDM or in a web services client.
This section provides sample files of mefa.xml
and matchConfigFile.cfg
.
Note:
There are no changes in themefa.xml
and matchConfigFile.cfg
files.<match-system-object> <object-name>MPI1</object-name> <match-columns> <match-column> <column-name>Enterprise.SystemSBR.MPI1.FirstName_Std</column-name> <match-type>FirstName</match-type> </match-column> <match-column> <column-name>Enterprise.SystemSBR.MPI1.LastName_Std</column-name> <match-type>LastName</match-type> </match-column> </match-columns> <match-system-object>
match-columns
, which is defined in the current mefa.xml
file, forms a default MatchSet.
If the agreements and disagreements for any match-columns are not defined in mefa_ext.xml
, they would be used from matchConfigFile.cfg
.
Sample of Existing matchConfigFile.cfg
PrimaryName 30 0 us 0.9 0.001 13 -2 StreetName 25 0 us 0.9 0.001 11 0 HouseNumber 8 0 un 0.9 0.001 11 0 StreetDir 15 0 u 0.9 0.001 7 -2 StreetType 10 0 u 0.9 0.001 7 -2 FirstName 15 0 uf 0.99 0.001 10 -4 LastName 15 0 ul 0.99 0.001 10 -4 String 25 0 us 0.99 0.001 10 -10
The mefa.xml
and matchconfig.cfg
files contain the default set of agreement and disagreement weights. Each individual MatchSet has its own set of agreement weights which override the default weights in the mefa.xml
file.
The comparators for match types are stored in MatchEngine and are not configurable from mefa_ext.xml
.
Note:
There is no direct mapping betweenmefa_ext.xml
source field (match column) such as Enterprise.SystemSBR.Person.FirstName_Std
and the comparator defined in matchConfigFile.cfg
. This mapping is provided via the matchType column in mefa_ext.xm
l and match type in matchConfigFile.cfg
.Any project created in a previous release of OHMPI will work seamlessly with this release.
To use a project from a previous release without any of the new matching features for a component in this release:
Import the previous project and re-deploy it.
To use a project from a previous release with matching in this release:
Import the previous project, configure it for the new matching, regenerate, clean and build, and then re-deploy it.
Note:
When you import (that is copy) a project from a previous release of OHMPI into this release of OHMPI, the project from the previous release has matching and standardization as created in the previous release. To use the matching and standardization features that come with this release you need to create a new project, which in turn creates matching and standardization. You then need to copy the new matching and standardization files that you just created into the appropriate directory of the project that you have imported into this release of OHMPI.If you import a previous project, you also need to copy the sbyn_frequency table that is associated with the previous project, as the sbyn_frequency table is created when a project is created.