Understanding Operational Processes

Master index applications created by Sun Master Index use a custom Java API library to transform and route data into and out of the master index database. In order to customize the way the Java methods transform the data, it is helpful to understand the logic of the primary processing functions and how messages are typically processed through the master index system.

This chapter describes and illustrates the processing flow of messages to and from the master index application, providing background information to help design and create custom processing rules for your implementation. This chapter discusses the following topics:

• Learning About Message Processing

• Inbound Message Processing

• Outbound Message Processing

• Inbound Message Processing Logic

• Primary Function Processing Logic

Learning About Message Processing

This section provides a summary of how inbound and outbound messages can be processed in a master index application. A master index application cross-references records stored in various computer systems of an organization and identifies records that might represent or do represent the same object. The master index application uses web services to connect to and share data with these external systems.

Figure 1 illustrates the flow of information through a master index application that includes a JMS Topic to which updates to the index are published.

Figure 1 Master Index Processing Flow

Inbound Message Processing

An inbound message refers to the transmission of data from external systems to the master index database. These messages can be sent into the database via a number of web services. Inbound messages can be stored in journal files and tracked in the log files. The steps below describe how inbound messages are processed.

1. Messages are created in an external system, and the message is transmitted to the Enterprise Service Bus (ESB) via that system's binding service.

2. The ESB identifies the message and the appropriate process to which the message should be sent. The message is then routed for processing.

3. The message is modified into the appropriate format for the master index database, and validations are performed against the data elements of the message to ensure accurate delivery. The message is typically validated using the business process logic and other information stored in the master index configuration files.

4. If the message was successfully transmitted to the database, the appropriate changes to the database are processed.

5. After the master index application processes the message, an enterprise-wide universal identifier (EUID) is returned (for a new or updated record). That EUID can be sent back out through a different web service to the external system. Alternatively, the entire updated message can be published using the generated outbound message (see Outbound Message Processing).

Figure 2 below illustrates the flow of a message inbound to an Sun Master Index application.

Figure 2 Inbound Message Processing Data Flow

About Inbound Messages

The format for the bulk of the inbound message is defined by the object structure of the master index, located in the object.xml configuration file. The inbound messages can either conform to the required format for the master index application, or they can be mapped to the correct format in a business process.

In addition to the objects and fields defined in object.xml, you can include standard master index application fields. For example, you must include the system and local ID fields, and you can also include transaction information, such as the date and time of the transaction, the transaction type, user ID, and so on. Transaction fields include the following:

• Message Id

• EventTypeCode

• UserId

• AssigningSystem

• Source

• Department

• TerminalId

• DateOfEvent

• TimeOfEvent

  If you do not send these fields into the master index application, default values are used (for example, the date and time fields default to the date and time the transaction is processed by the master index application).

Outbound Message Processing

An outbound message refers to the transmission of data from the master index database to any external system. Messages can be transmitted from the master index application in two ways. The first way is by transmitting the output of executeMatch (an EUID). This is described in Inbound Message Processing and is only used for messages received from external systems.

The second way is by publishing updates from the master index application to a JMS Topic, which allows you to publish complete, updated single best records (SBRs) to any system subscribing to that topic. When updates are made to the database from either external systems or the Master Index Data Manager (MIDM), the master index application generates outbound messages in the format of the outbound XSD.

A Sun master index application only publishes the outbound message to JMS Topics and not to JMS Queues.

This section describes how the second type of outbound message is processed. A JMS Topic must be defined in the application server for this type of processing to occur.

1. When a message is received from an external system or data is entered through the MIDM, the master index application processes the information and generates an XML message, which is sent to the JMS Topic that is configured to publish messages from the master index application.

2. Messages published by the JMS Topic are processed through a business process (or other client application), which uses the master index outbound XSD. The business process transforms the message into the appropriate format.

3. The message is routed using the appropriate binding component and sent to the appropriate external systems.

Figure 3 below illustrates the flow of data for a message outbound from a master index application.

Figure 3 Outbound Message Processing Data Flow

About Outbound Messages

When you customize the master index object structure and generate the master index application, an outbound schema file named outbound.xsd is created based on the object structure. This XSD is used to publish changes in the master index database to external systems via a JMS Topic. The output of the executeMatch process described earlier is an EUID of the new or updated record. You can use this EUID to obtain additional information and configure a business process to output the data, or you can process all updates in the master index application through a JMS Topic using the outbound XSD.

Outbound XSD Structure

The outbound XSD is located in the file structure for the master index application under the files-generated folder. The XSD includes transaction information along with the updated record from the master index database, and includes the following primary elements: OutMsg, SBR, SystemObject, the parent object, and any child objects. The OutMsg element defines the Event and ID. The Event field is populated with the type of transaction that created the outbound message, and the ID field is populated with the unique identification code of that transaction. The SBR element is created from object.xml. Table 1 describes the components of the SBR portion of the outbound OTD.

Outbound Message Trigger Events

When outbound messaging is enabled, the following transactions automatically generate an outbound message that is sent to the JMS Topic (if a JMS Topic has been incorporated into the master index Project).

• Activating a system record

• Activating an enterprise record

• Adding a system record

• Creating an enterprise record

• Deactivating a system record

• Deactivating an enterprise record

• Merging an enterprise record

• Merging a system record

• Transferring a system record

• Unmerging an enterprise record

• Unmerging a system record

• Updating an enterprise record

• Updating a system record

Sample Outbound Message

The following text is a sample outbound message for a master index application based on a master person index. Your outbound messages will appear differently depending on how you configure the client Project connectivity components.

<?xml version="1.0" encoding="UTF-8"?>
<OutMsg Event="UPD" ID="00000000000000004010">
  <SBR EUID="0000006501"  Status="active" CreateFunction="Add" ChildType="Customer" 
   CreateSystem="System" UpdateFunction="Update" RevisionNumber="3" CreateUser="mdm" 
   UpdateSystem="System" UpdateDate="" CreateDate="" UpdateUser="mdm">
    <SystemObject SystemCode="ORACLE" LID="2372385720" Status="active"></SystemObject>
    <SystemObject SystemCode="ORACLE" LID="2837482562" Status="active"></SystemObject>
    <SystemObject SystemCode="SAP" LID="8327423434" Status="active"></SystemObject>
    <Customer PersonCatCode="MEMBER" FirstName="ELIZABETH" FirstName_Std="ELIZABETH" 
     FirstName_Phon="E421" MiddleName="ANN" LastName="WARNER" LastName_Std="WARNER" 
     LastName_Phon="WARNAR" Suffix="" Title="DR" SSN="555999222" 
     DOB="04/14/1964 00:00:00"Death="" Gender="F" MStatus="M" Race="C" Ethnic="31" 
     Religion="AG" Language="EN" SpouseName="CRAIG" MotherName="JEN" MotherMN="SMITH" 
     FatherName="MARK" Maiden="MILLER" PobCity="MILFORD" PobState="ONTARIO" 
     PobCountry="CAN" VIPFlag="NONE" VetStatus="NONE" Status="" 
     DriverLicense="CT12941284" DriverLicenseSt="CT" Dod="" DeathCertificate="" 
     Nationality="CAN" Citizenship="CAN">
      <Alias FirstName="LIZ" MiddleName="" LastName="MILLER"></Alias>
      <Address AddressType="H" AddressLine1="1330 BLOSSOM ST." 
       AddressLine1_HouseNo="1330"AddressLine1_StDir="" AddressLine1_StName="BLOSSOM" 
       AddressLine1_StPhon="BLASAN" AddressLine1_StType="St" AddressLine2="" 
       City="SHEFFIELD" StateCode="CT" PostalCode="09876" PostalCodeExt="" 
       County="CAPE BURR" CountryCode="USA"></Address>
      <Phone PhoneType="CC" Phone="9894774477" PhoneExt=""></Phone>
    </Customer>
  </SBR>
</OutMsg>

Inbound Message Processing Logic

When records are transmitted to the master index application, one of the “execute match” methods is usually called and a series of processes are performed to ensure that accurate and current data is maintained in the database. The execute match methods include executeMatch, executeMatchUpdate, executeMatchDupRecalc, and executeMatchUpdateDupRecalc. The MIDM uses executeMatchGui. For more information about how these methods differ, refer to the Javadocs provided with Sun Master Index.

The steps performed by the standard executeMatch method are outlined below, and the diagrams on the following pages illustrate the message processing flow. The processing steps performed in your environment might vary from this depending on how you customize the business process and environment.

The steps outlined below refer to the following parameters in the master.xml file. They are described in Sun Master Index Configuration Reference ).

• OneExactMatch parameter

• SameSystemMatch parameter

• MatchThreshold parameter

• DuplicateThreshold parameter

• Update mode

  ---

  Note –

  There are several decision points in the match process that can be defined by custom logic using custom plug-ins (for more information, see Sun Master Index Configuration Reference ). These decision points are not listed in the below steps, which describe the default processing logic. Inbound Message Processing with Custom Logic provides the same steps as below with the decision points included.

  ---

  1. When a message is received by the master index application, a search is performed for any existing records with the same local ID and system as those contained in the message. This search only includes records with a status of A, meaning only active records are included. If a matching record is found, an existing EUID is returned.

  2. If an existing record is found with the same system and local ID as the incoming message, it is assumed that the two records represent the same object. Using the EUID of the existing record, the master index application performs an update of the record’s information in the database.

     • If the update does not make any changes to the object’s information, no further processing is required and the existing EUID is returned.

     • If there are changes to the object’s information, the updated record is inserted into database, and the changes are recorded in the sbyn_transaction table.

     • If there are changes to key fields (that is, fields used for matching or for the blocking query) and the update mode is set to pessimistic, potential duplicates are reevaluated for the updated record.

  3. If no records are found that match the record’s system and local identifier, a second search is performed using the blocking query. A search is performed on each of the defined query blocks to retrieve a candidate pool of potential matches.

     Each record returned from the search is weighted using the fields defined for matching in the inbound message.

  4. After the search is performed, the number of resulting records is calculated.

     • If a record or records are returned from the search with a matching probability weight above the match threshold, the master index application performs exact match processing (see Step 5).

     • If no matching records are found, the inbound message is treated as a new record. A new EUID is generated and a new record is inserted into the database.

  5. If records were found within the high match probability range, exact match processing is performed as follows:

     • If only one record is returned from this search with a matching probability that is equal to or greater than the match threshold, additional checking is performed to verify whether the records originated from the same system (see Step 6).

     • If more than one record is returned with a matching probability that is equal to or greater than the match threshold and exact matching is set to false, then the record with the highest matching probability is checked against the incoming message to see if they originated from the same system (see Step 6).

     • If more than one record is returned with a matching probability that is equal to or greater than the match threshold and exact matching is true, a new EUID is generated and a new record is inserted into the database.

     • If no record is returned from the database search, or if none of the matching records have a weight in the exact match range, a new EUID is generated and a new record is inserted into the database.

       ---

       Note –

       Exact matching is determined by the OneExactMatch parameter, and the match threshold is defined by the MatchThreshold parameter. For more information about these parameters, see Sun Master Index Configuration Reference .

       ---

  6. When records are checked for same system entries, the master index application tries to retrieve an existing local ID using the system of the new record and the EUID of the record that has the highest match weight.

     • If a local ID is found and same system matching is set to true, a new record is inserted, and the two records are considered to be potential duplicates. These records are marked as same system potential duplicates.

     • If a local ID is found and same system matching is set to false, it is assumed that the two records represent the same object. Using the EUID of the existing record, the master index application performs an update, following the process described in Step 2 earlier.

     • If no local ID is found, it is assumed that the two records represent the same object and an assumed match occurs. Using the EUID of the existing record, the master index application performs an update, following the process described in Step 2 earlier.

  7. If a new record is inserted, all records that were returned from the blocking query are weighed against the new record using the matching algorithm. If a record is updated and the update mode is pessimistic, the same occurs for the updated record. If the matching probability weight of a record is greater than or equal to the potential duplicate threshold, the record is flagged as a potential duplicate (for more information about thresholds and the update mode, see Sun Master Index Configuration Reference ).

The following flow charts provide a visual representation of the processes performed in the default configuration. Figure 4 and Figure 5 represent the primary flow of information. Figure 6 expands on update procedures illustrated in Figure 4 and Figure 5.

Figure 4 Inbound Message Processing using executeMatch

Figure 5 Inbound Message Processing (continued)

Figure 6 Record Update Expansion

Primary Function Processing Logic

The primary functions of a master index application can be performed from the MIDM or can be called from business processes, Java clients, web services, and so on. Whether potential duplicates are evaluated after a call to any of these functions is dependent on the update mode settings. Potential duplicates are only processed against the single best record (SBR) and not the system records. These functions are all defined in the MasterController and MasterControllerEJB classes, and are fully described in the Sun Master Index Javadocs. In the following diagrams, significant fields for potential duplicate processing include fields defined for matching and fields included in the blocking query used for matching. In all of the methods described below, an entry is made in the transaction history table (sbyn_transaction).

activateEnterpriseObject

This method reactivates an enterprise record. The MIDM calls this method when you reactivate a previously deactivated EUID. Since all potential duplicates were deleted when the EUID was originally deactivated, potential duplicates are always recalculated, regardless of the update mode. Figure 7 illustrates the processing steps.

Figure 7 activateEnterpriseObject Processing

activateSystemObject

This method reactivates a system record. The MIDM calls this method when you reactivate a previously deactivated system record. If the update mode is set to “pessimistic”, the application checks whether any key fields were updated in the SBR. If key fields were updated, potential duplicates are recalculated for the enterprise record. Figure 8 illustrates the processing steps.

Figure 8 activateSystemObject Processing

addSystemObject

This method adds a system record to an enterprise record. The MIDM calls this method when you add a system record to an existing enterprise record from the Record Details window. If the update mode is set to “pessimistic”, the application checks whether any key fields were updated in the SBR. If key fields were updated and the update mode is set to pessimistic, potential duplicates are recalculated for the enterprise record. Figure 9 illustrates the processing steps.

Figure 9 addSystemObject Processing

createEnterpriseObject

There are two createEnterpriseObject methods, both of which add a new enterprise record to the database and bypass any potential duplicate processing. One method takes only one system record as a parameter and the other takes an array of system records. These methods cannot be called from the MIDM and are designed for use in business processes, web services, or Java clients.

deactivateEnterpriseObject

This method deactivates an enterprise record specified by its EUID. The MIDM calls this method when you deactivate an EUID from the Record Details page. When an enterprise record is deactivated, all potential duplicate listings for that record are deleted.

deactivateSystemObject

This method deactivates a system record in an enterprise record. The MIDM calls this method when you deactivate a system record. If the enterprise record containing this system record has no active system records remaining, the enterprise record is deactivated and all potential duplicate listings are deleted. (Note that if the system record is reactivated, then the enterprise record is recreated.) If the enterprise record has active system records after the transaction and the update mode is set to “pessimistic”, the application checks whether any key fields were updated in the SBR. If key fields were updated, potential duplicates are recalculated for the enterprise record. Figure 10 illustrates the processing steps.

Figure 10 deactivateSystemObject Processing

deleteSystemObject

Unlike deactivateSystemObject, this method permanently removes a system record from an enterprise record. This method cannot be called from the MIDM. If the enterprise record containing the deleted system record has no active system records remaining, the enterprise record is deactivated (even if the enterprise record does have deactivated system records). If the enterprise record has no remaining system records after the system object is deleted, the enterprise record is also deleted. In both cases, any potential duplicate listings for that enterprise record are removed. If the enterprise record has active system records after the transaction and the update mode is set to “pessimistic”, the application checks whether any key fields were updated in the SBR. If key fields were updated, potential duplicates are recalculated for the enterprise record. Figure 11 illustrates the processing steps.

Figure 11 deleteSystemObject Processing

mergeEnterpriseObject

There are four mergeEnterpriseObject methods that merge two enterprise records (see the Javadocs provided with Sun Master Index for more information about each). The MIDM calls a merge method twice during a merge transaction. When you select records to merge and then click Preview, the method is called with the calculateOnly parameter set to true in order to display the merge result record for you to view. When you confirm the merge, the MIDM calls this method with the calculateOnly parameter set to false in order to commit the changes to the database and recalculate potential duplicates if needed. The method called by the MIDM checks the SBRs of the records involved in the merge against their corresponding SBRs in the database. If the SBRs differ, the merge is not performed since that means the records were changed by someone else during the merge process.

When this method is called with calculateOnly set to false, the application changes the status of the merged enterprise record to merged and deletes all potential duplicate listings for the merged enterprise record. If the update mode is set to pessimistic, the application checks whether any key fields were updated in the SBR of the surviving enterprise record. If key fields were updated, potential duplicates are recalculated for the enterprise record. Figure 12 illustrates the processing steps, and includes the check for SBR differences, which only occurs in two of the merge methods.

Figure 12 mergeEnterpriseObject Processing

mergeSystemObject

There are four methods that merge two system records that are either from the same enterprise record or from two different enterprise records (for more information about each method, see the Javadocs provided with Sun Master Index). The system records must originate from the same external system. The MIDM calls this method twice during a system record merge transaction. When you first click Keep LID# (where # is the heading number of the LID), the method is called with the calculateOnly parameter set to true in order to display the merge result record for you to view. When you confirm the merge, the MIDM calls this method with the calculateOnly parameter set to false in order to commit the changes to the database and recalculate potential duplicates if needed. Two of the merge methods compare the SBRs of the records with their corresponding SBRs in the database to ensure that no updates were made to the records before finalizing the merge.

When this method is called with calculateOnly set to “false”, the application changes the status of the merged system record to “merged”. If the system records were merged within the same enterprise record and the update mode is set to “pessimistic”, the application checks whether any key fields were updated in the SBR. If key fields were updated, potential duplicates are recalculated for the enterprise record.

If the system records originated from two different enterprise records and the enterprise record that contained the unkept the system record no longer has any active system records but does contain inactive system records, that enterprise record is deactivated and all associated potential duplicate listings are deleted. (Note that if the system records are unmerged, the enterprise record is reactivated.) If the enterprise record that contained the unkept system record no longer has any system records, that enterprise record is deleted along with any potential duplicate listings.

If both enterprise records are still active and the update mode is set to pessimistic, the application checks whether any key fields were updated in the SBR for each enterprise record. If key fields were updated, potential duplicates are recalculated for each enterprise record. Figure 13 illustrates the processing steps, and includes the check for SBR differences, which only occurs in two of the merge methods.

Figure 13 mergeSystemObject Processing

transferSystemObject

This transfers a system record from one enterprise record to another. This method is not called from the MIDM. If the enterprise record from which the system record was transferred no longer has any active system records (but still contains deactivated system records), that enterprise record is deactivated and any associated potential duplicate listings are removed. If the enterprise record from which the system record was transferred no longer has any system records, that enterprise record is deleted along with all associated potential duplicate listings. If both enterprise records are still active and the update mode is set to pessimistic, the application checks whether any key fields were updated in the SBR for each enterprise record. If key fields were updated, potential duplicates are recalculated for each enterprise record. Figure 14 illustrates the processing steps.

Figure 14 transferSystemObject Processing

undoAssumedMatch

This method reverses an assumed match made by the master index application, using the information from the system record that created the assumed match to create a new enterprise record. The MIDM calls this method when you confirm the transaction after selecting Undo Assumed Match. Potential duplicates are calculated for the new record regardless of the update mode. If the update mode is set to pessimistic, the application checks whether any key fields were updated in the SBR of the original enterprise record. If key fields were updated, potential duplicates are recalculated for the enterprise record. Figure 15 illustrates the processing steps.

Figure 15 undoAssumedMatch Processing

unmergeEnterpriseObject

There are two methods that unmerge two enterprise records that were previously merged. One method unmerges the record without checking to make sure the SBR of the active record was not changed by another process before finalizing the merge and one method performs the SBR check (see the Javadocs provided with Sun Master Index for more information). The MIDM calls this method twice during an unmerge transaction. When you first click Unmerge, the method is called with the calculateOnly parameter set to true in order to display the unmerge result records for you to view. When you confirm the unmerge, the MIDM calls this method with the calculateOnly parameter set to false in order to commit the changes to the database and recalculate potential duplicates.

When this method is called with calculateOnly set to false, the application changes the status of the merged enterprise record back to active and recalculates potential duplicate listings for the record. If the update mode is set to “pessimistic”, the application checks whether any key fields were updated in the SBR of the enterprise record that was still active after the merge. If key fields were updated, potential duplicates are recalculated for that enterprise record. Figure 16 illustrates the processing steps and includes the check for SBR updates.

Figure 16 unmergeEnterpriseObject Processing

unmergeSystemObject

There are two methods that unmerge two system records that had previously been merged. One method unmerges the record without checking to make sure the SBR of the active record was not changed by another process before finalizing the merge and one method performs the SBR check (see the Javadocs provided with Sun Master Index for more information). The MIDM calls this method twice during a system record unmerge transaction. When you first click Unmerge, the method is called with the calculateOnly parameter set to true in order to display the unmerge result record for you to view. When you confirm the unmerge, the MIDM calls this method with the calculateOnly parameter set to false in order to commit the changes to the database and recalculate potential duplicates if needed.

When this method is called with calculateOnly set to false, the application changes the status of the merged system record back to active. If the source enterprise record (the record that contained the merge result system record after the merge) has more than one active system record after the unmerge and the update mode is set to pessimistic, the application checks whether any key fields were updated in that record. If key fields were updated, potential duplicates are recalculated for the source enterprise record.

If the source enterprise record has only one active system, potential duplicate processing is performed regardless of the update mode and of whether there were any changes to key fields. If the update mode is set to pessimistic, the application checks whether any key fields were updated in the SBR for destination enterprise record. If key fields were updated, potential duplicates are recalculated for each enterprise record. Figure 17 illustrates the processing steps, assuming the system record unmerge involves two enterprise records and including the check for SBR updates.

Figure 17 unmergeSystemObject Processing

updateEnterpriseDupRecalc

This method updates the database to reflect new values for an enterprise record. It processes records in the same manner as updateEnterpriseObject, but provides an override flag for the update mode that allows you to defer potential duplicate processing. The MIDM does not call this method.

If the enterprise record is deactivated during the update, potential duplicates are deleted for that record. If the enterprise record was changed during the transaction but is still active and the performPessimistic parameter is set to true, the application checks whether any key fields were updated in the SBR of the enterprise record. If key fields were updated, potential duplicates are recalculated. Figure 18 illustrates the processing steps.

Figure 18 updateEnterpriseDupRecalc Processing

updateEnterpriseObject

This method updates the database to reflect new values for an enterprise record, and is called from the MIDM when you commit changes to an existing record. If the enterprise record is deactivated during the update, potential duplicates are deleted for that record. If the enterprise record is still active, was changed during the transaction, and the update mode is set to pessimistic, the application checks whether any key fields were updated in the SBR of the enterprise record. If key fields were updated, potential duplicates are recalculated. Figure 19 illustrates the processing steps.

Figure 19 updateEnterpriseObject Processing

updateSystemObject

There are two methods that update the database to reflect new values for a system record. One method updates the record without checking that there were no concurrent changes to the record, and the other method compares the SBR of the associated enterprise object in the transaction with that in the database to be sure there were no concurrent changes (see the Javadocs provided with Sun Master Index for more information). The MIDM calls the method that checks for SBR changes when you commit changes to an existing system record.

If the enterprise record is deactivated during the update, potential duplicates are deleted for that record. If the enterprise record was changed during the transaction and is still active, and the update mode is set to pessimistic, the application checks whether any key fields were updated in the SBR of the enterprise record. If key fields were updated, potential duplicates are recalculated. Figure 20 illustrates the processing steps and includes the check for SBR changes though it only occurs with one of the methods.

Figure 20 updateSystemObject Processing