1.3.4.3 Deduplicate

The Deduplicate processor is used to identify duplicate records in a single data stream; that is, records that represent the same entity, using sophisticated matching that does not require the original records to be precisely the same.

For an introduction to matching in EDQ, see the Matching Concept Guide. More

Use the Deduplicate processor to identify duplicate records in a data stream. Like all matching processors, Deduplicate offers the ability to match records using both automatic rules and manual decisions.

If required, a combination of automatic rules and manual decisions may also be used to create a de-duplicated data stream with all duplicate records removed. Alternatively, the output of the Deduplicate processor may be used to link duplicate records together in a system.

Deduplicate is a type of matching processor. Matching processors consist of several sub-processors, where each sub-processor performs a different step of matching, and requires separate configuration. The following sub-processors make up the Deduplicate processor, each performing a distinct function as described below.

Sub-processor	Description
Input	Select the attributes from the data streams included in the matching process.
Identify	Create identifiers to use in matching, and map them to attributes.
Cluster	Divide the data streams into clusters.
Match	Choose which comparisons to perform, and how to interpret them with match rules.
Merge	Optionally, use rules to merge matching records, to create a 'best' set of output records

Inputs

Any attributes that you want to include in the matching process.

The inputs are configurable in the Input sub-processor.

Options

All options are configured within the sub-processors above, except for the Advanced Options for Match Processors.

Outputs

The output data streams, and their attributes are configured in the Match and Merge sub-processors above.

Execution

It is possible to use a De-duplicate processor in a Real time Response process, provided the process contains only one match processor.

Calling a de-duplication match processor in this way results in special behavior on the response interface. See the Real time matching concept guide. More

Execution Mode	Supported
Batch	Yes
Real time Monitoring	Yes
Real time response	Yes

Note:

The Deduplicate processor always appears with a re-run marker, indicating that it will be completely re-executed each time the process is run, regardless of whether or not its configuration has changed. This will also mean that processors that are downstream of the Deduplicate processor will also need to be rerun.

Results Browsing

The Deduplicate processor produces a number of views of results as follows. Any of the views may be seen by clicking on the Deduplicate processor in the process. The views may also be seen by expanding the Deduplicate processor to view its sub-processors, and selecting the sub-processor that produces the view.

Input View (produced by Input)

The Input View shows a simple view of the input data stream (to be de-duplicated), and its selected attributes.

Cluster Views (produced by Cluster)

A Cluster View is displayed for each configured cluster. Use these views to assess the sensitivity of your clustering, to ensure you are not making too many redundant comparisons, and not missing any potential matches. See the Clustering concept guide for further information. More

Statistic	Meaning
Cluster	Each distinct cluster key value
Group size	The total number of records in the cluster; that is, the number of records with the same distinct cluster key value
Processed?	Indicates whether or not this cluster was actually processed. Values can be: Yes Skipped - cluster size limit Skipped - comparison limit
[Data stream name]	For each input data stream: A drillable count of the records in each cluster from each input data stream

Matching View (produced by Match) [Match Review only]

The Matching View summarizes how many duplicate records were found in the data stream:

Statistic	Meaning
Matching records	The total number of duplicate records; that is, records that were matched to other records with Match relationships. Note that this does not include records matched to others with Review relationships, unless the advanced option to Use Review relationships in Match Groups is ticked. See Use review relationships in match groups [Match Review only].
Non-matching records	The total number of records that were not matched to other records (not identified as duplicates).

Rules View (produced by Match)

The Rules View displays a summary of the number of relationships created by each automatic match rule:

Statistic	Meaning
Rule id	The numeric identifier of the match rule.
Rule name	The name of the match rule.
Relationships	The number of relationships between records that were created by the match rule. Note that each distinct relationship between a pair of records (A and B) can only be created by a single rule. If a higher rule creates the relationship, lower rules will not apply. One of the records in a relationship may be related to another record (for example, A and C) by another rule.

Review Status View (produced by Match)

The Review Status view summarizes relationships by their review status:

Statistic	Meaning
Review Status	The review status. A row is displayed for each possible review status, as follows: Automatic match Manual match Pending Awaiting review Manual No match
Relationships	The number of relationships between records of the given review status. See note below.

Note:

The statistics in this view will update automatically based on decisions made during the review process, so the top-level statistics will always provide an up-to-date view of the review status of each relationship. However, the drilldowns to the data are generated on each run of the match processor, and will not update based on review decisions made since the last time the match processor was run. When this happens, the Results Browser informs you that the generated data that you are looking at is out-of-date.

Match Groups View (produced by Match) [Match Review only]

The Match Groups view summarizes the groups of matching (duplicate) records:

Statistic	Meaning
Match groups	The total number of groups of matching records. Drill down to see a summary of the groups by group size (in number of records). Note that the match groups will not include records matched to others with Review relationships only, unless the advanced option to Use Review relationships in Match Groups is ticked. See Use review relationships in match groups [Match Review only].
Unmatched output records	The total number of unmatched records from working tables (non-duplicate records) that were output.

Alert Groups View (produced by Match) [Case Management only]

The Alert Groups view summarizes the groups of matching records:

Statistic	Meaning
Alert groups	The total number of alert groups. Drill down to see a summary of the groups by group size (in number of records).
Records not in alerts	The total number of records from the working data that were not included in any alerts. Note that unmatched records from reference sources are not output.

Groups Output (produced by Match) [Match Review only]

The Groups Output is a Data View of the match groups created by the match processor. The groups that are output in the data view, and the attributes of the view, may vary depending on the options for the Groups Output in the Match sub-processor. For example, the data view may or may not include 'groups' which contain a single record.

Alerts Output (produced by Match) [Case Management only]

The Alerts Output is a Data View of the alerts created by the match processor. The alerts that are output in the data view, and the attributes of the view, may vary depending on the options for the Alerts Output in the Match sub-processor.

Relationships Output View (produced by Match)

The Relationships Output is a Data View of the distinct relationships (links) between pairs of records created by the match processor. The relationships that are output in the data view, and the attributes of the view, may vary depending on the options for the Relationships Output in the Match sub-processor. For example, the view may or may not include relationships formed by particular rules.

Merge Summary View (produced by Merge)

The Merge Summary view summarizes the Merge stage of match processing.

Statistic	Meaning
Succeeded	The number of groups that were merged and output successfully, without any errors in the merge process. Drill down to see a summary of the successful groups by group size (in number of records). Note that this will include 'groups' with a single record, if the Merge configuration was set up to output unrelated records.
Contained errors	The number of groups that were not merged successfully, due to errors in automatic output selection that require manual resolution. Drill down to see a summary of the unsuccessful groups by group size (in number of records).

Merged Output View (produced by Merge)

The Merged Output is a Data View of the merged output from the match processor; that is, the record set after duplicate records have been merged together. The records that are output, and their attributes, will vary depending on the options set in the Merge sub-processor.

Output Filters

The following output filters are available from the Deduplicate processor:

• Groups

• Relationships

• Deduplicated

• Clustered

• Decisions

The Groups, Relationships and Deduplicated output filters correspond with the Groups Output, Relationships Output, and Merged Output, as above.

The Clustered output filter outputs the input record(s) and the cluster values using the clustering configuration, in added ARRAY attributes. This is normally only useful for real time matching.

Decisions Inputs and Outputs

The decisions input has the following purposes:

• Importing historic match decisions that have been made in other products into EDQ. This is a one-time process. When complete, the data should be unwired from the decisions input.

• Importing match decisions that have been made (and are regularly being made) in an external review system. This should be a part of the normal run process.

The decisions output enables a full audit trail of match decisions to be stored externally.

Note:

External match review uses the relationships output, which contains the latest match decisions. The decisions output differs, as it contains all decisions that have ever been made, including old ones and those that are no longer associated with a current relationship. For this reason the Decisions output is better suited for audit purposes.

See "Importing Match Decisions" and "Exporting Match Decisions" in Oracle Fusion Middleware Using Oracle Enterprise Data Quality for additional information about using the decision inputs and outputs.