Data Replication
Introduction
The focus of this part of the document is synchronization or replication of data between various Oracle Health Insurance applications.
An example of this is the replication of person data in Policies to Claims. Both applications make use of the same application or sub system for maintaining that data. Customers could use the Persons HTTP API message based service or the Data File Set interface for maintaining person data in both Policies and Claims.
To prevent customers from being burdened with having to synchronize person data across multiple Oracle Health Insurance applications, Oracle provides capabilities for replicating these changes to other (Oracle Health Insurance) applications in an automated fashion. In this scenario, Policies acts as the system of record or source system for the person data. Whenever a person or a detail of a person, like an address, is changed in Policies (the source system), it creates a replication event for that person. Target systems, like Claims in this example, subscribe to receive these replication events and use these as a trigger to automatically retrieve the changes.
Entities for which Oracle Health Insurance supports Data Replication between Applications
The following table provides an overview of the entities for which Data Replication can be enabled within the Oracle Health Insurance applications/components.
Entity | Source System | Target System |
---|---|---|
Person |
Oracle Health Insurance Enterprise Policy Administration |
Oracle Health Insurance Authorizations |
Person |
Oracle Health Insurance Enterprise Policy Administration |
Oracle Health Insurance Claims Adjudication and Pricing |
Person |
Oracle Health Insurance Enterprise Policy Administration |
Oracle Health Insurance Value-Based Payments |
Authorization |
Oracle Health Insurance Authorizations |
Oracle Health Insurance Claims Adjudication and Pricing |
Oracle identifies the systems that qualify as source and target systems and which versions of these systems are certified to exchange data using the data replication mechanism.
In the following paragraphs the concepts of this data replication mechanism are explained as well as the way to configure eligible Oracle Health Insurance applications to act as either source or target systems.
Data Replication Overview and Concepts
This paragraph provides a high-level overview of the data replication mechanism, which can be broken down into the following major areas:
- Change detection
-
The source system is responsible for detecting changes to entities that need be replicated to target systems. These replication events include insert, update and delete operations.
- Event logging
-
The source system maintains a log of the replication events that need to be replicated.
- Event retrieval / publishing
-
Events are returned by the source system per request of target systems. For that purpose, the source system exposes an HTTP API resource that target systems can use to retrieve replication events.
- Event processing
-
The target system processes the events to update its local representation of the entities.
The following pictures provide a high-level overview of the solution, identifying processing in source and target systems:
![Event Detection and Publishing](../_images/event-detection-and-publishing.png)
![Event Processing](../_images/event-processing.png)
Change Detection and Change Logging in Source Systems
Changes are tracked at the ‘root aggregate’ entity level, that is for a person and not for a person’s address.
Changes to the details propagate up to the root aggregate. Changes are tracked for each database transaction in the system and for each transaction new replication events are created; replication events that were created in previous transactions are never updated.
The following table lists examples of changes and the resulting data replication events:
Change in the source system in one transaction | Data Replication events created by the system |
---|---|
A new person is added with an address |
One Insert event for the person |
The last name of an existing person is changed |
One Update event for the person |
The last name for the same person is changed again |
One Update event for the person |
For three persons the last names are changed |
Three Update events, one for each person |
A new address is added for an existing person |
One Update event for the person |
The name of an existing person is changed and a new address is added for the same person |
One Update event for the person |
An address is removed from the person’s list of addresses |
One Update event for the person |
A person and all its associated data (like addresses and, marital statuses) is removed |
One Delete event for the person |
Three persons and their associated data are removed |
Three Delete events, one for each person |
Configuration options in the Source System
Change detection and logging in a source system is automatically activated when the system is started. It is driven by seeded configuration that is provided by Oracle. Source systems expose an HTTP API resource 'api/replicationentities' that can be used to inspect and change the configuration. It supports the following operations:
Operation | Description |
---|---|
GET api/replicationentities |
Returns an overview of the entities and their configuration details for which the system is able to track changes |
GET api/replicationentities/{entity} |
Returns configuration details for a specific business entity |
PUT api/replicationentities/{entity} |
To change the configuration for the given entity. It can be used to enable or disable replication event logging for the given entity. Any other elements cannot be changed, attempts to change are simply ignored |
For example, to disable change detection and logging for the "persons" entity execute the following:
curl -H 'Content-Type: application/xml' -H 'Accept: application/xml' -X PUT -d '<sourceReplicationEventLogConfiguration enabled="N" />' http://localhost:8001/api/replicationentities/Person
To re-enable change detection and logging for the "persons" entity execute the following:
curl -H 'Content-Type: application/xml' -H 'Accept: application/xml' -X PUT -d '<sourceReplicationEventLogConfiguration enabled="Y" />' http://localhost:8001/api/replicationentities/Person
Remarks:
-
Oracle assumes that this configuration is very static. It may take up to 15 minutes before the system picks up a change to this configuration
-
Once change detection and logging for an entity is disabled the HTTP API resource must be used to enable it again, for example, if it is disabled restarting the system does not re-enable change detection and logging
Retrieving Replication Events that were logged by a Source System
Source systems expose an HTTP API resource 'api/replicationevents' that target systems use to retrieve replication events (that is, the source system does not publish or push replication events). A GET operation on the 'api/replicationevents/{entity}' resource provides access to the Replication Events that a source system logged for a given entity. It accepts the following parameters:
Parameter | Description |
---|---|
Timestamp (default: null) |
Identification of the last replication event entry that was already received by the target system. The source system returns replication event entries with a timestamp that is larger than the given timestamp |
Limit (default: null) |
For pagination of replication events |
Timestamp Threshold (default: null) |
The same timestamp could occur multiple times (that is, multiple events were logged at the same time). It might be the case that the target system did not receive all events logged for that timestamp The Timestamp Threshold informs the source system exactly how many replication events that were logged for the business entity at the given timestamp were already received by the target system If more replication events were logged for the business entity at the given timestamp the source system returns all these replication events again |
Remarks:
-
The source system paginates responses. If the page size limit is not specified in the request it returns a maximum of 1000 replication events (configurable by setting system property 'ohi.ws.replicationevents.pagination.limit'). If the number of events that were logged for the business entity for a specific timestamp exceeds the pagination limit, the latter is ignored. If additional results are available, the response contains a link that the client can use to get the next set
-
The media type / payload format is either 'application/json' or 'application/xml'
-
For each business entity, the response contains (relative) URIs that point to the source system’s resource for retrieving the entity, for example '/persons/123' for retrieving the person that can be identified with id '123' in the source system. Using the configured system property 'ohi.ws.sourcesystem.<entity_name>.baseurl' (see below), the target system constructs a fully-qualified URI and use that to get the updated business entity from the source system
Configuring an Oracle Health Insurance Application Target System to Retrieve Replication Events from a Source System
An Oracle Health Insurance application that can retrieve replication events from a source system (or from multiple source systems) comes with the basic setup and processing logic for that purpose. However, by default, retrieving replication events from a source system is not enabled. Customers that want to enable replication of data to a target system must configure the base URI for each entity for which data replication is supported in the target system. Note that the base URI must include the context root for accessing HTTP API services; by default that context root is "/api" but it might be changed with a deployment plan.
The following table lists the system properties for currently supported data replication use cases:
Entity | Source System | Target System | System Property | Sample Value |
---|---|---|---|---|
Authorization |
Oracle Health Insurance Authorizations |
Claims |
ohi.ws.sourcesystem |
http://host:port/api/ |
Person |
Policies |
Oracle Health Insurance Authorizations |
ohi.ws.sourcesystem. + person.baseurl |
http://host:port/api/ |
Person |
Policies |
Claims |
ohi.ws.sourcesystem |
http://host:port/api/ |
Person |
Policies |
Oracle Health Insurance Value-Based Payments |
ohi.ws.sourcesystem |
http://host:port/api/ |
The values in the System Property and Sample Value columns is formatted for readability, these must normally be specified as a single string, without line breaks |
An Oracle Health Insurance application that is pre-configured to retrieve change events from a source system checks for each entity if the associated system property is given a valid URI value. If that is the case, the configuration for retrieving replication events for that specific entity from a source system is enabled.
For every entity for which the configuration is enabled, the system starts a 'DataReplicationTargetSynchronization' task that frequently runs to collect replication events and make sure that these are processed. The interval for running the Target Synchronization task is configurable, it can be influenced by setting system property 'ohi.datareplication.event.retrieval.interval' (value in seconds, by default it is 300). The task is queued with the specified interval as delay, it is eligible to be de-queued and executed when the interval expires.
The Target Synchronization task for an enabled entity executes the following workflow:
-
It retrieves replication events from the source system verbatim, that is, it does not check for duplicates. The entity for which this activity instance must retrieve changes is passed to it as input parameter. The Target Synchronization task retrieves replication events from the configured source system, using the GET operation on the source system’s 'api/replicationevents/{entity}' resource. The target system again uses 'ohi.ws.sourcesystem.<entity_name>.baseurl' for constructing a fully-qualified URI. It stores the replication events returned by the source system in a local events table from which these are processed. As long as the response message indicates that additional replication events are available the Target Synchronization task that continues to retrieve and store these.
-
If none is running yet, the Target Synchronization task spawns an instance of the Replication Event Processor Global activity. The Replication Event Processor processes events for a given entity. The processing algorithm is explained in the following paragraph.
Task status (failures) can be tracked through the View Technical Error screen whereas activities and their status can be tracked from the Activities page in Oracle Health Insurance Components applications. Instances of the Replication Event Processor Global activity that completed successfully is periodically removed if the Data Purge routine (see below) is activated.
Processing Replication Events in a Target System
The source system created new replication events for any change to a specific entity. The Replication Event Retriever Global activity retrieved all these replication events from the configured source system.
An overview of the data attributes in a target system is provided in the following table:
Attribute | Data Type | Required? | Description |
---|---|---|---|
Entity |
varchar2(30) |
Yes |
Entity. |
Source System |
varchar2(30) |
Yes |
The source system. |
Source System Version |
varchar2(30) |
Yes |
Source system version. |
Source Subject UUID |
varchar2(36,0) |
Yes |
Unique identifier of the entity as provided in the source system at the time it was created. |
Logged Timestamp |
timestamp |
Yes |
Identifies when the replication event was logged in the source system. |
Operation |
varchar2(1) |
Yes |
As logged by the source system. Possible values: I; U; D (Insert, Update, Delete respectively). |
Retrieved Timestamp |
timestamp |
Yes |
Identifies when the replication event was retrieved. |
Business Entity URI |
varchar2(200) |
No |
URI for retrieving the up to date entity from the source system. Empty in case of Delete events. |
Activity Id |
number(30,0) |
No |
Reference to the activity that processes the entity. |
Processed Timestamp |
timestamp |
No |
Identifies when the replication event was processed. |
Processed Result |
varchar2(2) |
No |
Superseded (SU) / Completed (CO) / Errored (ER) |
The replication events in the target system’s database may contain duplicate entries or events that make no sense to even process. An example of the latter is a series of update events for a specific entity that are followed by a delete event; in that case it makes no sense to update the entity in the target system because it is already clear that it deletes eventually. Therefore processing replication events in a target system requires consolidation ('rolling up the changes') to prevent replication of the same entity multiple times and to prevent doing unnecessary work.
The algorithm is explained in terms of the data that Oracle Health Insurance applications store for replication events that are retrieved from a source system.
The algorithm is as follows:
-
First, 'roll up' changes by marking events for an entity as Superseded and set the Processed Timestamp using the following rules:
-
Any events for an entity with the same Source Subject UUID prior to a delete operation for that specific entity are marked as Superseded. Only the delete event for that entity remains to be processed
-
Any insert or update events except the last one for an entity with the same Source Subject UUID for which there is no delete event are marked as Superseded. Only the last insert or update event for that entity remains to be processed
-
-
Loop over (remaining) non-processed replication events (for which the Processed Timestamp is null) to process these:
-
Insert a new or update an existing local representation by using the Business Entity URI to retrieve the entity from the source system and calling existing import functions on existing services with the data that was retrieved
-
In case of a delete operation the entity is queried using the Source Subject UUID. Processing is as follows: If the entity could not be resolved using the Source Subject UUID then the processed result in the Target Replication Events entity will be set to Completed. If the object could be resolved it will be deleted along with all its details. If the entity could be resolved but the delete operation fails, for example because there are references to the business entity in the target system, then a non-fatal message will be added to the activity and the Processed Result will be set to Completed
-
If processing of an event results in an error then:
-
The processed result for that specific replication event is set to Errored
-
The status of the activity that processed the entity is set to Completed with Errors so that it can be recovered after the problem was investigated and fixed
-
Processing will continue for events of the same entity but with a different Business Entity URI. This is similar to importing a file: in that case every element is processed independent from other elements
Overcoming Model and Setup Differences between Oracle Health Insurance Applications
The data replication mechanism can handle differences between entities in various applications:
-
Entity models may have application specific differences and the models may evolve over time. Moreover, changes to the entity model may not be applied to all Oracle Health Insurance applications at the same time. Similarly, customers may uptake new versions of applications at different times
-
The setup may differ from application to application. For example, the person entity in Oracle Health Insurance Policies is likely to be extended with dynamic fields that are not comparable to the dynamic fields for a person in Oracle Health Insurance Claims. Other examples that may differ for the person entity include (but are not limited to) the code to identify a Dynamic Logic script for formatting a person’s name and the access restriction code for viewing a person’s (contact) details
Dynamic fields (and values) that are specified for the entity in the source system will be ignored when importing the entity in the target system. Dynamic field matching is based on the usage name of the dynamic field. The data type of the dynamic field is not taken into account. The following table lists how Oracle Health Insurance Applications deal with a number of sample set up scenarios:
Dynamic Field set up in Source System | Dynamic Field set up in Target System | Result when importing entity |
---|---|---|
Dynamic field with name 'foo' and value 'bar' |
Dynamic field with name 'foo' does not exist |
Dynamic field is ignored |
Dynamic field with name 'foo', type String and value 'bar' |
Dynamic field with name 'foo', but of type Date |
Import fails |
Dynamic field with name 'foo' does not exist |
Mandatory dynamic field with name 'foo' exists |
Import fails unless dynamic logic is used to populate the field |
A dynamic Logic function can be used to overcome differences between entities in source and target applications. If a Data Replication Transformation Dynamic Logic function is configured for the combination of Oracle Health Insurance applications (versions) that is certified to exchange data using the data replication mechanism then it will be applied.
The following tables list the Data Replication Transformation Dynamic Logic configuration for Oracle Health Insurance Authorizations:
Entity | Source System | Target System | Dynamic Logic Code | Dynamic Logic Signature |
---|---|---|---|---|
Person |
Policies 2.16.2.0.0 |
Oracle Health Insurance Authorizations 2.16.2.0.0 |
POL216200_AUT216200 |
Persons Data Replication Transformation |
and for Oracle Health Insurance Claims:
Entity | Source System | Target System | Dynamic Logic Code | Dynamic Logic Signature |
---|---|---|---|---|
Person |
Policies 2.16.2.0.0 |
Claims 2.16.1.0.0 |
POL216200_CLA216100 |
Persons Data Replication Transformation |
Authorization |
Oracle Health Insurance Authorizations 2.16.2.0.0 |
Claims 2.16.2.0.0 |
AUT216200_CLA216200 |
Authorizations Data Replication Transformation |
and for Oracle Health Insurance Capitation:
Entity | Source System | Target System | Dynamic Logic Code | Dynamic Logic Signature |
---|---|---|---|---|
Person |
Policies 3.18.2.0.0 |
Oracle Health Insurance Value-Based Payments 3.18.2.0.0 |
POL318200_CPN318200 |
Persons Data Replication Transformation |
Dynamic Logic resolution is done by the combination of signature and dynamic logic code.
Dynamic Logic Parameters
The target system provides the following input parameters / bind variables:
In / Out | Name | Type | Description |
---|---|---|---|
In |
payload |
StringReader |
The JSON payload representation of the entity that was retrieved from the source system. Oracle recommends to use a JsonSlurper. |
In |
persistable |
Oracle Health Insurance Domain class |
The OHI domain class that was constructed from the entity that was retrieved from the source system. |
The following sample Dynamic Logic demonstrates construction of a Map using a JsonSlurper. Subsequently it sets the value of (dynamic) field 'field' that is defined in the target system to the value of the (dynamic) field 'anotherField' that is defined in the source system:
def message = new groovy.json.JsonSlurper().parse(payload) assert message instanceof Map persistable.field = message.anotherField
Target System Processing Error Messages
During processing of replication events (in order: extract, transform and load) in a target systems the following errors could occur:
Code | Message |
---|---|
REP-TTET-001 |
Unable to extract entity {0} from source system |
REP-TTET-002 |
Error transforming entity {0} |
REP-TTET-003 |
Error loading entity with UUID {0} |
Data Purging
Data purging capabilities are available to periodically clean up
-
Source Replication Events
-
Target Replication Events
-
Activities that were used to import changed entities in the target system and that completed successfully
-
Activities that were used to import changed entities in the target system, that completed with errors, but that were not associated with any replication events yet (for example, because it failed fast when not all dynamic logic functions were properly configured)
In-line with existing data purging capabilities, the routine for cleaning up replication events is implemented as a PL/SQL database operation:
ohi_repl_purge_pkg (p_purge_days in integer );
The input parameter defines the retention period for the data expressed in days. The minimum retention period is 30 days. A value smaller than 30 for the input parameter results in an error.
The System Administration guide explains how to set up frequent and automated purging of technical data.