Data Replication

Introduction

Data replication is the process that synchronizes or replicates data between two applications.

For example, replicating a person’s data from the Policies application to the Claims application. Both applications use the same subsystem to maintain data. Here, the Persons API service or the Data File Set interface maintains a person’s data in both the Policies and the Claims application.

From the example, Policies is a source application when synchronizing a person’s data across applications. The source application creates a replication event for every change in a person’s detail. A target application subscribes to receive these replication events to replicate the changes.

Entities that Support Data Replication between Applications

The following table has a list of entities that replicate data between applications:

Table 1. Entities that Replicate Data between Applications
Entity	Source System	Target System
Person	Oracle Health Insurance Enterprise Policy Administration	Oracle Health Insurance Authorizations
Person	Oracle Health Insurance Enterprise Policy Administration	Oracle Health Insurance Claims Adjudication and Pricing
Person	Oracle Health Insurance Enterprise Policy Administration	Oracle Health Insurance Value-Based Payments
Authorization	Oracle Health Insurance Authorizations	Oracle Health Insurance Claims Adjudication and Pricing

Data Replication Overview and Concepts

The data replication process has the following steps:

Change Detection: The source application monitors changes to an entity. A change is an insert, update, or delete operation on an entity.
Event Logging: The source application logs the changes to replicate to a target application.
Event Retrieval: The source application returns events with change information on the request of a target application. The source application has an HTTP API resource that the target application uses to retrieve replication events.
Event Processing: The target application updates the local representation of the entities.

The following images show the process of data replication in the source and the target applications:

Change Detection and Change Logging in Source Applications

The source application tracks changes at the root aggregate level. This means the application tracks changes for the entity and not for the entity’s details.

Changes in the entity’s detail propagate up to the root aggregate level. The application tracks changes for each database transaction and creates new replication events for each transaction. The application does not update replication events of older transactions.

The following table lists examples of the resulting data replication events:

Table 2. Change Detection and Change Logging in Source Systems
Example	Replication Event that the Source Application Creates
Updates in the source application through a single transaction	An Update event for the transaction
Adding a new person with an address	A single insert event for the person
Updating the last name of a person	A single update event for the person
Re-updating the last name of a person	A single update event for the person
Changing the last names of three people	Three update events, one for each person
Adding a new address to a person	A single update event for the person
Changing the name and address of a person	One update event for the person
Removing an address from a person	One update event for the person
Removing a person with its details (like addresses, last name, etc.)	One delete event for the person
Removing three people and their data	Three delete events, one for each person

Replication Events for a Person Entity

The application logs replication events for the changes in the following details of a person entity:

Person itself
Address
Assigned Provider
Bank Account Number
Contract Alignment
Marital Status
Title
Relation Tag
Person Covered Service
Relation Link

Replication Event for Authorization Entity

The application logs an event in Authorizations for the following entities:

Authorization

The application logs replication events for the last version of an authorization in the Approved or Denied status.

Configuration in the Source Application

The source application detects and logs changes on startup. A source application uses the HTTP API resource: api/replicationentities to detect changes in entities. The resource supports the following operations:

Table 3. Configuration in the Source Application
Operation	Description
GET api/replicationentities	Returns entities and their configuration. Return details of the entities with changes.
GET api/replicationentities/{entity}	Returns configuration details of the specified entity.
PUT api/replicationentities/{entity}	Changes configuration of the specified entity. Enables or disables change tracking on the specified entity. Does not change any detail of the entity. The integration point ignores requests for changes to any entity’s details.

For example, you can use the following code to prevent tracking and logging changes to a person entity:

curl -H 'Content-Type: application/xml' -H 'Accept: application/xml' -X PUT -d '<sourceReplicationEventLogConfiguration enabled="N" />' http://[hostName]:[portNumber]/[api-context-root]}/replicationentities/Person

You can use the following code to enable change tracking and logging for the person entity:

curl -H 'Content-Type: application/xml' -H 'Accept: application/xml' -X PUT -d '<sourceReplicationEventLogConfiguration enabled="Y" />' http://[hostName]:[portNumber]/[api-context-root]/replicationentities/Person

Remarks:

The application requires the configuration to be static. It takes fifteen minutes for the application to pick up any change to the configuration.
You must use the HTTP API resource to enable the change tracking and logging for an entity. Restarting the application does not enable the change tracking and logging by itself.

Receiving Replication Events from the Source Application

The source application has an HTTP API resource api/replicationevents that a target application uses to receive replication events. The source application neither publishes nor pushes replication events on its own. A GET operation on the api/replicationevents/{entity} resource provides access to the replication events that a source application logs for a specific entity. It accepts the following parameters:

Table 4. Receiving Replication Events from the Source Application
Parameter	Default	Description
Timestamp	null	The source application returns replication event entries with a timestamp larger than the specified timestamp. It helps determine the last replication event entry that the target application receives.
Limit	null	This parameter helps with the pagination of replication events.
Timestamp Threshold	null	A timestamp can have multiple entries for different events that log at the same time. Sometimes the target application does not receive all events for a timestamp. In such cases, the timestamp threshold informs the source application of the number of replication events that the target application has already retrieved for an entity at the specified timestamp. If the source application finds that there are still many replication events left for a target application to retrieve for an entity at the specified timestamp, the source application resends all the replication events.

The source application paginates responses. If a request does not specify the page size limit, it returns a maximum of 1000 replication events. You can configure the number using an ohi.ws.replicationevents.pagination.limit system property. If the number of events exceeds the pagination limit for an entity for a specific timestamp, the response ignores the pagination limit. Then, the response contains a link to access the next set of results.
The media type must be application/json or application/xml.
For each entity, the response contains relative URIs that point to the source application’s resource for retrieving the entity. For example, /persons/123 retrieves the person with id 123 in the source application. The target application constructs a URI to retrieve the updated entity from the source application using the ohi.ws.sourcesystem.<entity_name>.baseurl system property.

Configuring a Target Application to Retrieve Replication Events from a Source Application

The target applications come with a basic setup and processing logic to retrieve replication events from a source application or multiple source applications. By default, the feature to retrieve replication events from a source application is disabled. To retrieve replication events from a source application, you must configure the source application’s base URI in the target application for each entity that supports data replication.

For example, set the following property to replicate a person entity from Policies to Claims: The property must hold the URI of the source application, or the Policies application in the example.

POST: http://example.claims.com:8001/api/generic/properties
{
"name": "ohi.ws.sourcesystem.person.baseurl",
"value": "example.policies.com:8001/api"
}

The base URI must include the context root for accessing HTTP API services. The default context root is /api which differs for unique deployment plans.

The following table lists the system properties for data replication between different applications:

Table 5. Supported System Properties
Entity	Source System	Target System	System Property
Authorization	Oracle Health Insurance Authorizations	Claims	ohi.ws.sourcesystem.authorization.baseurl
Person	Policies	Oracle Health Insurance Authorizations	ohi.ws.sourcesystem.person.baseurl
Person	Policies	Claims	ohi.ws.sourcesystem.person.baseurl
Person	Policies	Oracle Health Insurance Value-Based Payments	ohi.ws.sourcesystem.person.baseurl

If an application is pre-configured to retrieve replication events from the source application, it compares the URI of the business entity in the source application with the value of the system property. If they are similar, the configuration to retrieve replication events from the source application for that specific entity is successful.

For every successful configuration, the application starts a DataReplicationTargetSynchronization task that frequently runs to collect replication events and process them further. You can configure the interval for running the target synchronization task by setting the ohi.datareplication.event.retrieval.interval system property. This property specifies the waiting period between starting two activities in seconds with the default being 300. The application queues the task for the duration of the interval and de-queues it when the interval expires.

The target synchronization task for a configured entity has the following workflow:

The target synchronization task retrieves replication events from the configured source application using a GET operation on the api/replicationevents/{entity} resource of the source application. The target application uses ohi.ws.sourcesystem.<entity_name>.baseurl to construct a valid URI.
The task stores the replication events that the source application returns to a local events table to process later. As long as the response message shows that additional replication events are available, the target synchronization task continues to retrieve and store replication events.
If there are no more replication events, the target synchronization task spawns an instance of the Replication Event Processor global activity. The replication event processor processes events for a specific entity. The sections below explain the processing algorithm.

You can track failed tasks using the View Technical Error screen. You can track activities with their status from the Activities page in the application.

You can remove the completed instances of the replication event processor global activity using the Data Purging routine.

Processing Replication Events in a Target Application

The source application creates new replication events for any change to a specific entity. The target application retrieves all the replication events from the configured source application using the Replication Event Retriever Global activity.

The following table lists the data attributes in a target application:

Table 6. Processing Replication Events in a Target System
Attribute	Data Type	Required?	Description
Entity	varchar2(30)	Yes	Entity.
Source System	varchar2(30)	Yes	The source application.
Source System Version	varchar2(30)	Yes	The source application version.
Source Subject UUID	varchar2(36,0)	Yes	The unique identifier of the entity in the source application.
Logged Timestamp	timestamp	Yes	The time when the source application logs a replication event.
Operation	varchar2(1)	Yes	The change that the source application registers. It takes values: `I` for insert, `U` for update, or `D` for delete.
Retrieved Timestamp	timestamp	Yes	The time the target application retrieves the replication event.
Business Entity URI	varchar2(200)	No	The URI to retrieve the updated entity from the source application. The value for this attribute is empty for delete events.
Activity Id	number(30,0)	No	A reference to the activity that processes the entity.
Processed Timestamp	timestamp	No	The time when the target application processes the replication event.
Processed Result	varchar2(2)	No	The resulting status that can be: Superseded(SU), Completed(CO), or Errored(ER).

In some cases the replication events in the target application’s database contain duplicate entries or events with unnecessary values. For example, a series of update events that follow a delete event. Updating the target application seems pointless as the entity deletes, eventually. Therefore, the target application needs to consolidate or round up the replication events to avoid replicating the same entity multiple times or performing unnecessary actions.

The following algorithm explains the processing steps that consider the data that a target application retrieves from a source application:

The target application rounds up replication events by marking each event as Superseded for an entity and sets the Processed Timestamp using the following rules:
- The application marks any events prior to a delete operation and with similar Source Subject UUIDs as Superseded. The application supersedes these events from processing so that only the delete event remains for processing.
- The application marks all the insert and updates events with similar Source Subject UUIDs that have no eventual delete operation as Superseded, but leaves the last insert or update event for processing.
The application loops over the remaining, non-processed replication events with the Processed Timestamp as null to process:
- The target application inserts a new or updates the local representation of an entity using the Business Entity URI from the source application and runs import functions to update existing services with the new data.
- For delete operation, the target application queries the entity using the Source Subject UUID. The process is:
  - If the entity does not return when the target application queries the Source Subject UUID, then the Processed Result in the Target Replication Events entity sets to a Completed status.
  - If the object does not return a value when queries, the target application deletes it along with all its details.
  - If the entity returns a value but the delete operation fails in the target application, for example, when there are references to an entity, then the application adds a non-fatal message to the activity and sets the Processed Result to a Completed status.
If processing of an event results in an error, then:
- The target application sets the Processed Result for that specific replication event to Errored.
- The target application sets the status of the activity that processes the entity as Completed with Errors to recover after fixing the error.
- The target application keeps processing events for the same entity, but with a different Business Entity URI. This resembles importing a file where the application processes every element independent of the other elements.

Overcoming Model and Setup Differences between Applications

The data replication process handles differences in entities between applications:

Entity models can differ between applications. For example, some changes in the entity model take time to apply to all the applications. Similarly, adopting newer versions of different applications at different times makes the entity model different between applications.
Also, applications can have a unique setup. For example, the person entity in Policies extends differently than the person entity in the Claims application. Other examples include different codes to identify a dynamic logic to format a person’s name or the different access restriction codes to view a person’s contact details.

The application uses the usage name of the dynamic field to compare the dynamic fields and values of an entity in the source application to that of the target system. The application does not consider the data types of the dynamic fields when comparing dynamic fields and values.

The following is a sample of how an application deals with different setups:

Table 7. Overcoming Model and Setup Differences between Applications
Dynamic Field Setup in the Source Application	Dynamic Field Setup in the Target Application	Result when Importing Entity
Dynamic field with the name 'foo' and 'bar' value	Dynamic field with the name 'foo' does not exist	The application ignores the dynamic field.
Dynamic field with the name 'foo', string type and 'bar' value	Dynamic field with the name 'foo' of date type	Import fails.
Dynamic field with the name 'foo' does not exist	Mandatory dynamic field with the name 'foo' exists	Import fails unless there is a dynamic logic to populate the field.

The application uses a dynamic logic function to overcome differences between entities in the source and the target applications. If an application holds a Data Replication Transformation dynamic logic function with the right combination of applications, versions, and object, it applies the function to the data.

Table 8. List of Data Replication Transformation Dynamic Logic Configuration for Oracle Health Insurance Authorizations
Entity	Source System	Target System	Dynamic Logic Code	Dynamic Logic Signature
Person	Policies 3.22.2.0.0	Oracle Health Insurance Authorizations 2.16.2.0.0	POL216200_AUT216200	Persons Data Replication Transformation

Table 9. List of Data Replication Transformation Dynamic Logic Configuration for Claims
Entity	Source System	Target System	Dynamic Logic Code	Dynamic Logic Signature
Person	Policies 2.16.2.0.0	Claims 2.16.1.0.0	POL216200_CLA216100	Persons Data Replication Transformation
Authorization	Oracle Health Insurance Authorizations 2.16.2.0.0	Claims 2.16.2.0.0	AUT216200_CLA216200	Authorizations Data Replication Transformation

Table 10. List of Data Replication Transformation Dynamic Logic Configuration for Capitation:
Entity	Source System	Target System	Dynamic Logic Code	Dynamic Logic Signature
Person	Policies 3.18.2.0.0	Oracle Health Insurance Value-Based Payments 3.18.2.0.0	POL318200_CPN318200	Persons Data Replication Transformation

A application resolves dynamic logic by comparing the signature and the dynamic logic code.

Dynamic Logic Parameters

The target application provides the following input parameters:

Table 11. Dynamic Logic Parameters
In or Out	Name	Type	Description
In	payload	StringReader	The JSON payload representation of the entity that the target application retrieves from the source application. Oracle recommends using a JsonSlurper.
In	persistable	OHI Domain class	The domain class that the target application constructs from the entity that it retrieves from the source application.

The following sample dynamic logic shows the construction of a Map using a JsonSlurper:

def message = new groovy.json.JsonSlurper().parse(payload)
assert message instanceof Map
persistable.field = message.anotherField

Subsequently, the code sets the value of the dynamic field as field in the target application to the value of the dynamic field as anotherField in the source application.

Target Application Processing Error Messages

When the target application processes the replication events following errors can occur:

Table 12. Target System Processing Error Messages
Code	Message
REP-TTET-001	Unable to extract entity {0} from source system
REP-TTET-002	Error transforming entity {0}
REP-TTET-003	Error loading entity with UUID {0}

Data Purging

You can clean up or purge data from replication events using a PL/SQL database operation. See Purging Data from Replication Technical Tables for more details.