Data Replication
Introduction
Data replication is the process that synchronizes or replicates data between two applications.
For example, replicating a person’s data from the Policies application to the Claims application. Both applications use the same subsystem to maintain data. Here, the Persons API service or the Data File Set interface maintains a person’s data in both the Policies and the Claims application.
From the example, Policies is a source application when synchronizing a person’s data across applications. The source application creates a replication event for every change in a person’s detail. A target application subscribes to receive these replication events to replicate the changes.
Operations
This API supports the following operations:
|
|
|
|
|
|
|
|
Entities that Support Data Replication between Applications
The following table has a list of entities that replicate data between applications:
Entity | Source System | Target System |
---|---|---|
Person |
Oracle Health Insurance Enterprise Policy Administration |
Oracle Health Insurance Authorizations |
Person |
Oracle Health Insurance Enterprise Policy Administration |
Oracle Health Insurance Claims Adjudication and Pricing |
Person |
Oracle Health Insurance Enterprise Policy Administration |
Oracle Health Insurance Value-Based Payments |
Authorization |
Oracle Health Insurance Authorizations |
Oracle Health Insurance Claims Adjudication and Pricing |
Data Replication Overview and Concepts
The data replication process has the following steps:
-
Change Detection: The source application monitors changes to an entity. A change is an insert, update, or delete operation on an entity.
-
Event Logging: The source application logs the changes to replicate to a target application.
-
Event Retrieval: The source application returns events with change information on the request of a target application. The source application has an HTTP API resource that the target application uses to retrieve replication events.
-
Event Processing: The target application updates the local representation of the entities.
The following images show the process of data replication in the source and the target applications:
Change Detection and Change Logging in Source Applications
The source application tracks changes at the root aggregate level. This means the application tracks changes for the entity and not for the entity’s details.
Changes in the entity’s detail propagate up to the root aggregate level. The application tracks changes for each database transaction and creates new replication events for each transaction. The application does not update replication events of older transactions.
The following table lists examples of the resulting data replication events:
Example | Replication Event that the Source Application Creates |
---|---|
Updates in the source application through a single transaction |
An Update event for the transaction |
Adding a new person with an address |
A single insert event for the person |
Updating the last name of a person |
A single update event for the person |
Re-updating the last name of a person |
A single update event for the person |
Changing the last names of three people |
Three update events, one for each person |
Adding a new address to a person |
A single update event for the person |
Changing the name and address of a person |
One update event for the person |
Removing an address from a person |
One update event for the person |
Removing a person with its details (like addresses, last name, etc.) |
One delete event for the person |
Removing three people and their data |
Three delete events, one for each person |
Configuration in the Source Application
The source application detects and logs changes on startup.
The source application provides information on the tracked resources through the /replicationentities
API.
Messages
Fetch The Resources That Log Events
Sending a GET
request to /replicationentities
fetches the entities that have changes.
The response message returns the following response message (for example):
{
"hasMore": false,
"sourceReplicationEventLogConfiguration": [
{
"description": "For detecting and logging changes to members",
"enabled": true,
"entityName": "Person",
"logAnyChange": true
}
]
}
The above response message shows that only one entity, Person, logs change events.
Send a GET
request to /replicationentities/{entity}
to fetch information on one specific entity.
Switch Off The Event Logging For Person
Sending a PUT
request to replicationentities/Person
with the below payload prevents tracking and logging changes to the person resource.
{ "enabled": false }
Use the same operation with payload "enabled" : true
to enable tracking on Person
again.
Restarting the application does not enable the change tracking and logging by itself.
The application requires the configuration to be static. It takes fifteen minutes for the application to pick up any change to the configuration.
Receiving Replication Events from the Source Application
The source application has provides the /replicationevents` API that a target application can use to fetch replication events.
The source application neither publishes nor pushes replication events on its own.
A GET operation on the /replicationevents/{entity}
resource provides access to the replication events that a source application logs for a specific entity.
It accepts the following parameters:
Parameter | Default | Description |
---|---|---|
Timestamp |
null |
The source application returns replication event entries with a timestamp larger than the specified timestamp. It helps determine the last replication event entry that the target application receives. |
Limit |
null |
This parameter helps with the pagination of replication events. |
Timestamp Threshold |
null |
A timestamp can have multiple entries for different events that log at the same time. |
-
The source application paginates responses. If a request does not specify the page size limit, it returns a maximum of 1000 replication events. You can configure the number using an
ohi.ws.replicationevents.pagination.limit
system property. If the number of events exceeds the pagination limit for an entity for a specific timestamp, the response ignores the pagination limit. Then, the response contains a link to access the next set of results. -
The media type must be
application/json
orapplication/xml
. -
For each entity, the response contains relative URIs that point to the source application’s resource for retrieving the entity. For example,
/persons/123
retrieves the person with id 123 in the source application. The target application constructs a URI to retrieve the updated entity from the source application using theohi.ws.sourcesystem.<entity_name>.baseurl
system property.
Configuring a Target Application to Retrieve Replication Events from a Source Application
The target applications come with a basic setup and processing logic to retrieve replication events from a source application or multiple source applications. By default, the feature to retrieve replication events from a source application is disabled. To retrieve replication events from a source application, you must configure the source application’s base URI in the target application for each entity that supports data replication.
For example, set the following property in the target environment to replicate a person entity from Policies to Claims. The property must hold the URI of the source application, or the Policies application in the example.
Send a POST
request to {api}/generic/properties with the payload
{
"name": "ohi.ws.sourcesystem.person.baseurl",
"value": "example.policies.com:8001/api"
}
The base URI must include the context root for accessing HTTP API services. The default context root is /api which differs for unique deployment plans.
The following table lists the system properties for data replication between different applications:
Entity | Source System | Target System | System Property |
---|---|---|---|
Authorization |
Oracle Health Insurance Authorizations |
Claims |
ohi.ws.sourcesystem.authorization.baseurl |
Person |
Policies |
Oracle Health Insurance Authorizations |
ohi.ws.sourcesystem.person.baseurl |
Person |
Policies |
Claims |
ohi.ws.sourcesystem.person.baseurl |
Person |
Policies |
Oracle Health Insurance Value-Based Payments |
ohi.ws.sourcesystem.person.baseurl |
If an application is pre-configured to retrieve replication events from the source application, it compares the URI of the business entity in the source application with the value of the system property. If they are similar, the configuration to retrieve replication events from the source application for that specific entity is successful.
For every successful configuration, the application starts a DataReplicationTargetSynchronization task that frequently runs to collect replication events and process them further.
You can configure the interval for running the target synchronization task by setting the ohi.datareplication.event.retrieval.interval
system property.
This property specifies the waiting period between starting two activities in seconds with the default being 300.
The application queues the task for the duration of the interval and de-queues it when the interval expires.
The target synchronization task for a configured entity has the following workflow:
-
The target synchronization task retrieves replication events from the configured source application using a GET operation on the
api/replicationevents/{entity}
resource of the source application. The target application usesohi.ws.sourcesystem.<entity_name>.baseurl
to construct a valid URI.
The task stores the replication events that the source application returns to a local events table to process later. As long as the response message shows that additional replication events are available, the target synchronization task continues to retrieve and store replication events. -
If there are no more replication events, the target synchronization task spawns an instance of the Replication Event Processor global activity. The replication event processor processes events for a specific entity. The sections below explain the processing algorithm.
You can track failed tasks using the View Technical Error screen. You can track activities with their status from the Activities page in the application.
You can remove the completed instances of the replication event processor global activity using the Data Purging routine.
Processing Replication Events in a Target Application
The source application creates new replication events for any change to a specific entity. The target application retrieves all the replication events from the configured source application using the Replication Event Retriever global activity.
The following table lists the replication event data attributes in a target application:
Attribute | Data Type | Required? | Description |
---|---|---|---|
Entity |
varchar2(30) |
Yes |
Entity. |
Source System |
varchar2(30) |
Yes |
The source application. |
Source System Version |
varchar2(30) |
Yes |
The source application version. |
Source Subject UUID |
varchar2(36,0) |
Yes |
The unique identifier of the entity in the source application. |
Logged Timestamp |
timestamp |
Yes |
The time when the source application logs a replication event. |
Operation |
varchar2(1) |
Yes |
The change that the source application registers. It takes values: |
Retrieved Timestamp |
timestamp |
Yes |
The time the target application retrieves the replication event. |
Business Entity URI |
varchar2(200) |
No |
The URI to retrieve the updated entity from the source application. The value for this attribute is empty for delete events. |
Activity Id |
number(30,0) |
No |
A reference to the activity that processes the entity. |
Processed Timestamp |
timestamp |
No |
The time when the target application processes the replication event. |
Processed Result |
varchar2(2) |
No |
The resulting status that can be: Superseded(SU), Completed(CO), or Errored(ER). |
In some cases the replication events in the target application’s database contain duplicate entries or events with unnecessary values. For example, a series of update events that follow a delete event. Updating the target application seems pointless as the entity deletes, eventually. Therefore, the target application needs to consolidate or round up the replication events to avoid replicating the same entity multiple times or performing unnecessary actions.
The following algorithm explains the processing steps that consider the data that a target application retrieves from a source application:
-
The target application rounds up replication events by marking each event as Superseded for an entity and sets the Processed Timestamp using the following rules:
-
The application marks any events prior to a delete operation and with similar Source Subject UUIDs as Superseded. The application supersedes these events from processing so that only the delete event remains for processing.
-
The application marks all the insert and updates events with similar Source Subject UUIDs that have no eventual delete operation as Superseded, but leaves the last insert or update event for processing.
-
-
The application loops over the remaining, non-processed replication events with the Processed Timestamp as
null
to process:-
The target application inserts a new or updates the local representation of an entity using the Business Entity URI from the source application and runs import functions to update existing services with the new data.
-
For delete operation, the target application queries the entity using the Source Subject UUID. The process is:
-
If the entity does not return when the target application queries the Source Subject UUID, then the Processed Result in the Target Replication Events entity sets to a Completed status.
-
If the object does not return a value when queries, the target application deletes it along with all its details.
-
If the entity returns a value but the delete operation fails in the target application, for example, when there are references to an entity, then the application adds a non-fatal message to the activity and sets the Processed Result to a Completed status.
-
-
-
If processing of an event results in an error, then:
-
The target application sets the Processed Result for that specific replication event to Errored.
-
The target application sets the status of the activity that processes the entity as Completed with Errors to recover after fixing the error.
-
The target application keeps processing events for the same entity, but with a different Business Entity URI. This resembles importing a file where the application processes every element independent of the other elements.
-
Overcoming Model and Setup Differences between Applications
The data replication process handles differences in entities between applications:
-
Entity models can differ between applications. For example, some changes in the entity model take time to apply to all the applications. Similarly, adopting newer versions of different applications at different times makes the entity model different between applications.
-
Also, applications can have a unique setup. For example, the person entity in Policies extends differently than the person entity in the Claims application. Other examples include different codes to identify a dynamic logic to format a person’s name or the different access restriction codes to view a person’s contact details.
The application uses the usage name of the dynamic field to compare the dynamic fields and values of an entity in the source application to that of the target system. The application does not consider the data types of the dynamic fields when comparing dynamic fields and values.
The following is a sample of how an application deals with different setups:
Dynamic Field Setup in the Source Application | Dynamic Field Setup in the Target Application | Result when Importing Entity |
---|---|---|
Dynamic field with the name 'foo' and 'bar' value |
Dynamic field with the name 'foo' does not exist |
The application ignores the dynamic field. |
Dynamic field with the name 'foo', string type and 'bar' value |
Dynamic field with the name 'foo' of date type |
Import fails. |
Dynamic field with the name 'foo' does not exist |
Mandatory dynamic field with the name 'foo' exists |
Import fails unless there is a dynamic logic to populate the field. |
The application uses a dynamic logic function to overcome differences between entities in the source and the target applications. If an application holds a Data Replication Transformation dynamic logic function with the right combination of applications, versions, and object, it applies the function to the data.
Entity | Source System | Target System | Dynamic Logic Code | Dynamic Logic Signature |
---|---|---|---|---|
Person |
Policies 3.22.2.0.0 |
Oracle Health Insurance Authorizations 2.16.2.0.0 |
POL216200_AUT216200 |
Persons Data Replication Transformation |
Entity | Source System | Target System | Dynamic Logic Code | Dynamic Logic Signature |
---|---|---|---|---|
Person |
Policies 2.16.2.0.0 |
Claims 2.16.1.0.0 |
POL216200_CLA216100 |
Persons Data Replication Transformation |
Authorization |
Oracle Health Insurance Authorizations 2.16.2.0.0 |
Claims 2.16.2.0.0 |
AUT216200_CLA216200 |
Authorizations Data Replication Transformation |
Entity | Source System | Target System | Dynamic Logic Code | Dynamic Logic Signature |
---|---|---|---|---|
Person |
Policies 3.18.2.0.0 |
Oracle Health Insurance Value-Based Payments 3.18.2.0.0 |
POL318200_CPN318200 |
Persons Data Replication Transformation |
A application resolves dynamic logic by comparing the signature and the dynamic logic code.
Dynamic Logic Parameters
The target application provides the following input parameters:
In or Out | Name | Type | Description |
---|---|---|---|
In |
payload |
StringReader |
The JSON payload representation of the entity that the target application retrieves from the source application. Oracle recommends using a JsonSlurper. |
In |
persistable |
OHI Domain class |
The domain class that the target application constructs from the entity that it retrieves from the source application. |
The following sample dynamic logic shows the construction of a Map using a JsonSlurper:
def message = new groovy.json.JsonSlurper().parse(payload) assert message instanceof Map persistable.field = message.anotherField
Subsequently, the code sets the value of the dynamic field as field in the target application to the value of the dynamic field as anotherField in the source application.
Target Application Processing Error Messages
When the target application processes the replication events following errors can occur:
Code | Message |
---|---|
REP-TTET-001 |
Unable to extract entity {0} from source system |
REP-TTET-002 |
Error transforming entity {0} |
REP-TTET-003 |
Error loading entity with UUID {0} |
Data Purging
You can clean up or purge data from replication events using a PL/SQL database operation. See Purging Data from Replication Technical Tables for more details.