Synchronizing Data In A Multi-Data Center

19 Synchronizing Data In A Multi-Data Center

The Multi-Data Center infrastructure can be configured to keep Access Manager data synchronized across multiple data centers. This can be done using the Automated Policy Synchronization Replication Service or data can be replicated manually.

The following topics describe how to synchronize across data centers and replicate data:

19.1 Understanding the Multi-Data Center Synchronization

The Multi-Data Center (MDC) infrastructure can be configured to keep Access Manager data synchronized across multiple data centers.

Policy, system configuration, and partner metadata are all synchronized with APS.

Note:

For creating a Clone (also referred to as a Consumer) from a Master (also referred to as a Supplier), MDC Admin REST APIs are used. Once the MDC infrastructure is deployed, APS can be enabled to automatically sync any changes from the Master to the Clones.

See Before Setting Up a Multi-Data Center.

APS (also referred to as the Replication Service) is a set of REST API. The binaries are installed as part of the Access Manager application and deployed in the AdminServer. It is enabled by default. After enabling the service, create a pull model Replication Agreement between the Clone data center and the Master. The Clone polls for changes from the Master as long as the Replication Agreement is valid for it. Conversely, the Master will respond to the Clone's request as long as it finds a valid replication agreement. The Clone applies the changes locally.

Note:

APS is optional; administrator-initiated import and export based replication is still available using WLST command procedure used previous to this release.

When setting up the Replication Service, the following may occur:

Replication Agreement is established. One data center is registered as a Replication Clone and another one is registered in a separate geographical location as its Master. From the Master data center, the changes are pulled and applied to the Clone data center.
Data center-specific configurations that may not be replicated across data centers are defined.
Access Manager configuration changes are tracked in each data center; the current replication state can be queried in any of the data centers.
Change Log is generated; it can be applied in the context of a similar setup running in another data center.
A pull from the Master data center can be triggered, if required; for example, if there is a failure of automated replication.
Access Manager configuration artifacts in the Master-Clone model are replicated.

Note:

APS does not sync IDS Profiles, OAM Keystores, and Oracle Platform Security Services artifacts (jps-config.xml changes, credential store configuration and the like).

This section describes the following topics:

19.1.1 How Replication Works

Replication works in a Master-Clone topology. In this topology, multiple Clones pull changes from a single Master. One data center is defined by the administrator as the Master and one or more other data centers are Clones.

The administrator makes changes to the Master that are replicated to the Clones. Only Master to Clone replication is supported; changes to Clones are not replicated back to the Master.

Note:

Multi-master replication is not supported.

To partake in replication, the Master data center (initiator of the replication) and the Clone data center (receiver of the changes) must have a Replication Agreement stored in the Access Manager data store. Table 19-1 documents the states in which replication can be deployed.

Table 19-1 Replication States

State	Definition
Active	An Access Manager domain (including Admin and managed servers) is setup to serve access requests. In an active state, the Access Manager server provides the web access management functionality without additional MDC features.
Bootstrapping	This state is optional for some Data Centers; for example, the first one in an MDC topology. A Data Center goes through this intermediate state when added to an existing MDC topology. The new DC contacts the master and bootstraps itself to the same state. The bootstrap includes synchronizing the server keys, policy artifacts, partners, and system configuration. After completion of bootstrapping, the DC will be Replication Ready.
Replication Ready	In this state, MDC is enabled, the DC is made part of the topology, and the replication service is enabled. Once enabled, a clone can be registered with the master via a Replication Agreement. Once established, clone DCs can query and start pulling changelogs from the master.

Figure 19-1 illustrates the replication flow.

Figure 19-1 Replication Flow

Description of "Figure 19-1 Replication Flow"

Each Clone pulls changes from the Master. A replication thread runs on the Clone after the pre-configured interval of time and fetches changes from the Replication Service (REST endpoint) running on the Master environment.

For every cloned environment, the Master keeps track of a change sequence number indicating when it was last synced. The Master also keeps track of the list of changes (Create/Update/Delete) that have been pulled by the Clones in a changelog using specific change sequence numbers. When updating configuration metadata, the Clone can also change environment specific parameter values depending on transformation rules.

See Customizing Transformation Rules.

19.1.2 Understanding the Replication Agreement

Configuration changes (defined as journals) are replicated from a Master node to Clone nodes. On receiving the journals, each node updates it's configuration to match the journal and remain in a synchronized state. The nodes, though, need to enter a Replication Agreement to receive change journals.

When a new data center is added to an existing MDC topology, it has to bootstrap itself to be in sync with the existing data centers. This bootstrap operation will get the current Access Manager policies, system configuration, partner metadata, and server keys for the existing MDC topology. After the bootstrap operation, the new data center captures the last change sequence number from the topology's Master so that during replication it can be used to determine the current state.

Note:

Automated bootstrap is the ideal scenario but you can execute diagnostic MDC REST APIs first to ensure the Master and Clone data centers are in the same state. If the data centers are not in same state, then you have to execute MDC ADMIN REST APIs.

To establish a Replication Agreement, the Clone data center must know the Master's changelog sequence number. If the data center is added to the topology on 'day–0' and the Replication Agreement was created on 'day–1', there is a need to bootstrap again. To avoid this and to keep the flow simple, creating a Replication Agreement should take care of the bootstrap and actual replication agreement creation.

See Setting Up a Master and a Clone in Multi-Data Center.

19.1.3 About Synchronizing Data Manually in a Multi-Data Center

Data in an MDC topology can also be synced manually. The manual option uses WLST export and import calls to migrate the data from one data center to another. Partner profiles and policies should both be exported and imported using WLST.

See Access Manager WLST Commands in Oracle Fusion Middleware WebLogic Scripting Tool Command Reference for Identity and Access Management.

19.2 Enabling Data Replication

Data Replication on the Master and Clone data centers is enabled by default. Setting the flag -Doracle.oam.EnableMDCReplication to true is not mandatory.

Validate the Replication REST endpoints in Master and Clone data centers as follows:

Validate that the REST endpoints are enabled in the Master data center.

curl -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/hello

Validate that the REST endpoints are enabled in the Clone data center.

curl -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/hello

Repeat the process of validation on all Clone data centers.

19.3 Synchronizing Master and Clone Metadata

The process for syncing metadata across an MDC involves first syncing Access Manager UDM metadata and then creating a replication agreement.

See Understanding the Replication Agreement.

The following topics describe how to synchronize master and clone metadata:

19.3.1 Synchronizing the UDM Metadata

You must synchronize the UDM Metadata before you can create the replication agreement.

To sync Access Manager UDM metadata stored in the Master to all Clones:

Execute the exportAccessStore WLST command on the Master Data Center to create a ZIP file containing the UDM metadata.
```
exportAccessStore(toFile="/master/location/dc1metadata.zip", 
   namePath="/")
```
Copy dc1metadata.zip to the Clone DC location.
Execute the importAccessStore WLST command on the Clone Data Center to import the UDM metadata.
```
importAccessStore(fromFile="/clone/location/dc1metadata.zip", 
   namePath="/")
```
Repeat on all Clone DCs.

19.4 Using REST API for Replication Agreements

Use REST APIs provided by Access Manager to manage replication agreements.

This section describes the following topics:

19.4.1 Querying for Replication Agreement Details

Execute a REST request at the Master's endpoint to query the details of the Replication Agreement (including the polling interval) between a Master and a Clone data center.

If the replication agreement identifier is unknown, run the following command to list all the replication agreement identifiers, else skip this step.

curl -k -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/agreements'
 
Sample output 1:
{"featureEnabled":"true","identifiers":"201411211137273612","ok":"true"}
Sample output 2:
{"featureEnabled":"true","identifiers":["201411211137273612","201411211137273900"],"ok":"true"}

Query the details of a Clone data center's replication agreement (including the polling interval) as follows:

curl -k -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231329353668?type=consumer'

Sample output:
{"enabled":"true","identifier":"201409231329353668","ok":"true",
  "pollInterval":"60","startingSequenceNumber":"110","state":"READY"}

Query the details of a Master data center's replication agreement (including the polling interval) as follows:

curl -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231329353668'
Sample output:
{"enabled":"true","identifier":"201409231329353668","lastSequenceNumber":"110","ok":"true","pollInterval":"3600","startingSequenceNumber":"110","state":"ACTIVE"}

Note:

The poll interval of the Master data center's replication agreement does not affect the actual replication.

19.4.2 Creating a Replication Agreement

Creating a Replication Agreement is a one time operation which will enable the Clone data center(s) to pull changes from the Master data center.

The replication agreement can be created using any REST client. In this procedure, we use the standard Curl utility.

After you execute this command, the following results occur:

Insert an entry in the Master's Replication Agreement store containing details regarding the Clone that wants to pull changes.
Insert an entry in the Clone's Replication Agreement store containing details regarding the Master from which it will pull changes. Replication configuration values like the poll interval will also be set.

To create a replication agreement:

Ensure the Master and Clone DC REST endpoints are up and running.
Execute the following command on the Master DC.
This command will use the repluser specified for replication queries from the Master to the Clone. repluser is expected to be available in the default identity stores for all involved DCs.
```
curl -u <repluser> -H 'Content-Type: application/json' -X POST
 'https://supplier.example.com:7002/oam/services/rest/
  _replication/setup' -d '{"name":"DC12DC2", 
 "source":"DC1","target":"DC2","documentType":"ENTITY"}'
```
The following is an example of output for the command.
```
{"enabled":"true","identifier":"201409231329353668","ok":"true",
  "pollInterval":"900","startingSequenceNumber":"110","state"  :"READY"}
```
Be sure to note the values of the replication identifier, pollInterval and startingSequence Number. The identifier is a reference specific to this Replication Agreement and is used for replication related queries. The pollInterval is a value (in seconds) after which the Clone will poll for changes against the Master. (Typically policy and configuration are not changed often so this number can be as high as the default value of 900 seconds.) The startingSequenceNumber is the value before which all records will be unavailable. In the example, all records before the value of 110 are unavailable. It is implicit that bootstrapping happened before creating the Replication Agreement thus the Clone can start pulling changes from sequence number 110. The Clone also has an entry created in its local replication table which keeps track of the last sequence number. The starting sequence process is illustrated in Figure 19-2.

Figure 19-2 Starting Sequence Illustrated

Description of "Figure 19-2 Starting Sequence Illustrated"

The create replication agreement command will return details of an already existing replication agreement if applicable. In this case, the value of ok will be false.
```
{"enabled":"true","identifier":"201409231329353668","ok":"false",
  "pollInterval":"900","startingSequenceNumber":"110",
  "state":"READY"}
```
Note:

If a specific user needs to be used for replication, the user's credentials can be provided in the command in the format "BASIC base64(user:password)".For example, "BASIC base64(weblogic:welcome1)" is specified as "BASIC d2VibG9naWM6d2VsY29tZTE=" in the following command.
```
curl -u <repluser> -H 'Content-Type: application/json' -X POST 
  'https://supplier.example.com:7002/oam/services/rest/
  _replication/setup' -d 
  '{"source":"DC1","target":"DC2","documentType":"ENTITY","config":
  {"entry":{"key":"authorization","value":"BASIC 
  d2VibG9naWM6d2VsY29tZTE="}}}''
```
Basic Authorization is supported for replication REST API.
Restart the Master and Clone AdminServers.

Once the replication agreement is created and the AdminServers restarted, the Clone will start polling for changes. See Modifying Polling Interval.

19.4.3 Modifying a Replication Agreement

Using the Replication Agreement identifier, changes can be made to the Replication Agreement configuration. Replication Agreement properties (enabled status, poll interval and the like) can be updated by executing a REST request at the Master's endpoint. Either the Master or Clone Replication Agreement will be updated as specified by the value of the replicaType parameter. The clone will poll for changes, apply them and wait the duration specified as the pollInterval.

In this example, the value of poll Interval will be changed to 60 seconds.

Service responds back with JSON object that is the status of replication agreement before making the change. You need to fetch replication agreement status again to see updated configuration.

Query the existing replication agreements using the following command and obtain the replication Identifier, replId that needs to be used in the following steps.
```
curl -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/agreements'
```
Note:
If there are multiple replication agreements, select the identifier for which replication agreement needs to be modified by querying the corresponding Clone data center.

Execute the following command to get the current status of the Replication Agreement in Clone machine:

curl -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/201409040157218184?type=consumer'

The JSON response would be:

{“enabled":"true","identifier":"201409040157218184","ok":"true","pollInterval":"900","startingSequenceNumber":"101","state":"READY"}

Execute the following command to modify the value of pollInterval in Clone machine:

curl -k -u weblogic:password 'Content-Type: application/json' -X PUT 
'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/201409040157218184' -d '{“pollInterval":"60","replicaType":"CONSUMER"}'

The JSON response would be:

{“enabled":"true","identifier":"201409040157218184","ok":"true","pollInterval":"60","startingSequenceNumber":"101","state":"READY"}

Restart the AdminServer on Clone machine.

Execute the following command to get the current status of the Replication Agreement.

This will validate that the change has been made. Note the value of pollInterval in the JSON Response is different from the value returned in the first step of this procedure.

curl -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/201409040157218184?type=consumer'

The JSON response would be:

{“enabled":"true","identifier":"201409040157218184?,"ok":"true","pollInterval":"60","startingSequenceNumber":"101","state":"READY"}

Table 19-2 Modifying Replication Agreement Properties

Property	Modification Command
BatchSize	Number of change records (journals) returned by the master as a result of a getChanges query by clone. Ideally the default batch size of 32 is sufficient as all changes are pulled in multiple batches as part of fetching. However if the setup needs a large batch size, execute the following command: curl -k -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/<replid>' -d '{"batchSize":"100","replicaType":"SUPPLIER"}'
User Context	In rare instances, the user context for replication poll may need to be modified. curl -k -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/201409231329353668' -d '{"replicaType":"CONSUMER", "config":{"entry":{"key":"authorization","value":" BASIC cG9sbHVzZXI6c2VjcmV0"}}}' 'cG9sbHVzZXI6c2VjcmV0' is a base 64 encoded value for polluser credentials. Any user credentials can be used here instead of the repluser which is used to execute the command.

Property

Modification Command

BatchSize

Number of change records (journals) returned by the master as a result of a getChanges query by clone. Ideally the default batch size of 32 is sufficient as all changes are pulled in multiple batches as part of fetching. However if the setup needs a large batch size, execute the following command:

curl -k -u weblogic:password -H 'Content-Type: application/json' -X PUT 
 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/<replid>' 
 -d '{"batchSize":"100","replicaType":"SUPPLIER"}'

User Context

In rare instances, the user context for replication poll may need to be modified.

curl -k -u weblogic:password -H 'Content-Type: application/json' -X PUT 
 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/201409231329353668' 
 -d '{"replicaType":"CONSUMER",
"config":{"entry":{"key":"authorization","value":" 
 BASIC cG9sbHVzZXI6c2VjcmV0"}}}'

'cG9sbHVzZXI6c2VjcmV0' is a base 64 encoded value for polluser credentials. Any user credentials can be used here instead of the repluser which is used to execute the command.

19.4.4 Deleting a Replication Agreement

A Replication Agreement can be deleted by executing the REST API at the Master data center's endpoint. Replication Agreements that are currently active and in use cannot be deleted until the Master and Clone have been disabled.

Disable the replication agreement before you remove it from on the Clone and Master data centers.

Disable the replication agreement on the Clone as follows:

curl -u <repluser> -H 'Content-Type: application/json' -X PUT
  'https://supplier.example.com:7002/oam/services/rest/_replication/201409231
  329353668' -d '{"enabled":"false","replicaType":"CONSUMER"}''

Disable the replication agreement on the Master as follows:

curl -u <repluser> -H 'Content-Type: application/json' -X PUT
  'https://supplier.example.com:7002/oam/services/rest/_replication/201409231
  329353668' -d '{"enabled":"false","replicaType":"SUPPLIER"}''

Delete the replication agreement as follows:

curl -u weblogic:welcome1 -H 'Content-Type: application/json' -X DELETE
  'https://supplier.example.com:7002/oam/services/rest/_replication/201409231
  329353668'

19.5 Customizing Transformation Rules

Transformation rules are used by APS and you can modify rules, as required.

The transformation rules illustrated in the following example are the default rules provided by Access Manager. A Clone can be configured to override these OOTB rules. This section documents how some of these rules can be modified and how to configure Access Manager to recognize these custom rules.

Specify the custom transformation rules in a file (say, transformationRules.txt). It is recommended to copy the default OOTB transformation rules specified as follows and modify rules as required.

<?xml version="1.0" encoding="UTF-8"?>
<mdc-transform-rule>
<changes-to-include entity-path="/policy"/>
<changes-to-include entity-path="/oauth"/>
<changes-to-include entity-path="/IDM"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance" />
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/AuthenticationModules"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/oamproxy"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/Sme/SessionConfigurations"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile">
<ignore attribute-match="/OAMSERVER/serverprotocol"/>
<ignore attribute-match="/OAMSERVER/serverhost"/>
<ignore attribute-match="/OAMSERVER/serverport"/>
<ignore attribute-match="/OAMSERVER/serversslterminated"/>
<ignore attribute-match="/HostAlias/oamserverHttps/serverprotocol"/>
<ignore attribute-match="/HostAlias/oamserverHttps/serverhost"/>
<ignore attribute-match="/HostAlias/oamserverHttps/serverport"/>
</changes-to-include>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER">
<ignore attribute-match="/serverprotocol"/>
<ignore attribute-match="/serverhost"/>
<ignore attribute-match="/serverport"/>
<ignore attribute-match="/serversslterminated"/>
</changes-to-include>
<changes-to-include
entity-path="/config/NGAMConfiguration/DataCenterConfiguration/Cluster">
<ignore attribute-match="/DataCenterType"/>
<ignore attribute-match="/ClusterId"/>
<ignore attribute-match="/WriteEnabledFlag"/>
</changes-to-include>
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Descriptors/OAMSEntityDescriptor" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Federation/IdentityProvider" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Federation/ServiceProvider" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedattributeprofiles" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedpartnerprofiles" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedserverconfig" />
</mdc-transform-rule>

The rule document will mention the XPath of system configuration artifacts to be replicated for a clone. If there is any transformation to be done during replication for the entry in XPath, it can be provided as replacement rule for that clone. To add a new XPath for replication to a clone, create a new transformation XML file, using the above XML document as a template. Add and remove XPaths as required. For example, adding the following XPath as the child of an <mdc-transformation-rule> node and saving the file to the clone's file system will modify Available Services.

Add the following environmental variable to the EXTRA_JAVA_PROPERTIES of setDomainEnv.sh (On linux or Unix) or setDomainEnv.cmd (On Windows) file available at $MW_HOME/user_projects/domains/OAMDomain/bin.
```
-Doracle.oam.MDCRuleFile=/scratch/transformationRules.txt
```
Each Clone can use different transformation rules. Figure 19-3 illustrates how to apply these custom rules.

Figure 19-3 Applying Custom Transformation Rules

Description of "Figure 19-3 Applying Custom Transformation Rules"
Restart Clone Admin and Managed Servers.

This transformation rule will not change the WebGate agent definitions. The following information details how you can modify these changes for the PrimaryServerList and logoutRedirectUrl attributes.

You can use the transformation rule in the following example to update the PrimaryServerList and logoutRedirectUrl attributes with the Access Manager server host from the Clone environment. This change can be viewed in the oam-config.xml file; it replaces the value of the PrimaryServerList attribute with the value equal to ${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}; for example, oam1-lon.example.com. The limitation of this rule is that it updates all servers in the primary list.

<changes-to-include 
  entity-path="/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance">
  <replace attribute-match="/*/PrimaryServerList/*/host" value-match="(.*)">
   <replace-with 
n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with>
  </replace>
  <replace attribute-match="/*/UserDefinedParameters/logoutRedirectUrl" 
   value-match="(.*)<a target="_blank" href="://">://</a>(.*):(.*)/oam/server/logout">
  <replace-with 
n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverprotocol}</replace-with>
  <replace-with 
n="2">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with>
  <replace-with 
n="3">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverport}</replace-with>
  </replace>
  <replace attribute-match="/*/logoutRedirectUrl" 
   value-match="(.*)<a target="_blank" href="://">://</a>(.*):(.*)/oam/server/logout">
  <replace-with 
n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverprotocol}</replace-with>
  <replace-with 
n="2">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with>
  <replace-with 
n="3">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverport}</replace-with>
 </replace>
</changes-to-include>

You can use the transformation rule in the following example to update servers in PrimaryServerList with the different Clone servers.

<changes-to-include entity-path=
 "/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance">
    <replace attribute-match="/*/PrimaryServerList/0/host" value-match="(.*)">
        <replace-with n="1">${"/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/
 Instance/oam_server1/host"}
        </replace-with>
    </replace>
    <replace attribute-match="/*/PrimaryServerList/1/host" value-match="(.*)">
        <replace-with n="1">${"/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/
 Instance/oam_server2/host"}
        </replace-with>
    </replace>
</changes-to-include>

Note:

The OAM Managed Servers such as oam_server1 and oam_server2 must be updated with the names specified during deployment.

A load balancer is recommended between the WebGate and Access Manager server. In this case, you do not have to update the PrimaryServerList across data centers and can remove this transformation rule from the XML.

The following example illustrates how to change the transformation rule to update the PrimaryServerList only for WebgateAgent1 and WebgateAgent2 agents and not WebGate agents.

<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/
 Agent/WebGate/Instance">
    <replace attribute-match="/WebgateAgent1/PrimaryServerList/*/host" value-match="(.*)">
        <replace-with n="1">${"/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/
 Profile/OAMServerProfile/OAMSERVER/serverhost"}
        </replace-with>
    </replace>
    <replace attribute-match="/WebgateAgent2/PrimaryServerList/*/host" value-match="(.*)">
        <replace-with n="1">${"/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/
 Profile/OAMServerProfile/OAMSERVER/serverhost"}
        </replace-with>
    </replace>
</changes-to-include>

The logoutRedirectUrl attribute updates the logout URL protocol, host and port for all WebGate agents with respective values from the Clone. If a load balancer is used globally to define the logout URL for all WebGate agents in the Master environment, you don't need to replace the logout URL in the Clone environment and can remove the transformation rule. If you are using a DCC authentication scheme and a global load balancer host name to define the DCC login and logout URL, then again you don't need to replace the login and logout URL in the Clone environment and can remove the transformation rule.

19.6 Disabling Automated Policy Synchronization

Disable and then delete the replication agreements from the Master and Clone data centers.

Query the existing replication agreements using the following command and obtain the replication Identifier, replId; use it in Step 2 through Step 4.
```
curl -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/agreements'
```
Note:
If there are multiple replication agreements, select the identifier for which APS needs to be disabled by querying the corresponding Clone Data Center.

Disable the replication agreement on Clone data center.

curl -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/replId' -d '{"enabled":"false","replicaType":"CONSUMER"}'

Disable the replication agreement on Master data center.

curl -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/replId' -d '{"enabled":"false","replicaType":"SUPPLIER"}'

Delete the replication agreement from Master and Clone data centers.

curl -u weblogic:password -H 'Content-Type: application/json' -X DELETE 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/replId'

19.7 Best Practices for Replication

Here are some best practices to help you set up data replication.

The following points and the information in the sections should be taken into account when setting up data replication.

It is recommended that as many Policy Domain artifacts are created as possible before cloning. This will help the replication manager work efficiently during incremental updates.
The OAM server instance list will be used as the Well Known Addresses (WKA) to create a Coherence cluster so do not add other data center servers to the server instance list.
To allow for a WebGate profile to point to a remote data center in the secondary server list, use the Other option to provide OAP with the host and port details of the remote data center.
See Enabling Replication Logs
See Changing the User Identifier

19.7.1 Enabling Replication Logs

To get detailed logs on replication agreement and replication poll-related issues, enable the logger ‘oracle.oam.replication' by executing the WLST command.

setLogLevel(logger="oracle.oam.replication", level="TRACE:32", persist="0",
target="AdminServer")

This enables logger only till next shutdown of AdminServer. To keep the logger state across restart, set the persist attribute to “1”.

19.7.2 Changing the User Identifier

While creating replication agreement if you have not specified any authorization header of the user to be used for replication if the user's password got changed at later point, you can edit the replication agreement with the latest user identity and password.

Run the following command:

curl -u <repluser> -H 'Content-Type: application/json' -X PUT
'https://supplier.example.com:7002/oam/services/rest/_replication/201409231
329353668' -d '{"replicaType":"CONSUMER","config":{"entry":
{"key":"authorization","value":"BASIC d2VibG9naWM6d2VsY29tZTE="}}}'