19 Synchronizing Data In A Multi-Data Center
The Multi-Data Center infrastructure can be configured to keep Access Manager data synchronized across multiple data centers. This can be done using the Automated Policy Synchronization Replication Service or data can be replicated manually.
The following topics describe how to synchronize across data centers and replicate data:
19.1 Understanding the Multi-Data Center Synchronization
The Multi-Data Center (MDC) infrastructure can be configured to keep Access Manager data synchronized across multiple data centers.
Policy, system configuration, and partner metadata are all synchronized with APS.
Note:
For creating a Clone (also referred to as a Consumer) from a Master (also referred to as a Supplier), MDC Admin REST APIs are used. Once the MDC infrastructure is deployed, APS can be enabled to automatically sync any changes from the Master to the Clones.
APS (also referred to as the Replication Service) is a set of REST API. The binaries are installed as part of the Access Manager application and deployed in the AdminServer. It is enabled by default. After enabling the service, create a pull model Replication Agreement between the Clone data center and the Master. The Clone polls for changes from the Master as long as the Replication Agreement is valid for it. Conversely, the Master will respond to the Clone's request as long as it finds a valid replication agreement. The Clone applies the changes locally.
Note:
APS is optional; administrator-initiated import and export based replication is still available using WLST command procedure used previous to this release.
When setting up the Replication Service, the following may occur:
-
Replication Agreement is established. One data center is registered as a Replication Clone and another one is registered in a separate geographical location as its Master. From the Master data center, the changes are pulled and applied to the Clone data center.
-
Data center-specific configurations that may not be replicated across data centers are defined.
-
Access Manager configuration changes are tracked in each data center; the current replication state can be queried in any of the data centers.
-
Change Log is generated; it can be applied in the context of a similar setup running in another data center.
-
A pull from the Master data center can be triggered, if required; for example, if there is a failure of automated replication.
-
Access Manager configuration artifacts in the Master-Clone model are replicated.
Note:
APS does not sync IDS Profiles, OAM Keystores, and Oracle Platform Security Services artifacts (jps-config.xml changes, credential store configuration and the like).
This section describes the following topics:
19.1.1 How Replication Works
Replication works in a Master-Clone topology. In this topology, multiple Clones pull changes from a single Master. One data center is defined by the administrator as the Master and one or more other data centers are Clones.
The administrator makes changes to the Master that are replicated to the Clones. Only Master to Clone replication is supported; changes to Clones are not replicated back to the Master.
Note:
Multi-master replication is not supported.
To partake in replication, the Master data center (initiator of the replication) and the Clone data center (receiver of the changes) must have a Replication Agreement stored in the Access Manager data store. Table 19-1 documents the states in which replication can be deployed.
Table 19-1 Replication States
State | Definition |
---|---|
Active |
An Access Manager domain (including Admin and managed servers) is setup to serve access requests. In an active state, the Access Manager server provides the web access management functionality without additional MDC features. |
Bootstrapping |
This state is optional for some Data Centers; for example, the first one in an MDC topology. A Data Center goes through this intermediate state when added to an existing MDC topology. The new DC contacts the master and bootstraps itself to the same state. The bootstrap includes synchronizing the server keys, policy artifacts, partners, and system configuration. After completion of bootstrapping, the DC will be Replication Ready. |
Replication Ready |
In this state, MDC is enabled, the DC is made part of the topology, and the replication service is enabled. Once enabled, a clone can be registered with the master via a Replication Agreement. Once established, clone DCs can query and start pulling changelogs from the master. |
Figure 19-1 illustrates the replication flow.
Each Clone pulls changes from the Master. A replication thread runs on the Clone after the pre-configured interval of time and fetches changes from the Replication Service (REST endpoint) running on the Master environment.
For every cloned environment, the Master keeps track of a change sequence number indicating when it was last synced. The Master also keeps track of the list of changes (Create/Update/Delete) that have been pulled by the Clones in a changelog using specific change sequence numbers. When updating configuration metadata, the Clone can also change environment specific parameter values depending on transformation rules.
19.1.2 Understanding the Replication Agreement
When a new data center is added to an existing MDC topology, it has to bootstrap itself to be in sync with the existing data centers. This bootstrap operation will get the current Access Manager policies, system configuration, partner metadata, and server keys for the existing MDC topology. After the bootstrap operation, the new data center captures the last change sequence number from the topology's Master so that during replication it can be used to determine the current state.
Note:
Automated bootstrap is the ideal scenario but you can execute diagnostic MDC REST APIs first to ensure the Master and Clone data centers are in the same state. If the data centers are not in same state, then you have to execute MDC ADMIN REST APIs.
To establish a Replication Agreement, the Clone data center must know the Master's changelog sequence number. If the data center is added to the topology on 'day–0' and the Replication Agreement was created on 'day–1', there is a need to bootstrap again. To avoid this and to keep the flow simple, creating a Replication Agreement should take care of the bootstrap and actual replication agreement creation.
19.1.3 About Synchronizing Data Manually in a Multi-Data Center
Data in an MDC topology can also be synced manually. The manual option uses WLST export and import calls to migrate the data from one data center to another. Partner profiles and policies should both be exported and imported using WLST.
See Access Manager WLST Commands in Oracle Fusion Middleware WebLogic Scripting Tool Command Reference for Identity and Access Management.
19.2 Enabling Data Replication
-
Validate that the REST endpoints are enabled in the Master data center.
curl -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/hello
-
Validate that the REST endpoints are enabled in the Clone data center.
curl -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/hello
-
Repeat the process of validation on all Clone data centers.
19.3 Synchronizing Master and Clone Metadata
The process for syncing metadata across an MDC involves first syncing Access Manager UDM metadata and then creating a replication agreement.
See Understanding the Replication Agreement.
The following topics describe how to synchronize master and clone metadata:
19.4 Using REST API for Replication Agreements
Use REST APIs provided by Access Manager to manage replication agreements.
This section describes the following topics:
19.4.1 Querying for Replication Agreement Details
Execute a REST request at the Master's endpoint to query the details of the Replication Agreement (including the polling interval) between a Master and a Clone data center.
-
If the replication agreement identifier is unknown, run the following command to list all the replication agreement identifiers, else skip this step.
curl -k -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/agreements' Sample output 1: {"featureEnabled":"true","identifiers":"201411211137273612","ok":"true"} Sample output 2: {"featureEnabled":"true","identifiers":["201411211137273612","201411211137273900"],"ok":"true"}
-
Query the details of a Clone data center's replication agreement (including the polling interval) as follows:
curl -k -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231329353668?type=consumer' Sample output: {"enabled":"true","identifier":"201409231329353668","ok":"true", "pollInterval":"60","startingSequenceNumber":"110","state":"READY"}
-
Query the details of a Master data center's replication agreement (including the polling interval) as follows:
curl -u weblogic:password 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231329353668' Sample output: {"enabled":"true","identifier":"201409231329353668","lastSequenceNumber":"110","ok":"true","pollInterval":"3600","startingSequenceNumber":"110","state":"ACTIVE"}
Note:
The poll interval of the Master data center's replication agreement does not affect the actual replication.
19.4.2 Creating a Replication Agreement
Creating a Replication Agreement is a one time operation which will enable the Clone data center(s) to pull changes from the Master data center.
The replication agreement can be created using any REST client. In this procedure, we use the standard Curl utility.
After you execute this command, the following results occur:
-
Insert an entry in the Master's Replication Agreement store containing details regarding the Clone that wants to pull changes.
-
Insert an entry in the Clone's Replication Agreement store containing details regarding the Master from which it will pull changes. Replication configuration values like the poll interval will also be set.
To create a replication agreement:
19.4.3 Modifying a Replication Agreement
In this example, the value of poll Interval will be changed to 60 seconds.
Service responds back with JSON object that is the status of replication agreement before making the change. You need to fetch replication agreement status again to see updated configuration.
19.4.4 Deleting a Replication Agreement
Disable the replication agreement before you remove it from on the Clone and Master data centers.
-
Disable the replication agreement on the Clone as follows:
curl -u <repluser> -H 'Content-Type: application/json' -X PUT 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231 329353668' -d '{"enabled":"false","replicaType":"CONSUMER"}''
-
Disable the replication agreement on the Master as follows:
curl -u <repluser> -H 'Content-Type: application/json' -X PUT 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231 329353668' -d '{"enabled":"false","replicaType":"SUPPLIER"}''
-
Delete the replication agreement as follows:
curl -u weblogic:welcome1 -H 'Content-Type: application/json' -X DELETE 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231 329353668'
19.5 Customizing Transformation Rules
Transformation rules are used by APS and you can modify rules, as required.
The transformation rules illustrated in the following example are the default rules provided by Access Manager. A Clone can be configured to override these OOTB rules. This section documents how some of these rules can be modified and how to configure Access Manager to recognize these custom rules.
-
Specify the custom transformation rules in a file (say,
transformationRules.txt
). It is recommended to copy the default OOTB transformation rules specified as follows and modify rules as required.<?xml version="1.0" encoding="UTF-8"?> <mdc-transform-rule> <changes-to-include entity-path="/policy"/> <changes-to-include entity-path="/oauth"/> <changes-to-include entity-path="/IDM"/> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance" /> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/AuthenticationModules"/> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/oamproxy"/> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/Sme/SessionConfigurations"/> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile"> <ignore attribute-match="/OAMSERVER/serverprotocol"/> <ignore attribute-match="/OAMSERVER/serverhost"/> <ignore attribute-match="/OAMSERVER/serverport"/> <ignore attribute-match="/OAMSERVER/serversslterminated"/> <ignore attribute-match="/HostAlias/oamserverHttps/serverprotocol"/> <ignore attribute-match="/HostAlias/oamserverHttps/serverhost"/> <ignore attribute-match="/HostAlias/oamserverHttps/serverport"/> </changes-to-include> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER"> <ignore attribute-match="/serverprotocol"/> <ignore attribute-match="/serverhost"/> <ignore attribute-match="/serverport"/> <ignore attribute-match="/serversslterminated"/> </changes-to-include> <changes-to-include entity-path="/config/NGAMConfiguration/DataCenterConfiguration/Cluster"> <ignore attribute-match="/DataCenterType"/> <ignore attribute-match="/ClusterId"/> <ignore attribute-match="/WriteEnabledFlag"/> </changes-to-include> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Descriptors/OAMSEntityDescriptor" /> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Federation/IdentityProvider" /> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Federation/ServiceProvider" /> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedattributeprofiles" /> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedpartnerprofiles" /> <changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedserverconfig" /> </mdc-transform-rule>
The rule document will mention the XPath of system configuration artifacts to be replicated for a clone. If there is any transformation to be done during replication for the entry in XPath, it can be provided as replacement rule for that clone. To add a new XPath for replication to a clone, create a new transformation XML file, using the above XML document as a template. Add and remove XPaths as required. For example, adding the following XPath as the child of an <mdc-transformation-rule> node and saving the file to the clone's file system will modify Available Services.
-
Add the following environmental variable to the EXTRA_JAVA_PROPERTIES of
setDomainEnv.sh
(On linux or Unix) orsetDomainEnv.cmd
(On Windows) file available at$MW_HOME/user_projects/domains/OAMDomain/bin
.-Doracle.oam.MDCRuleFile=/scratch/transformationRules.txt
Each Clone can use different transformation rules. Figure 19-3 illustrates how to apply these custom rules.
Figure 19-3 Applying Custom Transformation Rules
Description of "Figure 19-3 Applying Custom Transformation Rules" -
Restart Clone Admin and Managed Servers.
This transformation rule will not change the WebGate agent definitions. The following information details how you can modify these changes for the PrimaryServerList and logoutRedirectUrl attributes.
-
You can use the transformation rule in the following example to update the PrimaryServerList and logoutRedirectUrl attributes with the Access Manager server host from the Clone environment. This change can be viewed in the oam-config.xml file; it replaces the value of the PrimaryServerList attribute with the value equal to ${DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}; for example, oam1-lon.example.com. The limitation of this rule is that it updates all servers in the primary list.
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance"> <replace attribute-match="/*/PrimaryServerList/*/host" value-match="(.*)"> <replace-with n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with> </replace> <replace attribute-match="/*/UserDefinedParameters/logoutRedirectUrl" value-match="(.*)<a target="_blank" href="://">://</a>(.*):(.*)/oam/server/logout"> <replace-with n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverprotocol}</replace-with> <replace-with n="2">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with> <replace-with n="3">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverport}</replace-with> </replace> <replace attribute-match="/*/logoutRedirectUrl" value-match="(.*)<a target="_blank" href="://">://</a>(.*):(.*)/oam/server/logout"> <replace-with n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverprotocol}</replace-with> <replace-with n="2">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with> <replace-with n="3">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverport}</replace-with> </replace> </changes-to-include>
You can use the transformation rule in the following example to update servers in PrimaryServerList with the different Clone servers.
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance"> <replace attribute-match="/*/PrimaryServerList/0/host" value-match="(.*)"> <replace-with n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Instance/oam_server1/host}</replace-with> </replace> <replace attribute-match="/*/PrimaryServerList/1/host" value-match="(.*)"> <replace-with n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Instance/oam_server2/host}</replace-with> </replace> </changes-to-include>
Note:
The OAM Managed Servers such asoam_server1
andoam_server2
must be updated with the names specified during deployment.A load balancer is recommended between the WebGate and Access Manager server. In this case, you do not have to update the PrimaryServerList across data centers and can remove this transformation rule from the XML.
The following example illustrates how to change the transformation rule to update the PrimaryServerList only for WebgateAgent1 and WebgateAgent2 agents and not WebGate agents.
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance"> <replace attribute-match="/WebgateAgent1/PrimaryServerList/*/host" value-match="(.*)"> <replace-with n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with> </replace> <replace attribute-match="/WebgateAgent2/PrimaryServerList/*/host" value-match="(.*)"> <replace-with n="1">${/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER/serverhost}</replace-with> </replace> </changes-to-include>
-
The logoutRedirectUrl attribute updates the logout URL protocol, host and port for all WebGate agents with respective values from the Clone. If a load balancer is used globally to define the logout URL for all WebGate agents in the Master environment, you don't need to replace the logout URL in the Clone environment and can remove the transformation rule. If you are using a DCC authentication scheme and a global load balancer host name to define the DCC login and logout URL, then again you don't need to replace the login and logout URL in the Clone environment and can remove the transformation rule.
19.6 Disabling Automated Policy Synchronization
Disable and then delete the replication agreements from the Master and Clone data centers.
19.7 Best Practices for Replication
Here are some best practices to help you set up data replication.
The following points and the information in the sections should be taken into account when setting up data replication.
-
It is recommended that as many Policy Domain artifacts are created as possible before cloning. This will help the replication manager work efficiently during incremental updates.
-
The OAM server instance list will be used as the Well Known Addresses (WKA) to create a Coherence cluster so do not add other data center servers to the server instance list.
-
To allow for a WebGate profile to point to a remote data center in the secondary server list, use the Other option to provide OAP with the host and port details of the remote data center.
19.7.1 Enabling Replication Logs
To get detailed logs on replication agreement and replication poll-related issues, enable the logger ‘oracle.oam.replication' by executing the WLST command.
setLogLevel(logger="oracle.oam.replication", level="TRACE:32", persist="0", target="AdminServer")
This enables logger only till next shutdown of AdminServer. To keep the logger state across restart, set the persist attribute to “1”.
19.7.2 Changing the User Identifier
While creating replication agreement if you have not specified any authorization header of the user to be used for replication if the user's password got changed at later point, you can edit the replication agreement with the latest user identity and password.
Run the following command:
curl -u <repluser> -H 'Content-Type: application/json' -X PUT 'https://supplier.example.com:7002/oam/services/rest/_replication/201409231 329353668' -d '{"replicaType":"CONSUMER","config":{"entry": {"key":"authorization","value":"BASIC d2VibG9naWM6d2VsY29tZTE="}}}'