Setting Up the Multi-Data Center: A Sequence

20 Setting Up the Multi-Data Center: A Sequence

The sequence of steps in this chapter will help you to setup a Multi-Data Center with four nodes using Oracle Access Manager 12.2.1.3.0. The configuration spans two Data Centers with two nodes per Data Center. The nodes are configured in Active/Active Mode.

This chapter contains the following section.

20.1 Before You Begin

Before you proceed with MDC configuration ensure the system level requirements are met.

Read the following chapters before beginning the steps documented in this sequence for an understanding of Multi-Data Center and its features.

Confirm the following before you begin the Multi-Data Center set-up sequence.

Check that your operating system is up-to-date with all necessary patches applied.
Verify that each machine has more than 30 GB space available and more than 8GB of memory available.
Mount the binaries you will be using. The applicable Oracle software includes:
- Oracle Fusion Middleware Identity and Access Management 12c (12.2.1.3.0)
- Oracle WebLogic Server 12c (12.2.1.3.0)
- Oracle Database 12c (12.2.x.x)
- Oracle Fusion Middleware Repository Creation Utility 12c (12.2.1.3.0)
Add /etc/hosts entries on all four nodes being configured.
Verify that the Oracle Database is connected and accessible.
OAM Admin server in the Master and Clone data center should be SSL-enabled.
The OAM managed servers in the Master and Clone data centers should be SSL-enabled or the load balancer front-ending the OAM managed servers should be SSL-terminated or both. Before restarting the servers, ensure that the load balancer is configured in the OAM Admin Console of the Master data center.
All the managed servers in the Master and Clone data centers should be configured with the same security mode.
- Use SIMPLE mode to secure communication between OAM Servers and WebGates using out-of-box certificates.
- Use CERT mode, if you have access to a trusted third-party Certificate Authority (CA).
The ID Stores are configured for Master and Clone data centers and they have the same name.

20.2 Setting Up a Multi-Data Center

For a successful set-up of a Multi-Data Center with data replication, the configuration spans two data centers with two nodes per data center. The nodes are configured in Active-Active Mode. MDC Admin REST APIs are used for diagnostics and configuration of Master and Clone data centers.

Install the Java Development Kit (JDK) 1.8.0 on Data Center 1 Node 1 and set the appropriate environment variables.
Install WebLogic Server 12c (12.2.1.3.0) on Data Center 1, Node 1.

This process creates the Middleware Home (<MW_HOME>).
Install the Oracle Identity and Access Management 12c (12.2.1.3.0) software on Data Center 1, Node 1.

Oracle Identity and Access Management contains the Oracle Access Management suite which includes Oracle Access Manager.
Run the Repository Creation Utility (RCU) 12c (12.2.1.3.0) on Data Center 1, Node 1.

It creates and loads the appropriate database schemas for Oracle Identity and Access Management products. And you can run it from $MW_HOME/oracle_common/bin.
Configure Oracle Access Management on Data Center 1, Node 1 using the Oracle Fusion Middleware Configuration Wizard script, $MW_HOME/oracle_common/common/bin/config.sh script (on Linux or UNIX), or $MW_HOME\oracle_common\common\bin\config.cmd (on Windows). Minimally, you will be configuring:
- a new WebLogic domain
- an Oracle Access Management Administration Server
- an Oracle Access Management Managed Server
- Oracle Access Manager
Modify the following WebLogic scripts on Data Center 1, Node 1:

On Linux or UNIX:
1. Open startWeblogic.sh and startManagedWeblogic.sh using vi and enter the appropriate value for WLS_USER.
  
  Enter the password, if prompted, while starting the servers; do not hard code it.
2. Save startWeblogic.sh and startManagedWeblogic.sh.
On Windows:
1. Open startWeblogic.cmd and startManagedWeblogic.cmd using notepad and enter the appropriate value for WLS_USER.
  
  Enter the password, if prompted, while starting the servers; do not hard code it.
2. Save startWeblogic.cmd and startManagedWeblogic.cmd
Start the Administration and Managed Servers.
Repeat Step 1 through Step 7 to prepare other data centers until you have added all the required nodes of your Master and Clone data centers (Data Center 1, Node 2; Data Center 2, Node 1; Data Center 2, Node 2) to the MDC setup.
Run pack and unpack scripts within the same data center to create a Managed Server JAR and to copy the JAR between nodes.
1. Run pack.sh located in the <MW_HOME>/oracle_common/common/bin directory to create the JAR file on Data Center 1, Node 1.
```
./pack.sh -domain=$MW_HOME/user_projects/domains/OAMDomain 
-template=OAMManagedServer.jar -template_name=“OAM Domain" -managed=true
```
2. Copy OAMManagedServer.jar to the MW_HOME/oracle_common/common/bin directory on Data Center 1, Node 2.
3. Run unpack.sh to unpack the Managed server JAR on Data Center 1, Node 2. The JAR is used as a template to create the OAMDomain Domain Structure on Data Center 1, Node 2.
```
mkdir -p $MW_HOME/user_projects/domains/OAMDomain
cd <MW_HOME>/oracle_common/common/bin
./unpack.sh -domain=$MW_HOME/user_projects/domains/OAMDomain   -template=OAMManagedServer.jar
```
4. Repeat the same procedure (Step a through Step c) to create OAMManagedServer.jar on Data Center 2, node 1 and to copy it to MW_HOME/oracle_common/common/bin directory on Data Center 2, node 2.
At this point in the sequence, the Data Center 1 cluster, Data Center 2 cluster and its four nodes are configured and ready for Multi-Data Center configurations. Perform a validation check at this point:
- OAM Admin server in the Master and Clone data center should be SSL-enabled.
- The OAM managed servers in the Master and Clone data centers should be SSL-enabled or the load balancer front-ending the OAM managed servers should be SSL-terminated or both. Before restarting the servers, ensure that the load balancer is configured in the OAM Admin Console of the Master data center.
- All the managed servers in the Master and Clone data centers should be configured with the same security mode.
  - Use SIMPLE mode to secure communication between OAM Servers and WebGates using out-of-box certificates.
  - Use CERT mode, if you have access to a trusted third-party Certificate Authority (CA).
- The ID Stores are configured for Master and Clone data centers and they have the same name.
Optionally, You can run the diagnostic REST APIs on the Master and the Clone Data Centers to view the MDC configuration settings:
```
curl -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/configuration'
curl -k -u weblogic:password 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/configuration'
```
Verify the following from the output of the command:
- In dcConfigMap entry, MultiDataCenterEnabled should be false and MultiDataCenterPartners list should be empty.
- agentMap entry should be empty.
Note: If MultiDataCenterEnabled is true (MDC is already enabled) and the data center has to be setup again for some reasons, see Overwriting the Existing MDC Configuration or Recovering from an Inconsistent State

See MDC Diagnostic REST API in REST API for Multi Data Center in Oracle Access Manager.
Run the following command with appropriate values to configure the Master data center.
```
curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/master' -d '{"mdcTopologyType":"value", "masterMDCAgentID":"value","cloneMDCAgentID":"value", "accessClientPassword":"value","artifactPassword":"value","cloneServerURL":"value","agentKeyPassword":"value","certModeKeystorePassword":"value","masterServerURL":"value", "cloneAdminUserNamePassword":"value","trustStorePath":"value", "keyStorePath":"value", "artifactsZipLocation":"value"}'
```
- mdcTopologyType: Choose one of the two topology types available for MDC configuration, ACTIVE_ACTIVE or DISASTER_RECOVERY.
- masterMDCAgentID: Enter the MDC NAP Agent Name for the Master data center.
- cloneMDCAgentID: Enter the MDC NAP Agent Name for the Clone data center.
- accessClientPassword: Provide the password required to be used by the MDC NAP agents in Master and Clone data centers.
- artifactPassword: Provide the password that is used to protect cloning artifacts.
- cloneServerURL: Enter the URL of the Clone Admin server or the URL of the reverse proxy front ending the Clone Admin server.
- (Only for CERT mode) agentKeyPassword: Enter the agent key password used to register partners in the CERT mode.
- (Only for CERT mode) certModeKeystorePassword: Enter the keystore password used to protect clientTrustStore.jks and clientKeyStore.jks.
- (Optional) masterServerURL: Enter the URL of the Master Admin server or the URL of the reverse proxy front ending the Master Admin Server.
- (Optional) cloneAdminUserNamePassword: Enter the user credentials of the Clone data center’s Administrator if the username and password of the Administrator for Master and Clone data centers are different.
- (Optional)trustStorePath: Enter the following depending on SIMPLE or CERT mode:
  - For SIMPLE mode : Provide the path to oamclient-truststore.jks file if this file is available in folders other than $MW_HOME/user_projects/domains/OAMDomain/output/webgate-ssl-SHA-256/
  - For CERT mode : Provide the path to clientTrustStore.jks file if this file is available in folders other than $MW_HOME/user_projects/domains/OAMDomain/config/fmwconfig/oam-mdc-cert-artifacts/
- (Optional)keyStorePath: Enter the following depending on SIMPLE or CERT mode:
  - For SIMPLE mode : Provide the path to oamclient-keystore.jks file if this file is available in folders other than $MW_HOME/user_projects/domains/OAMDomain/output/webgate-ssl-SHA-256/
  - For CERT mode : Provide the path to clientKeyStore.jks file if this file is available in folder other than $MW_HOME/user_projects/domains/OAMDomain/config/fmwconfig/oam-mdc-cert-artifacts/
- (Optional) artifactsZipLocation: Provide the location where cloning artifacts has to be stored; specify only if cloning artifacts need to be stored in any location other than /tmp
Here are the sample Curl commands for configuring a Master data center in SIMPLE and CERT modes using Active-Active MDC topology:
- Using CERT mode:
```
curl  -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/master' -d '{"mdcTopologyType":"ACTIVE_ACTIVE", "masterMDCAgentID":"MDCmasterNAPagent","cloneMDCAgentID":"MDCcloneNAPagent", "accessClientPassword":"password","artifactPassword":"password","cloneServerURL":"https://oamadmin1-dc2.poc.com:7002","cloneAdminUserNamePassword":"weblogic:password","agentKeyPassword":"password", "certModeKeystorePassword":"password"}'
```
- Using SIMPLE mode:
```
curl  -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/master' -d '{"mdcTopologyType":"ACTIVE_ACTIVE", "masterMDCAgentID":"MDCmasterNAPagent","cloneMDCAgentID":"MDCcloneNAPagent", "accessClientPassword":"password","artifactPassword":"password","cloneServerURL":"https://oamadmin1-dc2.poc.com:7002","cloneAdminUserNamePassword":"weblogic:password"}'
```
See MDC Master REST API in REST API for Multi Data Center in Oracle Access Manager.
Run the following command with appropriate values to configure the Clone data center.
```
curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/clone' -d '{"masterServerURL":"value","artifactPassword":"value","masterAdminUserNamePassword":"value", "artifactsZipLocation":"value", "masterArtifactsZipLocation":"value"}'
```
- masterServerURL: Enter the URL of the Master Admin server or the URL of the reverse proxy front ending the Master Admin Server.
- artifactPassword: Provide the same password that protects cloning artifacts and used while setting up the Master data center
- (Optional) masterAdminUserNamePassword: Enter the user credentials of the Master data center’s Administrator if the username and password of the Administrator for Master and Clone data centers are different.
- (Optional)artifactsZipLocation: Provide the location where backup artifacts should be stored in Clone data center (artifacts present in Clone data center are backed up before replacing it with Master artifacts); specify only when the backup artifacts need to be stored in any location other than /tmp.
- (Optional) masterArtifactsZipLocation: Provide the location where cloning artifacts are present in Master data center; specify only when artifactsZipLocation was used in input while configuring the Master data center.
Here is the sample Curl command for configuring a Clone data center:
```
curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/clone' -d '{"masterServerURL":"https://oamadmin1-dc1.poc.com:7002/","artifactPassword":"password","masterAdminUserNamePassword":"password"}'
```
See MDC Clone REST API in REST API for Multi Data Center in Oracle Access Manager.
Run the following command to reconfigure the Clone Data Center:
```
curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST ' https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/clone/configuration'
```
Note:
This command does not require any input parameters. It updates the flag, DataCenterType to Clone. To make the clone write-protected, execute the WLST command setMultiDataCenterWrite(WriteEnabledFlag="false"). It ignores any update to clone configuration.

See MDC Reconfigure Clone REST API in REST API for Multi Data Center in Oracle Access Manager.
Restart Clone Administration and managed servers.

Run the following diagnostic REST API on the Master and the Clone Data Centers to verify MDC configurations:

curl -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/configuration'
curl -k -u weblogic:password 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/configuration'

Export the partner and policy information from Data Center 1, Node 1 and then import it to Data Center 2, Node 1.
1. To export, change to the $MW_HOME/oracle_common/common/bin directory and run WLST to export from Data Center 1, Node 1.
```
./wlst.sh
connect()
exportAccessStore(toFile=”<name and location of the master metadata ZIP file>”, namePath=”/”)
exit()
```
2. Copy the exported file (that is, <name and location of the master metadata ZIP file>) from Data Center 1, Node 1 to Data Center 2, Node 1. To import, change to the $MW_HOME/oracle_common/common/bin directory and run WLST to import on Data Center 2, Node 1.
```
./wlst.sh
connect()
importAccessStore(fromFile=”<name and location of master metadata ZIP file>”, namePath=”/”)
exit()
```

20.3 Enabling Automated Policy Synchronization

Enabling the Automated Policy Synchronization (APS) feature for automated data synchronization among the servers includes commands for testing the REST services as well as details on adding custom transformation rules to the synchronization.

See Synchronizing Data In A Multi-Data Center for details on APS and transformation rules.

After exporting the partner and policy information from Master data center to Clone data center, perform the following steps to enable APS:

Validate the REST services using the following commands:

curl -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/hello'
curl -u weblogic:password 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/_replication/hello'

Run the following command in the Master and Clone Data Centers to get the clusterName:

curl -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/configuration'
curl -k -u weblogic:password 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/dc/configuration'

Run the following command with appropriate values to setup replication agreement:
```
curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/setup' -d '{"name":"value","source":"value","target":"value","documentType":"ENTITY","config": {"entry":{"key":"authorization","value":"authzValue"}}}'
```
- name: Enter a name for the replication agreement.
- source: Enter the cluster name of the Master data center (clusterName obtained as a result of first command in step 2).
- target: Enter the cluster name of the Clone data center (clusterName obtained as a result of second command in step 2).
- documentType: Default value for this parameter is ENTITY.
- Config: The map that contains key value pairs.
  1. (Optional) authzValue: If the username and password of the Administrator for Master and Clone data centres are different, then enter the value of Authorization Header (authzValue) to be used for contacting the Clone data center. Authorization Header will be Base 64 encoded value of CloneAdminUser:CloneAdminPassword.
For example,
```
curl -k -u weblogic:password  -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/setup' -d '{"name":"DC12DC2","source":"e60ef-oamadmin1-dc1.u","target":"70d7b-oamadmin1-dc2.u","documentType":"ENTITY","config": {"entry":{"key":"authorization","value":"Basic b2FtQWRtaW5Vc2VyOldlbGNvbWUy"}}}'
```
Tip:
The configuration changes made to the Master data center will take at least 900 Seconds to get propagated to the Clone data center as the default POLLINTERVAL is 900 Seconds. However, you can modify this polling interval in the Clone data center using SQLDeveloper in the database after setting up the Replication Agreement. See Modifying Polling Interval in Clone Data Centers.

Note:
To enable Replication, it is not mandatory in 12c to set the flag -Doracle.oam.EnableMDCReplication to true.

Optionally, create transformation rules, /tmp/transformationrules.xml file. Use the following content and modify rules, as required:

<?xml version="1.0" encoding="UTF-8"?>
<mdc-transform-rule>
<changes-to-include entity-path="/policy"/>
<changes-to-include entity-path="/oauth"/>
<changes-to-include entity-path="/IDM"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Agent/WebGate/Instance" />
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/AuthenticationModules"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/oamproxy"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/Sme/SessionConfigurations"/>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile">
<ignore attribute-match="/OAMSERVER/serverprotocol"/>
<ignore attribute-match="/OAMSERVER/serverhost"/>
<ignore attribute-match="/OAMSERVER/serverport"/>
<ignore attribute-match="/OAMSERVER/serversslterminated"/>
<ignore attribute-match="/HostAlias/oamserverHttps/serverprotocol"/>
<ignore attribute-match="/HostAlias/oamserverHttps/serverhost"/>
<ignore attribute-match="/HostAlias/oamserverHttps/serverport"/>
</changes-to-include>
<changes-to-include
entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/OAMServerProfile/OAMSERVER">
<ignore attribute-match="/serverprotocol"/>
<ignore attribute-match="/serverhost"/>
<ignore attribute-match="/serverport"/>
<ignore attribute-match="/serversslterminated"/>
</changes-to-include>
<changes-to-include
entity-path="/config/NGAMConfiguration/DataCenterConfiguration/Cluster">
<ignore attribute-match="/DataCenterType"/>
<ignore attribute-match="/ClusterId"/>
<ignore attribute-match="/WriteEnabledFlag"/>
</changes-to-include>
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Descriptors/OAMSEntityDescriptor" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Federation/IdentityProvider" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Federation/ServiceProvider" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedattributeprofiles" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedpartnerprofiles" />
<changes-to-include entity-path="/config/NGAMConfiguration/DeployedComponent/Server/NGAMServer/Profile/STS/fedserverconfig" />
</mdc-transform-rule>

Add -Doracle.oam.MDCRuleFile=/tmp/transformationRules.txt to $MW_HOME/user_projects/domains/OAMDomain/bin/setDomainEnv.sh (on Linux on Unix) or $MW_HOME\user_projects\domains\OAMDomain\bin\setDomainEnv.cmd (on Windows) on Data Center 2, Node 1 only and save the file.

Restart the Administration and Managed Servers.

This completes the Multi-Data Center configuration and APS configuration! You can test the APS function by creating an agent and a policy on data center 1 and verifying that it auto migrates to data center 2.

20.4 Troubleshooting the Multi-Data Center Setup

These troubleshooting tips may help you diagnose and fix some common problems encountered during MDC configuration.

This section analyzes the following issues:

20.4.1 Unauthorized Error Displayed When the Authorization Header is Correct

When Authorization Header provided is correct, ‘Error 401–Unauthorized’ is displayed while executing the REST command.

You may see this error when the WebLogic user or password do not match with the OAM Admin users or passwords and/or when OAM Admin users or passwords are different for the Master and Clone data centers. You can fix this by disabling WebLogic authentication and then verify the status of the EnforceValidBasicAuthCredentials parameter.

Run the following REST commands in both, the Master and Clone data centers to disable WebLogic authentication as shown in the following example:

connect('weblogicUser','weblogicPassword','t3://localhost:7001')
edit()
startEdit()
cd('SecurityConfiguration/Your_Domain')
set('EnforceValidBasicAuthCredentials','false')
save()
activate()

Restart the Master and Clone servers.

Verify the status of EnforceValidBasicAuthCredentials parameter.

connect('weblogicUser','weblogicPassword','t3://localhost:7001') 
cd('SecurityConfiguration/Your_Domain')
ls()

From the list, confirm that the EnforceValidBasicAuthCredentials parameter is set to false.

20.4.2 Curl Command Returns Curl: (35) SSL Connect Error

You can fix the SSL connect error in two ways.

Verify your Curl version and update it to the latest version available.

Force specify the TLS version to 1.2 as follows:

 curl --tlsv1.2 -k -u weblogic:password 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/configuration'

Use any REST Client (available as extensions to web browsers).

20.4.3 APS Synchronization Failed With 401-UnAuthorized Error

When Admin Users and Password are different for Master and Clone data centers, the APS Synchronization fails. Update the Authorization Header in the replication agreement present in Clone data center to fix this issue.

Query the existing replication agreements using the following command and obtain the replication Identifier, replId

curl -k -u weblogic:welcome1 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/agreements'

Note: If there are multiple identifiers, select the identifier for which the replication agreement needs to be updated by querying the corresponding Clone data center.

Run the following command with appropriate values to update the Authorization Header in the replication agreement of Clone data center.

curl -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/"replId"'-d '{"replicaType":"CONSUMER","config":{"entry":{"key":"authorization","value":"authzvalue"}}}'

replId : Identifier obtained from the above command.
authzvalue : Enter the value of Authorization Header to be used for contacting the Master data center. Authorization Header will be Base 64 encoded value of MasterAdminUser:MasterAdminPassword.

For Example,

curl -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/201706200405204694' -d '{"replicaType":"CONSUMER","config":{"entry":{"key":"authorization","value":Basic d2VibG9naWM6d2VsY29tZTE="}}}'

20.4.4 Fail to Decrypt oamkeystore Data with Cipher Key from OAM Config

After executing the REST API for setting up the clone, OAM server logs show exceptions due to internal synchronization of keys. Restarting the Clone data center sets all the required internal keys.

The following exception is shown in OAM server logs after setting up the Clone data center:

<Error> <oracle.oam.config> 
<OAMSSA-08032> <Configuration event dispatch failed. oracle.security.am.common.utilities.exception.AmRuntimeException: 
Fail to decrypt oamkeystore data with cipher key from OAM config(/DeployedComponent/Server/NGAMServer/Profile/ssoengine/CipherKey) 
at oracle.security.am.engines.sso.adapter.OAMSessionConfiguration$ConfigListener.configurationChanged(OAMSessionConfiguration.java:295) 
at oracle.security.am.admin.config.BasicFileConfigurationStore$ListenerDispatcher.run(BasicFileConfigurationStore.java:961) 
Caused By: javax.crypto.BadPaddingException: Given final block not properly padded

Setup clone REST API internally synchronizes some keys from the Master data center. After executing the REST API for reconfiguring Clone data center, restart the Clone and this exception is not shown in OAM server logs. You can safely ignore this exception in this scenario.

20.4.5 Modifying the Polling Interval in Clone Data Centers

The configuration changes made to the Master data center will take at least 900 Seconds to get propagated to the Clone data center as the default POLLINTERVAL is 900 Seconds. However, the polling interval can be modified, if required.

Run the following command to modify pollInterval parameter:

Query the existing replication agreements using the following command and obtain the replication Identifier, replId

curl -k -u weblogic:welcome1 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/agreements'

Note: If there are multiple identifiers, select the identifier for which the replication agreement needs to be updated by querying the corresponding Clone data center.

curl -k -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/replId' -d '{"pollInterval":"value","replicaType":"CONSUMER"}'

Where

replId is the Identifier obtained from the above command

PollInterval is the time in seconds for the Clone data center to send the query to the Master data center for the latest updates

For example,

curl -u weblogic:password -H 'Content-Type: application/json' -X PUT 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/_replication/replId' -d '{"pollInterval":"60","replicaType":"CONSUMER"}'

20.4.6 Overwriting the Existing MDC Configuration or Recovering from an Inconsistent State

Set the forceOverWrite parameter to overwrite the existing MDC configuration.

Disable APS, if configured.
See Disabling APS.

Specify the forceOverWrite parameter as follows while configuring the Master data center:

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/master' -d'{"mdcTopologyType":"ACTIVE_ACTIVE","masterMDCAgentID":"MasterNAPAgent","cloneMDCAgentID":"CloneNAPAgent1","accessClientPassword":"Welcome123","artifactPassword":"password","cloneServerURL":"https://oamadmin1-dc2.poc.com:7002/","cloneAdminUserNamePassword":"weblogic:password","agentKeyPassword":"password", "certModeKeystorePassword":"password","forceOverWrite":"true"}'

Specify the forceOverWrite parameter as follows while configuring the Clone data center:

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/clone' -d '{"masterServerURL":"https://oamadmin1-dc1.poc.com:7002/", "artifactPassword":"password","masterAdminUserNamePassword":"oamAdminUser:password","forceOverWrite":"true"}'

Note:

After overwrite the existing MDC configuration, proceed with reconfiguring Clone data center and setting up replication agreement. See, Setting Up a Multi-Data Center and Enabling Automated Policy Synchronization.

20.4.7 Changing the Security Mode of Managed Servers in Working MDC Environment

The secure communication mode of the servers can be changed to CERT or SIMPLE.

Disable APS. See Disabling Automated Policy Synchronization

Change the WriteEnabled flag to true in Clone data center using WLST commands.

connect('weblogic','password','t3://localhost:7001')

domainRuntime()
setMultiDataCenterWrite(WriteEnabledFlag="true")

Modify all the Managed Server instances to the required security mode in Master and Clone data centers.
In CERT Mode, follow the additional steps in Master:
1. Setup OAM Servers in CERT mode. See
2. Copy the MDC Partner Certificates generated to %DOMAIN_HOME%/config/fmwconfig/oam-mdc-cert-artifacts/. See
  
  Note:
  Its not necessary to import the CERT mode certificates into .oamkeystore and to configure PEM KeyStore Alias and PEM KeyStore Alias Password in Clone data center.
3. Run the REST commands to configure Master and Clone data centers with an additional parameter forceOverWrite:truein the request.
  Example: While configuring the Master data center:
```
curl  -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/master' -d '{"mdcTopologyType":"ACTIVE_ACTIVE","masterMDCAgentID":"MasterNAPAgent","cloneMDCAgentID":"CloneNAPAgent1","accessClientPassword":"Welcome123","artifactPassword":"password","cloneServerURL":"https://oamadmin1-dc2.poc.com:7002/","cloneAdminUserNamePassword":"weblogic:password","agentKeyPassword":"password", "certModeKeystorePassword":"password","forceOverWrite":"true"}'
```
  Example: While configuring Clone data center:
```
curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/clone' -d '{"masterServerURL":"https://oamadmin1-dc1.poc.com:7002/", "artifactPassword":"password","masterAdminUserNamePassword":"oamAdminUser:password","forceOverWrite":"true"}'
```
4. Re-configure the Clone data center command and Setup Replication Agreement command. See
5. Restart Master and Clone Admin and Managed Servers.
6. Verify SSO between Master and Clone data centers.

20.4.8 MDC Session Adoption Issues in 11g–12c OAM Setup with SIMPLE Mode Servers

When the OAM servers are configured in SIMPLE mode, MDC session adoption issues are encountered in an MDC environment where 11g Master and 12c Clone (or vice versa) data centers are deployed.

Relax the java security posture for compatibility if SSL communication fails between a 11g Master data center and a 12c Clone data center.

Perform the following on the machine that has the 12c OAM Server:

Locate and edit the file $JAVA_HOME/jre/lib/security/java.security.

key - jdk.tls.disabledAlgorithms - remove TLSv1, TLSv1.1, MD5withRSA
key - jdk.certpath.disabledAlgorithms - remove MD5

Restart Admin and Managed Servers.
Verify MDC Session adoption use-case.

20.4.9 Request Failed When the Input Parameters Passed are Valid

The Curl command fails and displays 'Request failed' error where the input parameters are valid. Verify the syntax and remove any white spaces in your input.

Following is the command that has the correct format with no white spaces before or after the parameters:

curl  -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc2.poc.com:7002/oam/services/rest/mdc/clone' -d '{"masterServerURL":"https://oamadmin1-dc1.poc.com:7002/","artifactPassword":"password","masterAdminUserNamePassword":"oamAdminUser:password","forceOverWrite":"true"}'

20.4.10 Modifying Session Control Parameters

Modify and specify custom values for Session control parameters.

Run the following command separately in the Master and Clone data centers. When APS is enabled, changes made to the MDC Configuration using this commands will not be propagated to Clone data center(s).

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/mode' -d '{"config":{"entry":[{"key":"SessionMustBeAnchoredToDataCenterServicingUser","value":<<true (for Invalidate) or false (for No Invalidation)>>},{"key":"SessionDataRetrievalOnDemand","value":<<true (for Cross DC Retrieval) or false (for No Cross DC Retrieval)>>},{"key":"SessionContinuationOnSyncFailure","value":<<true (for Invalidation/Retrieval should succeed) or false (for Ignore failure)>>},{"key":"Reauthenticate","value":<<true (for Force Reauthentication) or false (for No Reauthentication)>>},{"key":"SessionDataRetrievalOnDemandMax_retry_attempts","value":<<the value equal to the binary that represents the number of attempts for data retrieval when it fails. DEFAULT: 2>>},{"key":"SessionDataRetrievalOnDemandMax_conn_wait_time","value":<<the value equal to the binary that represents the total amount of time in seconds to wait for a connection. DEFAULT: 1000>>}, {"key":"MDCGitoCookieDomain","value":<<the domain in which OAM_GITO cookie should be set. OPTIONAL: Set it in MDC Deployments where a common domain hierarchy can be derived>>}]}}'

For example,

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/mode' -d '{"config":{"entry":[{"key":"SessionMustBeAnchoredToDataCenterServicingUser","value":"true"},{"key":"SessionDataRetrievalOnDemand","value":"true"},{"key":"SessionContinuationOnSyncFailure","value":"true"},{"key":"Reauthenticate","value":"true"},{"key":"SessionDataRetrievalOnDemandMax_retry_attempts","value":"3"},{"key":"SessionDataRetrievalOnDemandMax_conn_wait_time","value":"80"}]}}'

20.4.11 Modifying Backward Compatibility Flag

Set isBackwardCompatible parameter to true to enable or false to disable backward compatibility in a data center. Use this parameter only when the Master and Clone data centers are running in different versions of OAM.

Note:

If all the data center are using 12.2.1.3.0 binary, this flag should not be enabled.

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/compatibility' -d '{"isBackwardCompatible":<<"true" to enable or "false" to disable backward compatibility if Master and Clone DCs are running different versions of OAM such as 11g and 12c respectively>>}'

For example,

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/compatibility' -d '{"isBackwardCompatible":"true|false"}

20.4.12 Disabling MDC

Set the isMultiDataCenterEnabled to false to disable MDC.

When APS is enabled, changes made to the MDC Configuration using this commands will not be propagated to Clone data center(s). Run the command separately in the Master and Clone data centers.

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/mode' -d '{"isMultiDataCenterEnabled":"false"}'

Note:

Do not set isMultiDataCenterEnabled to true as its not supported. See Modifying Session Control Parameters.

20.4.13 Backup Existing Artifacts in a Data Center

Specify appropriate values for artifactPassword and artifactsZipLocation to take a backup of existing artifacts in the data center.

When APS is enabled, changes made to the MDC Configuration using this commands will not be propagated to Clone data center(s). Run the command separately in the Master and Clone data centers.

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/backup' -d '{"artifactPassword":<<password used for protecting the cloning artifacts>>,"artifactsZipLocation":<<Location where Artifacts has to be stored. (OPTIONAL: specify if the Artifacts need to be stored in any location other than /tmp)>>}'

For example,

curl -k -u weblogic:password -H 'Content-Type: application/json' -X POST 'https://oamadmin1-dc1.poc.com:7002/oam/services/rest/mdc/dc/backup' -d '{"artifactPassword":"password","artifactsZipLocation":"/scratch"}'