The following standby OMS implementation is used for Enterprise Manager Release 12.1.0.2 and earlier installations. For Enterprise Manager Release 12.1.0.3 and later, see Chapter 18, "Enterprise Manager Disaster Recovery."
IMPORTANT:
Standby OMSs using Standby WebLogic Domain are still supported but have been deprecated and may be de-supported in a future release (see My Oracle Support Note 1563541.1 for details).BI Publisher server configuration is not supported on standby OMSs using Standby Weblogic Domain for Cloud Control 12.1.0.4 and later.
Consider the following before installing the standby Management Services.
Oracle recommends that this activity be done during a lean period or during a planned maintenance window. When new Oracle Management Service instances are installed on the standby site, they are initially configured to connect to the Management Repository database on the primary site. Some workload will be taken up by the new Management Service. This could result in temporary loss in performance if the standby site Management Services are located far away from the primary site Management Repository database. However there would be no data loss and the performance would recover once the standby Management Services are shutdown post configuration.
The primary site must be configured as per Cloud Control MAA guidelines described in previous sections. This includes Management Services fronted by an SLB and all Management Agents configured to upload to Management Services by the SLB.
The standby site must be similar to the primary site in terms of hardware and network resources to ensure there is no loss of performance when failover happens.
Configure storage used by the software library to be replicated at the primary and standby site. In the event of a site outage, the contents of this storage must be made available on the standby site using hardware vendor disk level replication technologies.
The shared storage used for the software library must be made available on the standby site using the same paths as the primary site.
For complete redundancy in a disaster recovery environment, a second load balancer must be installed at the standby site. The secondary SLB must be configured in the same fashion as the primary. Some SLB vendors (such as F5 Networks) offer additional services that can be used to pass control of the Virtual IP presented by the SLB on the primary site to the SLB on the standby site in the event of a site outage. This can be used to facilitate automatic switching of Management Agent traffic from the primary site to the standby site.
Oracle Platform Security Services (OPSS) is the underlying security platform that provides security to Oracle Fusion Middleware including products like WebLogic Server, SOA, and WebCenter and serves as the single security framework for both Oracle and third-party environments. After upgrading Enterprise Manager, you many encounter configuration problems when attempting to set up a standby site if the Weblogic domain name and the OPSS farm name do not match. Specifically, you will run into this issue after switching over from primary to standby, upgrading the new primary site, and then attempting to create new standby sites. The name used for the primary domain is the original name and can be found in the jps-config.xml file of the original primary site.
Install the first standby Management Service using the following steps:
Copy the emkey to the Management Repository by running the following command on the first Management Service on the primary site:
emctl config emkey -copy_to_repos
Export the configuration from the first Management Service on the primary site using:
emctl exportconfig oms -dir <location for the export file>
After the configuration is exported, do not make any configuration changes to the primary site till the standby management service is configured.
Install a Management Agent on the standby host if one does not already exist.
Perform a software-only install of the Enterprise Manager software using a modified version of the ”Add Management Service” Deployment Procedure.
From the Enterprise menu, select Provisioning and Patching, then select Procedure Library.
Select Add Oracle Management Service procedure and click Create Like.
Go to the Procedure Steps tab and select and disable the steps - ”Configure Oracle Management Service”, ”Targets Discovery” and ”Post Configuration Tasks”.
Save the modified deployment procedure and use it to install the Enterprise Manager software on the standby OMS host.
After the Deployment Procedure completes, delete the file emInstanceMapping.properties from <OMS Oracle Home>/sysman/config on the standby OMS host.
Configure the Management Service by running omsca in standby mode. Choose a different domain name for the standby. For example, if the primary WebLogic domain is GCDomain, choose GCDomainStby.
omsca standby -EM_PRIMARY_DOMAIN_NAME GCDomain -EM_DOMAIN_NAME GCDomainStby -NM_USER nodemanager -AS_USERNAME weblogic -nostart
When prompted for the Administration Server host and EM Instance host, enter the standby OMS hostname (or accept the default).
When prompted for the passwords, provide the same passwords as the primary site.
When prompted for Management Repository details, provide the primary database details.
Important:
When runningomsca standby, if the OMS Oracle Home has been installed in software-only mode, then you must first copy the ORACLE_HOME/sysman/config/farmkey from the primary OMS machine to this new Oracle Home at the same location (ORACLE_HOME/sysman/config).Configure the required plug-ins by running the following command:
pluginca -action deploy -isFirstOMS true -plugins <plugin-list> -oracleHome <oms oracle home> -middlewareHome <wls middleware home>
where plugin-list is the list of plug-ins returned by the SQL query
SELECT epv.plugin_id, epv.version FROM em_plugin_version epv, em_current_deployed_plugin ecp WHERE epv.plugin_type NOT IN ( 'BUILT_IN_TARGET_TYPE' , 'INSTALL_HOME') AND ecp.dest_type='2' AND epv.plugin_version_id = ecp.plugin_version_id;
and is a comma separated list in the following format: <plugin-id>=<plugin-version>,<plugin-id>=<plugin-version>,…
Example: oracle.sysman.empa=12.1.0.1.0,oracle.sysman.mos=12.1.0.1.0,oracle.sysman.emas=12.1.0.1.0,oracle.sysman.emfa=12.1.0.1.0,oracle.sysman.db=12.1.0.1.0,oracle.sysman.emct=12.1.0.1.0,oracle.sysman.vt=12.1.0.1.0,oracle.sysman.ssa=12.1.0.1.0
Copy over the configuration exported from the primary Management Service in step 2 above to the standby Management Service host. Import the exported configuration on the standby Management Service using:
emctl importconfig oms -file <full path of the export file>
Note this command will start the Management Service.
Stop the Management Service but leave the Administration Server running using:
emctl stop oms
Add the standby Weblogic Domain and associated targets:
The standby Weblogic Domain and associated targets can be added using the Guided Discovery process.
From the Setup menu, select Add Target, then select Add Targets Manually.
Select Add Targets Using Guided Process.
Select Oracle Fusion Middleware/WebLogic Domain from the Target Types menu.
Use the secure port (typically 7101) and, under Advanced, set the JMX Protocol to t3s.
Note:
The WebLogic targets, except the Administration Server, will be shown as down as the standby OMS is down at this stage.If you have Single Sign On configured on the primary site, follow the same steps to configure SSO on the standby OMS.
If you have Real User Experience Insight, AD4J Manager or BI Publisher configured on the primary site, follow the same steps to configure them on the standby OMS.
Copy the emkey to the Management Repository by running the following command on the first OMS on the primary site:
emctl config emkey -copy_to_repos
Export the configuration from the first OMS on the primary site.
emctl exportconfig oms -dir <location for the export file>
After the configuration is exported, do not make any configuration changes to the primary site until the standby OMS is configured.
On the remote host, perform a software-only installation of the standby OMS as described in the Oracle® Enterprise Manager Cloud Control Advanced Installation and Configuration Guide chapter "Installing Enterprise Manager Software Now and Configuring Later" (Installing in Graphical Mode). Ensure that you install the software binaries in the same middleware location as that of the primary OMS.
Note:
Make sure you select same set of plug-ins as the primary OMS.If you have installed extra plug-ins which are not part of the installation, then complete the following steps.
Connect to repository and execute the following query to obtain the list of installed plug-ins:
SELECT epv.plugin_id "plugin id", epv.version "version", epv.rev_version "revision", epv.su_entity_id "update id"
FROM em_plugin_version epv, em_current_deployed_plugin ecp
WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE', 'INSTALL_HOME')
AND ecp.dest_type='2'
AND epv.plugin_version_id = ecp.plugin_version_id;
Write down the returned list of plug-ins installed.
Extract the plug-in archives from the primary site.
emcli export_update -id=<update id> -host=<standby OMS host> -dir=<directory to export archives> <host credential options>
where <update id> is the value returned by the four column of query above.
Note that this command generates a file with a .zip extension. Rename the file from <file_name>.zip to <file_name>.opar .
Repeat above steps for all plugins. In standby OMS host, place all .opar files in one directory <plugin dir >.
Invoke runInstaller and choose software-only install. This will install only the middleware home and OMS Oracle homes.
To install the required plug-ins, run the plug-in install script.
<OMS_HOME>/sysman/install/PluginInstall.sh -pluginLocation <plugin dir>
where <plugin dir> is the directory where you placed extra plug-ins in step two above. Running this command displays a screen that will let you select the plug-ins to install. Verify that all the plug-ins listed in step 1 appear and that you select them. If there is a mismatch, OMS configuration may fail at a later step.
On the software-only installation, apply all the patches you applied on the primary OMS so that both the primary and standby OMSs are identical and are in sync.
Configure the OMS by running omsca in standby mode. Choose a different domain name for the standby. For example, if the primary WebLogic domain is GCDomain, choose GCDomainStby.
omsca standby -EM_DOMAIN_NAME GCDomainStby -NM_USER nodemanager -AS_USERNAME weblogic -nostart
When prompted for the Administration Server host and Enterprise Manager Instance host, enter the standby OMS hostname (or accept the default).
When prompted for the passwords, provide the same passwords as the primary site.
When prompted for Management Repository details, provide the primary database details.
Configure the required plug-ins by running the following command:
pluginca -action deploy -isFirstOMS true -plugins <plug-in list> -oracleHome <oms oracle home> -middlewareHome <wls middleware home>
where plug-in list is the list of plug-ins returned by the SQL query.
SELECT epv.plugin_id, epv.version FROM em_plugin_version epv, em_current_deployed_plugin ecp WHERE epv.plugin_type NOT IN ( 'BUILT_IN_TARGET_TYPE' , 'INSTALL_HOME') AND ecp.dest_type='2' AND epv.plugin_version_id = ecp.plugin_version_id;
The plug-in list must be a comma separate list in the following format:
<plugin-id>=<plugin-version>,<plugin-id>=<plugin-version>,…
Example:
oracle.sysman.empa=12.1.0.1.0,oracle.sysman.mos=12.1.0.1.0,oracle.sysman.emas=12.1.0.1.0,oracle.sysman.emfa=12.1.0.1.0,oracle.sysman.db=12.1.0.1.0,oracle.sysman.emct=12.1.0.1.0,oracle.sysman.vt=12.1.0.1.0,oracle.sysman.ssa=12.1.0.1.0
Copy over the configuration exported from the primary OMS in step 2 above to the standby OMS host. Import the exported configuration on the standby OMS:
emctl importconfig oms -file <full path of the export file>
Note that this command generates a warning about a failed export and prompts for confirmation in order to proceed. The warning can be ignored by entering "y" to proceed.
This command will automatically start the OMS.
Stop the Management Service but leave the Administration Server running using:
emctl stop oms
Configure the Management Agent on the second OMS host by running the following command from the OMS home:
$<AGENT_HOME>/sysman/install/agentDeploy.sh AGENT_BASE_DIR=<middleware_home>/agent OMS_HOST=<oms_host_name> EM_UPLOAD_PORT=<oms_port> AGENT_REGISTRATION_PASSWORD=<password> -configOnly
Note: If you have an SLB configured, then directly specify the host and port of the primary load balancer.If no SLB is configured, then use the secure upload port of primary OMS.
Example:
./agentDeploy.sh AGENT_BASE_DIR=$MIDDLEWARE_BASE/agent OMS_HOST=prim_oms_hhost.domain.com EM_UPLOAD_PORT=4900 AGENT_REGISTRATION_PASSWORD=password -configOnly
Deploy the required plug-ins on the Management Agent.
For information about deploying plug-ins, refer to the chapter on Managing Plug-Ins in the of the Enterprise Manager Cloud Control Administrator's Guide.
Add the standby WebLogic Domain and associated targets:
The standby WebLogic Domain and associated targets can be added using the Guided Discovery process.
From the Setup menu, select Add Target , then select Add Targets Manually. Select Oracle Fusion Middleware from the Target Types menu. Use the secure port (typically 7101) and, under Advanced, set the JMX Protocol to t3s.Note that the WebLogic targets, except the Administration Server, will be shown as "down" since the standby OMS is down at this stage.
If you have Single Sign-On configured on the primary site, follow the same steps to configure SSO on the standby OMS.
If you have Real User Experience Insight, AD4J Manager or BI Publisher configured on the primary site, follow the same steps to configure them on the standby OMS.
It is recommended that your standby site be similar in configuration as your primary site. This means configuring multiple OMS on your standby site, similar to your primary site.
Start the standby Administration Server by running the following command on the first standby Management Service:
emctl start oms –admin_only
Export the configuration from the first Management Service on the primary site using:
emctl exportconfig oms -dir <location for the export file>
After the configuration is exported, do not make any configuration changes to the primary site until the standby management service is configured.
Install a Management Agent on the standby host.
Perform a software-only install of the Enterprise Manager software using a modified version of ”Add Oracle Management Service” Deployment Procedure.
From the Enterprise menu, select Provisioning and Patching, then select Procedure Library.
Select Add Oracle Management Service procedure and then click Create Like.
Go to the Procedure Steps tab and select and disable the steps - ”Configure Management Service”, ”Targets Discovery” and ”Post Configuration Tasks”.
Save the modified deployment procedure and use it to install the Enterprise Manager software on the standby OMS host.
After the Deployment Procedure completes, delete the file emInstanceMapping.properties from <OMS Oracle Home>/sysman/config on the standby OMS host.
Configure the Management Service by running omsca.
omsca add -nostart
When prompted for Management Repository details, provide the primary database details.When prompted for Administration Server details, provide the standby administration server details.
Configure the required plug-ins by running the following command:
pluginca -action deploy –isFirstOMS false -plugins <plugin-list> -oracleHome <oms oracle home> -middlewareHome <wls middleware home>
where plugin-list is the list of plug-ins returned by the SQL query
SELECT epv.plugin_id, epv.version FROM em_plugin_version epv, em_current_deployed_plugin ecp WHERE epv.plugin_type NOT IN ( 'BUILT_IN_TARGET_TYPE' , 'INSTALL_HOME') AND ecp.dest_type='2' AND epv.plugin_version_id = ecp.plugin_version_id;
and is a comma separated list in the following format: <plugin-id>=<plugin-version>,<plugin-id>=<plugin-version>,…
Example oracle.sysman.empa=12.1.0.1.0,oracle.sysman.mos=12.1.0.1.0,oracle.sysman.emas=12.1.0.1.0,oracle.sysman.emfa=12.1.0.1.0,oracle.sysman.db=12.1.0.1.0,oracle.sysman.emct=12.1.0.1.0,oracle.sysman.vt=12.1.0.1.0,oracle.sysman.ssa=12.1.0.1.0
Copy over the configuration exported from the primary Management Service in step 1 above to the standby Management Service host. Import the exported configuration on the standby Management Service using:
emctl importconfig oms -file <full path of the export file>
Note this command will start the Management Service.
Stop the Management Service using:
emctl stop oms
Refresh the standby domain target from the console. This will present a guided workflow to discover and add the new managed server and associated targets.
If you have Single Sign On configured on the primary site, follow the same steps to configure SSO on the standby OMS.
If you have Real User Experience Insight, AD4J Manager or BI Publisher configured on the primary site, follow the same steps to configure them on the standby OMS.
Start the standby Administration Server by running the following command on the first standby Management Service:
emctl start oms –admin_only
Export the configuration from the first Management Service on the primary site using:
emctl exportconfig oms -dir <location for the export file>
After the configuration is exported, do not make any configuration changes to the primary site still the standby management service is configured.
On the remote host, perform a software-only installation of the standby OMS as described in the Oracle® Enterprise Manager Cloud Control Advanced Installation and Configuration Guide chapter "Installing Enterprise Manager Software Now and Configuring Later" (Installing in Graphical Mode). Ensure that you install the software binaries in the same middleware location as that of the primary OMS.
Note:
Make sure you select same set of plug-ins as the primary OMS.If you have installed extra plug-ins which are not part of the installation, then complete the following steps.
Connect to repository and execute the following query to obtain the list of installed plug-ins:
SELECT epv.plugin_id "plugin id", epv.version "version", epv.rev_version "revision", epv.su_entity_id "update id"
FROM em_plugin_version epv, em_current_deployed_plugin ecp
WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE', 'INSTALL_HOME')
AND ecp.dest_type='2'
AND epv.plugin_version_id = ecp.plugin_version_id;
Write down the returned list of plug-ins installed.
Extract the plug-in archives from the primary site.
emcli export_update -id=<update id> -host=<standby OMS host> -dir=<directory to export archives> <host credential options>
where <update id> is the value returned by the four column of query above.
Note that this command generates a file with a .zip extension. Rename the file from <file_name>.zip to <file_name>.opar .
Repeat above steps for all plugins. In standby OMS host, place all .opar files in one directory <plugin dir >.
Invoke runInstaller and choose software-only install. This will install only the middleware home and OMS Oracle homes.
To install the required plug-ins, run the plug-in install script.
<OMS_HOME>/sysman/install/PluginInstall.sh -pluginLocation <plugin dir>
where <plugin dir> is the directory where you placed extra plug-ins in step two above. Running this command displays a screen that will let you select the plug-ins to install. Verify that all the plug-ins listed in step 1 appear and that you select them. If there is a mismatch, OMS configuration may fail at a later step.
On the software-only installation, apply all the patches you applied on the primary OMS so that both the primary and standby OMSs are identical and are in sync.
Configure the Management Service by running omsca.
omsca add –nostart
When prompted for Management Repository details, provide the primary database details. When prompted for Administration Server details, provide the standby Administration Server details.To keep the ports consistent with your primary site, provide all the ports on the command line itself. For example:
omsca add -MSPORT 7202 -MS_HTTPS_PORT 7301 -EM_NODEMGR_PORT 7403 -EM_UPLOAD_PORT 4889 -EM_UPLOAD_HTTPS_PORT 4900 -EM_CONSOLE_PORT 7788 -EM_CONSOLE_HTTPS_PORT 7799 -nostart
Configure the required plug-ins by running the following command:
pluginca -action deploy –isFirstOMS false -plugins <plugin-list> -oracleHome <oms oracle home> -middlewareHome <wls middleware home>
where the plugin-list is the list of plug-ins returned by the SQL query above and is a comma separated list in the following format: <plugin-id>=<plugin-version>,<plugin-id>=<plugin-version>,…
Example oracle.sysman.empa=12.1.0.1.0,oracle.sysman.mos=12.1.0.1.0,oracle.sysman.emas=12.1.0.1.0,oracle.sysman.emfa=12.1.0.1.0,oracle.sysman.db=12.1.0.1.0,oracle.sysman.emct=12.1.0.1.0,oracle.sysman.vt=12.1.0.1.0,oracle.sysman.ssa=12.1.0.1.0
Copy over the configuration exported from the primary Management Service in step 1 above to the standby Management Service host. Import the exported configuration on the standby Management Service using:
emctl importconfig oms -file <full path of the export file>
Note this command emits a warning about a failed export and prompts for confirmation to proceed. The warning can be ignored by entering "y" to proceed.
Note this command will automatically start the OMS.
Stop the OMS using:
emctl stop oms
Configure the Management Agent on the second OMS host by running the following command from the OMS home:
$<AGENT_HOME>/sysman/install/agentDeploy.sh AGENT_BASE_DIR=<middleware_home>/agent OMS_HOST=<oms_host_name> EM_UPLOAD_PORT=<oms_port> AGENT_REGISTRATION_PASSWORD=<password> -configOnly
Note:
If you have an SLB configured, then directly specify the host and port of the primary load balancer.Deploy the required plug-ins on the Management Agent.
For information about deploying plug-ins, refer to the Updating Cloud Control chapter of the Enterprise Manager Cloud Control Administrator's Guide.
Refresh the standby domain target from the console. This will present a guided workflow to discover and add the new managed server and associated targets.
If you have Single Sign On configured on the primary site, follow the same steps to configure SSO on the standby OMS.
If you have Real User Experience Insight, AD4J Manager or BI Publisher configured on the primary site, follow the same steps to configure them on the standby OMS.
Update the standby SLB configuration by adding the standby Management Service(s) to the different pools on the SLB. Setup monitors for the new Management Service.
After the initial setup of the standby site, the standby site has to be kept in sync with the changes done on primary site. Transactions on the primary Management Repository get propagated to the Standby Management Repository automatically through Data Guard but the OMS side changes have to be redone manually on the standby site. The following sections describe this procedure for typical activities.
When patches are applied on the primary site Management Services, they have to be applied on the standby site Management Services too. Note that patches typically update the Oracle Homes (via the OPatch apply command) and optionally might require scripts to be run against the Management Repository. On the standby site, it is sufficient to update the Oracle Homes (via the OPatch apply command) and skip the running of scripts on the Management Repository because database changes are automatically propagated to the standby site using Data Guard.
When new plug-ins are deployed on the primary site or existing plug-ins upgraded or un-deployed on the primary site, the following procedures needs to be run on the standby site too to keep the Standby Management Services in sync. Note if the Standby Management Services are not kept in sync, they would fail to start when a switchover or failover operation is attempted.
The procedure below assumes that the standby site was setup as per the documented process and the standby management services are currently down and point to the primary repository. The plug-in(s) deployment on the primary site has been completed successfully.
Deploying a New Plug-in or Upgrading a Plug-in on a Standby Site
Extract the plug-in archives from the primary site
Go to the Self Update Home, click on Plug-ins, select the required plug-in and select export from the Action table menu. Note the EM CLI command from the popup that gets displayed.
emcli export_update -id=<update id> -deep -host=<standby OMS host> -dir=<directory to export archives> <host credential options>
Note that an additional option ”-deep” is required. This command would create 4 files on the destination directory specified. The filename <version>_OMS_<platform>_<revision>.zip is the one to be used in next step.
Start the Standby Administration Server, if it is down.
emctl start oms -admin_only
Run prerequisite checks for the plug-in and apply the required patch reported by after running the following command.
./emctl plugin deploy -action prereqcheck -isFirstOMS false -archives <exported_plugin_archive>
where <exported_plugin_archive> is the zip file that was exported in Step 1: <version>_OMS_<platform>_<revision>.zip
Important:
Deploying Plug-in RevisionsIf the plug-ins you are attempting to deploy have revision versions, you must add the -revisions <list_of_revisions> option when executing the emctl plugin deploy command. Without the -revisions option, the plug-in CA attempts to copy the plug-in XML for the base revision over the existing correct plug-in XML, thus overwriting revision information.
For example, the commands to check prerequisites and deploy 12.1.0.3.0 DB plug-in revision 20130402 on a stand-by OMS are as follows:
./emctl plugin deploy -action prereqcheck -isFirstOMS false -archives 12.1.0.3.0_OMS_2000_20130402.zip -revisions 20130402 ./emctl plugin deploy -action deploy -isFirstOMS false -archives 12.1.0.3.0_OMS_2000_20130402.zip -revisions 20130402
Configure the plug-in on the first standby OMS Oracle Home.
./emctl plugin deploy -action deploy -isFirstOMS false -archives <exported_plugin_archive>
where exported_plugin_archive represents the zip file <version>_OMS_<platform>_<revision>.zip
Repeat steps 3 and 4 for each standby additional OMS/
./emctl plugin deploy -action prereqcheck -isFirstOMS true -archives <exported_plugin_archive> ./emctl plugin deploy -action deploy -isFirstOMS true -archives <exported_plugin_archive>
This completes the plug-in deployment on the standby site.
When administrators modify sysman credentials on the primary site, the following procedure must be run on the standby site in order to keep the standby management service in sync with respect to sysman credentials.
If the sysman credentials for the primary and standby sites are not kept in sync, they will fail to start when switchover or failover operations are attempted. In addition, the administrator will not be able to log into the standby OMS once the switchover operation is complete.
The procedure below assumes that the standby site was set up according to the documented process and the standby management services are currently down and point to the primary repository. The plug-in(s) deployment on the primary site has been completed successfully.
Perform following steps on the standby management service:
Stop all Enterprise Manager processes.
emctl stop oms -all &
Stop any remaining Enterprise Manager processes using the "kill" command on UNIX platforms or by using the Process Explorer or similar process monitoring tool on Windows platforms.
Change the datasource password by running the following command:
emctl config oms -update_ds_pwd -ds_name sysman-opss-ds [-ds_pwd <new_pwd>]
Start the AdminServer by itself by running the following command:
emctl start oms -admin_only
Change the SYSMAN password:
emctl config oms -change_repos_pwd [-use_sys_pwd]
Restart all Enterprise Manager processes:
emctl stop oms -all &
emctl start oms
Outages can be planned as might be the case when performing upgrades or periodic maintenance, or unplanned as can happen in the event of hardware/software failure, or perhaps some environmental catastrophe. Regardless of the type of outage, you want to ensure that your IT infrastructure can be restored and running as soon as possible.
This section covers:
Warning:
The following switchover instructions are only for use with the Standby OMSs using Standby WebLogic Domain disaster recovery approach as originally used in Enterprise Manager Release 12.1.0.2 and earlier installations as detailed in this appendix.For switchover instructions for the Standby OMSs using Storage Replication disaster recovery approach used in Enterprise Manager Release 12.1.0.3 and later, see Chapter 18, "Enterprise Manager Disaster Recovery."
Switchover is a planned activity where operations are transferred from the Primary site to a Standby site. This is usually done for testing and validation of Disaster Recovery (DR) scenarios and for planned maintenance activities on the primary infrastructure.
This section describes the steps to switchover to the standby site. The same procedure is applied to switchover in either direction.
Enterprise Manager Console cannot be used to perform switchover of the Management Repository database. Use the Data Guard Broker command line tool DGMGRL instead.
Prepare the Standby Database
Verify that recovery is up-to-date. Using the Enterprise Manager Console, you can view the value of the ApplyLag column for the standby database in the Standby Databases section of the Data Guard Overview Page.
Shut down the Primary Enterprise Manager Application Tier.
Shutdown all the Management Service instances in the primary site by running the following command on each Management Service:
emctl stop oms -all
Verify Software Library Availability
Ensure all files from the primary site are available on the standby site.
Switch over to the Standby Database
Use DGMGRL to perform a switchover to the standby database. The command can be run on the primary site or the standby site. The switchover command verifies the states of the primary database and the standby database, affects switchover of roles, restarts the old primary database, and sets it up as the new standby database.
SWITCHOVER TO <standby database name>;
Verify the post switchover states. To monitor a standby database completely, the user monitoring the database must have SYSDBA privileges. This privilege is required because the standby database is in a mounted-only state. A best practice is to ensure that the users monitoring the primary and standby databases have SYSDBA privileges for both databases.
SHOW CONFIGURATION;
SHOW DATABASE <primary database name>;
SHOW DATABASE <standby database name>;
Start the AdminServer if it is not already running on the standby site.
emctl start oms -admin_only
Make the standby Management Services point to the Standby Database which is now the new Primary by running the following on each standby Management Service.
emctl config oms -store_repos_details -repos_conndesc <connect descriptor of new primary database> -repos_user sysman
Startup the Enterprise Manager Application Tier.
Startup all the Management Services on the standby site:
emctl start oms
Relocate Management Services and Management Repository target
The Management Services and Management Repository target is monitored by a Management Agent on one of the Management Services on the primary site. To ensure that the target is monitored after switchover/failover, relocate the target to a Management Agent on the standby site by running the following command on one of the Management Service standby sites.
emctl config emrep -agent <agent name> -conn_desc <connect descriptor of new primary database>
Switch over to the Standby SLB.
Make appropriate network changes to failover your primary SLB to standby SLB that is, all requests should now be served by the standby SLB without requiring any changes on the clients (browser and Management Agents).
Establish the old primary Management Services as the new standby Management Services to complete the switchover process.
Start the Administration Server on old primary site
emctl start oms -admin_only
This completes the switchover operation. Access and test the application to ensure that the site is fully operational and functionally equivalent to the primary site. Repeat the same procedure to switchover in the other direction.
Warning:
The following failover instructions are only for use with the Standby OMSs using Standby WebLogic Domain disaster recovery approach as originally used in Enterprise Manager Release 12.1.0.2 and earlier installations as detailed in this appendix.For failover instructions for the Standby OMSs using Storage Replication disaster recovery approach used in Enterprise Manager Release 12.1.0.3 and later, see Chapter 18, "Enterprise Manager Disaster Recovery."
This section describes the steps to failover to a standby database, recover the Enterprise Manager application state by re-synchronizing the Management Repository database with all Management Agents, and enabling the original primary database as a standby using flashback database.
Shut down all OMS components at the primary site if running.
Verify Software Library Availability.
Ensure all files from the primary site are available on the standby site.
Failover to Standby Database.
Shutdown the database on the primary site. Use DGMGRL to connect to the standby database and execute the FAILOVER command:
FAILOVER TO <standby database name>;
Verify the post-failover states:
SHOW CONFIGURATION;
SHOW DATABASE <primary database name>;
SHOW DATABASE <standby database name>;
Note that after the failover completes, the original primary database cannot be used as a standby database of the new primary database unless it is re-enabled.
Start the AdminServer on the standby site if it is not already running.
emctl start oms -admin_only
Make the standby Management Services point to the Standby Database which is now the new Primary by running the following on each standby Management Service.
emctl config oms -store_repos_details -repos_conndesc <connect descriptor of new primary database> -repos_user sysman
Resync the New Primary Database with Management Agents.
Skip this step if you are running in Data Guard Maximum Protection or Maximum Availability level as there is no data loss on failover. However, if there is data loss, synchronize the new primary database with all Management Agents.
On any one Management Service on the standby site, run the following command:
emctl resync repos -full -name "<name for recovery action>"
This command submits a resync job that would be executed on each Management Agent when the Management Services on the standby site are brought up.
Repository re-synchronization is a resource intensive operation. A well tuned Management Repository will help significantly to complete the operation as quickly as possible. Specifically if you are not routinely coalescing the IOT/indexes associated with Advanced Queueing tables as described in My Oracle Support note 271855.1, running the procedure before re-synchronization will significantly speed up the re-synchronization process.
Start up the Enterprise Manager Application Tier
Start up all the Management Services on the standby site by running the following command on each Management Service.
emctl start oms
Relocate Management Services and Management Repository target.
The Management Services and Management Repository target is monitored by a Management Agent on one of the Management Services on the primary site. To ensure that target is monitored after switchover/failover, relocate the target to a Management Agent on the standby site by running the following command on one of the standby site Management Service.
emctl config emrep -agent <agent name> -conn_desc <connect descriptor of new primary database>
Switchover to the Standby SLB.
Make appropriate network changes to failover your primary SLB to the standby SLB, that is, all requests should now be served by the standby SLB without requiring any changes on the clients (browser and Management Agents).
Establish Original Primary Database as Standby Database Using Flashback.
Once access to the failed site is restored, and if you had flashback database enabled, you can reinstate the original primary database as a physical standby of the new primary database using the following procedure.
Shut down all the Management Services in the original primary site.
emctl stop oms -all
Restart the original primary database in mount state:
shutdown immediate;
startup mount;
Reinstate the Original Primary Database
Use DGMGRL to connect to the old primary database and execute the REINSTATE command
REINSTATE DATABASE <old primary database name>;
The newly reinstated standby database will begin serving as standby database to the new primary database.
Verify the post-reinstate states.
SHOW CONFIGURATION;
SHOW DATABASE <primary database name>;
SHOW DATABASE <standby database name>;
Establish Original Primary Management Service as the standby Management Service.
Start the Administration Server on old primary site:
emctl start oms -admin_only
Monitor and complete Repository Re-synchronization
Navigate to the Management Services and Repository Overview page of Cloud Control Console. Under Related Links, click Repository Synchronization. This page shows the progress of the re-synchronization operation on a per Management Agent basis. Monitor the progress.
Operations that fail should be resubmitted manually from this page after fixing the error mentioned. Typically, communication related errors are caused by Management Agents being down and can be fixed by resubmitting the operation from this page after restarting the Management Agent.
For Management Agents that cannot be started due to some reason, for example, old decommissioned Management Agents, the operation should be stopped manually from this page. Re-synchronization is deemed complete when all the jobs have a completed or stopped status.
This completes the failover operation. Access and test the application to ensure that the site is fully operational and functionally equivalent to the primary site.
Perform a switchover procedure if the site operations have to be moved back to the original primary site.