4 Getting Started with Upgrading Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4), 12c Release 3 (12.1.0.3), or 12c Release 2 (12.1.0.2) to 12c Release 5 (12.1.0.5)

This chapter describes the high-level process for upgrading your Enterprise Manager 12c Release 4 (12.1.0.4), 12c Release 3 (12.1.0.3), or 12c Release 2 (12.1.0.2) to 12c Release 5 (12.1.0.5).

In particular, this chapter covers the following:

Note:

  • If you have Oracle Management Service 12c Release 1 (12.1.0.1) [with or without Bundle Patch 1], then first upgrade it to either 12c Release 2 (12.1.0.2), 12c Release 3 (12.1.0.3), or 12c Release 4 (12.1.0.4), and then upgrade to 12c Release 5 (12.1.0.5).

    For instructions to upgrade to 12c Release 2 (12.1.0.2), 12c Release 3 (12.1.0.3), or 12c Release 4 (12.1.0.4) refer to the Oracle Enterprise Manager Cloud Control Upgrade Guide for the respective release, available in the Enterprise Manager documentation library:

    http://docs.oracle.com/cd/E24628_01/index.htm

  • The Oracle Management Agent releases that are supported for Enterprise Manager Cloud Control 12c Release 5 (12.1.0.5) are 12c Release 5 (12.1.0.5), 12c Release 4 (12.1.0.4), 12c Release 3 (12.1.0.3), and 12c Release 2 (12.1.0.2). If you have any earlier releases of Oracle Management Agent, then before upgrading the Oracle Management Service to 12c Release 5 (12.1.0.5), make sure you upgrade your Oracle Management Agent to either 12c Release 4 (12.1.0.4), 12c Release 3 (12.1.0.3), or 12c Release 2 (12.1.0.2) using the Agent Upgrade Console present within the Enterprise Manager Cloud Control Console. For instructions, refer to Oracle Enterprise Manager Cloud Control Upgrade Guide.

  • If you want to take any additional preparatory steps for a successful upgrade, see My Oracle Support note 1682332.1.

  • If you want to see a list of known issues before starting the upgrade process, then see My Oracle Support note 2022505.1.

WARNING:

Do not upgrade 12c Release 4 (12.1.0.4), 12c Release 3 (12.1.0.3), or 12c Release 2 (12.1.0.2) while it is undergoing a 2-system upgrade from its earlier release (10.2.0.5 or 11.1.0.1). Wait until the upgrade completes fully because there might be some standalone Management Agents in status pending state while the upgrade is in progress.

4.1 Upgrading Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4), 12c Release 3 (12.1.0.3), or 12c Release 2 (12.1.0.2) to 12c Release 5 (12.1.0.5) in a Single-OMS or Multi-OMS Non-HA Environment

Table 4-1 describes the steps for upgrading your Enterprise Manager to 12c Release 4 (12.1.0.4) in a single-OMS or multi-OMS non-HA (or non-high availability) environment.

If you want to see a list of known issues before starting the upgrade process, then see My Oracle Support note 2022505.1.

Table 4-1 Upgrading Enterprise Manager Cloud Control to 12c Release 5 (12.1.0.5) in a Single-OMS Non-HA Environment

Step No. Step Procedure

Step 1

Prepare Yourself

  

(a)

Learn about the 1-System upgrade approach.

Section 2.1

(b)

Review the important facts you need to know before you begin.

Chapter 3

Step 2

Upgrade Oracle Management Service and Oracle Management Repository

  

(a)

Meet the following prerequisites:

  • Meet the Oracle Management Service-related prerequisites described in the chapter on installing Enterprise Manager Cloud Control, in the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

  • Upgrade to 12c Release 5 (12.1.0.5) is an out-of-place upgrade, therefore make sure your host meets the hardware requirements specific to 12c Release 5 (12.1.0.5) as described in Oracle Enterprise Manager Cloud Control Basic Installation Guide. Here, host refers to the host on which your current Enterprise Manager, which you want to upgrade, is running.

  • Ensure that you are upgrading only on the supported platforms as listed in Section 3.3.

  • If you want to upgrade your database, which houses the Management Repository, to 12c Release 1(12.1.0.2), then make sure you upgrade the database ONLY after you upgrade the Enterprise Manager system.

  • Ensure that the ports used by Enterprise Manager are not set to a value lower than or equal to 1024. If they are, then the upgrade will fail. Ports up to 1024 are typically reserved for root users (super users). Therefore, make sure the ports are greater than 1024.

  • Back up the OMS (the middleware home and the inventory), the Management Repository, and the Software Library. In case the upgrade fails, you can always restore using the backup. For instructions to back up, refer to Oracle Enterprise Manager Cloud Control Administrator's Guide.

  • Make a copy (using the Create Like action) of the out-of-box rule sets you are using to receive e-mail notifications. Otherwise, you will lose the e-mail subscriptions made to the rule sets.

    To make a copy, from the Setup menu, select Incidents, then select Incident Rules. On the Incident Rules - All Enterprise Rules page, in the table, select the out-of-box rule set you want to copy. Then, from the Actions menu, select Create Like Rule Set. In the Create Like Rule Set page, provide the required details and click Save.

  • Ensure that the contents of the following directory across OMS hosts are the same. If the number of plug-ins and/or their revisions are different across OMS hosts, then ensure that you copy them over from one pluginextract_with_rev directory to another to make them consistent.

    <OMS_HOME>/sysman/install/pluginextract_with_rev

    For example, on the first OMS host, OMS_Host1, you may have the following plug-ins and revisions:

    oracle.sysman.emfa 12.1.0.1.0 and 12.1.0.2.0; oracle.sysman.bda 12.1.0.1.0, 12.1.0.2.0, and 12.1.0.3.0; and oracle.em.sidb 12.1.0.1.0 and 12.1.0.2.0.

    On the additional OMS host, OMS_Host2, you may have the following plug-ins: oracle.sysman.emfa 12.1.0.1.0 and 12.1.0.2.0; oracle.sysman.bda 12.1.0.3.0; and oracle.em.sehc 12.1.0.1.0.

    In this case, ensure that you copy oracle.sysman.bda 12.1.0.1.0 and 12.1.0.2.0, and oracle.em.sidb 12.1.0.1.0 and 12.1.0.2.0 from the first OMS host to the additional OMS host, and ensure that you copy oracle.em.sehc 12.1.0.1.0 from the additional OMS host to the first OMS host. This way, the plug-ins and their revisions in both OMS hosts will be consistent.

  • If you have done any of the following customization to the OMS, then ensure that you remove all of them before starting the upgrade process. Once the upgrade is complete, you can redo the customization.

    • Configuring additional data source parameters in Weblogic.

    • Additional third-party SSL certificate.

    • Smart card authentication for Enterprise Manager login.

  

(b)

If you have Oracle BI Publisher installed on the Enterprise Manager Cloud Control 12c installation that you are upgrading, then do one of the following:

  • If you are upgrading from Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4), then run emctl stop oms -all on all the OMS instances, including the ones running a BI Publisher Server.

  • If you are upgrading from Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3) or lower, then stop the primary BI Publisher Managed Server named BIP using the WebLogic Administration Console.

As part of the upgrade to Enterprise Manager Cloud Control 12c Release 5 (12.1.0.5), Oracle BI Publisher 11.1.1.7.0 with the latest bundle patches is automatically installed in the middleware home. However, although Oracle BI Publisher is installed by default, it is not configured or upgraded from the earlier version by default. Therefore, after upgrading to 12c Release 5 (12.1.0.5), you must manually configure Oracle BI Publisher. This configuration step also migrates all the reports from the old BI Publisher home to the new one. For instructions, see Step 4 (f).

  

(c)

Check for any outstanding database service instance creation requests. If there are any requests in progress, allow them to complete. For requests that are scheduled, suspend them.

To do so, follow these steps.

  1. In Cloud Control, from the Enterprise menu, select Cloud, then select Self Service Portal.

  2. On the Infrastructure Cloud Self Service Portal page, right under the page title, select My Databases to view only database requests.

  3. In the Requests table, for requests that are in progress, allow them to complete. For requests that are scheduled, suspend them.

    To suspend the scheduled requests, click the request name. Click the Deployment tab. Click the deployment procedure listed there, and suspend it.

  

(d)

Ensure that the tables in the Management Repository do not have any snapshots created.

To verify this, log in to the Management Repository and run the following SQL query as SYSMAN user:

select master , log_table from all_mview_logs where log_owner='<EM_REPOS_USER>

For example,

select master , log_table from all_mview_logs where log_owner='SYSMAN'

If there are snapshots created in a table, then the query displays the master table and the snapshot details. For example,

SQL> master log_table

em-violations em$violation_log

If there are snapshots, then drop them by running the following command as SYSMAN user:

SQL> Drop snapshot log on <master>

For example,

SQL> Drop snapshot log on em-violations

  

(e)

Ensure that you do not have any login or logoff triggers set in the Oracle Database that houses the Oracle Management Repository.

To verify this, log into the database and run the following query:

  • Verify if any login triggers are set:

    SQL> SELECT COUNT (trigger_name) FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGIN%' AND status='ENABLED';

    SQL> SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGIN%' AND status='ENABLED';

  • Verify if any logoff triggers are set:

    SQL> SELECT COUNT (trigger_name) FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED';

    SQL> SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED';

If the query results in anything other than zero or no rows selected, then manually disable the triggers. After completing the upgrade, you can enable them again.

To disable the triggers, run the following query:

SQL> alter trigger <trigger_name> disable;

For example,

SQL> alter trigger EXPFIL_ALTEREXPTAB_MAINT disable;

  

(f)

Enable auditing of the Delete Target operation.

  • To view a list of operations, run the following command:

    emcli show_operations_list

    Some of the Delete operations include Delete Target, Delete Named Credential, Delete Role, Delete Rule, Delete Monitoring Template, Delete User.

  • To verify the Delete operations that are currently enabled, run the following command:

    emcli show_audit_settings

  • If the Delete Target operation is not already enabled, then run the following command to enable it.

    emcli update_audit_settings

    -operations_to_enable="name_of_the_operations_to_enable._For_all_operations_use_ALL"

    -audit_switch="ENABLE

    -directory="db_directory_name' (Should_be_configured_with_an_OS_directory_where_the_export_service_archives_the_audit_data_files)

    -file_prefix="file_prefix" (To be used by the export service to create the file name where audit data has to be written. The default value is em_audit. You can change per your standards for all sites)

    -file_size="file_size (Bytes)" (Maximum value of each file size. The default value for this is 5000000 bytes)

    -data_retention_period="data_retention_period (Days)" (Maximum period the Enterprise Manager repository stores audit data. The default value is 365 days)

    The aforementioned parameters to the help you set up or configure an archive location for archiving the audit data from the Management Repository to a file system after the retention period. Standardize this on all sites.

  

(g)

(CRITICAL MANDATORY STEP)

Apply the following patches on the database. You can access My Oracle Support and search for these patches. For instructions, see the ReadMe file associated with the patch. If you do not apply these patches, you will run into upgrade failures that cannot be corrected.

For Oracle Database 11 Release 1 (11.1.0.7)

  • On Unix platforms, apply patch 17082366 (Patch Set Update 17). Then apply patch 9577583 and patch 8405205.

  • On Microsoft Windows (32 bit and 64 bit) platforms, apply patch 17363760 (Patch 54).

For Oracle Database 11g Release 2 (11.2.0.1)

  • On Unix platforms, apply patch 12419378 (Patch Set Update 6).

  • On Microsoft Windows (64 bit) platforms, apply patch 13423278 (Patch 16).

For Oracle Database 11g Release 2 (11.2.0.2)

  • On Unix platforms, apply patch 11061801 and patch 9748749.

  • On Microsoft Windows (32 bit) platforms, apply patch 11061801 and patch 12429530.

  • On Microsoft Windows (64-bit) platforms, apply patch 11061801 and patch 12429531.

For Oracle Database 11g Release 2 (11.2.0.3), 10g Release 2 (10.2.0.5)

On Unix as well as Microsoft Windows platforms, apply the patch 11061801.

For Oracle Database 11g Release 2 (11.2.0.4) and 12c Release 1 (12.1.0.2)

No patches required for these releases.

Note: Oracle also recommends that you apply patch 13496395. For information on what database releases you can apply the patch, see the ReadMe that is packaged with the patch.

  

(h)

[IN CASE OF MULTI-OMS UPGRADE, PERFORM THIS STEP ONLY FOR THE FIRST OMS UPGRADE]

Copy the emkey from the existing OMS to the existing Management Repository. To do so, run the following command on the old OMS home you are trying to upgrade:

$<OMS_HOME>/bin/emctl config emkey -copy_to_repos [-sysman_pwd <sysman_pwd>]

To verify whether the emkey is copied, run the following command:

$<OMS_HOME>/bin/emctl status emkey

If the emkey is copied, then you will see the following message:

The EMKey  is configured properly, but is not secure.
Secure the EMKey by running "emctl config emkey -remove_from_repos".
 

(i)

If you have changed the default, out-of-box memory settings for an OMS instance, then preserve the changes so that they are not lost during upgrade.

To preserve the changes, follow these steps:

  1. Run the following command on all the OMS instances:

    $<OMS_HOME>/bin/emctl set property -name 'JAVA_EM_MEM_ARGS' -value '<java_memory_arguments>'

    For example,

    $<OMS_HOME>/bin/emctl set property -name 'JAVA_EM_MEM_ARGS' -value '-Xms256m -Xmx1740m'

  2. Restart all the OMS instances:

    $<OMS_HOME>/bin/emctl stop oms -all

    $<OMS_HOME>/bin/emctl start oms

  

(j)

Selectively skip some job type updates for reduced downtime of your Enterprise Manager system.

While upgrading the Enterprise Manager system, job types are registered. As part of the job type registration process, all active executions corresponding to a job type are automatically upgraded to the newly registered version of the job type. This job type upgrade process is skipped for all queued and waiting executions, thereby reducing the overall downtime of the Enterprise Manager system. However, in some cases, the Enterprise Manager system might experience a considerable backlog, and if such a backlog is not cleared before initiating the upgrade, then the downtime of the Enterprise Manager system can be much longer. To circumvent this issue, you can selectively skip or postpone the upgrade of certain job types so that they are upgraded only after the downtime.

To skip or postpone some job types from being upgraded, follow these steps:

  1. Identify the job types that you want to exclude from being upgraded during the downtime.

    To do so, as a SYSMAN user, log in to the database that houses the Oracle Management Repository, and run the following query. Particularly look for the job types that have a large number of active executions.

    SELECT job_type, COUNT(1) as n_execs

    FROM MGMT_JOB_EXEC_SUMMARY

    JOIN MGMT_JOB_TYPE_INFO USING (job_type_id)

    WHERE status NOT IN (3,4,5,8,18,19,23)

    GROUP BY job_type

    HAVING COUNT(1) > 5000

    ORDER BY COUNT(1) DESC;

  2. Exclude the job types PropagateTarget and ExecLoaderCallbacks.

    INSERT INTO MGMT_PARAMETERS(parameter_name, parameter_value) VALUES ('mgmt_job_skip_job_type_upg.1', 'PropagateTarget');

    INSERT INTO MGMT_PARAMETERS(parameter_name, parameter_value) VALUES ('mgmt_job_skip_job_type_upg.2', 'ExecLoaderCallbacks');

    COMMIT;

  3. Exclude the other job types you identified in the first step.

    To do so, run the following query to exclude a job type from the MGMT_PARAMETERS table. For each job type you want to exclude, you must run one INSERT statement. For example, if you have three job types to exclude, then run the INSERT statement three times, each with a job type you want to exclude.

    INSERT INTO MGMT_PARAMETERS(parameter_name, parameter_value) VALUES ('mgmt_job_skip_job_type_upg.1', '<job type>');

  

(k)

Shut down the OMS you are about to upgrade and also the other OMS instances that connect to it.

IMPORTANT: If you are upgrading a multi-OMS environment using the software-only upgrade approach, then skip this step. You can stop the OMS instances after copying the software binaries as described in Section 5.3.1 or Section 5.4.1.

  1. As a prerequisite, stop the JVMD and ADP engines explicitly:

    • To stop them in silent mode, run the following commands:

      $<OMS_HOME>/bin/emctl extended oms jvmd stop -all

      $<OMS_HOME>/bin/emctl extended oms adp stop –all

    • To stop them in graphical mode, access the weblogic console, and stop the JVMD and ADP weblogic managed servers manually.

  2. Now shut down the OMS you are about to upgrade and also the other OMS instances that connect to it:

    $<OMS_HOME>/bin/emctl stop oms -all

  

(l)

Shut down the Management Agent that monitors the Management Services and Repository target, to prevent the Management Agent from connecting to the Management Repository for metric collections. Not shutting down this Management Agent might cause the OMS upgrade to fail.

IMPORTANT: If you are upgrading a multi-OMS environment using the software-only upgrade approach, then skip this step. You can stop the Management Agent after copying the software binaries as described in Section 5.3.1 or Section 5.4.1.

  

(m)

Upgrade the OMS and the Management Repository. You can choose to upgrade in graphical or silent mode. You can also choose to install the software binaries at one point and upgrade them later in graphical or silent mode.

If you see an error message stating that you have not copied the emkey, then follow these steps:

  1. Copy the emkey from the old OMS to the old Management Repository. To do so, run the following command from the old OMS home you are trying to upgrade:

    $<OMS_HOME>/bin/emctl config emkey -copy_to_repos_from_file -repos_conndesc '"<conndesc>"' -repos_user <username> [-repos_pwd <pwd>] -emkey_file <OMS_HOME>/sysman/config/emkey.ora

    For example,

    /u01/software/oracle/middleware/oms/bin/emctl config emkey -copy_to_repos_from_file -repos_conndesc '"(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dbhost.mydomain.com)(PORT=1521)))(CONNECT_DATA=(SID=emrepos12)))"' -repos_user sysman -emkey_file /u01/software/oracle/middleware/oms/sysman/config/emkey.ora

    Note: Here, the Management Repository details are the details of the existing Management Repository you are trying to upgrade. And <OMS_HOME> is the OMS home you are trying to upgrade.

  2. Try upgrading the OMS and the Management Repository again.

IMPORTANT: Upgrade the Management Agent that was installed with the old OMS (that is, central agent) immediately after upgrading the old OMS. To upgrade this Management Agent, use the Agent Upgrade Console.

Section 5.1, Section 5.2, Section 5.3, or Section 5.4

Step 3

Upgrade Oracle Management Agent

  

(a)

Review the important facts you need to know before you begin upgrading the Management Agents.

Section 6.2

(b)

Meet the prerequisites.

Section 6.3

(c)

Ensure that you restart the Management Agent that you shut down in Step 2 (l).

  

(d)

Upgrade the Management Agents.

IMPORTANT: Upgrade the Management Agent that was installed with the old OMS (that is, central agent) immediately after upgrading the old OMS. To upgrade this Management Agent, use the Agent Upgrade Console.

Section 6.4

Step 4

Perform Postupgrade Task

  

(a)

Perform postupgrade tasks.

Section 13.6.1

(b)

Reconfigure Oracle WebLogic Server with custom certificates

Section 13.6.14

(c)

Track the status of deferred data migration jobs.

Section 13.7.2

(d)

Delete unwanted central agents

Section 13.13.2

(e)

Upgrade Application Dependency and Performance (ADP) engines and JVM Diagnostics (JVMD) engines

Chapter 7

(f)

As part of the upgrade to Enterprise Manager Cloud Control 12c Release 5 (12.1.0.5), Oracle BI Publisher 11.1.1.7.0 with the latest patch set is automatically installed. Therefore, do not perform a software-only install of any version of Oracle BI Publisher in the middleware home that contains Enterprise Manager Cloud Control 12c Release 5 (12.1.0.5). However, although Oracle BI Publisher is installed by default, it is not configured or upgraded from the earlier version by default.

  • If you did not have Oracle BI Publisher in your earlier release, then configure the newly installed one.

  • If you already had Oracle BI Publisher in your earlier release, then upgrade the earlier version using the configureBIP -upgrade command. As part of the upgrade, all reports from the old Oracle BI Publisher are migrated to the upgraded one.

For configuring the newly installed Oracle BI Publisher, see Oracle Enterprise Manager Advanced Installation and Configuration Guide.

For upgrading Oracle BI Publisher and migrating the reports, see Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide.

(g)

Delete old OMS home

Section 13.15

4.2 Upgrading Enterprise Manager Cloud Control 12c Release 4 (12.1.0.4), 12c Release 3 (12.1.0.3), or 12c Release 2 (12.1.0.2) to 12c Release 5 (12.1.0.5) in an HA Environment (Primary and Standby Sites)

This section describes how you can upgrade your primary as well as standby Enterprise Manager sites to 12c Release 5 (12.1.0.5) in an HA (or high availability) environment. In particular, this section covers the following:

4.2.1 Upgrading Primary and Standby OMS Instances When the Standby OMS Is Created Using Storage Replication

Table 4-2 describes the steps for upgrading your primary and standby OMS instances when the standby OMS is created using Storage Replication.

Note:

You do not have to remove the standby OMS as part of this procedure.

Table 4-2 Upgrading Primary and Standby OMS Instances When the Standby OMS Is Created Using Storage Replication

Step No. Step Procedure

Step 1

Upgrade the Primary OMS

  
 

Upgrade the primary Enterprise Manager site, both OMS and Management Agents.

There will be some downtime as the primary OMS will go down during upgrade process.

Section 4.1

Step 2

Verify the New Middleware Home on the Standby Storage Server

  
 

Contact your system administrator to make sure that the new middleware home is also replicated to the standby storage server.

  

4.2.2 Upgrading Primary and Standby OMS Instances When the Standby OMS Is Created Using Standby WebLogic Domain

Table 4-3 describes the steps for upgrading your primary and standby OMS instances when the standby OMS is created using Standby WebLogic Domain.

Table 4-3 Upgrading Primary and Standby OMS Instances When the Standby OMS Is Created Using Standby WebLogic Domain

Step No. Step Procedure

Step 1

Remove the Standby OMS

  

(a)

Remove all additional standby OMS instances.

To remove all additional standby OMS instances, see the section on removing additional standby OMS instances in Oracle Enterprise Manager Advanced Installation and Configuration Guide.

(b)

Remove the first standby OMS.

To remove the first standby OMS, see the section on removing the first standby OMS in Oracle Enterprise Manager Advanced Installation and Configuration Guide.

Step 2

Upgrade the Primary OMS

  
 

Upgrade the primary Enterprise Manager site, both OMS and Management Agents.

Section 4.1

Step 3

Redeploy the Standby OMS

  
 

Re-create the standby OMS environment using Standby WebLogic Domain.

To re-create the standby OMS environment using Standby WebLogic Domain, see the chapter on disaster recovery in Oracle Enterprise Manager Cloud Control Administrator's Guide.