Patching a RAC Database Using Fleet Maintenance

Before you Begin

This tutorial shows you how to patch an Oracle RAC Database using Oracle Enterprise Manager's Fleet Maintenance.

Background

Database Fleet Maintenance, part of the Database Lifecycle Management Pack, allows administrators to maintain Oracle Homes and associated databases up to date by applying database updates and patches.

Description of Image 1 follows — Description of illustration 1

For more information on Fleet Maintenance see Database Fleet Maintenance in Oracle Enterprise Manager Cloud Control Database Lifecycle Management Administrator's Guide.

What Do You Need?

Enterprise Manager (EM) version 13.2 and above. The latest Release Update of EM 13.4 is recommended. This tutorial uses EM 13.4. For the latest Enterprise Manager Release Updates, see Enterprise Manager 13.4.
An Oracle RAC database discovered in Enterprise Manager. This tutorial uses a two-node RAC, Oracle 19c GI (19.3.0) and Oracle 19c (19.3.0) Database, running on Oracle Linux 8.0, that will be patched to Oracle version 19.8.0.
A reference Oracle database home, which is a database Oracle Home already patched to the desired patch level. This reference home will be used to create the Gold Image. This tutorial is using an existing Oracle Home, patched to level version 19.8.0 and discovered in EM. For more information on how to discover an oracle home see this Discovery section. For concepts and supported image creation tasks, see also Image and Subscription Tasks.
Access to the target database and its host. In this example, the EM administrator has root access to the host the database resides on and privileged credentials set up for the database. For more information on how to set up your access see Checking Host Readiness Before Provisioning or Patching.
Access to the Enterprise Manager command line interface utility, emcli. For more information on how to set up EM CLI see Getting Started with EM CLI.

The following table contains the description of some Input Variables that will be used across this tutorial:

Variable	Description
Name	User specified unique name of the operation
Normal Credential	Enter in the format of <Named Credential: Credential Owner>
	<Named Credential>: Named credential for the host where new Oracle home will be deployed
	<Credential Owner>: The Enterprise Manager user who owns this Named Credential
Privilege Credential	Enter in the format of <Named Credential: Credential Owner>
	<Named Credential>: Named credential for the host where new Oracle home will be deployed
	<Credential Owner>: The Enterprise Manager user who owns this Named Credential
	This credential is used to run scripts as root

Step 1. Create a Gold Image

Create a Gold Image from the Oracle RAC Database reference environment and upload it to the Software library. This Oracle Home will be used as a reference target for patching the remaining RAC Databases.

Create a response file patch_rac.inp with this content:
```
IMAGE_NAME=RACDB_19.8.0
IMAGE_DESCRIPTION=RACDB_19.8.0
REF_TARGET_NAME=dbhome_1_my.example.com_6741 
IMAGE_SWLIB_LOC=Oracle Home Provisioning Profiles/19.8.0/linux_x64
STORAGE_TYPE_FOR_SWLIB=OmsShared
STORAGE_NAME_FOR_SWLIB=default_loc
REF_HOST_CREDENTIALS=MYORACLE:SYSMAN
WORKING_DIRECTORY=/u01/app/tmp
VERSION_NAME=RACDB_19.8.0_Jul20
                        
```
This response file was created keeping this in mind:

IMAGE_NAME, IMAGE_DESCRIPTION, VERSION_NAME must be unique names. These are values you determine as an Administrator, based on what you know about your environment.

REF_TARGET_NAME: This field is an the Oracle Home name as shown in Enterprise Manager. This home must already have the needed patches applied and it will be used as a reference Oracle Home.

IMAGE_SWLIB_LOC: The path in software library where the gold image will be stored. Navigate to Enterprise > Provisioning and Patching > Software Library to view available locations . This location should be created by a Superadministrator, in this example the SYSMAN user.

STORAGE_TYPE_FOR_SWLIB: The storage type configured for the software library. Possible options are OmsShared, OmsAgent, Http, Nfs, ExtAgent. Use "OmsShared" in this case.

STORAGE_NAME_FOR_SWLIB: The name provided for software library. Navigate to Setup > Provisioning and Patching> Software Library. Review column 'name' provided based on the storage type above.

REF_HOST_CREDENTIALS: The credentials created in EM for connecting to the host target where the RAC database is installed. The format is:

credential name:credential owner

WORKING_DIRECTORY: The path to the temporary directory if other than /tmp. Note that the 'emStageDir' value provided in the target <Agent Base>/agent_inst/sysman/config/emd.properties file will not be used as a staging directory in this case.
Create a Gold image of the RAC Home which already has the needed patches applied and upload it to Software Library using the following commands:

emcli login -username=SYSMAN

emcli db_software_maintenance -createSoftwareImage -input_file="data:patch_rac.inp"

Note:

Once above command is submitted you will be able to track progress detail by navigating to Enterprise > Provisioning and Patching > Procedure Activity Page.

Verify the available images:

emcli db_software_maintenance -getImages

Output:


==============================================================================================================================

IMAGE ID         IMAGE NAME          DESCRIPTION       VERSION            CREATION DATE         OWNER 

==============================================================================================================================


 C1300CEA54E75EF0E0537235920BE85D  RACDB_19.8.0       RACDB_19.8.0      19.8.0.0.0      2020-11-03 12:08:01.0     SYSMAN


------------------------------------------------------------------------------------------------------------------------------

Total Rows: 1

List the version of your image to ensure this is the correct patch level you are looking for:

emcli db_software_maintenance -getVersions -image_id=C1300CEA54E75EF0E0537235920BE85D

Output:

===========================================================================================================POSITION    VERSION ID    VERSION NAME   DATE CREATED     EXTERNAL ID      HASH CODE                 STATUS =========================================================================================================== 1    C1300CEA54E75EF0E0537235920BE85D      19.8.0  2020-11-03 12:08:01.0 oracle:default[...]C2891668972:B5D785B CURRENT

Verify the Subscription:

emcli db_software_maintenance -getImageSubscriptions -image_id=C1300CEA54E75EF0E0537235920BE85D

Check the applicability of your gold image to your target:

emcli db_software_maintenance -checkApplicability -image_id=C1300CEA54E75EF0E0537235920BE85D -target_list=orcl -target_type=rac_database

Step 2. Subscribe Your Target to the Gold Image

Subscribe the RAC database target to the Gold Image you just created. If you already have a couple of images created and have some targets already subscribed, you can issue the following commands to validate before subscribing the target to the image.

Subscribe your target to the Gold Image:

emcli db_software_maintenance -subscribeTarget -target_name=orcl -target_type=rac_database -image_id=C1300CEA54E75EF0E0537235920BE85D

Output:

Target orcl subscribed successfully.

Verb subscribeTarget completed successfully

Check if the target is subscribed to this image:

emcli db_software_maintenance -getTargetSubscriptions -target_name=orcl -target_type=rac_database

Step 3. Stage and Deploy the Software

This step copies the Gold image from software library to a new Oracle Home. In other words, this command allows you to stage and deploy the new Oracle Home on the subscribing Oracle Homes. The Deploy command automatically uses the "CURRENT" version of the subscribed image while creating the new Oracle Home.

Create a Response file deploy_rac.rsp. The response file contains the new Oracle Home location where you would like to have the database software configured:

NEW_ORACLE_HOME_LIST=/u01/app/oracle/product/19c/dbhome_2
workingDir=/u01/app/oracle/tmp
dispatchLoc=/u01/app/oracle/tmp/dispatchloc

Note:

workingDir: Path of temp directory if location needs to be changed from /tmp. 'emStageDir' provided in Target <AgentBase>/agent_inst/sysman/config/emd.properties file will not be picked as staging directory.

Deploy the software:

emcli db_software_maintenance -performOperation -name='Deploy DB Home' -purpose=DEPLOY_RAC_SOFTWARE -target_type=rac_database -target_list=orcl -normal_credential=ORA_HOST:SYSMAN -privilege_credential=ORA_ROOT:SYSMAN -input_file='data:/home/oracle/deploy_rac.txt'

Note:

Once above command is submitted you will be able to track progress detail by navigating to Enterprise > Provisioning and Patching > Procedure Activity Page.

Description of command specific Input Variables:

Variable	Description
Purpose	Use 'DEPLOY_RAC_SOFTWARE' for RAC database
Target_type	Type of target being provided. For RAC Database it is 'rac_database'
Target_list	Comma separated list of homogenous targets which need to be patched. You can get target names using emctl config agent listtargets
Input File	Location of input response file, parameters of this file are explained above

Output:

Processing target orcl...

Checking image subscription...

Check image subscription passed.

Oracle home target list is orcl

Operation 'Deploy DB Home'' created successfully.

Step 4. Update the RAC Database

This step migrates the RAC database and its instances targets running from old Oracle Home to the newly patched Oracle Home:

emcli db_software_maintenance -performOperation -name='Update DB Home' -purpose=UPDATE_RACDB -target_type=rac_database -target_list=orcl -normal_credential=ORA_HOST:SYSMAN -privilege_credential=ORA_ROOT:SYSMAN -rolling=true

Output:

Processing target orcl...

Checking if target is already running from the current version of the image...

Check Passed.

Checking image subscription...

Check image subscription passed.

Operation 'Update DB Home' created successfully.

Note:

Once above command is submitted you will be able to track progress detail by navigating to Enterprise > Provisioning and Patching > Procedure Activity Page.

Description of command specific Input Variables:

Variable	Description
Purpose	Use 'UPDATE_RACDB' for RAC Database
Target_type	Type of target being provided. For RAC Database it is 'rac_database'
Target_list	Comma separated list of RAC Databaes targets which need to be migrated. You can get target names using emctl config agent listtargets

Note:

'Rolling Mode' refers to the patching methodology where the nodes of the cluster are patched individually, that is, one by one. For example, if you are patching a clusterware target that has five nodes, then the first node is shut down, patched, and restarted, and then the process is rolled over to the next node until all the nodes are patched successfully.

Note:

'-rolling' is an optional flag with the default value as true. The update procedure works in 'Rolling Patch' mode by default but you can override by false.

Step 5. Migrate Listeners

Use this command to migrate listeners running from Oracle database homes. Listeners running from Grid Infrastructure homes are automatically migrated during the GI update process:

emcli db_software_maintenance -performOperation -name="Update Listener" -purpose=MIGRATE_LISTENER -target_type=rac_database -target_list=orcl -normal_credential=ORA_HOST:SYSMAN -privilege_credential=ORA_ROOT:SYSMAN

Output:

Processing target orcl...

Checking if target is already running from the current version...

Check Passed.

Checking image subscription...

Check image subscription passed.

Operation 'Update Listener' created successfully.

Note:

Once above command is submitted you will be able to track progress detail by navigating to Enterprise > Provisioning and Patching > Procedure Activity Page.

Description of command specific Input Variables:

Variable	Description
Purpose	Use 'MIGRATE_LISTENER' for RAC Database
Target_type	Type of target being provided. For RAC Database it is 'rac_database'
Target_list	This is a comma separated list of targets which are to be migrated

Step 6. Rollback the RAC Database oracle homes (optional)

If needed, you may rollback to the old Oracle Home.

This step migrates the RAC Database target running from the NEW patched Oracle home to OLD Oracle home:

emcli db_software_maintenance -performOperation -name='Rollback RAC DB' -purpose=ROLLBACK_RACDB -target_type=rac_database -target_list=orcl -normal_credential=ORA_HOST:SYSMAN -privilege_credential=ORA_ROOT:SYSMAN -rolling=true

Output:

Processing target orcl...

Checking image subscription...

Check image subscription passed.

Operation 'Rollback RAC DB' created successfully.

Note:

Once above command is submitted you will be able to track progress detail by navigating to Enterprise > Provisioning and Patching > Procedure Activity Page.

Description of command specific Input Variables:

Variable	Description
Purpose	Use 'ROLLBACK_RACDB' for RAC Database
Target_type	Type of target being provided. For RAC Database it is 'rac_database'
Target_list	Comma separated list of RAC Database targets which need to be migrated. You can get target names using emctl config agent listtargets

Note:

'-rolling' is an optional flag with the default value as true.The update procedure works in "Rolling Patch" mode by default but you can override by false.

Step 7. Clean Up Oracle Homes

Since Oracle Homes consume storage, they can be cleaned up to release space after the successful switch of databases to new oracle home.

Note:

Oracle homes will be deleted only if there are no targets running from them.

Set below property to delete old oracle homes. The value is the number of Oracle Homes that must be retained. The default value is 3. For example, if there are four versions of an Oracle Home (with the target running from the fourth version), only the earliest of them can be deleted if the value is set to 3. Similarly if the value is set to 1, only the current Oracle home from which the target is running will remain and the other three homes will be deleted.

emctl set property -name oracle.sysman.emInternalSDK.db.gis.lineageLength -sysman_pwd $em_sysman_pwd -value 1

Get a list of Oracle Homes that can be cleaned:

emcli db_software_maintenance -performOperation -name='Cleanup Old OH' -purpose=CLEANUP_SOFTWARE -target_type=rac_database -target_list=orcl -normal_credential=ORA_HOST:SYSMAN -privilege_credential=ORA_ROOT:SYSMAN -workDir='/u01/app/oracle/tmp' -reportOnly=true

Clean-up Oracle Homes:

emcli db_software_maintenance -performOperation -name='Cleanup Old OH' -purpose=CLEANUP_SOFTWARE -target_type=rac_database -target_list=orcl -normal_credential=ORA_HOST:SYSMAN -privilege_credential=ORA_ROOT:SYSMAN -workDir='/u01/app/oracle/tmp'

Note:

Once above command is submitted you will be able to track progress detail by navigating to Enterprise > Provisioning and Patching > Procedure Activity Page.

Description of command specific Input Variables:

Variable	Description
Purpose	Use 'CLEANUP_SOFTWARE' for cleanup operation
Target_type	Type of target being provided. For RAC Database it is 'rac_database'
Target_list	This is a comma separated list of targets which are to be cleaned
reportOnly	The default value is false. When set to ‘true’, it prints the eligible homes for deletion but the homes are not deleted

Want to Learn More?

Title and Copyright Information

Patching a RAC Database Using Fleet Maintenance

F36781-01

January 2021

This tutorial shows you how to patch an Oracle RAC Database using Oracle Enterprise Manager's Fleet Maintenance.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract. The terms governing the U.S. Government's use of Oracle cloud services are defined by the applicable contract for such services. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Epyc, and the AMD logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.