6 Manual Multi-Node Patching of Grid Infrastructure and RAC DB Environment Using OPatchAuto

Patch orchestration is the automated execution of the patching steps, such as the execution of pre-patch checks, stopping services, applying the binary patches, and starting the services. Patch orchestration for Oracle Database 12c applies the patch to the GI/RAC configuration on that machine, including all of its databases. The OPatchAuto patch orchestration utility has been made available with version 12.1 of the OPatch utility.

This chapter covers the following topics:

Note:

This chapter applies to Oracle Database 12c only.

6.1 Running OPatchAuto on a Single Node

Prerequisites: Node Availability During Patching (Rolling vs. Non-rolling)

In order to start a new patching session, the following conditions must be met.

OPatchauto in a single node can be executed in two different ways:

OPatchauto can be executed as a root user.
OPatchauto can be executed as an oracle home owner. In this case -wallet needs to be provided which contains the credential to become root during execution.

Note:

Latest OPatch must be present on all the homes requested for patching on all the nodes.
If opatchauto is executed as Oracle home owner, then sudo must be configured for the oracle home owner to become root and owners of the other homes, if the other homes have different owner.

Note:

The following conditions apply only for the first node, such as when the session is first started on the cluster.

Local node must be up for both Rolling and Non-rolling modes.
At least one of the remote nodes must be up in order to start a Rolling mode session.
ALL the remote nodes must be down in order to start a Non-rolling session.

To ensure successful patching, Oracle recommends that a patching session always consist of the four following steps:

Run opatch lsinventory

When to run: During cluster uptime.

Before you begin a new patching session you must ensure that all nodes of the cluster are at the same patch level.
```
$GRID_HOME/OPatch/opatch lsinventory
```
Example Output
```
Patch level status of Cluster nodes :
Patching Level           Nodes
--------------           -----
12345678                 node123, node456
```
Run opatchauto apply -analyze

When to run: During cluster uptime.

The apply -analyze command simulates an opatchauto apply session by running all prerequisite checks, when possible, without making changes to the system (either bits or configurations).
Run opatchauto apply

When to run: During cluster downtime.

The apply command applies all patches within a specified system patch to the GI or RAC home from which the opatchauto command is run.
Run opatch lsinventory

When to run: During cluster uptime.

After successfully running opatchauto apply on each individual node, you must run opatch lsinventory on that node to confirm that the patch level has been correctly updated.

6.2 Patching Session Output

The following patching session output examples illustrate successful OPatchAuto apply and rollback sessions. Note: This session applies to both single and multi-node environments.

Example 6-1 OPatchAuto Apply/Rollback Session in Analyze Mode

--------------------------------Summary--------------------------------
Analysis for applying patches has completed successfully:
Host:myhostq
CRS Home:/scratch/aime_ordb_myhostq/crso1/crshome_crso1
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /tmp/patch_gipsu_12024/patch/22191349/21436941
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log
Patch: /tmp/patch_gipsu_12024/patch/22191349/21948341
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log
Patch: /tmp/patch_gipsu_12024/patch/22191349/21948344
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log
Patch: /tmp/patch_gipsu_12024/patch/22191349/21948354
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log
 
Host:myhostr
CRS Home:/scratch/aime_ordb_myhostq/crso1/crshome_crso1
 
==Following patches were SUCCESSFULLY analyzed to be applied:
 
Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21436941
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log
 
Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948341
 
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log
 
Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948344
 
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log
 
Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948354
 
Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log
 
OPatchAuto successful.

Example 6-2 OPatchAuto Apply Session

--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:mymachineemq
CRS Home:/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /tmp/patch_gipsu_12024/patch/22191349/21436941
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log
Patch: /tmp/patch_gipsu_12024/patch/22191349/21948341
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log
Patch: /tmp/patch_gipsu_12024/patch/22191349/21948344
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log
Patch: /tmp/patch_gipsu_12024/patch/22191349/21948354
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log
Host:mymachineemr
CRS Home:/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21436941
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log
Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948341
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log
Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948344
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log
Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948354
Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log
OPatchAuto successful.

Example 6-3 OPatchAuto Rollback Session

--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:mymachineemm
CRS Home:/scratch/aime_ordb_mymachineemm/crso1/crshome_crso1
Summary:
==Following patches were SUCCESSFULLY rolled back:
Patch: /tmp/patch_gipsu_12019/patch/22191492/17077442
Log: /scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_19-25-46PM_1.log
Patch: /tmp/patch_gipsu_12019/patch/22191492/17303297
Log: /scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_19-25-46PM_1.log
Patch: /tmp/patch_gipsu_12019/patch/22191492/21951844
Log: /scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_19-25-46PM_1.log
OPatchAuto successful.

Example 6-4 OPatchAuto Apply/Rollback Failure

---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /scratch/aime/app/aime/product/11.2.0/dbhome_2, host: mymachineemg.
Command failed:  /bin/sh -c 'ORACLE_HOME=/scratch/aime/app/aime/product/11.2.0/dbhome_2 /scratch/aime/app/aime/product/11.2.0/dbhome_2/bin/srvctl stop home -o /scratch/aime/app/aime/product/11.2.0/dbhome_2 -n mymachineemg -f -t TRANSACTIONAL -s /scratch/aime/app/aime/product/11.2.0/dbhome_2/cfgtoollogs/opatchautodb/statfile/mymachineemg/OracleHome-eca39d53-5b51-4cdf-9c79-ce9d9312d86a_mymachineemg.stat'
Command failure output:
PRCH-1000 : Failed to stop resources running from Oracle home /scratch/aime/app/aime/product/11.2.0/dbhome_2
PRCH-1029 : One or more resources failed to stop: PRCH-1006 : Failed to stop Listener
PRCR-1014 : Failed to stop resource ora.LISTENER2.lsnr
PRCR-1065 : Failed to stop resource ora.LISTENER2.lsnr
CRS-5016: Process "/scratch/aime/app/aime/product/11.2.0/dbhome_2/bin/lsnrctl" spawned by agent "/scratch/aime_ordb_mymachineemg/crso1/crshome_crso1/bin/oraagent.bin" for action "stop" failed: details at "(:CLSN00010:)" in "/scratch/aime_ordb_mymachineemg/crso1/crshome_crso1/log/mymachineemg/agent/crsd/oraagent_aime/oraagent_aime.log"
CRS-2675: Stop of 'ora.LISTENER2.lsnr' on 'mymachineemg' failed
 
After fixing the cause of failure Run opatchauto resume with session id "J5A3"
]
OPATCHAUTO-68061: The orchestration engine failed.
 
OPATCHAUTO-68061: The orchestration engine failed with return code 1
 
OPATCHAUTO-68061: Check the log for more details.

6.3 Sample Console Output

Example 6-5 opatchauto apply -analyze

System initialization log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_01-55-58PM.log.

Session log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/opatchauto2016-05-05_01-56-18PM.log

WARNING: the option -ocmrf is deprecated and no longer needed. OPatch no longer checks for OCM configuration. It will be removed in a future release.

The id for this session is MDAN
[init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1

Executing OPatch prereq operations to verify patch applicability on CRS Home........
 
[init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
[init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Executing prereq operations before applying on CRS Home........
 
[init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
Analysis for applying patches has completed successfully:
 
Host:mymachineelu
 
CRS Home:/scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
==Following patches were SUCCESSFULLY analyzed to be applied:
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/17077442
 
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_13-56-25PM_1.log
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/17303297
 
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_13-56-25PM_1.log
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/22291141
 
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_13-56-25PM_1.log

Example 6-6 opatchatuo apply

System initialization log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_02-22-02PM.log.
 
Session log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/opatchauto2016-05-05_02-22-19PM.log
 
WARNING: the option -ocmrf is deprecated and no longer needed. OPatch no longer checks for OCM configuration. It will be removed in a future release.
 
The id for this session is WLR9
 
[init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Executing OPatch prereq operations to verify patch applicability on CRS Home........
 
[init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
[init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Executing prereq operations before applying on CRS Home........
 
[init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
[shutdown:shutdown] Executing GIShutDownAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Performing prepatch operations on CRS Home........
 
Prepatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_02-22-52PM.log
 
[shutdown:shutdown] GIShutDownAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Start applying binary patches on CRS Home........
 
[offline:binary-patching] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
[startup:startup] Executing GIStartupAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Performing postpatch operations on CRS Home........
 
Postpatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_02-27-03PM.log
 
[startup:startup] GIStartupAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Verifying patches applied on CRS Home.
[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
 Patching is completed successfully. Please find the summary as follows:
 
Host:mymachineelu
CRS Home:/scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
Summary:
 
==Following patches were SUCCESSFULLY applied:
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/17077442
 
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_14-23-38PM_1.log
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/17303297
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_14-23-38PM_1.log
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/22291141
 
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_14-23-38PM_1.log

Example 6-7 opatchauto rollback

System initialization log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_04-34-39PM.log.
 
Session log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/opatchauto2016-05-05_04-35-00PM.log
The id for this session is K5BA
[init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Executing OPatch prereq operations to verify patch applicability on CRS Home........
 
[init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
[init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Executing prereq operations before rolling back on CRS Home........
 
[init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
 
[shutdown:shutdown] Executing GIShutDownAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Performing prepatch operations on CRS Home........
 
Prepatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_04-35-22PM.log
 
[shutdown:shutdown] GIShutDownAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfull 
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Start rolling back binary patches on CRS Home........
 
[offline:binary-patching] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully
[startup:startup] Executing GIStartupAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1
 
Performing postpatch operations on CRS Home........
 
Postpatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_04-38-59PM.log
 
[startup:startup] GIStartupAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully 
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
 
Patching is completed successfully. Please find the summary as follows:
 
Host:mymachineelu
CRS Home:/scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 
Summary:
 
==Following patches were SUCCESSFULLY rolled back:
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/17077442
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_16-36-04PM_1.log
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/17303297
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_16-36-04PM_1.log
 
Patch: /tmp/patch_gipsu_12019/patch/22654153/22291141
Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_16-36-04PM_1.log

Example 6-8 opatchauto apply -analyze (Multi-node)

System initialization log file is /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_10-19-07PM.log.
 
Session log file is /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/opatchauto2016-05-05_10-19-19PM.log
The id for this session is KMKT 
[init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1
[init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
 
Executing OPatch prereq operations to verify patch applicability on RAC Home........
 
Executing OPatch prereq operations to verify patch applicability on CRS Home........
 
[init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully
<mymachineelh> : [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1
<mymachineelh> : [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
<mymachineelh> :
Executing OPatch prereq operations to verify patch applicability on CRS Home........
 
<mymachineelh> :
Executing OPatch prereq operations to verify patch applicability on RAC Home........
 
<mymachineelh> : [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully
[init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully 
<mymachineelh> : [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully
[init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 
[init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
 
Executing prereq operations before applying on CRS Home........ 
 
Executing prereq operations before applying on RAC Home........
 
[init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully
[init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully
<mymachineelh> : [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
<mymachineelh> : [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1
<mymachineelh> :
Executing prereq operations before applying on RAC Home........
 
<mymachineelh> :
Executing prereq operations before applying on CRS Home........
 
<mymachineelh> : [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully
<mymachineelh> : [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully 
[init:init] Executing RACDBPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1
 
Executing prereq operations before applying on RAC Home........
 
<mymachineelh> : [init:init] Executing RACDBPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1
<mymachineelh> :
Executing prereq operations before applying on RAC Home........
 
<mymachineelh> :
Following step failed during analysis:
/bin/sh -c 'ORACLE_HOME=/scratch/aime_ordb_mymachineelg/obase/rachome_racr1 ORACLE_SID=racr12 /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/OPatch/datapatch -prereq'
 
<mymachineelh> : [init:init] RACDBPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully
[init:init] RACDBPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully
[finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
[finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully
[finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1
[finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully
<mymachineelh> : [finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
<mymachineelh> : [finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully
<mymachineelh> : [finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1
<mymachineelh> : [finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully
OPatchAuto successful.
 
--------------------------------Summary--------------------------------
 
Analysis for applying patches has completed successfully:
 
Host:mymachineelg
CRS Home:/scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
 
==Following patches were SUCCESSFULLY analyzed to be applied:
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/21436941
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/22291127
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/22502518
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/22502555
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log
 
Host:mymachineelg
 
RAC Home:/scratch/aime_ordb_mymachineelg/obase/rachome_racr1
 
==Following patches were SKIPPED:
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/21436941
Reason: This patch is not applicable to this specified target type - "rac_database"
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/22502518
Reason: This patch is not applicable to this specified target type - "rac_database"
 
==Following patches were SUCCESSFULLY analyzed to be applied:
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/22291127
Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-39PM_1.log
 
Patch: /tmp/patch_gipsu_12024/patch/22646084/22502555
Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-39PM_1.log
 
Host:mymachineelh
RAC Home:/scratch/aime_ordb_mymachineelg/obase/rachome_racr1
 
==Following patches were SKIPPED:
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/21436941
Reason: This patch is not applicable to this specified target type - "rac_database"
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502518
Reason: This patch is not applicable to this specified target type - "rac_database"
 
==Following patches were SUCCESSFULLY analyzed to be applied:
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22291127
Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-25-55PM_1.log
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502555
Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-25-55PM_1.log
 
Host:mymachineelh
CRS Home:/scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1
 
==Following patches were SUCCESSFULLY analyzed to be applied:
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/21436941
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22291127
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502518
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log
 
Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502555
Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log

6.4 OPatchAuto Apply

When you run OPatchAuto's apply command, numerous operations are performed to implement the complete patch application cycle. These operations vary depending on the environment to be patched. The following environment is representative of the vast majority of patching environments in which OPatchAuto is used. For example, a typical patching environment would be one GI Home managing two RAC Homes. When you run opatchauto apply, OPatchAuto will perform the operations shown in Figure 6-1.

Figure 6-1 Patching with OPatchAuto: Process Flow

6.4.1 OPatchAuto: System Reboot Request

Depending on the patch or the home directory configuration, you may encounter a request to reboot the system. After a reboot during the patching process, you need to invoke the opatchauto utility again so that it seamlessly continues with rest of the patch application process.

Typically an error message, as shown in the following example, will be displayed when a problem arises.

Example 6-9 OPatchAuto Console Error

# OPatch/opatchauto apply /scratch/aime/sh/RDBMS_12.1.0.1.0_LINUX.X64_130418/patches/v2/nosql/gipsu/11111111 -ocmrf /tmp/ocm.rsp 
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
 ... 
 
CLSRSC-400: A system reboot is required to continue installing.
 
...

Apply Summary:
Following patch(es) are successfully installed:
GI_HOME=/u01/GI12/app/12.1.0/grid:13852018, 22222222, 123456788
DB_HOME=/scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1:13852018, 123456788
DB_HOME=/scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1:13852018, 123456788
 
opatchauto failed with error code 1.

When you receive an error like this, follow the reboot instructions specified in the console. The following example shows a system reboot request issued by the user.

Example 6-10 Rebooting the System

# OPatch/opatchauto resume -reboot
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
 
 
OPatchauto version : 12.1.0.1.1
OUI version        : 12.1.0.1.0
Running from       : /u01/GI12/app/12.1.0/grid
Log file location  : /u01/GI12/app/12.1.0/grid/cfgtoollogs/opatch/opatch2013-05-16_13-36-59PM_1.log
 
Opatchauto will attempt to resume from reboot patching session. This might take several minutes...
 
Command "/usr/bin/perl /u01/GI12/app/12.1.0/grid/crs/install/rootcrs.pl -postpatch" is successfully resumed.
Command "/scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1/bin/srvctl start home -o /scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1 -n slc00epi -s /scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1/OracleHome-50b8f1a0-e220-4b8e-98d7-49177979991f.stat " is successfully resumed.
Command "/scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1/bin/srvctl start home -o /scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1 -n slc00epi -s /scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1/OracleHome-58232a10-3130-4930-b588-0c8594cf8c87.stat " is successfully resumed.
Opatchauto was able to resume from the previous reboot patching session and complete successfully.
 
opatchauto succeeded.

6.5 Patching a Sharded Database

The session has to be initiated from the catalog host by providing details of the catalog database. It should provide a top level view to the end user about patching a SDB, making it easier to understanding the flow of opatchautoSDB.

Sharding is an application-managed scaling technique using many (hundreds /thousands of) independent databases. With sharding, data is split into multiple databases (shards) with each database holding a subset of data. Shards can be replicated for high availability and scalability.

OPatchAuto supports end-to-end patching of sharded databases across multiple regions, along with the grid infrastructure (GI) that supports the clustered databases/shards and shards that are managed by Oracle Golden Gate or by Oracle DataGuard.

OPatchAuto supports all sharding and replication methods.

Supported Configurations:

Shard Types: Databases running on GI/SIHA and standalone databases
Different versions across Data Guards (currently, only version 12.2 is available)
Multiple shards sharing the same cluster/grid
Multiple versions of shards in an OGG replicated SDB

6.5.1 Selectively Patching Subset Entities

A sharded database can span multiple regions and clusters. To manage the patching cycle more efficiently, you may want to scale down the patching effort by patching only specific subset entities. OPatchAuto provides three options for selecting the subset entities:

1. Data Guards

2. Shard Groups

3. Shard Spaces

You can select instances of any one of the above entities. Note: You can only specify a single entity for a given patching cycle. For all of the subset entities, OPatchAuto identifies the targets on the basis of their names:

1. –dg: Comma-separated list of names of the primary database

2. –shardgroup: Comma-separated list of names of the shard groups

3. –shardspace: Comma-separated list of names of the shard spaces

For all of the above options, the Grid homes of the databases will also be patched. The CRS/RAC homes will be patched across all their nodes in a rolling manner. Similarly, the Golden Gate home will also be patched.

6.5.1.1 Data Guard

By selecting the -dg <primaryDB name> option, only the databases (along with their cluster homes) that belong to the selected Data Guard will be patched. All these databases will be patched in a rolling manner starting with the standby shards and ending with the primary shard.

The entire list of databases in a sharded database can be collected from the catalog database table GSMADMIN_INTERNAL.DATABASE under the column NAME.

6.5.1.2 Shard Group

By selecting the -shardgroup <shardgroup name> option, the databases of the selected shard group will be patched in a rolling manner. In a Data Guard-replicated configuration, a shard group can host ONLY one member of each Data Guard. Hence, this option is NOT supported in a sharded database that employs Data Guard replication because an entire Data Guard, while being spread across multiple shard groups, needs to be patched together in a defined sequence starting with standby databases and ending with the primary database. Patching individual shard groups poses a major risk of breaking that sequence.

The entire list of shard groups in a sharded database can be collected from the catalog database table GSMADMIN_INTERNAL.SHARD_GROUP under the column NAME.

6.5.1.3 Shard Space

By specifying the -shardspace <shardspace name> option, the databases of the entire shard space are patched in a rolling manner. These databases could be spread across multiple shard groups, depending upon the configuration of the sharded database.

The entire list of shard spaces in a sharded database can be collected from the catalog database table GSMADMIN_INTERNAL.SHARD_SPACE under the column NAME.

6.5.2 Sharded Database Command Option

The OPatchAuto sdb command option allows you to patch sharded databases. This operation patches all the shards of the Sharded database.

In a sharded database with Data Guard replication, patching a Data Guard involves patching all its standby databases first, followed by its primary database.In Oracle Golden Gate based replication, for a user-defined configuration, the operation involves patching around the shard spaces. For system-managed and composite configuration, the operation involves patching around the shard groups.The values of -host, -port and -sid are used to connect to the catalog database and hence they should form the required connect string for the catalog database.All the database(s) are patched first.Thereafter, the Grid/HAS homes that are host to any of the databases belonging to the sharded database are patched.The catalog database and the GSM's will not be patched. In order to patch these, OPatchAuto needs to be run separately on these databases.

The following syntax illustrates command line usage:

opatchauto apply <patch-location>
    -sdb
    -wallet <wallet>
    [ -phBaseDir <patch.base.directory> ] 
    [ -logLevel <log_priority> ]
    [ -analyze ]
    [ -host <tns-host> ]
    [ -dg <primary.database.name> ]    
    [ -shardgroup <shardgroup> ]
    [ -shardspace <shardspace> ]
    [ -rolling ]
    [ -service <service> ]
    [ -inplace ]
    [ -sid <sid> ]
    [ -port <port> ]

Parameters

patch-location

The patch location.

Options

phBaseDir <patch.base.directory>

The location of base patch directory.
logLevel <log_priority>

The log level (defaults to "INFO").

Supported values: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
analyze

If this option is selected, the environment will be analysed for suitability of the patch on each home, without affecting the home.

The patch will not be applied or rolled back, and targets will not be shut down.
host <tns-host>

The tns-host of the catalog database. This should match the 'HOST' used in the network configuration file of the catalog database. The default host is set as the local 'hostname' without appending the domain.
wallet <wallet location> (Required)

The location of the wallet file.

The entries made in the wallet file must satisfy these requirements:
- It is mandatory to provide the credentials of all home owners of every node that would be patched as part of the session in the wallet. The homes that would be patched on the node also includes GI/SIHA when installed. The list of nodes refers to all hostnames/ip-address listed in GSMADMIN_INTERNAL.SHA_DATABASES.DB_HOST column of the Sharded database catalog.
- It must also contain the credentials for the database user with 'sysdba' privilege for the catalog database.
Additionally, the host user provided in the wallet must meet the following requirements:
- The user must be able to change to root using ‘sudo’ if the node belongs to GI/SIHA environment.
dg <primary.database.name>

This is used to restrict patching to databases of the selected dataguard. All the standby databases of the dataguard are patched first, followed by its primary database.
shardgroup <shardgroup>

This is used to restrict patching to databases of the selected shard group.
shardspace <shardspace>

This is used to restrict patching to databases of the selected shard space.
sdb (Required)

To signify patching sharded database. Run 'opatchauto <apply|rollback> -sdb -help' to get more help on patching a sharded database.
rolling

Enables sdb rolling mode where database(s) are patched one after the other.
service <service>

Service name of the catalog database.
inplace

This option can be used to perform in place patching through opatchauto. Here opatchauto performs the patching operation on the original Oracle Home, so there will be down time and high availability of services will get affected. The default patching mode from opatchauto is inplace.
sid <sid>

This option can be used for both Shard patching as well as standalone SIDB patching. In context to standalone SIDB patching it will take the instance name of the database as its value. In context to shard patching it will take the catalog database name. If -service or -sid is not provided in the commandline, the default sid is used from the environment variable 'ORACLE_SID'.
port <port>

This signifies the port for connecting to the catalog database. The default port is set as'1521'.

The following examples demonstrate how to use the various OPatchAuto command options when patching a sharded database.

Example 6-11 Patching a Sharded Database

<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>

Example 6-12 Patching a Data Guard in Sharded Database

<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -dg <primary_database_name1,primary_database_name2,...> -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>

Example 6-13 Patching a Shardgroup in a Sharded Database

<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -shardgroup <shardgroup_name1,shardgroup_name2,...> -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>

Example 6-14 Patching a Shardspace in a Sharded Database

<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -shardspace <shardspace_name1,shardspace_name2,...> -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>

Example 6-15 Listing Wallet Content

<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet.location> -list

Example 6-16 Adding a New Host Credential Entry to the Wallet

<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create <username>:<hostname>:ssh
<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create oracle:myhost:ssh
<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create oracle:127.50.50.50:ssh

Example 6-17 Adding a New Catalog Database Credential Entry to the Wallet

<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create <username>:<sid of catalog db>:jdbc

6.5.3 About the Wallet File

Sharded database patching requires credentials to access the targets. The entries made in the wallet file must satisfy the following requirements:

The wallet file must contain credentials for each home owner in the entire sharding setup. This includes all the shard homes as well as the GI home owners.
It must also contain the credentials for the database user with ‘sysdba’ privilege for the catalog database.
The GI home owner must have a privilege to run commands as root using sudo.

For information on adding credentials to a wallet file and listing down the stored credentials in a wallet file see, OPatchAuto Wallet Tool

6.5.4 Log Files

Summary on the console shows the location of the following log files that can be accessed for further details about the patching process and troubleshooting:

Main log file of the sharding session
Log file for each shard group
Log file for each dataguard/shard space
Log file for each individual database
Log file for each individual grid home

In addition to appearing in the console, these log files will be available on their respective nodes.