Patch orchestration is the automated execution of the patching steps, such as the execution of pre-patch checks, stopping services, applying the binary patches, and starting the services. Patch orchestration for Oracle Database 12c applies the patch to the GI/RAC configuration on that machine, including all of its databases. The OPatchAuto patch orchestration utility has been made available with version 12.1 of the OPatch utility.
This chapter covers the following topics:
Note:
This chapter applies to Oracle Database 12c only.
Prerequisites: Node Availability During Patching (Rolling vs. Non-rolling)
In order to start a new patching session, the following conditions must be met.
OPatchauto can be executed as a root user.
OPatchauto can be executed as an oracle home owner. In this case -wallet needs to be provided which contains the credential to become root during execution.
Note:
Latest OPatch must be present on all the homes requested for patching on all the nodes.
If opatchauto is executed as Oracle home owner, then sudo must be configured for the oracle home owner to become root and owners of the other homes, if the other homes have different owner.
Note:
The following conditions apply only for the first node, such as when the session is first started on the cluster.
Local node must be up for both Rolling and Non-rolling modes.
At least one of the remote nodes must be up in order to start a Rolling mode session.
ALL the remote nodes must be down in order to start a Non-rolling session.
To ensure successful patching, Oracle recommends that a patching session always consist of the four following steps:
Run opatch lsinventory
When to run: During cluster uptime.
Before you begin a new patching session you must ensure that all nodes of the cluster are at the same patch level.
$GRID_HOME/OPatch/opatch lsinventory
Example Output
Patch level status of Cluster nodes : Patching Level Nodes -------------- ----- 12345678 node123, node456
Run opatchauto apply -analyze
When to run: During cluster uptime.
The apply -analyze command simulates an opatchauto apply session by running all prerequisite checks, when possible, without making changes to the system (either bits or configurations).
Run opatchauto apply
When to run: During cluster downtime.
The apply command applies all patches within a specified system patch to the GI or RAC home from which the opatchauto
command is run.
Run opatch lsinventory
When to run: During cluster uptime.
After successfully running opatchauto apply
on each individual node, you must run opatch lsinventory
on that node to confirm that the patch level has been correctly updated.
The following patching session output examples illustrate successful OPatchAuto apply and rollback sessions. Note: This session applies to both single and multi-node environments.
Example 6-1 OPatchAuto Apply/Rollback Session in Analyze Mode
--------------------------------Summary-------------------------------- Analysis for applying patches has completed successfully: Host:myhostq CRS Home:/scratch/aime_ordb_myhostq/crso1/crshome_crso1 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /tmp/patch_gipsu_12024/patch/22191349/21436941 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22191349/21948341 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22191349/21948344 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22191349/21948354 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-21-21PM_1.log Host:myhostr CRS Home:/scratch/aime_ordb_myhostq/crso1/crshome_crso1 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21436941 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948341 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948344 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log Patch: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948354 Log: /scratch/aime_ordb_myhostq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-24-56PM_1.log OPatchAuto successful.
Example 6-2 OPatchAuto Apply Session
--------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:mymachineemq CRS Home:/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /tmp/patch_gipsu_12024/patch/22191349/21436941 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22191349/21948341 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22191349/21948344 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22191349/21948354 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-41-38PM_1.log Host:mymachineemr CRS Home:/scratch/aime_ordb_mymachineemq/crso1/crshome_crso1 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21436941 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948341 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948344 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log Patch: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/OPatch/auto/dbtmp/22191349/21948354 Log: /scratch/aime_ordb_mymachineemq/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_23-59-15PM_1.log OPatchAuto successful.
Example 6-3 OPatchAuto Rollback Session
--------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:mymachineemm CRS Home:/scratch/aime_ordb_mymachineemm/crso1/crshome_crso1 Summary: ==Following patches were SUCCESSFULLY rolled back: Patch: /tmp/patch_gipsu_12019/patch/22191492/17077442 Log: /scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_19-25-46PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22191492/17303297 Log: /scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_19-25-46PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22191492/21951844 Log: /scratch/aime_ordb_mymachineemm/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-03-08_19-25-46PM_1.log OPatchAuto successful.
Example 6-4 OPatchAuto Apply/Rollback Failure
---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /scratch/aime/app/aime/product/11.2.0/dbhome_2, host: mymachineemg. Command failed: /bin/sh -c 'ORACLE_HOME=/scratch/aime/app/aime/product/11.2.0/dbhome_2 /scratch/aime/app/aime/product/11.2.0/dbhome_2/bin/srvctl stop home -o /scratch/aime/app/aime/product/11.2.0/dbhome_2 -n mymachineemg -f -t TRANSACTIONAL -s /scratch/aime/app/aime/product/11.2.0/dbhome_2/cfgtoollogs/opatchautodb/statfile/mymachineemg/OracleHome-eca39d53-5b51-4cdf-9c79-ce9d9312d86a_mymachineemg.stat' Command failure output: PRCH-1000 : Failed to stop resources running from Oracle home /scratch/aime/app/aime/product/11.2.0/dbhome_2 PRCH-1029 : One or more resources failed to stop: PRCH-1006 : Failed to stop Listener PRCR-1014 : Failed to stop resource ora.LISTENER2.lsnr PRCR-1065 : Failed to stop resource ora.LISTENER2.lsnr CRS-5016: Process "/scratch/aime/app/aime/product/11.2.0/dbhome_2/bin/lsnrctl" spawned by agent "/scratch/aime_ordb_mymachineemg/crso1/crshome_crso1/bin/oraagent.bin" for action "stop" failed: details at "(:CLSN00010:)" in "/scratch/aime_ordb_mymachineemg/crso1/crshome_crso1/log/mymachineemg/agent/crsd/oraagent_aime/oraagent_aime.log" CRS-2675: Stop of 'ora.LISTENER2.lsnr' on 'mymachineemg' failed After fixing the cause of failure Run opatchauto resume with session id "J5A3" ] OPATCHAUTO-68061: The orchestration engine failed. OPATCHAUTO-68061: The orchestration engine failed with return code 1 OPATCHAUTO-68061: Check the log for more details.
Example 6-5 opatchauto apply -analyze
System initialization log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_01-55-58PM.log. Session log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/opatchauto2016-05-05_01-56-18PM.log WARNING: the option -ocmrf is deprecated and no longer needed. OPatch no longer checks for OCM configuration. It will be removed in a future release. The id for this session is MDAN [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Executing OPatch prereq operations to verify patch applicability on CRS Home........ [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Executing prereq operations before applying on CRS Home........ [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully OPatchAuto successful. --------------------------------Summary-------------------------------- Analysis for applying patches has completed successfully: Host:mymachineelu CRS Home:/scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /tmp/patch_gipsu_12019/patch/22654153/17077442 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_13-56-25PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22654153/17303297 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_13-56-25PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22654153/22291141 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_13-56-25PM_1.log
Example 6-6 opatchatuo apply
System initialization log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_02-22-02PM.log. Session log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/opatchauto2016-05-05_02-22-19PM.log WARNING: the option -ocmrf is deprecated and no longer needed. OPatch no longer checks for OCM configuration. It will be removed in a future release. The id for this session is WLR9 [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Executing OPatch prereq operations to verify patch applicability on CRS Home........ [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Executing prereq operations before applying on CRS Home........ [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [shutdown:shutdown] Executing GIShutDownAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Performing prepatch operations on CRS Home........ Prepatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_02-22-52PM.log [shutdown:shutdown] GIShutDownAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [offline:binary-patching] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Start applying binary patches on CRS Home........ [offline:binary-patching] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [startup:startup] Executing GIStartupAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Performing postpatch operations on CRS Home........ Postpatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_02-27-03PM.log [startup:startup] GIStartupAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Verifying patches applied on CRS Home. [finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:mymachineelu CRS Home:/scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /tmp/patch_gipsu_12019/patch/22654153/17077442 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_14-23-38PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22654153/17303297 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_14-23-38PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22654153/22291141 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_14-23-38PM_1.log
Example 6-7 opatchauto rollback
System initialization log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_04-34-39PM.log. Session log file is /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/opatchauto2016-05-05_04-35-00PM.log The id for this session is K5BA [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Executing OPatch prereq operations to verify patch applicability on CRS Home........ [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Executing prereq operations before rolling back on CRS Home........ [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [shutdown:shutdown] Executing GIShutDownAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Performing prepatch operations on CRS Home........ Prepatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_04-35-22PM.log [shutdown:shutdown] GIShutDownAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfull [offline:binary-patching] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Start rolling back binary patches on CRS Home........ [offline:binary-patching] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully [startup:startup] Executing GIStartupAction action on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Performing postpatch operations on CRS Home........ Postpatch operation log file location: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/crsconfig/crspatch_mymachineelu_2016-05-05_04-38-59PM.log [startup:startup] GIStartupAction action completed on home /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 successfully OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:mymachineelu CRS Home:/scratch/aime_ordb_mymachineelu/crso1/crshome_crso1 Summary: ==Following patches were SUCCESSFULLY rolled back: Patch: /tmp/patch_gipsu_12019/patch/22654153/17077442 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_16-36-04PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22654153/17303297 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_16-36-04PM_1.log Patch: /tmp/patch_gipsu_12019/patch/22654153/22291141 Log: /scratch/aime_ordb_mymachineelu/crso1/crshome_crso1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_16-36-04PM_1.log
Example 6-8 opatchauto apply -analyze (Multi-node)
System initialization log file is /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchautodb/systemconfig2016-05-05_10-19-07PM.log. Session log file is /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/opatchauto2016-05-05_10-19-19PM.log The id for this session is KMKT [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 Executing OPatch prereq operations to verify patch applicability on RAC Home........ Executing OPatch prereq operations to verify patch applicability on CRS Home........ [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully <mymachineelh> : [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 <mymachineelh> : [init:init] Executing OPatchAutoBinaryAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 <mymachineelh> : Executing OPatch prereq operations to verify patch applicability on CRS Home........ <mymachineelh> : Executing OPatch prereq operations to verify patch applicability on RAC Home........ <mymachineelh> : [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully <mymachineelh> : [init:init] OPatchAutoBinaryAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 Executing prereq operations before applying on CRS Home........ Executing prereq operations before applying on RAC Home........ [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully <mymachineelh> : [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 <mymachineelh> : [init:init] Executing GIRACPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 <mymachineelh> : Executing prereq operations before applying on RAC Home........ <mymachineelh> : Executing prereq operations before applying on CRS Home........ <mymachineelh> : [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully <mymachineelh> : [init:init] GIRACPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully [init:init] Executing RACDBPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 Executing prereq operations before applying on RAC Home........ <mymachineelh> : [init:init] Executing RACDBPrereqAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 <mymachineelh> : Executing prereq operations before applying on RAC Home........ <mymachineelh> : Following step failed during analysis: /bin/sh -c 'ORACLE_HOME=/scratch/aime_ordb_mymachineelg/obase/rachome_racr1 ORACLE_SID=racr12 /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/OPatch/datapatch -prereq' <mymachineelh> : [init:init] RACDBPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully [init:init] RACDBPrereqAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully [finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 [finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully [finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 [finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully <mymachineelh> : [finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 <mymachineelh> : [finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 successfully <mymachineelh> : [finalize:finalize] Executing DeletePatchAction action on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 <mymachineelh> : [finalize:finalize] DeletePatchAction action completed on home /scratch/aime_ordb_mymachineelg/obase/rachome_racr1 successfully OPatchAuto successful. --------------------------------Summary-------------------------------- Analysis for applying patches has completed successfully: Host:mymachineelg CRS Home:/scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /tmp/patch_gipsu_12024/patch/22646084/21436941 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22646084/22291127 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22646084/22502518 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22646084/22502555 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-37PM_1.log Host:mymachineelg RAC Home:/scratch/aime_ordb_mymachineelg/obase/rachome_racr1 ==Following patches were SKIPPED: Patch: /tmp/patch_gipsu_12024/patch/22646084/21436941 Reason: This patch is not applicable to this specified target type - "rac_database" Patch: /tmp/patch_gipsu_12024/patch/22646084/22502518 Reason: This patch is not applicable to this specified target type - "rac_database" ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /tmp/patch_gipsu_12024/patch/22646084/22291127 Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-39PM_1.log Patch: /tmp/patch_gipsu_12024/patch/22646084/22502555 Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-24-39PM_1.log Host:mymachineelh RAC Home:/scratch/aime_ordb_mymachineelg/obase/rachome_racr1 ==Following patches were SKIPPED: Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/21436941 Reason: This patch is not applicable to this specified target type - "rac_database" Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502518 Reason: This patch is not applicable to this specified target type - "rac_database" ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22291127 Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-25-55PM_1.log Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502555 Log: /scratch/aime_ordb_mymachineelg/obase/rachome_racr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-25-55PM_1.log Host:mymachineelh CRS Home:/scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1 ==Following patches were SUCCESSFULLY analyzed to be applied: Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/21436941 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22291127 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502518 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log Patch: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/OPatch/auto/dbtmp/22646084/22502555 Log: /scratch/aime_ordb_mymachineelg/crsr1/crshome_crsr1/cfgtoollogs/opatchauto/core/opatch/opatch2016-05-05_22-28-24PM_1.log
When you run OPatchAuto's apply command, numerous operations are performed to implement the complete patch application cycle. These operations vary depending on the environment to be patched. The following environment is representative of the vast majority of patching environments in which OPatchAuto is used. For example, a typical patching environment would be one GI Home managing two RAC Homes. When you run opatchauto apply
, OPatchAuto will perform the operations shown in Figure 6-1.
Figure 6-1 Patching with OPatchAuto: Process Flow
Depending on the patch or the home directory configuration, you may encounter a request to reboot the system. After a reboot during the patching process, you need to invoke the opatchauto
utility again so that it seamlessly continues with rest of the patch application process.
Typically an error message, as shown in the following example, will be displayed when a problem arises.
Example 6-9 OPatchAuto Console Error
# OPatch/opatchauto apply /scratch/aime/sh/RDBMS_12.1.0.1.0_LINUX.X64_130418/patches/v2/nosql/gipsu/11111111 -ocmrf /tmp/ocm.rsp OPatch Automation Tool Copyright (c) 2013, Oracle Corporation. All rights reserved. ... CLSRSC-400: A system reboot is required to continue installing. ... Apply Summary: Following patch(es) are successfully installed: GI_HOME=/u01/GI12/app/12.1.0/grid:13852018, 22222222, 123456788 DB_HOME=/scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1:13852018, 123456788 DB_HOME=/scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1:13852018, 123456788 opatchauto failed with error code 1.
When you receive an error like this, follow the reboot instructions specified in the console. The following example shows a system reboot request issued by the user.
Example 6-10 Rebooting the System
# OPatch/opatchauto resume -reboot OPatch Automation Tool Copyright (c) 2013, Oracle Corporation. All rights reserved. OPatchauto version : 12.1.0.1.1 OUI version : 12.1.0.1.0 Running from : /u01/GI12/app/12.1.0/grid Log file location : /u01/GI12/app/12.1.0/grid/cfgtoollogs/opatch/opatch2013-05-16_13-36-59PM_1.log Opatchauto will attempt to resume from reboot patching session. This might take several minutes... Command "/usr/bin/perl /u01/GI12/app/12.1.0/grid/crs/install/rootcrs.pl -postpatch" is successfully resumed. Command "/scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1/bin/srvctl start home -o /scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1 -n slc00epi -s /scratch/aime1/DB12N/app/aime1/product/12.1.0/dbhome_1/OracleHome-50b8f1a0-e220-4b8e-98d7-49177979991f.stat " is successfully resumed. Command "/scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1/bin/srvctl start home -o /scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1 -n slc00epi -s /scratch/aime/DB12_2/app/aime/product/12.1.0/dbhome_1/OracleHome-58232a10-3130-4930-b588-0c8594cf8c87.stat " is successfully resumed. Opatchauto was able to resume from the previous reboot patching session and complete successfully. opatchauto succeeded.
The session has to be initiated from the catalog host by providing details of the catalog database. It should provide a top level view to the end user about patching a SDB, making it easier to understanding the flow of opatchautoSDB.
Sharding is an application-managed scaling technique using many (hundreds /thousands of) independent databases. With sharding, data is split into multiple databases (shards) with each database holding a subset of data. Shards can be replicated for high availability and scalability.
OPatchAuto supports end-to-end patching of sharded databases across multiple regions, along with the grid infrastructure (GI) that supports the clustered databases/shards and shards that are managed by Oracle Golden Gate or by Oracle DataGuard.
OPatchAuto supports all sharding and replication methods.
Supported Configurations:
Shard Types: Databases running on GI/SIHA and standalone databases
Different versions across Data Guards (currently, only version 12.2 is available)
Multiple shards sharing the same cluster/grid
Multiple versions of shards in an OGG replicated SDB
A sharded database can span multiple regions and clusters. To manage the patching cycle more efficiently, you may want to scale down the patching effort by patching only specific subset entities. OPatchAuto provides three options for selecting the subset entities:
1. Data Guards
2. Shard Groups
3. Shard Spaces
You can select instances of any one of the above entities. Note: You can only specify a single entity for a given patching cycle. For all of the subset entities, OPatchAuto identifies the targets on the basis of their names:
1. –dg: Comma-separated list of names of the primary database
2. –shardgroup: Comma-separated list of names of the shard groups
3. –shardspace: Comma-separated list of names of the shard spaces
For all of the above options, the Grid homes of the databases will also be patched. The CRS/RAC homes will be patched across all their nodes in a rolling manner. Similarly, the Golden Gate home will also be patched.
By selecting the -dg <primaryDB name>
option, only the databases (along with their cluster homes) that belong to the selected Data Guard will be patched. All these databases will be patched in a rolling manner starting with the standby shards and ending with the primary shard.
The entire list of databases in a sharded database can be collected from the catalog database table GSMADMIN_INTERNAL.DATABASE under the column NAME.
By selecting the -shardgroup <shardgroup name>
option, the databases of the selected shard group will be patched in a rolling manner. In a Data Guard-replicated configuration, a shard group can host ONLY one member of each Data Guard. Hence, this option is NOT supported in a sharded database that employs Data Guard replication because an entire Data Guard, while being spread across multiple shard groups, needs to be patched together in a defined sequence starting with standby databases and ending with the primary database. Patching individual shard groups poses a major risk of breaking that sequence.
The entire list of shard groups in a sharded database can be collected from the catalog database table GSMADMIN_INTERNAL.SHARD_GROUP under the column NAME.
By specifying the -shardspace <shardspace name>
option, the databases of the entire shard space are patched in a rolling manner. These databases could be spread across multiple shard groups, depending upon the configuration of the sharded database.
The entire list of shard spaces in a sharded database can be collected from the catalog database table GSMADMIN_INTERNAL.SHARD_SPACE under the column NAME.
The OPatchAuto sdb command option allows you to patch sharded databases. This operation patches all the shards of the Sharded database.
In a sharded database with Data Guard replication, patching a Data Guard involves patching all its standby databases first, followed by its primary database.In Oracle Golden Gate based replication, for a user-defined configuration, the operation involves patching around the shard spaces. For system-managed and composite configuration, the operation involves patching around the shard groups.The values of -host
, -port
and -sid
are used to connect to the catalog database and hence they should form the required connect string for the catalog database.All the database(s) are patched first.Thereafter, the Grid/HAS homes that are host to any of the databases belonging to the sharded database are patched.The catalog database and the GSM's will not be patched. In order to patch these, OPatchAuto needs to be run separately on these databases.
The following syntax illustrates command line usage:
opatchauto apply <patch-location> -sdb -wallet <wallet> [ -phBaseDir <patch.base.directory> ] [ -logLevel <log_priority> ] [ -analyze ] [ -host <tns-host> ] [ -dg <primary.database.name> ] [ -shardgroup <shardgroup> ] [ -shardspace <shardspace> ] [ -rolling ] [ -service <service> ] [ -inplace ] [ -sid <sid> ] [ -port <port> ]
Parameters
patch-location
The patch location.
Options
phBaseDir <patch.base.directory>
The location of base patch directory.
logLevel <log_priority>
The log level (defaults to "INFO").
Supported values: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
analyze
If this option is selected, the environment will be analysed for suitability of the patch on each home, without affecting the home.
The patch will not be applied or rolled back, and targets will not be shut down.
host <tns-host>
The tns-host of the catalog database. This should match the 'HOST' used in the network configuration file of the catalog database. The default host is set as the local 'hostname' without appending the domain.
wallet <wallet location> (Required)
The location of the wallet file.
The entries made in the wallet file must satisfy these requirements:
It is mandatory to provide the credentials of all home owners of every node that would be patched as part of the session in the wallet. The homes that would be patched on the node also includes GI/SIHA when installed. The list of nodes refers to all hostnames/ip-address listed in GSMADMIN_INTERNAL.SHA_DATABASES.DB_HOST column of the Sharded database catalog.
It must also contain the credentials for the database user with 'sysdba' privilege for the catalog database.
Additionally, the host user provided in the wallet must meet the following requirements:
The user must be able to change to root using ‘sudo’ if the node belongs to GI/SIHA environment.
dg <primary.database.name>
This is used to restrict patching to databases of the selected dataguard. All the standby databases of the dataguard are patched first, followed by its primary database.
shardgroup <shardgroup>
This is used to restrict patching to databases of the selected shard group.
shardspace <shardspace>
This is used to restrict patching to databases of the selected shard space.
sdb (Required)
To signify patching sharded database. Run 'opatchauto <apply|rollback> -sdb -help' to get more help on patching a sharded database.
rolling
Enables sdb rolling mode where database(s) are patched one after the other.
service <service>
Service name of the catalog database.
inplace
This option can be used to perform in place patching through opatchauto. Here opatchauto performs the patching operation on the original Oracle Home, so there will be down time and high availability of services will get affected. The default patching mode from opatchauto is inplace.
sid <sid>
This option can be used for both Shard patching as well as standalone SIDB patching. In context to standalone SIDB patching it will take the instance name of the database as its value. In context to shard patching it will take the catalog database name. If -service or -sid is not provided in the commandline, the default sid is used from the environment variable 'ORACLE_SID'.
port <port>
This signifies the port for connecting to the catalog database. The default port is set as'1521'.
The following examples demonstrate how to use the various OPatchAuto command options when patching a sharded database.
Example 6-11 Patching a Sharded Database
<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>
Example 6-12 Patching a Data Guard in Sharded Database
<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -dg <primary_database_name1,primary_database_name2,...> -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>
Example 6-13 Patching a Shardgroup in a Sharded Database
<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -shardgroup <shardgroup_name1,shardgroup_name2,...> -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>
Example 6-14 Patching a Shardspace in a Sharded Database
<CATALOG_DB_HOME>/OPatch/opatchauto apply <patch location> -sdb -shardspace <shardspace_name1,shardspace_name2,...> -wallet <wallet file location> -sid <sid of catalog db> -port <sid configured port>
Example 6-15 Listing Wallet Content
<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet.location> -list
Example 6-16 Adding a New Host Credential Entry to the Wallet
<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create <username>:<hostname>:ssh <CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create oracle:myhost:ssh <CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create oracle:127.50.50.50:ssh
Example 6-17 Adding a New Catalog Database Credential Entry to the Wallet
<CATALOG_DB_HOME>/OPatch/auto/core/bin/patchingWallet.sh -walletDir <wallet location> -create <username>:<sid of catalog db>:jdbc
Sharded database patching requires credentials to access the targets. The entries made in the wallet file must satisfy the following requirements:
The wallet file must contain credentials for each home owner in the entire sharding setup. This includes all the shard homes as well as the GI home owners.
It must also contain the credentials for the database user with ‘sysdba’ privilege for the catalog database.
The GI home owner must have a privilege to run commands as root using sudo.
For information on adding credentials to a wallet file and listing down the stored credentials in a wallet file see, OPatchAuto Wallet Tool
Summary on the console shows the location of the following log files that can be accessed for further details about the patching process and troubleshooting:
Main log file of the sharding session
Log file for each shard group
Log file for each dataguard/shard space
Log file for each individual database
Log file for each individual grid home
In addition to appearing in the console, these log files will be available on their respective nodes.