Completing an Oracle Clusterware Upgrade when Nodes Become Unreachable

If some nodes become unreachable in the middle of an upgrade, then you cannot complete the upgrade, because the upgrade script (rootupgrade.sh) did not run on the unreachable nodes. Because the upgrade is incomplete, Oracle Clusterware remains in the previous release.

1. Confirm that the upgrade is incomplete by entering the following command:

   $ crsctl query crs activeversion

   The active release should be the source release.
2. To resolve this problem, run the rootupgrade.sh command with the -force flag using the following syntax:

   Grid_home/rootupgrade -force

   For example:

   # /u01/app/23.0.0/grid/rootupgrade -force
   

   This command forces the upgrade to complete. The force cluster upgrade has the following limitations:

   • All active nodes must be upgraded to the newer release.

   • All inactive nodes (accessible or inaccessible) may be either upgraded or not upgraded.

   • For inaccessible nodes, after patch set upgrades, you can delete the node from the cluster. If the node becomes accessible later, and the patch version upgrade path is supported, then you can upgrade it to the new patch version.

3. Verify that the upgrade has completed by using the following command:

   $ crsctl query crs activeversion -f
   Oracle Clusterware active version on the cluster is [23.0.0.0.0]. 
   The cluster upgrade state is [FORCED]. The cluster active patch level is [0].

   The active release should be the upgraded release. The upgrade state FORCED indicates that Oracle Clusterware upgrade was completed forcibly on all reachable nodes, while one or more cluster nodes were unreachable. You can either upgrade or delete the unreachable cluster nodes to change the cluster upgrade state from FORCED to NORMAL.