3 Multi-Node Patch Orchestration Using OPatchauto

OPatchauto is Oracle's strategic tool for binary and configuration patching. For the supported environments, OPatchauto sequences and executes all required steps, on all nodes, for comprehensive patch application. Because OPatchauto can patch full systems in one invocation, it removes the burdens of:

  • The physical effort of going host to host and executing commands

  • The mental effort of remembering the sequence of commands across the nodes in your system

Your product's patching documentation (Database, Fusion Middleware, Enterprise Manager Cloud Control) explains how to use OPatchauto to patch your specific product. This book augments those guides, by providing deeper conceptual and reference material for OPatchauto, in a product independent manner.


Beginning with the Fusion Middleware 12.2.1 release, OPatchauto has evolved in capability from a single node to multi-node tool. Other products will have multi-node support; once included, this document will be updated with the new link.

This chapter covers the following topics:

3.1 Remote Host Execution using SSH

OPatchauto executes across all hosts in the associated system, using SSH as its remote execution mechanism for requisite commands, such as opatch apply. Of course, the user must supply the SSH credentials to permit this execution. This can be done in three ways:

  • By supplying, via –host, the username and password for a single remote host

  • By supplying a wallet with the SSH credentials for one or more hosts (so required for multi-host patching).

Tip: For understanding the basic principles of SSH equivalence, try the Web search ”generating SSH keys” or ”SSH-keygen” to understand how SSH manages the keys that enable SSH authentication.

Remote Host Patching on Windows

OPatchauto only supports SSH on Windows servers if OpenSSH is set up as part of Cygwin. See "Installing Cygwin and Starting the SSH Daemon" in the Oracle® Enterprise Manager Cloud Control Basic Installation Guide.

3.2 Oracle Wallet for Credential Input

OPatchauto accepts credentials, in oracle wallet format, for accessing run-time entities, such as databases and admin servers. You input a wallet on the command line; if you do not supply one, and OPatchauto needs one, it will prompt you for one on the command line. Successful usage depends on the user possessing both the wallet and the wallet password.

For convenience, you can simply create and configure a wallet for patching purposes using the commands configWallet.sh or configWallet.cmd.

These convenience tools in turn build on the underlying Oracle Wallet mkstore command, which allows the user to create, modify, list, and delete credentials. For more information on mkstore, see "Managing Password Store Credentials" in the Database Security Guide.

The command line syntax of "username&hostname:type" is expected for each credential. The SSH credentials should be defined for the each host in your cluster. The WLS credential only needs to be defined for the Admin Server host.

The tool will prompt you to enter/confirm the password for each credential. For the SSH credentials provide the SSH password of your current user. For the WLS credential provide the same value as PASSWORD from the common.sh.

Here is an example invocation for wallet creation for an Fusion Middleware domain for the hosts outlined below.

./config-wallet.sh -create "${USER}&${HOSTNAME}:SSH" "${USER}&${HOST1}:SSH" "${USER}&${HOST2}:SSH" "${USERNAME}&${HOSTNAME}:wls"

3.3 Patch Orchestration Concepts

Applying a patch involves an orchestrated series of steps. As OPatchauto's name indicates, the customer does not need to understand these steps. They can just apply the patch.

However, the following underlying orchestration concepts are necessary for the customer to learn when the following are true:

  • The patching operation has failed and they need to trouble-shoot.

  • They see advantage to interleaving their own commands into the patching sequence in order to take advantage of their production system's downtime window. In this case, understanding phases is required.

3.3.1 Phases and Sessions

The conceptual related steps of patching operation is called a phase. Executing all phases leads to a completed patching operation on the target; skipping a phase means the patch is not correctly applied. For example, the phase/sub-phase of applying the bits to an Oracle Home is called offline:binary-patching in OPatchauto.

Each invocation of OPatchauto apply generates a new session, whether you are executing all phases in one go (default) or just a sub-phase (advanced).

Phase input to the command is both optional and an advanced feature. However, if the customer wishes to interleave commands between the phases, they will need to invoke OPatchauto multiple times, specifying the specific phase in the correct sequence so that all phases are executed. Phases are composed of sub-phases. The user may also invoke the tool at the sub-phase granularity. The –help option lists the available phases and sub-phases.

Phases are idem-potent; you may execute them repeatedly. However, the tool will not inform you if you do not follow the correct sequence of invocations. (See ER 21553825.)

The high-level phases follow. They can be specified as (a) book-keeping, (b), life-cycle operations, of (c) configuration change operations

  • Init: Bookeeping operation to initialize internal state needed for correction patching.

  • Shutdown: Life-cycle operation that brings down run-time entities to permit patching.

  • Offline: Configuration change operation to apply the patch content with the system down. Bits application happens here, for instance. So the opatch patch will be recorded to the homes OUI inventory in this phase.

  • Start-up: Life-cycle operation that brings the shutdown entities back up again.

  • Online: Configuration change operation apply patch content that requires that the systems be up. If these configuration changes have a system inventory, they will also be recorded to that system's inventory at this point.

  • Finalize: Book-keeping operation to record that patch operation is complete.

In the product's documentation, you might see sub-phases that include ”prepare” and ”binary/product” variants. Prepare means "ready materials but do not make a configuration change." Binary operations only change Oracle Homes. Product operations change the configured system, such as domain configurations or database dictionaries.

The specific content of the patch determines precisely what specific Oracle Home and configuration changes occur. Most Fusion Middleware patches, for example, only include offline content changes. But, of course, some include configuration changes as well.

In general, the session is an implicit parameter, set internally to the last session. It is visible to the user in the logs, communicated as a session ID, but there is no requirement for it to be supplied by the user. As a convenience for specifying the rollback parameters, you can specify the session ID. In this case, OPatchauto knows to query that session for the patch you wish to rollback.

3.3.2 Patch Plans

Patch plans describe, independent of a specific product instance's topology, the sequence of steps to execute in order to deploy the patch. Patch plans are life-cycle programs, developed by Oracle life-cycle management experts, specifically for the given product being patched.

They are optional and advanced inputs. However, internally, OPatchauto always selects a patch plan to guide its execution. For example, internally, OPatchauto apply and OPatchauto rollback automatically select different patch plans implicitly.

Users will input patch plans to OPatchauto when executing more complex life-cycle operation, such as Zero Downtime Patching. The product documentation for these life-cycle operations will list the names of valid patch plans.

3.3.3 Configuration Descriptions

OPatchauto needs to know the configuration (topology) of the system it is to patch. It must either discover programmatically or be given by the user the topology of said system. Whether OPatchauto uses self-discovery or user discovery depends on the underlying product capabilities.

Database patching uses self-discovery; Fusion Middleware requires user discovery. So, for Fusion Middleware, patching multi-node via OPatchauto requires the user to supply manually a topology file. For convenience, patching single-node, single domain Fusion Middlware, whether local or remote, allows the user to supply a domain, via -instance, as an alternative configuration description.

Therefore, to use and benefit from multi-node patching, you must author the topology file using the Fusion Middleware Composer tool, installed under oracle_common/bin.

For more information about Fusion Middleware Composer, see the section on "Building A Topology Using Fusion Middleware Composer" in the Oracle® Fusion Middleware Patching with OPatch guide.

The provided topology provides to OPatchauto information about domains, clusters, oracle_homes, node managers, hosts, and credential aliases (to credentials in the accompanying wallet). OPatchauto uses this data to fully automate the patch application.

3.3.4 Troubleshoot the Application of a Patch

You can find the OPatchauto log files in the following location:


For more information on troubleshooting OPatch, see Chapter 5, "Troubleshooting OPatch."

For both OPatch and OPatchauto, the log file location is also displayed in the console output.