8 Updating Exadata Software

There are different types of software used on an Oracle Exadata Database Machine that need be updated regularly.

8.1 About Updating Exadata Software

Exadata software updates apply to three major components:

  • Exadata storage servers
  • Exadata database servers
  • Exadata RDMA Network Fabric switches

Exadata storage server and Exadata database server updates generally contain updates for:

  • Oracle Linux operating system
  • Oracle Exadata System Software
  • Firmware (for example: disk, flash, RAID controller, ILOM, HCA)

The updates do not modify the Oracle Grid Infrastructure home, Oracle Database home (other than relinking during the dbnodeupdate.sh -c step), or customer-installed software.

While it is generally recommended for components to stay in step with the recommended minimum release, you can choose to update different components at different times. For example, you could update RDMA Network Fabric switches at a later time than Exadata storage servers and Exadata database servers. However, you must check My Oracle Support Doc ID 888828.1 for any dependencies.

It is not mandatory to apply each and every Oracle Exadata System Software update that comes out. For example, you can skip two or three releases and update directly to a newer release. Oracle recommends that you update database servers twice a year.

Upgrading is allowed under the following circumstances:

  1. The product version of the target release is higher than the installed software, and
  2. The date code of the target release is higher than the installed software.

For example, consider a system that is currently running image version 12.1.1.1.2.150411, which consists of product version 12.1.1.1.2 and date code 150411.

  • Upgrading to 12.1.2.1.2.150617.1 is allowed because both rules are satisfied.
  • Upgrading to 12.1.2.1.1.150316.2 is not allowed because date code 150411 of the installed software is higher than date code 150316 of the target release.

8.2 Planning for Software Maintenance

Before starting a software update, you should review best practices, determine the version to which you will upgrade, and obtain the proper patching software.

8.2.1 Understanding Exadata Database Machine Software and Updates

Understanding the different types of software updates required for Exadata Database Machine helps you plan an update schedule.

8.2.1.1 What Software Do You Update on Oracle Exadata Database Machine?

The software that runs on Oracle Exadata Database Machine is divided into two categories.

  • Oracle Exadata Database Machine infrastructure software

  • Oracle Grid Infrastructure and Oracle Database software

The following table describes the two primary categories of software that run on an Oracle Exadata Database Machine.

Software Components Installed On Software Content Example Software Version

Oracle Exadata Database Machine infrastructure software

  • Exadata storage servers
  • Exadata database servers
  • Exadata RDMA Network Fabric switches

 

  • Oracle Linux operating system
  • Oracle VM Server (on database servers for Oracle VM configurations)
  • Oracle Exadata System Software
  • Firmware (for example: disk, flash, RAID controller, ILOM, HCA, RDMA Network Fabric switch firmware)

Oracle Exadata System Software on storage servers and database servers:

  • 12.2.1.1.0
  • 12.1.2.3.4

InfiniBand Network Fabric switch firmware:

  • 2.2.4-3
  • 2.1.8-1

RoCE Network Fabric switch firmware:

  • 7.0(3)I7(6)

Oracle Grid Infrastructure and Oracle Database software

Exadata database servers

  • Oracle Grid Infrastructure (Grid home)
  • Oracle Database (Database home)

12.2.0.1.0

12.1.0.2.160117

11.2.0.4.161018

When an Oracle Exadata Database Machine is deployed, all of the software described in the table is installed and configured to deliver high performance and availability for Oracle Database. Extensive end-to-end testing ensures all software components supplied with Oracle Exadata Database Machine work seamlessly together.

My Oracle Support Doc ID 888828.1 is the primary source of information for software that runs on Oracle Exadata Database Machine. It contains the following information:

  • List of all current and previous releases
  • Minimum requirements for feature usage
  • Compatibility requirements between Oracle Exadata System Software version and Oracle Database software version
  • Compatibility requirements for specific hardware releases
  • Guidelines for related products when used with Oracle Exadata Database Machine
  • References to other pertinent information sources for Oracle Exadata Database Machine software maintenance
8.2.1.2 Software Release Types

The primary software categories Exadata Infrastructure software and Grid Infrastructure and Database software are further broken down by the release type.

Software Release type Description and Content

Exadata Infrastructure software

Feature release

The first four digits represent a Exadata feature release, for example, 12.2.1.1.0.

A feature release contains new features, bug fixes, and security fixes. It may be used to update any prior feature or sustaining release.

You can update directly to feature release 12.2.1.1.0 from any prior release from 11.2.3.3.1 to 12.1.2.3.4, inclusive.

Exadata Infrastructure software

Sustaining release

The fifth digit represents a cumulative update to a feature release, for example, 12.1.2.3.4.

A sustaining release contains bug fixes and security fixes. A given feature release will have approximately four sustaining releases. A sustaining release may be used to update any prior feature or sustaining release.

You can update directly to sustaining release 12.1.2.3.4 from any prior release from 11.2.3.3.1 to 12.1.2.3.3, inclusive.

Exadata Infrastructure software

Interim patch

An interim patch is one-off bug fix made available to customers who cannot wait until the fix is included in a sustaining release or feature release. Installation of an interim patch is done on an as-needed basis and is not a regularly scheduled planned maintenance event.

Oracle Grid Infrastructure and Oracle Database software

Major or maintenance release

The first and second digits represent a major and maintenance database release, for example, 12.2 or 12.1.

A major release or maintenance release contains new features, bug fixes, and security fixes.

Oracle Grid Infrastructure and Oracle Database software

Patch set release

The forth digit represents a patch set release, for example, 12.2.0.1 or 12.1.0.2. 

A patch set release contains primarily bug fixes. However, some minor new features and change in functionality may be included in a patch set release.

Oracle Grid Infrastructure and Oracle Database software

Proactive Bundle Patch

The fifth digit represents a cumulative update to a patch set release, for example, 12.1.0.2.170117 or 11.2.0.4.161018.

The proactive bundle patch contains bug fixes and security fixes. It is a superset of the standard Oracle Grid Infrastructure PSU (GIPSU). Only the Proactive Bundle Patch can be installed on Exadata systems, not the standard PSU (with the exception of Oracle Database 12.1.0.1, which uses the standard GIPSU).

For Oracle Database 11.2, the proactive bundle patch is called Database Patch for Exadata.

Oracle Grid Infrastructure and Oracle Database software

Interim patch

An interim patch is one-off bug fix made available to customers who cannot wait until the fix is included in a subsequent patch set release or database patch for Exadata. Installation of an interim patch is done on an as-needed basis and is not a regularly scheduled planned maintenance event.

8.2.1.3 Software Release Availability

Each software release type has a different frequency of availability.

Exadata sustaining releases and quarterly Grid Infrastructure and Database Proactive Bundle Patches are released on regular quarterly cycle as part of the Oracle Critical Patch Update (CPU) program.  Oracle provides these quarterly releases is to address proactive, critical fixes and security vulnerabilities.

Software Release type Release Frequency Importance of Maintaining Up-to-Date Versions Recommendation for Adopting New Releases

Exadata Infrastructure software

Feature release

6-18 months

Medium

Update to adopt new features and get critical fixes and security updates.

Exadata Infrastructure software

Sustaining release

Monthly

High

Update quarterly to get critical fixes and security updates. Do not lag more than 12 months.

Oracle Grid Infrastructure and Oracle Database software

Major or maintenance release

24+ months

Low

Update to adopt new features. Update prior to Lifetime Support end date for current release.

Oracle Grid Infrastructure and Oracle Database software

Patch set release

12-24 months

Medium

Update prior to Error Correction Support end date for current patch set, which is typically one year after next patch set release.  Details in My Oracle Support Document 742060.1.

Oracle Grid Infrastructure and Oracle Database software

Proactive Bundle Patch

3 months (quarterly)

High

Update quarterly to get critical fixes and security updates. Do not lag more than 12 months.

8.2.1.4 Software Update Frequency

You should plan to update your Exadata software on a regular basis.

These examples show three high-level quarterly software maintenance plans over a four year cycle using typical intervals between releases.

Note:

Release frequency may vary from what is documented here, particularly for releases that contain new features.
  • Example 8-1 Production System Software Maintenance Plan

  • Example 8-2 Production System Software Maintenance Plan with Reduced Updates

  • Example 8-3 Development and Test System Software Maintenance Plan

Example 8-1 Production System Software Maintenance Plan

Goal — Minimize risk by applying critical and security fixes as they become available, and adopting new feature releases as late as possible. Where possible do not adopt a new Exadata feature release and new Database feature release simultaneously.

Figure 8-1 Frequency of Production System Software Updates



Example 8-2 Production System Software Maintenance Plan with Reduced Updates

Goal — Minimize maintenance time by applying alternating quarterly updates. Where possible do not adopt a new Exadata feature release and new Database feature release simultaneously.

Figure 8-2 Frequency of Reduced Production System Software Updates



Example 8-3 Development and Test System Software Maintenance Plan

Goal — Adopt the latest features and software updates as soon as possible.

Figure 8-3 Frequency of Development and Test System Software Updates



8.2.1.5 Software Update Utilities

When updating software on Oracle Exadata Database Machine, you use specific utilities, depending on the component being updated.

Update Type Utility Components Description

Pre-update readiness check

Oracle EXAchk

Oracle Exadata Database Machine

Healthcheck tool used to qualify maintenance readiness, identify critical issue exposure, and provide version recommendations for updating software.

Obtain Oracle EXAchk from My Oracle Support Document 1070954.1.

Oracle Exadata Database Machine infrastructure software

patchmgr

  • Oracle Exadata Database Machine storage servers
  • Oracle Exadata Database Machine database servers
  • RDMA Network Fabric switches

Oracle Exadata Database Machine software update tool that updates all software (Oracle Linux, Oracle Exadata System Software, firmware) on Oracle Exadata Database Machine database servers, storage servers, or RDMA Network Fabric switches. For example, patchmgr updates Oracle Exadata Database Machine storage servers and database servers to Oracle Exadata System Software version 12.2.1.1.0, or InfiniBand Network Fabric switches to version 2.2.4-3.

Oracle Grid Infrastructure and Oracle Database software Proactive Bundle Patch

OPatch or opatchauto

Oracle Exadata Database Machine database servers

Software update tool that applies a Proactive Bundle Patch to an Oracle Grid Infrastructure home or Oracle Database home. For example, OPatch applies Proactive Bundle Patch 12.1.0.2.170117 to 12.1.0.2 Oracle Grid Infrastructure and Oracle Database homes.

Obtain OPatch from My Oracle Support Document 274526.1.

Oracle Grid Infrastructure and Oracle Database software patch set, major release, or maintenance release.

Oracle Universal Installer

Oracle Exadata Database Machine database servers

Software installation tool that installs an Oracle Grid Infrastructure and Oracle Database patch set, major, or maintenance release, and upgrades Oracle Grid Infrastructure. For example, Oracle Universal Installer installs Oracle Database 12.2.0.1, or installs and upgrades Oracle Grid Infrastructure from 12.1.0.2 to 12.2.0.1.

Use the version of Oracle Universal Installer bundled with the Oracle Grid Infrastructure and Oracle Database release to which you are upgrading.

Upgrade of an Oracle Database

Database Upgrade Assistant (DBUA)

Oracle Exadata Database Machine database servers

Database tool that upgrades a database to a later patch set, major, or maintenance release. For example, DBUA upgrades a database from 12.1.0.1 to 12.1.0.2, or from 12.1.0.2 to 12.2.0.1.

Use DBUA bundled with the Oracle Grid Infrastructure and Oracle Database release to which you are upgrading.

Oracle Grid Infrastructure and Oracle Database software

Fleet Patching and Provisioning (previously named Rapid Home Provisioning (RHP))

Oracle Exadata Database Machine database servers

Software maintenance tool that provisions, patches, and upgrades Oracle Grid Infrastructure and Oracle Database software on one or more clusters from a centralized server.

Use Fleet Patching and Provisioning as a feature of Oracle Clusterware.

Oracle Exadata Database Machine infrastructure software

Oracle Grid Infrastructure and Oracle Database software

Oracle Enterprise Manager Cloud Control

  • Exadata storage servers
  • Exadata database servers
  • Exadata InfiniBand switches

Oracle Enterprise Manager Cloud Control provides lifecycle management capability to apply software updates to Oracle Exadata Database Machine.

8.2.2 Configuration and Operational Best Practices for Software Maintenance

As part of planning your software update, you should review the different methods of performing updates and the best practices for updating the software.

8.2.2.1 Understanding Rolling and Non-Rolling Updates

Software updates can be performed in rolling manner while the database remains online and available, or non-rolling manner where Oracle Clusterware and Oracle Database are shutdown.

The manner an update is performed does not affect how often it should be done, but does determine how long it will take. In general there are two methods:

  • Rolling software updates — A rolling software update is one that is performed to one of a particular component at a time while the others remain online servicing requests. Note the following key points about rolling software updates:

    • Rolling updates have less application downtime compared to non-rolling updates.

    • The overall length of time to complete the update is longer because one component is offline and updated at a time, while all other components remain online and operational.

    • The impact to existing database connections differs depending on the component being updated.

      • Rolling updates for Oracle Exadata Database Machine storage servers or RDMA Network Fabric switches — All databases remain fully online and available for the duration of the rolling update. There is no disruption to database connections.

      • Rolling updates for Oracle Exadata Database Machine database servers, Oracle Grid Infrastructure, or Oracle Database — Multi-instance databases using Oracle Real Application Clusters (Oracle RAC) remain available for the duration of the update. However, database connections on the server being updated will be disrupted when the local database instance is shutdown. Use Oracle RAC features for client high availability to minimize application disruption, as described in Table 8-1.

  • Non-Rolling software updates — A non-rolling software update is one that is performed to all of a particular component while those components are offline. Note the following key points about non-rolling software updates:

    • Non-rolling updates are faster than rolling updates for overall maintenance time because multiple components are updated in parallel.
    • Because all of a particular component are offline for the duration of the update, a database (or all databases on a system) will also be offline for the duration of the update, resulting in a complete outage to the application
    • Important applications serviced by the Oracle Exadata Database Machine being updated in a non-rolling manner are often moved to a standby system in environments using Oracle Data Guard.
  • Combination of rolling and non-rolling software updates — When multiple components are updated in the same maintenance window, it is possible to use a combination of rolling and non-rolling methods to achieve the desired balance of application downtime and maintenance time. One typical combination used in the situation where an application does not handle connection disruption efficiently is to perform Oracle Exadata Database Machine storage server and RDMA Network Fabric switch updates in a rolling manner, then performing Oracle Grid Infrastructure, Oracle Database, and Oracle Exadata Database Machine database server updates in a non-rolling manner.

The patchmgr update utility manages the update orchestration in a rolling or non-rolling manner for Oracle Exadata Database Machine infrastructure components (storage servers, database servers, and RDMA Network Fabric switches).

The following table describes which update method is supported for each component type.

Table 8-1 Rolling versus Non-Rolling Upgrades

Method Component Support Database Availability Impact During Update

Rolling

Oracle Exadata Database Machine storage servers

No impact.

Oracle RAC and single instance databases remain fully available on all nodes in the cluster.

Rolling

RDMA Network Fabric switches

No impact.

Oracle RAC and single instance databases remain fully available on all nodes in the cluster.

Rolling

Oracle Exadata Database Machine database servers

Local connections are disconnected and all local instances are shutdown.

Oracle RAC databases remain available through other nodes in the cluster.

Rolling

Oracle Grid Infrastructure

Local connections are disconnected and all local instances are shutdown.

Oracle RAC databases remain available through other nodes in the cluster.

Rolling

Oracle Database - Proactive Bundle Patch

For the database home being updated, local connections are disconnected and local instances are shutdown.

Oracle RAC databases remain available through other nodes in the cluster.

Databases running from other Oracle Database software homes are unaffected.

Non-Rolling

  • Oracle Exadata Database Machine storage servers

  • Oracle Exadata Database Machine database servers

  • Oracle Grid Infrastructure

  • Oracle Database - Proactive Bundle Patch

  • Oracle Database - Patch Set

  • Oracle Database - Release

Databases unavailable

Move workload to Oracle Data Guard or Oracle GoldenGate standby systems to minimize impact.

8.2.2.2 Online Updates for Oracle Linux Kernel and Oracle Database Interim Fixes

 Online updates of qualified fixes are supported for Oracle Linux and Oracle Database.

Typically, planned software updates require that the component be restarted after the update. For example, an Exadata database server must be restarted after applying an Exadata software release to update the system firmware and to make active a new Oracle Linux kernel. Similarly, an Oracle database instance must be stopped and restarted to apply a software update to the database home.

Oracle supplies some interim fixes, however, that can be applied online, such that a component does not require a restart to apply the fix and make them active. Online updates of qualified fixes are supported for the following components:

  • Oracle Linux kernel on Exadata database servers using the Ksplice Offline Client.

  • Oracle Database using OPatch Online Patching.

Online updates are typically performed as a temporary measure when a critical fix must be applied to the system before the next scheduled planned software maintenance.

8.2.2.3 Configuration Practices for Optimal Software Maintenance

When configuring an Exadata Database Machine, it is important to adopt features that will lessen the impact and risk of performing software updates.

Configuration Practice Use for Software Maintenance Purpose

Oracle ASM high redundancy disk groups

Rolling updates - Exadata storage servers

When performing storage server updates in a rolling manner with the databases remaining online, it is highly recommended to configure Oracle ASM disk groups with high redundancy. High redundancy disk groups can tolerate the failure of a disk in another storage server during rolling updates.

During rolling, or online, storage server updates, the disks for the storage server being updated are taken offline on one storage server by patchmgr while it is updated. After the update completes the disks are resynchronized by Oracle ASM, and then patchmgr starts to update the next storage server.

While disks are offline the disk group has reduced redundancy. A normal redundancy disk group with reduced redundancy during a rolling update may dismount and have data loss if a disk fails in another storage server.

Oracle ASM disk group redundancy is typically set during initial system configuration. Therefore, consider how you plan to perform storage server updates, rolling or non-rolling, prior to system configuration.

An Oracle Data Guard physical standby system can also provide protection against disk failure during rolling storage server update.

Oracle RAC features for client high availability

Rolling updates - Exadata database servers, Oracle Grid Infrastructure or Oracle Database software

When performing Exadata database server, Oracle Grid Infrastructure, or Oracle Database software updates in a rolling manner on one database server at a time while database services remain running on the other servers in the cluster, the database services must be stopped on the Exadata database server being updated.

To minimize the impact to client applications connected to database instances that will be stopped during maintenance, configure client applications to use database services, Fast Application Notification (FAN), Fast Connection Failover (FCF), and Application Continuity.

Oracle Data Guard physical standby database

Any rolling or non-rolling update (except Oracle Database patch set and release updates)

Oracle Data Guard Standby-First Patch Apply provides support for different Exadata infrastructure, Oracle Grid Infrastructure, and Oracle Database software between a primary database and its physical standby database for the purpose of applying and validating Oracle patches in rolling fashion with minimal risk to the primary database.

Oracle Data Guard transient logical standby database

Any rolling or non-rolling update

Use Oracle Data Guard logical standby database to reduce database upgrade downtime for patch set and release updates. Database upgrade downtime is reduced by allowing the logical standby database to be upgraded to the new version and kept synchronized while the primary database remains online running the current version.

Oracle GoldenGate

Any rolling or non-rolling update

Use Oracle GoldenGate to reduce database upgrade downtime for patch set and release updates. Downtime during a database upgrade is reduced by allowing the target database to be upgraded to the new version and kept synchronized while the source database remains online running the current version.

8.2.2.4 Operational Practices for Optimal Software Maintenance

The following operational practices enable optimal software maintenance:

  • Run Oracle EXAchk regularly — As a general health check tool to ensure an Exadata system continues to meet the current and constantly evolving best practices, run Oracle EXAchk monthly. The exachk report should be utilized as follows:

    • Baseline comparison - Compare the current report against an accepted baseline report using the report comparison feature (-diff option).

    • Critical issue exposure - Review the report for exposure to critical issues. Take prompt action to resolve critical issues reported by Oracle EXAchk.

    • Version recommendation - Review the report for version recommendation. The MAA Scorecard section evaluates current software versions for consistency, compatibility, and whether or not it is current.

  • Qualify maintenance readiness with Oracle EXAchk — Prior to performing software maintenance, run Oracle EXAchk to ensure the system is in a healthy state. Correct any FAIL or WARNING checks before updating any software. After software maintenance is complete, run Oracle EXAchk again to confirm system health.

  • Update software at regular intervals — All software should be updated regularly. Maintaining software at current or recent releases provides the following benefits: better software security, more stable sustaining releases, continued compatibility with newer related software, better support and faster resolution of issues, and ability to supply fixes for newly discovered issues.

  • Use the latest versions of the update utilities — Use the latest version of software update utilities Oracle EXAchk, patchmgr, and OPatch.

    Oracle EXAchk is updated regularly to contain new features, fixes, best practice health checks, and version recommendations. Patchmgr for database servers is updated regularly to contain new features, fixes, and workarounds for known database server update issues. OPatch is updated regularly to contain new features and fixes.

  • Perform storage server updates separately from RDMA Network Fabric switch updatesDo not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. RDMA Network Fabric switch firmware upgrade requires switch reboot, which disrupts some connections on the RDMA Network Fabric network.

  • Avoid unsupported system changesOracle Exadata Database Machine is an integrated system and engineered to be the best platform for running Oracle Database. Oracle Exadata Database Machine storage servers and RDMA Network Fabric switches contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. Software updates to Oracle Exadata Database Machine storage servers and RDMA Network Fabric switches are performed using patchmgr. Configuration or installed software may not be altered manually (without using patchmgr) in any way unless the Oracle Exadata Database Machine documentation contains steps to perform the desired change. While making an unsupported manual change may have the desired immediate effect, there are many potential negative consequences, such as:

    • Failed future patchmgr software updates
    • Inability to rescue the system
    • Inability to diagnose a software defect efficiently

    Contact Oracle Support Services for further guidance if a desired Oracle Exadata Database Machine storage server or RDMA Network Fabric switch change is not documented.

  • Minimize database server customizationOracle Exadata Database Machine is an integrated system and engineered to be the best platform for running Oracle Database. Oracle Exadata Database Machine database servers contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. Software updates to Oracle Exadata Database Machine database servers are performed using patchmgr. However, it may be necessary to manually install additional, site-specific software, such as monitoring agents or backup agents.

    It is supported to manually customize database servers, but note that customizing the operating system by adding or updating packages may require additional actions when applying a future Oracle Exadata System Software update with patchmgr (for example, removing customization prior to updating Oracle Exadata Database Machine servers and reapplying the customization after the update completes) because the additional software may add new dependencies which are not provided by a future Oracle Exadata System Software update. It is recommended to minimize database server customization. See the topic Updating Oracle Exadata Database Machine Database Servers for additional details.

  • Test supported configuration changes — Database server site-specific customization must be tested completely, including verifying database servers reboot properly after making configuration changes. Custom configuration changes that prevent successful system reboot will cause future Oracle Exadata System Software updates to fail.

8.2.2.5 Version Compatibility and Mixed Version Support

It is recommended that software for all components be updated regularly to stay in step with the recommended minimum release in order to maintain the most stable and secure system.

However, it is supported to update only a subset of components during a software maintenance window while the remaining components remain at an earlier version. For example, the following scenarios are supported:

  • Update a subset of Oracle Exadata Database Machine storage servers to a higher version of the Oracle Exadata System Software while the remaining storage servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata Database Machine storage servers to a higher version of the Oracle Exadata System Software while the database servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata Database Machine database servers to a higher version of the Oracle Exadata System Software while the storage servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata Database Machine storage servers to a higher version of the Oracle Exadata System Software while RDMA Network Fabric switches remain at the earlier Oracle Exadata System Software version.

  • Update RDMA Network Fabric switches to a higher version of the Oracle Exadata System Software while Oracle Exadata Database Machine storage servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata Database Machine storage servers to a higher version of the Oracle Exadata System Software while Oracle Grid Infrastructure or Oracle Database software remains at an earlier release, patch set, and quarterly patch level.

While mixed versions are supported (within the same component or across different components), it is highly recommended that this be only a temporary configuration that exists for the purpose and duration of rolling upgrade.

If mixing versions the following rules and considerations must be observed:

  • A specific generation of Oracle Exadata Database Machine hardware will have a minimum required Oracle Exadata System Software version. For example, Oracle Exadata Database Machine X6 hardware requires Oracle Exadata System Software release 12.1.2.3.1 or higher.

  • A specific Oracle Database release requires a minimum Oracle Exadata System Software release to fully support Oracle Exadata Database Machine features. For example, Oracle Database 12c Release 2 (12.2) requires Oracle Exadata System Software release 12.2.1.1.0 or higher on storage servers to support all Oracle Exadata Database Machine offload features.

  • Features supplied with a new Oracle Exadata System Software release may require a minimum Oracle Grid Infrastructure or Oracle Database software release, patch set, or patch. For example, the Oracle Exadata Database Machine feature Smart Scan Offload for Compressed Index Scan requires Oracle Database 12.2 and Oracle Exadata System Software 12.2 on storage servers.

  • Oracle Grid Infrastructure supports mixed version for limited duration for the purposes of rolling update only (for example, updating from release 12.1.0.2 to 12.2.0.1, or updating from release 12.1.0.2.161018 to 12.1.0.2.170117). Some Oracle Clusterware functionality is restricted while the cluster is in rolling upgrade mode.

  • Oracle Database supports mixed version for Proactive Bundle Patch for the purposes of rolling update only (for example, updating from release 12.1.0.2.161018 to 12.1.0.2.170117).

8.2.2.6 Ordering of Updates

In general, when performing software updates, the updates may be applied to components in any order based on business needs and maintenance window requirements.

The recommended order for applying software updates is:

  1. Oracle Grid Infrastructure and Oracle Database software

  2. Oracle Exadata Database Machine database server software

  3. Oracle Exadata Database Machine storage server software

  4. RDMA Network Fabric switch software

There are exceptions where specific update ordering is required, which are documented in the Oracle Exadata System Software supplemental README for the version being updated to. Some examples of required update ordering are:

  • Oracle Grid Infrastructure and Oracle Database 18c require Oracle Exadata System Software version 18.1.4 or higher. When updating a system with Oracle Exadata System Software release 12.2 and Oracle Grid Infrastructure 12c to Oracle Exadata System Software release 18 and Oracle Grid Infrastructure 18c, respectively, all Oracle Exadata Database Machine components must be updated first to Oracle Exadata System Software release 18.1.4 or higher before updating Oracle Grid Infrastructure and Oracle Database to release 18c.
  • When updating a system with Oracle ASM Cluster File System (Oracle ACFS) configured, the Oracle Grid Infrastructure home must contain the fix for bug 22810422 before updating the database server (non-Oracle VM or Oracle VM) to Oracle Exadata System Software release 12.2.

  • When updating an Oracle VM configuration from Oracle Exadata System Software release 12.1 to release 12.2, all user domains must be updated to Oracle Exadata System Software release 12.2 before updating the management domain to Oracle Exadata System Software release 12.2.

  • When updating an Oracle VM configuration, the Oracle Grid Infrastructure home in user domain must be running the Proactive Bundle Patch release 12.1.0.2.161018 (October 2016) or later before updating a user domain to Oracle Exadata System Software release 12.2.

In general, software update ordering requirements occur when updating one or more components to a later version that contains new features, for example, when updating Oracle Exadata System Software from release 12.1 to release 12.2. Refer to the Oracle Exadata System Software supplemental README for the software version being updated to for additional details.

8.2.3 Understanding the Exadata Software Image Version

The Exadata image version number contains the product version, a date code, and a build number.

The Exadata version installed on an Exadata storage server or an Exadata database server is determined with the imageinfo command. The image version of an Exadata release consists of three components. For example, if the command imageinfo -ver returns 12.1.2.1.1.150316.2, then:

  • The product version is 12.1.2.1.1

  • The date code is 150316

  • The Oracle-internal build number is 2. This component is not always used.

In most circumstances it is sufficient to reference a release by the product version only, for example, 12.1.2.1.1. However, when upgrading, you need to consider both the product version and date code of the installed and target releases.

Starting with the 12.2.0.2.0 release, the product version no longer uses the legacy nomenclature such as 12.2.0.2. Instead, the product version is a three field format consisting of: Year.Update.Revision, for example 18.1.0.

8.2.4 Rules for Updating to Newer Exadata Version

When upgrading, you need to consider both the product version and date code of the installed and target releases.

In most circumstances it is sufficient to reference a release by the product version only, for example, 12.1.2.1.1.

Upgrading to a specific target version must adhere to the following rules, which are enforced by patchmgr:

  1. The product version of the target release must be higher than the installed software

    and

  2. The date code of the target release must be higher than the installed software.

For example, consider an Exadata Database Machine that is currently running image version 12.1.1.1.2.150411, which consists of product version 12.1.1.1.2 and date code 150411.

  • Upgrading to 12.1.2.1.2.150617.1 is allowed because both rules are satisfied.

  • Upgrading to 12.1.2.1.1.150316.2 is not allowed because date code 150411 of the installed software is higher than date code 150316 of the target release.

8.3 About Upgrading to Oracle Linux 7 on Exadata Servers

When you upgrade to Oracle Exadata System Software release 19.1.0, the Oracle Linux version is upgraded from Oracle Linux 6 to Oracle Linux 7.

Oracle Exadata System Software release 19.1.0 provides support for Oracle Database 19c, which runs on Oracle Linux 7. When you upgrade to Oracle Exadata System Software release 19.1.0 the operating system is upgraded at the same time.

If you use quorum disks on your Exadata Database Machine, then refer to My Oracle Support note 2453054.1.

8.4 Overview of Performing Exadata Software Updates

For each software update of the Exadata components, there are various actions required to complete the software update.

8.4.1 Actions to Perform Before Any Software Maintenance

Perform you update any software on your Exadata Database Machine, you should perform these actions.

  • Identify the target release — The target release can be determined by one of the following:

    • Recommendation in My Oracle Support Document 888828.1.

    • Version recommendations in the MAA Scorecard section of the Oracle ExaCHK report.

  • Run Oracle ExaCHK — Ensure software maintenance readiness of the system by running the latest version of Oracle ExaCHK (see My Oracle Support Document  1070954.1).  Review the report for the following:

    1. Correct any FAIL or WARNING checks that deviate from your baseline.

    2. Review the MAA Scorecard section for version recommendations. Oracle ExaCHK evaluates current software versions for consistency, compatibility, and whether or not it is current.

    3. Review the MAA Scorecard section for exposure to critical issues.  Target version choice should resolve any critical issue exposure.

    It is recommended to run the latest version of Oracle ExaCHK in the following times:

    • Every month to maintain a system that continues to adhere to Exadata best practices

    • The week before the software update to ensure maintenance readiness

    • The day before the software update to ensure maintenance readiness

    • Immediately after completing the software update

  • Run prerequisite checks

    1. Download the target software release(s) and the latest versions of the software update utilities (patchmgr, OPatch, and so on) from My Oracle Support.

    2. Run the prerequisite checks and correct any issues. The initial prerequisite check run should be performed days or weeks in advance of the maintenance window to allow time to correct issues. Run prerequisite checks again immediately before the update.

      Note:

      Special consideration must be given to running database server prerequisite checks and the -nomodify_at_prereq flag.  See Updating Exadata Database Servers for details.
  • Review the latest documentation

    1. Review My Oracle Support Document 1270094.1 for recently published critical issues that are not yet automatically checked by Oracle ExaCHK.

    2. Review the Supplemental README in My Oracle Support for the target software version for known issues discovered after the software was released.

8.4.2 Overview of Performing Exadata Storage Server Updates

A high level look at the process for updating Exadata storage servers.

Patchmgr manages the update orchestration in a rolling or non-rolling manner for storage servers. When using the -rolling option, Oracle recommends that you use Oracle ASM high redundancy disk groups to tolerate the failure of a disk in another storage server during the update.

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support document 888828.1.

  2. Create a file that contains the list of storage servers to update. This file will be specified as the component_list_file.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all storage servers in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. If storage servers will be updated non-rolling, then stop Oracle Clusterware and all databases accessing the storage servers.

  6. Run patchmgr to update storage servers in a non-rolling (default) or rolling (-rolling option) manner.

  7. If storage servers were updated non-rolling, then restart Oracle Clusterware and all databases.

8.4.3 Overview of Performing Exadata Database Servers Updates

When updating Exadata database servers, there are different steps for virtualized environments.

Note:

Patchmgr manages the update orchestration in a rolling or non-rolling manner for Exadata database servers. When using the -rolling option, Oracle recommends that you use Oracle RAC features for client high availability to minimize the impact to client applications connected to database instances that will be stopped during maintenance.
8.4.3.1 Steps for Updating Non-Virtualized Configurations

An overview of the steps involved in updating an Exadata database server that does not have Oracle VM configured.

  1. Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  2. Download the target Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support Document 888828.1.

  3. Create the component_list_file containing the list of database servers to update.

  4. Configure SSH equivalence from the user that will run patchmgr to root on all database servers in the component_list_file.

  5. Run patchmgr backup to backup database server root file system and operating system.

  6. Run patchmgr prerequisite check and correct any issues.

  7. Run patchmgr to update database servers in a non-rolling (default) or rolling (-rolling option) manner.

8.4.3.2 Steps for Updating Virtualized Configurations

The management domains (dom0s) and user domains (domUs) are updated independently using separate patchmgr executions.

Updating the Management domain (dom0)

  • Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  • Download the target dom0 Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support Patch Document 888828.1.

  • Create the component_list_file containing the list of database server dom0s to update.

  • Configure SSH equivalence from the user that will run patchmgr to root on all database server dom0s in the component_list_file.

  • Run patchmgr backup to backup database server dom0 root file system and operating system.

  • Run patchmgr prerequisite check and correct any issues.

  • Run patchmgr to update database server dom0s in a non-rolling (default) or rolling (-rolling option) manner.

Updating the User domains (domU)

  1. Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  2. Download the target domU Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support document 888828.1.

  3. Create the component_list_file containing the list of database server domUs to update.

  4. Configure SSH equivalence from the user that will run patchmgr to root on all database server domUs in the component_list_file.

  5. Run patchmgr backup to backup database server domU root file system and operating system.

  6. Run patchmgr prerequisite check and correct any issues.

  7. Run patchmgr to update database server domUs in a non-rolling (default) or rolling (-rolling option) manner.

8.4.4 Overview of Performing RoCE Network Fabric Switch Updates

Patchmgr manages the update orchestration in a rolling manner for RoCE Network Fabric switches

Note:

Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. RDMA Network Fabric switch firmware upgrade requires switch reboot, which disrupts some connections on the RDMA Network Fabric network.

The general steps for using patchmgr are:

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system. See My Oracle Support document 888828.1.

  2. Create the component_list_file containing the list of RoCE Network Fabric switches to update.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all RoCE Network Fabric switches in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. Run patchmgr to update RoCE Network Fabric switches. RoCE Network Fabric switches are always updated in a rolling manner.

8.4.5 Overview of Performing InfiniBand Network Fabric Switch Updates

Patchmgr manages the update orchestration in a rolling manner for InfiniBand Network Fabric switches

Note:

Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. RDMA Network Fabric switch firmware upgrade requires switch reboot, which disrupts some connections on the RDMA Network Fabric network.

The general steps for using patchmgr are:

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system. See My Oracle Support document 888828.1.

  2. Create the component_list_file containing the list of InfiniBand Network Fabric switches to update.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all InfiniBand Network Fabric switches in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. Run patchmgr to update InfiniBand Network Fabric switches. InfiniBand Network Fabric switches are always updated in a rolling manner.

8.4.6 Overview of Performing Oracle Grid Infrastructure and Oracle Database Updates

The steps for updating Oracle Grid Infrastructure or Oracle Database software depend on the type of software update you are performing.

Updating to a New Proactive Bundle Patch

Updates to a new Proactive Bundle Patch may be performed in-place or out-of-place. Both in-place and out-of-place methods support rolling and non-rolling updates.

  • In-place

    The update is applied to the current software home using OPatch while the Oracle Grid Infrastructure or Oracle Database software is shutdown on the node being updated.

    This is the default method of applying a Proactive Bundle Patch update for Oracle Grid Infrastructure or Oracle Database software, as described in the Proactive Bundle Patch README. The steps to apply the Proactive Bundle Patch must be performed on each node.

  • Out-of-place (recommended)

    A new software home is prepared and updated while the Oracle Grid Infrastructure and Oracle Database software remains running. Once the new homes are prepared, Oracle Grid Infrastructure and Oracle Database are quickly stopped, switched to the new home, and restarted.

    Out-of-place has significant advantage over in-place updates:

    • There is less risk because the new home can be prepared while the Oracle Grid Infrastructure and Oracle Database software remains online.
    • There is less downtime because switching to the new software home is faster than applying an update in-place.
    • Rollback is faster because it is possible to simply switch back to the original software home.

    The recommended way to adopt out-of-place updates is to use Fleet Patching and Provisioning (previously called Rapid Home Provisioning), which provides the following advantages:

    • Distributes software updates to all nodes in the cluster.
    • Orchestrates updates across the cluster in a rolling or non-rolling manner with a single command.
    • Provides control over database service relocation to maintain application availability.

    Out-of-place software updates are also supported by Oracle Enterprise Manager Cloud Control.

    Out-of-place software updates without Fleet Patching and Provisioning or Oracle Enterprise Manager Cloud Control may be accomplished by following My Oracle Support Document 2087150.1.

OJVM Patch Set Update

The OJVM patch set update (PSU) is a separate software update for database homes that addresses OJVM security vulnerabilities. It is installed separately from the standard Proactive Bundle Patch. The OJVM PSU patch may be installed in a rolling manner under certain situations.

Updating to a New Patch Set Release, Major Release, or Maintenance Release

To update to a higher patch set, maintenance, or major release for grid infrastructure or database, follow the step-by-step instructions in My Oracle Support.

8.4.7 Using sudo When Performing Software Updates

You can use sudo when running dbnodeupdate.sh and patchmgr (and dbnodeupdate Orchestration).

8.4.7.1 Running patchmgr (and dbnodeupdate Orchestration) Using sudo

You can run patchmgr (which is packaged in dbserver.patch.zip) using sudo to perform any of patchmgr's functionalities, such as patching cells, patching RDMA Network Fabric switches, or orchestrating dbnodeupdate.sh execution.

Patchmgr is packaged in dbserver.patch.zip. Perform the following steps to set up the /etc/sudoers file for running patchmgr using sudo:

  1. Log in as the root user and edit /etc/sudoers using visudo.
    # visudo
  2. Add the following entry to the bottom of the sudoers file to allow non-root users, such as the oracle user, to run patchmgr as root.

    Note:

    The first field in the line specifies the non-root user who is granted sudo access for the patchmgr command. The line below uses the oracle user as an example. You can specify a different user if necessary.
    oracle  ALL=(ALL)    NOPASSWD:SETENV: /u01/stage/patch/dbserverpatch/patchmgr
    
  3. As root, create the /u01/stage/patch/dbserverpatch directory and unzip dbserver.patch.zip:
    # mkdir -p /u01/stage/patch/dbserverpatch/
    # cp dbserver.patch.zip /u01/stage/patch/dbserverpatch/
    # cd /u01/stage/patch/dbserverpatch/
    # unzip dbserver.patch.zip
    
  4. Move everything under the /u01/stage/patch/dbserverpatch/dbserver_patch_x.yymmdd directory to /u01/stage/patch/dbserverpatch/.
    # mv /u01/stage/patch/dbserverpatch/dbserver_patch_x.yymmdd/* /u01/stage/patch/dbserverpatch/
    

Note:

  • Patchmgr expects root SSH equivalence on all database nodes that will be updated, even when run using sudo.
  • The above setup requires that the entire contents of /u01/stage/patch/dbserverpatch be owned by root.
  • If you update dbserver.patch.zip, then you must place the new version in the same location as specified by sudoers.

To verify that the setup is correct, run patchmgr in prereq check mode as the oracle user:

[oracle]$ cd /u01/stage/patch/dbserverpatch/

[oracle]$ sudo ./patchmgr -dbnodes dbgroup -precheck     \
  -yum_repo http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3/base/x86_64/  \
  -target_version 12.1.2.1.3.151021
8.4.7.2 Running dbnodeupdate.sh Using sudo

Before using dbnodeupdate.sh using sudo, you configure the /etc/sudoers file.

  1. Log in as the root user and edit /etc/sudoers using visudo.
    # visudo
  2. Add the following entry (all on one line) to the bottom of the sudoers file to allow non-root users, such as the oracle user, to run dbnodeupdate.sh as root:

    Note:

    The first field in the line specifies the non-root user who is granted sudo access for the dbnodeupdate.sh command. The line below uses the oracle user as an example. You can specify a different user if necessary.
    oracle  ALL=(ALL)    NOPASSWD:SETENV: /u01/stage/patch/dbnodeupdate/dbnodeupdate.sh
    
  3. As root, create the /u01/stage/patch/dbnodeupdate directory and unzip dbnodeupdate.zip:
    # mkdir -p /u01/stage/patch/dbnodeupdate
    # cp dbnodeupdate.zip /u01/stage/patch/dbnodeupdate
    # cd /u01/stage/patch/dbnodeupdate
    # unzip dbnodeupdate.zip
    

To verify that the setup is correct, run dbnodeupdate.sh in prereq check mode as the oracle user:

[oracle]$ cd /u01/stage/patch/dbnodeupdate

[oracle]$ sudo ./dbnodeupdate.sh -u -l http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3/base/x86_64/ -v

dbnodeupdate exits if it is run without root privileges.

Note:

  • The above setup requires that everything in /u01/stage/patch/dbnodeupdate be owned by the root user.
  • If you update the dbnodeupdate utility, then you must place the new version in the same location as specified by sudoers.

8.5 Exadata Patchmgr Update Utility

Patchmgr is the utility used to update software for Exadata infrastructure components.

8.5.1 About the Exadata Patchmgr Update Utility

Patchmgr is designed to simplify the software update process.

Patchmgr has the following capabilities:

  • With a single invocation updates all Oracle Exadata Database Machine storage servers, database servers, or RDMA Network Fabric switches.

  • Orchestrates the software update across components one at a time when performing a rolling update.

  • Parallelizes the software update to all components at the same time when performing a non-rolling update.

  • Updates all component software, as required, such as firmware, operating system, and Oracle Exadata System Software.

    • Database servers contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. However, it may be necessary to manually install additional, site-specific software, such as monitoring agents or backup agents. It is supported to manually customize database servers, but note that customizing the operating system by adding or updating packages may require additional actions when applying a future Oracle Exadata System Software update with patchmgr.

    • Storage server or RDMA Network Fabric switch configuration or installed software may not be altered manually (update without using patchmgr) in any way unless Oracle Exadata Database Machine documentation contains steps to perform the desired change.

  • Patchmgr may be run from an Engineered System running Oracle Linux, as the root user or a non-root user.

  • Multiple invocations of patchmgr may be run from the same software directory.

When updating database servers, patchmgr manages the following as needed:

  • Stops and starts databases and clusterware
  • Stops and starts user domains (domUs)
  • Stops and starts Oracle Enterprise Manager Cloud Control agents
  • Unmounts remote network mounts
  • Performs a root file system operating system backup that can be used for rollback
  • Relinks database home and Oracle Grid Infrastructure home binaries
  • Applies updated best practices configuration changes and workarounds for known issues

8.5.2 Obtaining Patchmgr

You can download the patchmgr utility from My Oracle Support.

Prior to Oracle Exadata System Software release 19.3.0, the storage server and RDMA Network Fabric switch updates use the version of patchmgr that is bundled with the Oracle Exadata System Software release to which you are updating. Starting with Oracle Exadata System Software release 19.3.0, there is a separate download for patchmgr for the RDMA Network Fabric switches and the storage servers, for example:

  • 19.3.0.0.0.190910.switch.patch.zip for both InfiniBand Transport Layer systems based on an InfiniBand Network Layer and RoCE Network Fabric switches.
  • 19.3.0.0.0.190910.patch.zip for cells

Patchmgr for database server is packaged in dbserver.patch.zip and is available as a separate download from My Oracle Support note 1553103.1 or patch 21634633.

It is recommended to always use the latest patchmgr from My Oracle Support when updating Oracle Exadata Database Machine database servers. The patchmgr utility is updated frequently to address known issues and best practices, and to support new hardware.

8.5.3 Patchmgr Syntax

Patchmgr is a utility used to update software for Oracle Exadata Database Machine infrastructure components.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata Database Machine database server or a non-Oracle Exadata Database Machine system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata Database Machine systems. If patchmgr is run from an Oracle Exadata Database Machine database server, then that database server cannot be in the group file supplied to patchmgr.

Syntax

Patchmgr syntax differs depending on the component to be updated, however, the general patchmgr syntax is:

./patchmgr -component component_list_file -action required_arguments [optional_arguments]

Options

Option Description
component Represents the component to update. Valid values are -cells, -dbnodes, -ibswitches, or -roceswitches
component_list_file A text file containing the host names of components to update. For example, if updating Oracle Exadata Database Machine storage servers (using -cells), then component_list_file will contain the list of all host names of the storage servers.
action The action to perform. In most cases the action is specific to the component being updated.
required_arguments The additional arguments required for certain actions.
optional_arguments Any optional arguments required for certain actions.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.
  • Multiple invocations of patchmgr may be run concurrently from the same software directory by using the -log_dir option. This allows patchmgr to update multiple Oracle Exadata Database Machine systems concurrently from the same software directory.

  • Patchmgr may be run as the root user or as a non-root user. The user running patchmgr must have root-level SSH equivalence configured to the servers or switches that patchmgr will update. To run patchmgr as a non-root user, the -log_dir option must be used.

  • Prior releases used the dbnodeupdate.sh utility to update database servers. dbnodeupdate.sh has been integrated with and is replaced by patchmgr.

8.5.3.1 Patchmgr Syntax for Storage Servers

You can use patchmgr to update software for Oracle Exadata Database Machine storage servers.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata Database Machine database server or a non-Oracle Exadata Database Machine system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata Database Machine systems

Patchmgr Syntax for Storage Servers

./patchmgr --cells cell_host_file [--patch_check_prereq |--rollback_check_prereq [--rolling] [--unkey] [ --ignore_alerts] 
 [--ignore_date_validations]] [--patch | --rollback [--rolling] [--unkey] [--ignore_alerts]] [--cleanup [--unkey]]

Main Arguments

Argument Description
--cells cell_host_file The cell_host_file is a text file containing the host names of the storage servers to update. The file contains one storage server host name or IP address per line. The storage server patching will fail if the list file is not specified.
--patch Apply the patch, including firmware updates, wherever possible (BIOS, Disk Controller and if possible disk drives) to all storage servers in the storage server list file.
--rollback Rolls back the patch.
--cleanup Cleans up all patch files and temporary content on all storage servers. Before cleaning up, collects logs and information for problem diagnostics and analysis. Cleaning up patch files can be done manually if patch fails by removing the directory /root/_cellupd_dpullec_ on each storage server.

Supported Options

The following options are supported for storage server patching and rollback:

Table 8-2 Patchmgr Options for Storage Servers

Option Description
--get property Get information about specific property. Property can be: log_dir - assigned directory for all log files.
--ignore_alerts Ignore any active hardware alerts on the Exadata storage server and proceed with the patching.
--ignore_date_validations Ignore date version validation. This allows upgrades to later releases with an earlier date stamp, for example 19.1.2.0.0.190306 -> 19.2.0.0.0.1.

--log_dir {log_directory| auto}

Specifies the absolute path to the log directory, or you can specify auto to use a log directory that is based on the launch directory and the content of the nodes list file.

Specifying the -log_dir option enables you to run multiple patch manager invocations and also to run patch manager as a non-root user.

--patch_check_prereq Runs prerequisite check on all the storage servers to determine if the patch can be applied to the storage servers.
--rollback_check_prereq Runs prerequisite check on all the storage servers to determine if the storage servers can be rolled back for this specific patch.

--rolling

Specifies that the update is to be done in a rolling fashion. If not specified, the update is done in a non-rolling fashion.

Environment variable EXA_PATCH_ACTIVATE_TIMEOUT_SECONDS controls the timeout value waiting for the grid disks to be activated. The default is set to 36000 (10 hours).

Note: Prerequisite checks and cleanups are always done in a non-rolling fashion, even if the -rolling option is specified.

--smtp_from "email_addr"

Specifies the from email address for the patchmgr notification.

--smtp_to "email_addr1 email_addr2 email_addr3 ..."

Specifies the to email addresses for the patchmgr notification.

--smtp_set_envelope_sender

Specifies that the same from address in Return-Path: mail header should be used.

--unkey Removes passwordless SSH access to the cells before exit.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.
  • Multiple invocations of patchmgr may be run concurrently from the same software directory by using the --log_dir option. This allows patchmgr to update multiple Oracle Exadata Database Machine systems concurrently from the same software directory.
  • Patchmgr may be run as the root user or as a non-root user. The user running patchmgr must have root-level SSH equivalence configured to the servers or switches that patchmgr will update. To run patchmgr as a non-root user, the -log_dir option must be used.

Example 8-4 Run storage server update pre-requisite checks, then update storage servers

./patchmgr -cells cell_group -patch_check_prereq 
./patchmgr -cells cell_group -patch

Example 8-5 Update storage servers in a rolling manner

The following command updates storage servers in a rolling manner, receive email notifications as the update proceeds, and remove passwordless SSH access to cells after the update is complete.

./patchmgr -cells cell_group -patch -rolling -unkey -smtp_from "dbm01@example.com" -smtp_to "admin1@example.com,admin2@example.com"

Example 8-6 Update multiple storage servers at one time

The following commands update storage servers on three Oracle Exadata Database Machine systems simultaneously from the same software directory; one update using a rolling update, the others updates are non-rolling.

In this example:

  • Each patchmgr command must be run from a separate terminal window.
  • Each patchmgr execution uses a unique logging directory name automatically generated based on the content of the component_list_file.
(terminal1) ./patchmgr -cells cell_group_exa01 -patch -rolling -log_dir auto 
(terminal2) ./patchmgr -cells cell_group_exa02 -patch -log_dir auto 
(terminal3) ./patchmgr -cells cell_group_exa03 -patch -log_dir auto

To have a subsequent patchmgr execution use an altered component_list_file with different content, yet use the same logging directory as a prior patchmgr execution, use the -get log_dir option to obtain the logging directory. For example:

  1. The logging directory for the initial patchmgr execution is generated automatically.

    ./patchmgr -cells cell_group -patch -log_dir auto
  2. Assume the last cell failed to update and patchmgr will be re-run for the last cell only, using the same logging directory as the initial patchmgr execution. Use the -get log_dir option to obtain the logging directory using the original component_list_file.

    ./patchmgr -cells cell_group -patch -log_dir auto -get log_dir log_dir=/tmp/patch_12.2.1.1.1.170419/log/dm01cel01_dm01cel02_cacce4ee
  3. Update the cell_group file to contain only the last cell, or use a different file that contains only the last cell. Specify the logging directory from the initial patchmgr execution so all logs for this group of cells are created in the same logging directory.

    ./patchmgr -cells cell_group -patch -log_dir /tmp/patch_12.2.1.1.1.170419/log/dm01cel01_dm01cel02_cacce4ee
8.5.3.2 Patchmgr Syntax for Database Servers

You can use patchmgr to update software for Oracle Exadata Database Machine database servers.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata Database Machine database server or a non-Oracle Exadata Database Machine system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata Database Machine systems. If patchmgr is run from an Oracle Exadata Database Machine database server, then that database server cannot be in the file that lists the servers to patch.

Patchmgr Syntax for Database Servers

./patchmgr --dbnodes database_node_file {
  --backup [--rolling] [--unkey] [--key_api key-api.sh] |
  --precheck {--yum_repo base_URL | --iso_repo zipped_iso_file} [--target_version version] 
     [--unkey] [--key_api key-api.sh] |
  --upgrade {--yum_repo base_URL | --iso_repo zipped_iso_file} [--target_version version] 
     [--rolling] [--unkey] [--key_api key-api.sh] | 
  --complete [--target_version version] [--unkey] [--key_api key-api.sh] |
  --rollback [--rolling] [--unkey] [--key_api key-api.sh] |
            --cleanup  [--unkey] [--key_api key-api.sh] }

Main Arguments

Argument Description
--dbnodes database_node_file The database_node_file is a text file containing the host names of the database servers to update. The file contains one database server host name or IP address per line. The database server patching will fail if the list file is not specified.
--backup Perform backup for the database nodes specified in the host list. Runs backup in non-rolling fashion unless the --rolling option is specified.
--precheck Runs the pre-upgrade validation checks in the database nodes specified in the host list in non-rolling fashion.
--upgrade Upgrades the database nodes specified in the host list. Runs upgrade in non-rolling fashion unless the --rolling option is specified.
--complete Runs 'completion-steps' only. In normal cases there is no need to run this separately as this already runs as part of --upgrade. Note: If the database stack or user domains are up, they will be shut down and restarted.
--rollback Rollback the database nodes specified in the host list. Runs rollback in non-rolling fashion unless the --rolling option is specified.
--cleanup Cleans up all temporary content on the database servers specified in the host list in non-rolling fashion.

Supported Options

The following options are supported for storage server patching and rollback:

Table 8-3 Patchmgr Options for Database Servers

Option Description

--allow_active_network_mounts

Allows dbnodeupdate to run with active NFS or SMB mounts.

This is equivalent to the dbnodeupdate.sh -A command.

--allow_non_signed_repo

Allow dbnodeupdate to run with a non-signed repository. (Oracle Exadata System Software release 19.2 and later)

--dbnode_patch_base

User preferred location on target database servers where patch iso image and dbnodeupdate archive files are to be unzipped.

Note: Provided location must be an absolute path of local file system and it should have sufficient amount of free space and inodes.

--force_remove_custom_rpms

Remove any custom RPMs when database server upgrades from Oracle Linux 5 to Oracle Linux 6.

--ignore_alerts Ignore any active hardware alerts on the Exadata server and proceed with the patching.

--iso_repo repo_location

The path to a zipped ISO file. It is recommended that the file be in the same directory as the dbnodeupdate.zip file.

This option or the -yum_repo option must be specified for -backup, -precheck and -upgrade actions.

This option cannot be used with the -yum_repo option.

--log_dir {log_directory| auto}

The absolute path to the log directory, or you can specify auto to use a log directory that is based on the directory you started patchmgr from and the content of the nodes list file.

Specifying the --log_dir option enables you to run multiple patch manager invocations and also to run patch manager as a non-root user.

--modify_at_prereq

RPM changes at prerequisite check.

--no_connection_draining

Disables database connection draining for Fleet Patching and Provisioning, formerly known as Rapid Home Provisioning (RHP). The connection draining availability depends on the Oracle Grid Infrastructure release. This option is applicable to only rolling updates.

--nomodify_at_prereq

No RPM changes at prerequisite check. This is the default behavior.

This is equivalent to the dbnodeupdate.sh -N command.

--nobackup

Do not backup the database servers before an upgrade.

--rolling

Specifies that the update is to be done in a rolling fashion, one server at a time. If not specified, the update is done in a non-rolling fashion.

Environment variable EXA_PATCH_ACTIVATE_TIMEOUT_SECONDS controls the timeout value waiting for the grid disks to be activated. The default is set to 36000 (10 hours).

Note: Prerequisite checks and cleanups are always done in a non-rolling fashion, even if the -rolling option is specified.

--rolling_backups

Directs patchmgr to backup each node prior to updating that node in a rolling manner. If this option is not specified (default), then patchmgr completes the backups of all nodes in parallel before updating the first node.

This option can only be used in conjunction with the --rolling option. Otherwise, this option is ignored and the default backup method is used.

This option is ignored if --nobackup is included in the command.

--skip_gi_db_validation

Skip certification of Oracle Grid Infrastructure and Oracle Database home compatibility with Oracle Linux 7.

--smtp_from "email_addr"

Specifies the from email address for the patchmgr notification.

--smtp_to "email_addr1 email_addr2 email_addr3 ..."

Specifies the to email addresses for the patchmgr notification.

--smtp_set_envelope_sender

Specifies that the same from address in Return-Path: mail header should be used.

--target_version version The patch version, for example 12.1.2.3.0.160207.3.
--unkey Removes passwordless SSH access to the cells before exit.
--yum_repo base_URL Base URL for the Exadata update repository. This option must be specified for --backup, --precheck and --upgrade actions if the --iso_repo option is not specified. This option cannot be combined with the --iso_repo option.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.
  • Prior releases used the dbnodeupdate.sh utility to update database servers. dbnodeupdate.sh has been integrated with and is replaced by patchmgr.

Examples

Example 8-7 Backup database servers then perform the update

The following commands backup the Oracle Exadata Database Machine database servers, run the prerequisite checks for the database server update, and then update the database servers in a rolling manner.

./patchmgr -dbnodes dbnode_group -backup -iso_repo /tmp/p25640941_122111_Linux-x86-64.zip 
-target_version 12.2.1.1.1.170419 

./patchmgr -dbnodes dbnode_group -precheck -iso_repo /tmp/p25640941_122111_Linux-x86-64.zip 
-target_version 12.2.1.1.1.170419 

./patchmgr -dbnodes dbnode_group -upgrade -nobackup -iso_repo /tmp/p25640941_122111_Linux-x86-64.zip 
-target_version 12.2.1.1.1.170419 -rolling
8.5.3.3 Patchmgr Syntax for RoCE Network Fabric Switches

You can use patchmgr to update software for RoCE Network Fabric switches.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata Database Machine database server or a non-Oracle Exadata Database Machine system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata Database Machine systems.

Note:

Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.

Patchmgr Syntax for RoCE Network Fabric Switches

./patchmgr --roceswitches [roceswitch_list_file] {
  --upgrade [--verify-config [yes|no] [ --newswitchlist new_list_file ]] [--roceswitch-precheck] [--force]  |
  --downgrade [--verify-config [yes|no] [ --newswitchlist new_list_file ]] [--roceswitch-precheck] [--force] | 
  --verify-config [yes|no] [ --newswitchlist new_list_file ] |
  --apply-config [--force]} [ -log_dir {absolute_path_to_log_directory | AUTO}]

Main Arguments

Argument Description
--roceswitches [roceswitch_list_file]

Specifies that patchmgr is acting on the RoCE Network Fabric switches.

If specified, the switch list file identifies the RoCE Network Fabric switches. The file has one switch host name or IP address on each line.

If no file name is provided, then the command acts on all RoCE Network Fabric switches discovered from the host that is running patchmgr.

--upgrade Upgrade the firmware on the RoCE Network Fabric switches to $EXADATA_IMAGE_ROCESWITCH_UPGRADE_VERSION.
--downgrade Downgrade the firmware on the RoCE Network Fabric switches to $EXADATA_IMAGE_ROCESWITCH_DOWNGRADE_VERSION.
--verify-config [yes|no] [ --newswitchlist new_list_file ]

Verify each switch configuration against the golden configuration. The default value is yes. If you specify no, then verification is skipped when using --upgrade or --downgrade.

Verification performed by default when using --upgrade or --downgrade. You can also use this option to perform verification as a standalone operation.

If specified, the --newswitchlist option generates a new switch list file with entries that match the current configuration of each switch.

--apply-config

Applies the golden configuration template to each switch.

This option relies on tags in the switch list file, which specify the configuration type for each switch. The following tags are supported:

  • leaf - Identifies a leaf switch in a single rack system. This configuration type is assumed if no tag is specified.
  • mleaf - Identifies a leaf switch in a multi-rack system.
  • sfleaf - Identifies a leaf switch in a single rack system that is enabled to support Exadata Secure RDMA Fabric Isolation.
  • msfleaf - Identifies a leaf switch in a multi-rack system that is enabled to support Exadata Secure RDMA Fabric Isolation.
  • mspine - Identifies a spine switch. Note that one spine switch configuration supports all spine switches on single and multi-rack systems, with and without Exadata Secure RDMA Fabric Isolation.
  • leaf23 - Identifies a leaf switch in a single rack system that is configured with 23 host ports. This configuration is required only for X8M-8 systems with 3 database servers and 11 storage servers.
  • mleaf23 - Identifies a leaf switch in a multi-rack system that is configured with 23 host ports. This configuration is required only for X8M-8 systems with 3 database servers and 11 storage servers.

To specify the configuration type for each switch, append a colon (:) and tag to each switch host name or IP address in the switch list file. For example:

switch123-rocea0:sfleaf          
switch123-roceb0:sfleaf

Supported Options

The following options are supported for RoCE Network Fabric switch configuration and firmware update:

Table 8-4 Patchmgr Options for RoCE Network Fabric Switches

Option Description
--roceswitch_precheck Performs switch firmware upgrade or downgrade simulation on the RoCE Network Fabric switches in the list file but does not perform the actual install. Use this option with --upgrade or --downgrade.
--force

In conjunction with --upgrade or --downgrade, this option proceeds with the upgrade or downgrade even if the switch is already on target firmware version or the RoCE Network Fabric switch is experiencing non-critical failures.

In conjunction with --apply-config, this option bypasses the check that determines if the current switch configuration matches the configuration type specified in the switch list file.

-log_dir {absolute_path_to_log_directory | AUTO}

When running patchmgr as a non-root user, use -log_dir to specify the absolute path to the log directory or use the keyword AUTO. If you specify AUTO, then patchmgr generates and sets the path to the log directory based on the directory patchmgr is launched from and the content of the nodes list file.

Note:

Specifying -log_dir enables multiple patch manager invocations and is required when running patch manager as a non-root user.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.

Example 8-8 Using patchmgr to Upgrade Firmware on RoCE Network Fabric Switches

This example runs the upgrade prerequisite checks on all detected switches, then upgrades the switches.

$ ./patchmgr --roceswitches --upgrade --roceswitch_precheck

$ ./patchmgr --roceswitches --upgrade

Example 8-9 Using patchmgr to Apply Golden Configurations to RoCE Network Fabric Switches

This example applies golden configuration settings to RoCE Network Fabric switches as specified in the switches.lst file. Logs for the operation are written to /tmp/switchlogs.

$ cat switches.lst
switch456-rocea0:leaf          
switch456-roceb0:leaf          

$ ./patchmgr --roceswitches switches.lst --apply-config –log_dir /tmp/switchlogs
8.5.3.4 Patchmgr Syntax for InfiniBand Network Fabric Switches

You can use patchmgr to update software for InfiniBand Network Fabric switches.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata Database Machine database server or a non-Oracle Exadata Database Machine system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata Database Machine systems.

Patchmgr Syntax for InfiniBand Network Fabric Switches

./patchmgr --ibswitches [ibswitch_list_file] {--upgrade | --downgrade} [--ibswitch_precheck] 
  [--unkey] [--force [yes|no]]

Main Arguments

Argument Description
--ibswitches ibswitch_list_file Specifies the name of the InfiniBand Network Fabric switch list file. The file has one switch host name or IP address per line. If no file name is provided, then it runs the command on all InfiniBand Network Fabric switches discovered from this host.
--upgrade Upgrade the InfiniBand Network Fabric switches in the list file to $EXADATA_IMAGE_IBSWITCH_UPGRADE_VERSION.
--downgrade Downgrade the InfiniBand Network Fabric switches in the list file to $EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION.

Supported Options

The following options are supported for InfiniBand Network Fabric switch configuration and firmware update:

Table 8-5 Patchmgr Options for InfiniBand Network Fabric Switches

Option Description
--force [yes | no] Specifies to proceed with the upgrade or downgrade even on non-critical failures. The value yes specifies to proceed by auto-answering prompts with yes. The value no specifies to proceed by auto-answering prompts with no. The default value is yes.
--ibswitch_precheck Runs the pre-update validation checks on the InfiniBand Network Fabric switches in the list file.

--smtp_from "email_addr"

Specifies the from email address for the patchmgr notification.

--smtp_to "email_addr1 email_addr2 email_addr3 ..."

Specifies the to email addresses for the patchmgr notification.

--smtp_set_envelope_sender

Specifies that the same from address in Return-Path: mail header should be used.

--unkey Removes passwordless SSH access to the InfiniBand Network Fabric switches before exit.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.

Example 8-10 Using patchmgr for InfiniBand Network Fabric Switches

This example runs the update prerequisite checks on all switches, then upgrades the switches specified in the ib_group file.

./patchmgr --ibswitches --ibswitch_precheck 

./patchmgr --ibswitches ib_group --upgrade

8.6 Updating Oracle Exadata Database Machine Database Servers

Use the following information and procedures when updating database servers within Oracle Exadata Database Machine.

8.6.1 Overview of Oracle Exadata Database Machine Database Server Updates

When updating database servers, there is more than one software that needs to be updated.

Oracle Exadata Database Machine database server release updates contain updates for the following components within a database server:

  • Oracle Linux operating system
  • Firmware (Disk, RAID controller, ILOM, HCA)
  • Oracle Exadata System Software

The software and firmware components that are updated for a specific release depend on the current Oracle Exadata System Software release the database server is running and the release you are updating to. Oracle Linux operating system packages and Oracle Exadata System Software are always updated while firmware may be updated for only a small selection of the components or not at all.

Updates for Oracle Exadata Database Machine database servers can be applied independently from the storage servers or RDMA Network Fabric switches unless otherwise specified in My Oracle Support Note 888828.1.

Updating database servers is always performed in-place. This means the active operating system is updated. The actual update is performed using YUM, but the command to do this is wrapped in an Oracle Exadata Database Machine utility called patchmgr.

The YUM command is wrapped in the update utility to maintain strict ordering of validation and preparation steps during the update process. Also by using a utility Oracle can enforce application of fixes for known issues and best practices.

Oracle Clusterware processes and Oracle Real Application Clusters (Oracle RAC) database instances must not be running on a database server that is being updated. To reduce application-level impact, follow the client failover best practices described in the "Client Failover Best Practices for Highly Available Oracle Databases" white paper.

If you cannot afford cluster-wide downtime, you can update database servers in a rolling fashion. This means updating one database server at a time. If you can afford cluster-wide downtime, you can update all database servers in parallel. Non-rolling updates reduce the overall time required to complete the upgrade at the expense of having a full database outage.

8.6.2 Update Paths for Exadata Database Servers

To update Exadata database servers running Oracle Linux 5.5 or later and Oracle Exadata software release 11.2.2.4.2 or later, you must use the “update utility” for applying Exadata software updates.

To update Exadata database servers running Exadata software releases older than 11.2.2.4.2, you should first update the servers to Oracle Linux 5.5 and Exadata Software release 11.2.2.4.2 or later before you can use the update utility. See the table below for details.

Table 8-6 Update Paths

Currently Installed Oracle Exadata Software Release Currently Installed Oracle Linux Release Recommended Action

Release 11.2.2.4.2 or later

Release 5.5. or later

Update to the target release using the update utility.

Releases earlier than 11.2.2.4.2

Release 5.5. or later

  1. Use patch 13513611 to update the database servers to release 11.2.2.4.2. Refer to the steps in section 8 of the patch 13513611 README.

  2. Update to a later release using the update utility.

Releases earlier than 11.2.2.4.2

Releases earlier than 5.5

  1. Update to release 11.2.3.2.0. Refer to My Oracle Support note 1284070.1 for details.

  2. Update to a later release using the Update utility.

8.6.3 Installing, Updating, and Managing Non-Oracle Software

Installing and updating non-Oracle branded RPMs on Oracle Exadata Database Machine database servers are allowed as long as kernel and RDMA Network Fabric packages remain untouched.

Note that customizing the operating system by adding or updating packages may introduce problems when applying an Oracle Exadata System Software update because the non-Oracle software may add new dependencies which will not be provided by the Oracle Exadata System Software update. For this reason it is recommended to stay close to the Oracle Exadata Database Machine image and customize as little as possible.

Note:

Oracle Exadata Database Machine does not ship with 32-bit software. This means any customization that introduces 32-bit RPMs on the system will break the Oracle Exadata System Software update and will need to be removed.

In case you have added or updated packages supplied by the Oracle Exadata Database Machine image, you can remove the exadata-sun-computenode-exact RPM when needed as follows:

[root@dm01 ]# rpm -e exadata-sun-computenode-exact

If you have installed customized packages, it is recommended that you have scripts to automate the removal (to run before updating Oracle Exadata System Software) and installation (to run after the Oracle Exadata System Software update) of those packages. After the Oracle Exadata System Software update, verify that the customized packages are still compatible and are still needed, before re-installing them.

Note:

Packages should not be forced in using the rpm -Uvh --nodeps command, unless directed by Oracle Support Services.

The same principle applies for customizations other than installation of additional software. Oracle recommends customizing the operating system as little as possible. Be aware that while changes on the database server are allowed within the earlier described boundaries, it is not possible for Oracle to anticipate each and every customization on every configuration item of the operating system.

There are certain common standards that Oracle Exadata Database Machine utilities depend on. These standards are based on best practices and have been validated over time. By moving away from Oracle Exadata Database Machine standards, systems might be missing optimal settings, and risk is introduced because it is difficult for Oracle to anticipate and validate software for one-off configurations. This is why customizations can cause unexpected results. As a result of adding software or customizing system configuration, you might require changes or additional steps to the standard Oracle Exadata Database Machine update process. With or without customizations, it is highly recommended to validate Oracle Exadata System Software updates on test systems before doing them on production systems.

8.6.4 Customization Levels and Impact

The table below provides some guidance in the risks associated with customizing Exadata database servers. While the prerequisite check checks for most customizations, certain configuration changes might need to be rolled back in order to make updating the database server possible. If still needed, the changes can be restored after the update process.

Table 8-7 Customizations and Their Risks

Item Customization Level Risk

Default provided Engineered System Software Stack

None

None

Setting up interactive shell profiles for system users (root/oracle)

Low

High

Using all free space in VGExaDb

Low

Low

Installing additional (non-Exadata) rpm packages

Medium

Low

Customizing file system with different mount points

Medium

Low

Updating packages shipped with the current Exadata image

Medium

Low

Customizing configuration files, changing basic operating system functionality

High

Medium

Installing additional (non-Exadata) non-rpm packages

High

Medium

Changing LVM layout

High

High

8.6.5 Update Utility for Exadata Database Servers

Patchmgr is the update utility for updating Oracle Exadata database servers.

Starting with release 12.2.1.1.0, Exadata software updates for the Exadata database server can be applied only through the patchmgr utility.

Note:

The patchmgr utility for updating Exadata database servers is not the same as the patchmgr script shipped with the Exadata software update.

The patchmgr utility supports all hardware generations and Exadata storage server releases starting with 11.2.3.1.0, Exadata database servers running Oracle Virtual Server (dom0), and Exadata Virtual Machines (domU). The README files for the Oracle Exadata System Software updates specify whether the update itself is applicable for a particular hardware generation or not. The README is not shipped with dbserver.patch.zip but with the Oracle Exadata System Software update zip file.

The utility also supports Exadata software updates from Oracle Linux 5 to Oracle Linux 6 or updates from Oracle Linux 6 to Oracle Linux 7.

The utility takes care of the orchestration. You can perform the update in a rolling or non-rolling fashion across one or multiple Exadata database servers.

The update utility performs the following tasks:

  • Automates all preparation, update, and validation steps, including:
    • Stopping the databases, Grid Infrastructure stack or domU’s
    • Stopping Oracle Enterprise Manager Cloud Control agents,
    • Un-mounting remote network mounts (when required)
  • Uses the built-in dbserver_backup.sh script to perform a backup of the file system hosting the operating system before updating the Exadata database server.
  • Applies Oracle best practices and fixes for the latest known issues.
  • Verifies that the update was successful, relinks the Oracle binaries, and starts the Oracle stack and domU’s

8.6.6 Update Tool Execution Host

If you are planning to update all Exadata database servers at once, it is a requirement to run the update utility from a Linux node outside the group of Exadata database servers being updated. This is because the update utility cannot update the Exadata database server it is currently running on. If you have no other systems running Oracle Linux or Oracle Solaris you can run the update utility from one of the Exadata database servers. In such cases be sure that the Exadata database server where the update utility is running is not listed in the dbs_group file you specify.

You need to set up ssh equivalence for the root user from the driving node to the root user of all Exadata database servers that will be updated.

8.6.7 Running the Update Utility as a Non-root User and Running Multiple Invocations Concurrently

By default the update utility assumes you want to run as root. It is however possible to run the update utility as a non-root user from a remote host. It is also possible to run multiple invocations at the same time. This allows you to update multiple logical groups of Exadata Database Servers concurrently. To do this, you run the update utility with the -log_dir flag.

Ensure that ssh equivalence is set up for the current user to the root user of the Exadata database servers to be updated.

The following example shows the option of running as a non-root user and running multiple-invocations.

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/ /p23557378_121223_Linux-x86-64.zip
 -target_version 12.1.2.4.0.160710 -log_dir auto

-log_dir specifies the absolute path to the log directory or the keyword autofor the utility to generate and set a path to the log directory that is based on the launch directory and content of nodes list file. If you want to make sure you are using the same log directory in later invocations where the list of Exadata database servers changes, use the -get log_dirflag to obtain the -log_dir location used on previous sessions. For example, if the following command:

[oracle@nonExadata ]$ ./patchmgr -dbnodes ~/dbs_group_test  -log_dir auto -get log_dir

This command returns output similar to the following:

log_dir=/u01/test/dbserver_patch_5.160715/log/dbm01_dbm02_e8f1f75

Use the log_dir value in subsequent commands. For example:

[oracle@nonExadata ]$ ./patchmgr -dbnodes ~/dbs_group_test -precheck 
-log_dir /u01/test/dbserver_patch_5.160715/log/dbm01_dbm02_e8f1f75 
-iso_repo /u01/test/dbserver_patch_5.160715/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts

8.6.8 Recommended Timeline for Updating Exadata Database Servers

Oracle recommends the following timeline for updating Exadata database servers. By following this approach, you allow yourself time to perform any necessary corrective actions.

Note:

Before making any changes, including prerequisite check, you should make a backup.

Table 8-8 Timeline for Performing Updates

When Tasks

Weeks to days before the update

  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.

  • Download the latest Exachk from My Oracle Support note 1070954.1.

  • Research release-specific My Oracle Support note for known issues.

  • Research Exadata Critical Issues from My Oracle Support note 1270094.1.

  • Run Exachk.

  • Perform a first prerequisite check.

    Note: Sometimes the update utility needs to make minimal changes to the Exadata database server to make YUM dependency checks work. If you wish to ensure no modifications are made, run the prerequisite check with the -nomodify_at_prereq flag.

    For details, see Running Prerequisite Checks.

  • Correct findings that need attention. You can ignore any dependency problems if you are unable to remove rpms at this point. Because -nomodify_at_prereq was specified, broken dependencies can be expected.

Just before the update

  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.

  • Download the latest Exachk from My Oracle Support note 1070954.1.

  • Research release-specific My Oracle Support note for known issues.

  • Research Exadata Critical Issues from My Oracle Support note 1270094.1.

  • Run Exachk and perform corrective actions as needed.

At update time

  • Perform a “backup only” run using the -backup flag.

    See Backing up Exadata Database Servers Before Planned Maintenance for details.

  • Perform a second prerequisite check. This time, omit the -nomodify_at_prereq flag to allow the update utility to make the required changes to get YUM dependencies to work. See Running Prerequisite Checks for details.

  • Remove any blocking rpms and re-run the prerequisite check to validate if all changes are complete.

  • Perform the update. Use the -nobackup flag to skip the backup because you already made a “backup only” run.

    See Running the Update for details.

After the update

  • Run Exachk.

  • Reinstall any non-Exadata rpms that you removed before the Exadata update.

8.6.9 Preparing and Populating the YUM Repository with the Oracle Exadata Channel Content

The Exadata software update procedure for Exadata database servers uses the Unbreakable Linux Network (ULN) for distribution. The updates are distributed as a set of packages (rpms) grouped in a channel. You need to ensure that these packages are available locally before performing the prerequisite check or update.

A channel can be subscribed to by a local non-Exadata database server. Channel content is then downloaded to the local server and the repository is made available as a YUM repository which is used to update Exadata database servers. If no Internet connection is available from the data center or if ULN synchronization is not possible, then a ready-to-use YUM repository can be downloaded as an ISO image.

Note:

Each Exadata Storage Server Update comes with its own channel/ISO image. This ISO image and ULN channel are not intended to be a complete replacement of a generic Linux ULN channel. The Exadata ISO and ULN channel contain only the Oracle Exadata channel content. If your environment uses other Linux packages for which updates are not contained in the Oracle Exadata ULN channel or ISO image, you should update those packages using Linux ULN channels as needed.

Use the following methods to build a YUM repository for hosting the Exadata software update. The update utility uses the local YUM repository to update the Exadata database servers.

8.6.9.1 Downloading the ISO Image as a YUM Repository from My Oracle Support and Passing the Location of the Compressed Image to the Update Utility

The Oracle Exadata channel holding the Oracle Exadata software update is available as a downloadable image (a compressed ISO file) from My Oracle Support and can be used in two ways:

  • Use the file location of the compressed file as an argument for the update utility using the -iso_repo flag

  • Publish using HTTP and use the URL for the update utility using the -yum_repo flag.

The file location of the compressed file can be passed directly to the update utility which is recommended for the following conditions:

  • There is no separate server available to be used as a repository YUM HTTP server.

  • The Exadata database servers do not have customized software that requires updates from additional ULN channels.

  • The simplest method is preferred.

Note:

The compressed ISO image should only exist on the node running the update utility. The distribution of the compressed ISO image is handled by the update utility

You can perform a prerequisite check (-precheck) to verify the repository is usable.

8.6.9.2 Setting Up a Local Mirror as a YUM HTTP Repository from the ULN Oracle Exadata Channel on a Separate Server

This method is recommended for the following conditions:

  • There is a large number of Exadata database servers to update.

  • The single repository server is accessible to all the Exadata database servers.

  • An infrastructure exists for building a local ULN mirror on a separate Linux server.

  • Exadata database servers have customized software that requires updates from other ULN channels.

The mirror server is a separate Linux server that holds the downloaded Exadata software update for Exadata database servers as downloaded from ULN. The Exadata database servers connect to this local mirror (YUM HTTP repository) to retrieve updates.

Note:

Do not use an Exadata database server as the local ULN mirror. Doing so may lead to dependency conflicts between the packages required by ULN to construct the repository, and the packages installed or updated on the Exadata database server by the Exadata software update. If no separate server is available, then use the ISO image method instead. See "Downloading the ISO Image as a YUM Repository from My Oracle Support and Passing the Location of the Compressed Image to the Update Utility".

When setting up a local repository for the first time, you may need additional Linux subscriptions, depending on the Linux release being used for the server that will be hosting the YUM repository. The server should subscribe to the following additional ULN channels in order for the repositories to be built:

  • For 64-bit systems: Enterprise Linux 5 systems: el5_x86_64_addons are required.

  • For 64-bit systems: Oracle Linux 5 systems: el5_x86_64_addons are required.

  • For 64-bit systems: Oracle Linux 6 systems: ol6_x86_64_addons are required.

Note:

After registration, depending on the operating system running your local ULN mirror, the system server automatically subscribes to el5_x86_64_latest, ol5_x86_64_latest, or ol6_x86_64_latest.

Review the prerequisites and server setup procedure as described in "How to create a local Unbreakable Linux Network mirror" at http://www.oracle.com/technetwork/articles/servers-storage-admin/yum-repo-setup-1659167.html to create and maintain a local ULN mirror as YUM repository. For step 5 in the above instructions, in addition to the listed channels, perform the following step for Exadata:

  • Add the channels listed in the Oracle Exadata patch README, such as exadata_dbserver_12.1.2.3.2_x86_64_base.

    The Oracle Exadata channel to subscribe to depends on the Oracle Exadata software release you want to update to. See the patch README for the release for details.

The YUM mirror creation procedure mentioned in the link above provides a script (uln-yum-mirror) which you can use to populate the local mirror. Running the script as root starts the rpm download from ULN to a local directory holding the repository.

You need to install a web server and configure its “document Root” directory to point to this repository location on disk so that the Oracle Exadata channel content is made available to the Oracle Exadata Linux database servers using the HTTP web server. This is described in step 7 from the mentioned instructions.

When you run the update utility, you specify the URL to the repository location. This URL should always be at the level of the “repodata” directory. You can run a prerequisite check (-precheck) to verify that the repository works. See the prerequisite check example using YUM repository in "Running Prerequisite Checks."

Note:

Exadata database servers must be running Exadata release 12.1.2.2.0 or later to access YUM repositories listening on IPv6 addresses only.

8.6.9.3 Downloading the ISO Image as a YUM Repository from My Oracle Support and Making It Available as a YUM HTTP Repository on a Web Server

You can download, uncompress, and place the ISO image channel holding the Oracle Exadata software update on a web server that has HTTP connectivity to every Oracle Exadata database server. This method is recommended for the following conditions:

  • There is a large number of database servers to update.

  • The single repository server is accessible from all the database servers.

    Note:

    If you run the update utility on a non-Exadata system against this ISO image, this is taken care of locally on each database server. In such cases there is no need to set up a local mirror.

  • The database servers do not have customized software that requires updates from additional ULN channels.

    Note:

    If the database servers have custom software installed it is better to use an HTTP YUM repository and not the ISOs, because this improves flexibility in adding back/installing rpms.

Use the following procedure to make the ISO image of the Oracle Exadata channel content available on a Linux-based server running an HTTP server. You can adapt the instructions for other operating systems. In this procedure, the ISO image is copied to the /u01/app/oracle/stage directory. The “Document Root” entry in the web server document is /var/www/html. The procedure uses release 12.1.1.1.0 as an example.

Note:

Oracle recommends that a separate server (non Exadata database server) be used as the web server.

  1. Create the ISO image mount point as the root user using the following commands:

    [root@nonExadataHost ]# mkdir -p /var/www/html/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base
    
    [root@nonExadataHost ]# mkdir –p /u01/app/oracle/stage
    
  2. Copy the compressed ISO image to the web server into /u01/app/oracle/stage.

  3. Uncompress and mount the ISO image as the root user using the following commands:

    [root@nonExadataHost ]# unzip p17997668_121110_Linux-x86-64.zip
    
    [root@nonExadataHost ]# mount -o loop /u01/app/oracle/stage /121110_base_repo.iso /var/www/html/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base
    
  4. Start the httpd service as the root user. This assumes httpd is installed.

    [root@nonExadataHost ]# service httpd start
    
  5. Identify and test the YUM HTTP repository by connecting to it using a web browser. The following is an example of the repository URL.

    http://yum-repo/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base/x86_64/
    
  6. Run the prerequisite check as the root user to verify that the repository is set up properly. See Running Prerequisite Checks.

8.6.10 Managing Exadata Obsoleted Packages

If you are updating to release 11.2.3.3.0 or later, some packages on the Exadata database server become obsolete. While updating an Exadata database server, the update utility prints the exclude rpm list and obsolete rpm list in the log file.

The following example shows the exclusion and obsolete lists from the log file. In this example, an exclusion list has not yet been created by the user.

RPM exclusion list  : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list   : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
                    : RPM obsolete list is extracted from exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64.rpm

To see which packages will become obsolete, review the contents of the obsolete.lst file. This file lists the packages defined to be obsolete by Exadata; these packages will be removed during the update when no action is taken. Packages manually added to this list are ignored. The following is a small sample of the obsolete.lst file:

[root@dm01 ]# cat /etc/exadata/yum/obsolete.lst
# Generated by dbnodeupdate.sh runid: 021213024645
at.x86_64
java-*-openjdk
rhino.noarch
jline.noarch
jpackage-utils.noarch

To prevent a package listed in the obsolete.lst file from being removed, create the /etc/exadata/yum/exclusion.lst file, and put in the rpm name (wildcards are allowed) for the packages you want to keep. Place the /etc/exadata/yum/exclusion.lst file on all Exadata database servers where you want to use it.

The following example shows a package added to the exclusion list:

[root@dm01 ]# cat /etc/exadata/yum/exclusion.lst
java-*-openjdk

After you have added an entry to the exclusion.lst file and re-run the update utility, the utility detects the exclusion list. The rpm packages on the exclusion list are still shown in the obsolete.lst file, but the listed packages in the exclusion.lst file will not be removed during the update.

8.6.11 Updating Individual Packages

Due to security findings or customization, you may need to update individual (general purpose) Linux packages supplied by the Exadata release. You can do this by removing the exadata-sun-computenode-exact rpm first.

Removing this rpm does not impact any functionality, but it removes a “lock”. By removing this “lock”, you are allowed to update specific individual Linux rpms.

If needed, you can remove the exadata-sun-computenode-exact rpm as follows:

[root@dm01 ]# rpm -e exadata-sun-computenode-exact

When updating to a new release, the update utility tries to “restore” the “lock” (exadata-sun-computenode-exact). For example, this is possible if newer packages are shipped in the newer release.

If the exadata-sun-computenode-exact rpm cannot be restored, the update utility falls back to exadata-sun-computenode-minimum.

Note:

Do not force in packages using the rpm -Uvh --nodeps command, unless directed by Oracle Support Services.

8.6.12 Running Prerequisite Checks

You should always run prerequisite checks before doing the actual update. The prerequisite checks do not require downtime and execute important validations such as:

  • Validation of the Exadata release (minimum is 11.2.2.4.2 running Oracle Linux 5.5)

  • Validation of user input

  • Validation of the installation media (YUM repository, either ISO or HTTP)

  • Validation of disk space and snapshots

  • Validation of YUM settings that are important for the update to finish successfully

  • Known issues / best practices

The most important validation executed by the update utility is the YUM dependency check. The YUM dependency check is a YUM update dry-run command (introduced in 11.2.3.3.0) that does not do the actual YUM update but does validate dependencies. This is a final test in determining whether or not the update can proceed. It is often due to customizations that prevent successful updates. For example, installation of additional RPMs might require dependent packages that are not in the YUM repository. If this happens, you need to take corrective action to resolve the conflict.

The YUM dependency check (dry-run) is validated against minimum and exact dependencies. These dependencies are enforced by non-functional Exadata RPMs and help administrators stay exactly at (or close to) the original Exadata release when customizing the system. The update utility uses the exadata-sun-computenode-exact and the exadata-sun-computenode-minimum RPMs as follows:

  • The exadata-sun-computenode-exact rpm ensures that only a specific release of Oracle Exadata branded packages is allowed during the update. (release = x)

  • The exadata-sun-computenode-minimum rpm ensures that a specific or later release of Oracle Exadata branded packages is allowed during the update. (release >= x)

With exadata-sun-computenode-exact rpm, the system appears as if it were freshly imaged to the newer release because all the Oracle Exadata packages are exactly the same as on a vanilla installation. The exadata-sun-computenode-minimum rpm, however, sets the minimum dependencies, and enforces that all packages are installed, but it also allows packages to be at a later version. A vanilla installation always starts with both RPMs. To allow customization or updates, you need to remove exadata-sun-computenode-exact.

By default, the update utility attempts to match the exact dependencies when updating to a later Exadata release. When exact dependencies conflict and cannot be enforced, the utility falls back and attempts to apply the exadata-sun-computenode-minimum rpm to enforce minimum dependencies. In such cases the exadata-sun-computenode-exact rpm is not installed.

Missing or not updating with exact dependencies is allowed and not a problem. If a system needs to be updated to the exact dependencies, then the conflict needs to be resolved first. Check the log file to see what packages conflict, remove them cautiously, and then re-run the update utility in prerequisite check mode.

If the prerequisite check fails due to dependency issues, you can view the errors on-screen. The update utility’s log file has more details and shows which dependencies failed. When both exact and minimum dependencies do not match, the update cannot proceed.

For such cases, check the log file to determine what caused the dependencies to fail. After removing the failed dependencies, re-run the update utility to ensure that at least the minimum dependencies can be enforced.

Note:

The update utility may need to remove certain packages during prerequisite check in order to make sure the YUM prerequisite check works as some Exadata storage server updates have known dependencies that need to be resolved. While the removal of such RPMs during the prerequisite check should be rare and should never impact Exadata functionality it is understood that not all systems allow a prerequisite check to make modifications. For such systems it is recommended to use the -nomodify_at_prereq flag. When this flag is used, the update utility will not remove packages from your system, but it will generate a list of packages that would have been removed (installed or not). When this flag is used, the dependency check may fail. For this reason it is recommended that you re-run the dependency check without the -nomodify_at_prereq flag after you have performed a backup-only run in your planned maintenance window.

When dependency errors occur during the prerequisite check or before the update starts, do the following to resolve the problem:

  • Analyze the YUM errors in the log file. Search for Error.

  • Depending on the issue, you may need to de-install, install, or update the rpm packages causing the dependency issue or conflict. The log file lists the failed dependencies.

After the update, you may re-install custom rpm packages that you de-installed, assuming you still require the packages and the packages are compatible with the updated system.

By default, prerequisite check warnings such as active NFS mounts result in a failed check. If you want to allow active NFS mounts, then starting in release 12.1.2.1.1 you can use the -allow_active_network_mounts flag.

For more options, see the update utility built-in help.

Prerequisite Check Examples

The following command shows an example of a prerequisite check that does not remove any RPMs using ISO. The command is run as root.

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip 
-target_version 12.1.2.3.0.160207.3 -nomodify_at_prereq

-dbnodes specifies the list of database nodes to be updated.

-precheck specifies the prerequisite check action.

-iso_repo specifies the location of the ISO (YUM) repository on the node running the updating utility. This can be replaced by -yum_repo <location>.

-target_version specifies the target release the database servers are being updated to.

-nomodify_at_prereq specifies no RPMs will be removed at prerequisite check time.

The following command shows an example of a prerequisite check that allows the removal of necessary RPMs using ISO. The command is run as a non-root user.

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip 
-target_version 12.1.2.3.0.160207.3 -log_dir auto

8.6.13 Backing up Exadata Database Servers Before Planned Maintenance

Oracle recommends making a backup of the Exadata database servers before making any changes when updating to a next release.

This means running the update utility in backup-only mode before running the prerequisite check without the -nomodify_at_prereq flag or before making any other (manual) changes to make the prerequisite / dependency check pass. The backup-only action backs up the active root and /boot file system only and is sufficient for rolling back (failed) updates.

Note:

The update utility’s built-in backup (dbserver_backup.sh) is insufficient for recovering from non-booting system failures. It is recommend having additional (validated) backup/restore procedures in place to recover the entire database server from failures such as double disk failures.

For regular and virtualized Exadata database servers (domU), when the active system image is running from a file system on /dev/mapper/VGExaDb-LVDbSys1, the backup is made onto /dev/mapper/VGExaDb-LVDbSys2 (and vice versa). For Exadata database servers running dom0 with /dev/mapper/VGExaDb-LVDbSys2 as the active image, the backup goes to /dev/mapper/VGExaDb-LVDbSys3 (and vice versa).

This means that upon rollback of an update the active system image for regular database servers and domU will become /dev/mapper/VGExaDb-LVDbSys2, and for dom0 will become /dev/mapper/VGExaDb-LVDbSys3.

The process of backing up an active system partition requires the default LVM scheme for the Sys* LVM’s and both LVM’s to be the same size. This is because it is impossible to back up a larger LVM partition to a smaller LVM partition. Resizing LVM VGExaDb-LVDbSys1 partition is allowed as long as VGExaDb-LVDbSys2 is resized to the same size.

While the backup is running, an LVM snapshot takes care of a consistent view of the file system. This LVM snapshot is maintained by the backup script and will always claim 1G of free VG space in VGExaDb.

Space for the snapshot is guaranteed by Oracle by using a placeholder LVM called /dev/VGExaDb/LVDoNotRemoveOrUse. When the backup script runs, it removes this placeholder, making sure there is always 1G of free space available for a snapshot. After the backup is completed, the snapshot is removed and the /dev/VGExaDb/LVDoNotRemoveOrUse LVM is re-created.

You can perform the backup while all system services are up and running. Starting with release 12.1.2.1.1, the utility supports active network mounts (NFS) with the -allow_active_network_mounts flag.

The time it takes to run the backup depends on how busy the system is and on the size and type of data that is backed up. For example, backing up millions of small files can take significantly longer than backing up a couple of larger files. For this reason it is recommended to make sure directories holding database .aud files are not found on the root file system. Note the following:

  • You can have only one backup. Running a new backup means overwriting the existing backup.

  • Re-running the update utility with the -backup flag (or in default updating mode) will overwrite existing backups.

  • All files in /boot and found on the active Sys LVM’s are backed up. Files in, for example, /u01, are not backed up.

Examples for “Backup Only”

The following example shows running “backup only” as root:

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -backup -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip 
-target_version 12.1.2.3.0.160207.3 -allow_active_network_mounts
  • -dbnodes specifies the list of database nodes to be updated.

  • - backup specifies the “backup only” action.

  • -iso_repo specifies the location of the ISO (YUM) repository on the node running the update utility. Alternatively, you can use the -yum_repo flag, which specifies the HTTP location of the YUM repository.

    -target_version specifies the target release for which the backup is run.

  • -allow_active_network_mounts ensures active network mounts remain active while performing the backup.

The following example shows a “backup only” action run as a non-root user:

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group_scab -backup -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -smtp_from "sender@somedomain.com" 
-smtp_to "recipient@example.com"

8.6.14 Running the Update

You can perform the actual update of Exadata database servers in a rolling (using the -rolling flag) or non-rolling fashion. The default is non-rolling.

You can also perform the update as root or as a non-root user (using the -log_dir flag), as described in "Running the Update Utility as a Non-root User and Running Multiple Invocations Concurrently".

The update proceeds only if the “minimum dependency check” succeeds. You may need to remove customizations for the update to proceed.

By default the update creates a backup on the inactive system image. If you have already taken a backup before running the prerequisite check without using the -nomodify_at_prereq flag, you can skip the backup using the –nobackup flag when performing the update.

Note:

Use the -nobackup flag only if a backup was already made before running the prerequisite check without the -nomodify_at_prereq flag.

The update action requires the following mandatory flags:

  • –upgrade to specify the update action

  • -iso_repo (for ISO image) or -yum_repo (for HTTP locations) to point to the YUM repository (see Example 8-12)

  • -target_version to specify the release you want to update to. The patch README always has this information.

You can specify additional flags to allow active remote network mounts during backup and updating (-allow_active_network_mounts) and specify mail recipients for updating status notification (-smtp_from "addr" and -smtp_to "addr1 addr2 addr3 ...")

Example 8-11 Running Update Using ISO Image for YUM Repository

The following example shows an update action run as root and using an ISO image for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[root@dm01 ]# ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -smtp_from "sender@somedomain.com" 
-smtp_to "receiver@somedomain.com" -nobackup

The following example shows an update action run as a non-root user from a remote host using an ISO YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -log_dir auto -smtp_from "sender@somedomain.com" 
-smtp_to "receiver@somedomain.com" -nobackup

Example 8-12 Running Update Using HTTP Location for YUM Repository

The following example shows an update action run as root using HTTP for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[root@dm01 ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -yum_repo http://yum-repo/yum/ol6/EXADATA/dbserver/12.1.2.2.3/base/x86_64/ 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -smtp_from "sender@somedomain.com" -smtp_to "receiver@somedomain.com"
-nobackup

The following example shows an update action run as a non-root user from a remote host using HTTP for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -yum_repo http://yum-repo/yum/ol6/EXADATA/dbserver/12.1.2.2.3/base/x86_64/ 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -log_dir auto -smtp_from "sender@somedomain.com" 
-smtp_to "receiver@somedomain.com" –nobackup

8.6.15 Rolling Back Updates

A backup enables you to roll back updates, regardless of whether the update failed or succeeded. This backup is stored on the inactive system partition, as described in "Backing up Exadata Database Servers Before Planned Maintenance".

When rolling back an update, the update utility performs the following actions:

  • Shuts down the stack and domU’s.
  • Deactivates the active system partition, and activates the inactive system partition.
  • Restores /boot from the inactive partition.
  • Updates the grub boot-loader.

Having only one inactive system partition limits the rollback options to only the previous active image.

Note:

  • For systems being updated to Oracle Linux 6, a backup must be performed before proceeding with the update. The backup is automatic when updating LVM-enabled systems from Oracle Linux 5 to Oracle Linux 6.
  • Database servers running as Oracle VM Server (dom0) switch between LVDbSys2 and LVDbSys3 as the active system partition when rolling back.
  • Database servers running as Oracle VM (domU) have smaller sizes for LVDbSys1 compared to physical hardware deployments.

Example 8-13 Rolling back an update using patchmgr

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -rollback -target_version 12.1.2.3.0.160207

-dbnodes specifies the list of database nodes to be updated.

-rollback specifies the rollback action.

-target_version specifies the target release to roll back to.

For more options see the update utility’s built-in help.

Note:

Firmware updates are not rolled back when rolling back to a previous image. The Oracle Exadata System Software releases support later firmware releases. After rolling back, run the following commands to apply older firmware versions when needed:

/etc/init.d/lsidiag stop

/etc/init.d/lsi_mrdsnmpd stop

/opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact

The last command only applies to releases 11.2.3.3.0 or later.

8.7 Updating Database Servers Running Oracle Exadata System Software Release 11.2.2.4.2

Updates for database servers running Oracle Exadata System Software release 11.2.2.4.2 include an update that prepares the servers to use yum.

The steps to prepare the servers are done using the dbnodeupdate.sh utility. For this reason, updates from Oracle Exadata System Software release 11.2.2.4.2 require two runs of the dbnodeupdate.sh utility with different arguments. The utility provides instructions about which command to run and when to it.

When updating the database servers, the following assumptions are made:

  • The Oracle Exadata System Software release is 11.2.2.4.2.
  • The database servers are running Oracle Linux 5.5 or later.
  • The database server update is available as a local ULN mirror or as a local ISO image.

8.7.1 Preparing to Use the dbnodeupdate.sh Utility on Database Servers with Release 11.2.2.4.2

This procedure describes how to download and prepare the utility on the server.

The dbnodeupdate.sh utility is available from My Oracle Support from Doc ID 1553103.1. The dbnodeupdate.sh utility can update releases running 11.2.2.4.2 directly to Oracle Linux 6.

  1. Download the dbnodeupdate.sh utility from My Oracle Support.
  2. Log in as the root user on the database server.
  3. Put the compressed file in the /u01/dbnodeupdate directory on the database server. Create this directory if it does not exist.
  4. Uncompress the p16486998_12xxxx_Linux-x86-64.zip package file in the /u01/dbnodeupdate directory using the following command:
    [root@dm01 ]# unzip p16486998_12xxxx_Linux-x86-64.zip
    

    Note:

    • When using a compressed ISO, upload it to the database server and place it in the /u01/dbnodeupdate directory, provided there is free space available. Create this directory if it does not exist.
    • When using a local ULN mirror, make sure the HTTP location is available.
    • When making ISO contents available on a web server, upload the contents to the web server, mount it, and verify the URL.
    • A user with sudo privileges can use sudo to run the dbnodeupdate.sh utility.

8.7.2 Running the dbnodeupdate.sh Utility

You can use the dbnodeupdate.sh utility on database servers running release 11.2.2.4.2.

  1. Log in as the root user on the database server.
  2. Run the dbnodeupdate.sh utility using the following command, where repo is the HTTP location of the ULN mirror or the location of the compressed ISO image:
    [root@dm01 ]# ./dbnodeupdate.sh -u -l repo
    

    The database server restarts automatically.

    Note:

    Just before the restart, instructions for the next step are provided onscreen.
  3. Run the following command after the restart mentioned in step 2. The utility automatically remounts the ISO image when a compressed ISO image is used.
    [root@dm01 ]# ./dbnodeupdate.sh -u -p 2 
    
  4. Run the following command to complete the update.
    [root@dm01 ]# ./dbnodeupdate.sh -c
    

    During the completion process, the utility performs post-update checks, applies best practices, relinks the Oracle homes, and starts the stack.

    Note:

    The update process might still be running when the dbnodeupdate.sh -c command is entered. When this happens, the utility waits until it can determine the image status. The system may restart while waiting to make pending updates effective. If this happens, then re-enter the dbnodeupdate.sh -c command when the system is back online.

8.8 Updating Software on Oracle Exadata Storage Servers

Use the following information and procedures when updating storage servers within Oracle Exadata Database Machine.

8.8.1 Overview of Oracle Exadata Database Machine Storage Server Updates

When updating storage servers, there is more than one type of software that needs to be updated and different methods of performing the updates.

Oracle Exadata System Software release updates contain updates for the following components within an Oracle Exadata Database Machine storage server:

  • Oracle Linux operating system
  • Firmware (Flash, Disk, RAID controller, Integrated Lights Out Manager (ILOM), HCA)
  • Oracle Exadata System Software

What software and firmware that is updated depends on the current Oracle Exadata System Software release the storage server is on and the release it is updated to. Oracle Linux operating system packages and Oracle Exadata System Software are always updated, while firmware updates are applied only on a small selection of the components, or not at all.

Updates for Oracle Exadata Database Machine storage server can be applied independently from the updates to Oracle Exadata Database Machine database server or RDMA Network Fabric switches unless specified otherwise. It is not mandatory to apply each and every Oracle Exadata System Software update that comes out. For example, you can skip two or three releases and update directly to a newer release.

Updating the Oracle Exadata System Software is always performed “out of place”. This means that a new version of the operating system including the Oracle Exadata System Software is installed on the inactive system partition. The utility to update the Oracle Exadata System Software ships with the update itself.

If you cannot afford cluster-wide downtime, you can update the storage servers in a rolling fashion. Rolling means updating one storage server at a time. If you can afford the cluster-wide downtime, you can update all storage servers in parallel. Non-rolling updates reduce the overall time required.

Starting with Oracle Exadata System Software release 18c (18.1.0), there is a more scalable alternative to using patchmgr for software updates. The storage servers automatically verify preconditions and download the update software from a URL that you specify. Each storage server downloads the software to its active partition, and then loads the software on its passive partition. At a specified time, the storage servers reboot to the new version. The storage servers use the Oracle Automatic Storage Management (Oracle ASM) disk deactivation status to determine when it is safe to deactivate the disks and reboot the storage server to the new software version. These scheduled updates invoke the same scripts that are currently used by the patchmgr process.

8.8.2 Scheduling Automated Updates of Storage Servers

Starting with Oracle Exadata System Software release 18.1.0.0.0, you can schedule software updates for the storage servers.

Perform the following steps from an external server, for example, an Oracle Exadata Database Server.

If you are accessing the Software Update store using the HTTPS protocol, then TLS certificate checks are required by default. If the certificate for the web server that hosts the software update cannot be validated, then the following error is returned:

CELL-00076: An error occurred during download of software update:
source https://example.com:port is not available. 
CELL-00092: The store's TLS certificate cannot be authenticated with 
known CA certificates
  1. Copy the software update ZIP file to a directory which is hosted by a web server.
  2. If you are using Oracle Exadata System Software 18c (18.1.0) or 18c (18.1.1), then the patch file must have a name like 18.1.1.0.0.171018.patch.zip.

    If the downloaded patch has a name like p26875767_181100_Linux-x86-64.zip, then rename the file to 18.1.1.0.0.17018.patch.zip. Rename the ZIP file to use the release number and date string that is used for the directory name within the ZIP file. For example, when you unzip the patch p26875767_181100_Linux-x86-64.zip it extracts the directory patch_18.1.1.0.0.171018.

    If you are using Oracle Exadata System Software release 18.1.2 or higher, then the patch downloaded from My Oracle Support is automatically renamed for you during the validation step.

  3. Determine approximate time to perform the software update of the storage servers.
  4. Create a file on the external server that contains the list of cells to be updated. Name this file cells.
  5. Use dcli to schedule the update of the cells.
    1. Setup passwordless access, if needed.
      $ dcli –g cells –k 
    2. Specify the location of the software update ZIP file to use during the update.
      $ dcli –g cells cellcli –e 'alter softwareUpdate store=\"https://host/exa-updates/cell\"'
    3. Specify the time to start the update of the Oracle Exadata System Software on the storage servers.

      If you specify the time before providing the software store location, then the software update download might start before the proper store location has been set.

      $ dcli –g cells cellcli –e 'alter softwareUpdate time=\"1 AM Thursday\"'
      
  6. Wait for the updates to occur.

    Management Server (MS) will the start file download and run pre-checks up to one week before the scheduled update. MS will generate an alert if any cell does not update as scheduled.

Related Topics

8.8.3 Update Utility for Oracle Exadata Storage Server

You use the patchmgr update utility for updating Oracle Exadata storage servers. For Exadata storage server updates the utility is packaged (and shipped) with the update itself and is available for download from My Oracle Support as storage server update.

Note:

The patchmgr used for updating Exadata storage servers is not the same as the patchmgr used for applying the Exadata database server software update.

Whether or not the utility supports the Exadata hardware you have depends on the Exadata storage server release you are trying to update to. The utility orchestrates the update process across the specified Exadata storage servers. The utility allows running the update in a rolling or non-rolling fashion. You can run the update utility from Exadata database servers or from other servers running Oracle Linux or Oracle Solaris.

The update utility performs the following tasks:

  • Automates the preparation, update, and validation steps

  • Automates rollbacks

The update utility supports multiple sessions: you can run multiple updates concurrently from the same server starting with release 12.1.2.3.2 for Exadata storage servers and starting with release 11.2.3.1.0 for Exadata database servers. This means multiple racks can be updated concurrently from the same server. The update utility can be run as root or as a non-root user. By default the update utility assumes it should run as the root user. If however you want to enable multiple session support or run as a non-root user, then you need to use the -log_dir flag. The -log_dir flag supports two types of arguments: either a location on disk or the keyword auto. If you specify auto, the update utility creates its own log directory based on the storage servers listed in the cell_group file. This behavior causes the update utility to create new directories for each run of updates in the same cluster where one or more clusters were added or removed from the cell_group file. In order to obtain (and reuse) such a directory, the update utility provides the -get flag to determine the log directory for your session. The-get flag scans the working directory for directories in the log directory and returns the directory for your cell_group. For example, the following command:

[oracle@nonExadataHost ]#./patchmgr -dbnodes ~/cell_group  -log_dir auto -get log_dir

The previous comment might return output similar to the following:

log_dir=/u01/test/patch_12.1.2.4.0.160802/log/dbm02celadm01_dbm02celadm02_9cfbc690

In a subsequent update session, you can re-use the log directory location:

[oracle@nonExadataHost ]# ./patchmgr -cells ~/cell_group -patch_check_prereq -log_dir /u01/test/patch_12.1.2.4.0.160802/log
/dbm02celadm01_dbm02celadm02_9cfbc690

8.8.4 Recommended Timeline for Updating Oracle Exadata Storage Server

Note:

It is highly recommended to validate Oracle Exadata System Software updates on test systems before doing them on production systems.

By following the approach below, you allow yourself time to perform any necessary corrective actions.

Table 8-9 Timeline for Performing Updates

When Tasks

Weeks to days before the update

  • Download the Oracle Exadata System Software update you require from My Oracle Support note 888828.1

  • Download the latest Oracle EXAchk from My Oracle Support note 1070954.1

  • Review the release-specific My Oracle Support note for known issues

  • Run Oracle EXAchk

  • Perform prerequisite check. See Preparing Exadata Storage Servers for Update for details.

  • Correct findings that need attention and re-run the above steps as needed.

At update time

  • Download the Oracle Exadata System Software update from My Oracle Support note 888828.1

  • Download the latest Oracle EXAchk from My Oracle Support note 1070954.1

  • Review the release-specific My Oracle Support note for known issues

  • Run Oracle EXAchk and perform corrective actions as needed

  • Perform prerequisite check and corrective actions as needed.

  • Perform the update. See Running the Update for Exadata Storage Servers for details.

After the update

  • Run Oracle EXAchk.

8.8.5 Preparing Exadata Storage Servers for Update

Perform these preparation steps before updating the Exadata storage servers.

You can perform the update (and rollback) action in a rolling or non-rolling method. You can also perform a prerequisite check in a rolling or non-rolling method. The default is non-rolling.

  1. Set up SSH equivalence for the user that is driving the update utility.
  2. Download and run Oracle ExaCHK. Review and address any open issues. See My Oracle Support note 1070954.1.
  3. Review the release-specific My Oracle Support note for any known issues and workarounds.
  4. Check prerequisites for your method of update or rollback.

    Prerequisites for performing a rolling update:

    1. Verify that your Grid Infrastructure home and Database home software versions and patch levels meet the minimum required for Exadata storage server rolling cell update as documented in My Oracle Support note 888828.1
    2. Verify failgroup_repair_time or disk_repair_time for each Oracle ASM disk group.

      When applying the update in a rolling manner, the update utility updates one server at a time, first taking all grid disks and ASM disks offline, then applying the update to the server, then bringing all ASM disks and grid disks back online. The Oracle ASM repair timeout attributes, disk_repair_time and failgroup_repair_time, need to be set to a value large enough to allow a single storage server update to complete. The default values of 3.6h and 24h, respectively, are the recommended values. Note that during a rolling storage server update disk groups with compatible.asm >= 12.1.0.2.0 will use the value of failgroup_repair_time, and disk groups with compatible.asm < 12.1.0.2.0 will use the value of disk_repair_time.

      Verify ASM repair timeout attributes are set to default or higher values. Use the following command to check repair time attributes for all mounted disk groups in the Oracle ASM instance.

      SQL> col attribute format a30
      SQL> col value format a10
      SQL> select dg.name as diskgroup, a.name as attribute, a.value
           from v$asm_diskgroup dg, v$asm_attribute a
           where dg.group_number=a.group_number
             and (a.name like '%repair_time' or a.name = 'compatible.asm');
      
      DISKGROUP                      ATTRIBUTE                      VALUE
      ------------------------------ ------------------------------ ----------
      DATA                           disk_repair_time               3.6h
      DATA                           failgroup_repair_time          24.0h
      DATA                           compatible.asm                 12.1.0.2.0
      RECO                           disk_repair_time               3.6h
      RECO                           failgroup_repair_time          24.0h
      RECO                           compatible.asm                 12.1.0.2.0
      
      

      If the Oracle ASM repair timer for any disk group is lower than the default value, then set the repair timer to the default value for the duration of the update. You may set it back to its current value after the update successfully finishes for all storage servers.

    Prerequisites for performing a non-rolling update:

    1. Shut down and stop the Oracle components on each Exadata database server using the following commands, where Grid_home is the directory where the Oracle Grid Infrastructure software is installed:
      [root@dm01 ]# dcli -g dbs_group -l root "Grid_home/bin/crsctl stop crs"
      

      If Oracle Clusterware was not stopped using the preceding command, then use the following command to force it to stop:

      [root@dm01 ]# crsctl stop crs -f
      
    2. Use the following command to check Oracle Clusterware status, where Grid_home is the directory where the Oracle Grid Infrastructure software is installed:
      [root@dm01 ]# dcli -g dbs_group -l root "Grid_home/bin/crsctl check crs"
      

      All Oracle Clusterware components must be offline. If you are performing a non-rolling update in a configuration running Oracle VM, then you must check the Oracle Clusterware state in all VM clusters.

  5. Check if your system meets the following conditions:
    • Oracle Exadata Database Machine hardware uses Sun servers.
    • The installed Oracle Exadata Software release is earlier than release 11.2.2.2.0.
    • The installed Linux ofa package is earlier than 1.5.1-4.0.28.

    If the system meets all of the preceding conditions, then Exadata storage servers running Oracle Linux may encounter a file system corruption that results in the root file system mounted as read-only after reboot. Follow the instructions in My Oracle Support note 1589868.1 before updating the storage servers and database servers.

  6. Unzip the update. It will extract into the patch_release.date_code directory. Change to this patch directory.

  7. Download any patchmgr plug-ins attached to the My Oracle Support note for your target release and install them as documented in the My Oracle Support note. Oracle recommends reviewing the My Oracle Support notes, issues, and workarounds listed in the release note just before starting to actually apply the update.
  8. Clean up any previous update utility runs using the -cleanup flag.

    Note:

    The first time the storage servers are updated the -reset_force flag should be used before running cleanup.

    Example using -reset_force as the root user:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group –reset_force
    

    Example using -reset_force as a non-root user:

    [oracle@nonExadataHost ]# ./patchmgr -cells ~/cell_group -log_dir auto –reset_force
    

    Example using -cleanup as the root user:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -cleanup
    

    Example using -cleanup as a non-root user:

    [oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto –cleanup
    
  9. Run prerequisite check.

    Example running prerequisite check for a rolling update as the root user from an Exadata database server:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -patch_check_prereq -rolling -smtp_from "sender@example.com"  
    -smtp_to receiver@example.com
    

    Example running prerequisite check for a non-rolling update as a non-root user from a non-Exadata database server:

    [oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto -patch_check_prereq -smtp_from "sender@example.com"
    -smtp_to "receiver@example.com"
    

8.8.6 Running the Update for Exadata Storage Servers

After performing the prerequisite steps in Preparing Exadata Storage Servers for Update, you can perform the actual update step.

Note the following when applying the update to the Exadata storage servers:

  • Do not use the serial console or the ILOM web-based console to start the update utility.

    There is a known issue of a system halt on the serial console when a write is attempted to stderr or stdout. If an update is started from the serial console, then it may halt.

    You are using the serial console if the output from the following command is serial.

    [root@dm01 ]# ./echo $consoletype
    
  • When needed, use the ILOM web-based console to monitor the storage server during the update. You will need to use the ILOM web-based console in case troubleshooting is required.

  • To obtain ILOM and serial console access for the storage servers, use SSH to the ILOM host name or IP address as the root user. Do the following to start the serial console:

    start /SP/console
    

    To stop it press the Escape key (ESC) followed by (.

  • Start a new login session for each update or rollback procedure. Do not run a rollback procedure from the same login session where an update was applied. Do not run an update from a login session where a rollback procedure was run.

  • Do not interrupt the update process.

  • If you must use a storage server as the patchmgr utility launch system, then do not use /opt/oracle as the staging area for the update. This causes the update to fail and corrupt the storage server. Use the /tmp directory as the staging area, that is, unzip the files for the update in /tmp.

  • Storage servers automatically reboot, as needed, during the update process. Do not reboot or power cycle storage servers while applying updates.

  • Do not edit or open log files in writable mode. You may use any of the following to view a log: view, less, more, or tail. You may cause the update process to be interrupted if you edit the log files during the update.

  • At the end of the patchmgr session, the patchmgr.stdout log file is divided into individual storage server log files with names in the format of cell_name.log. In addition, the /var/log/cellos content from the inactive cell partition is copied to the /var/log/cellos/inactive_partition directory. To locate the inactive partition, use the following command:

    [root@dm01 ]# ./imageinfo -inactive -sys
    

Examples running update:

Running the update in a rolling fashion as root from an Exadata database server:

[root@dm01 ]# ./patchmgr -cells ~/cell_group -patch -rolling -smtp_from "sender@somedomain.com" -smtp_to receiver@somedomain.com

Running the update in a non-rolling fashion as a non-root user from a non-Exadata database server:

[oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto -patch -smtp_from "sender@somedomain.com" -smtp_to "receiver@somedomain.com"

After the update is done, clean up the storage servers using the -cleanup option to clean up all the temporary update or rollback files. This option cleans the stale update and rollback states as well as cleaning up to 1.5 GB of disk space on the storage server. Use this option before retrying a halted or failed run of the update utility. See step 8 for details.

8.8.7 Rolling Back Updates for Exadata Storage Servers

You can roll back updated Exadata storage servers only when they are updated successfully. This means the imageinfo command must return success for active image status. Storage servers with incomplete or failed updates cannot be rolled back. Rollbacks can be done in a rolling or non-rolling fashion.

  1. Check the version that the storage servers will be rolled back to and the flashCacheMode setting with the following commands:

    [root@dm01 ]# dcli -l root -g cell_group imageinfo -ver -inactive
    
    [root@dm01 ]# dcli -l root -g cell_group cellcli -e 'list cell attributes flashCacheMode
    

    Note:

    If you need to roll back storage servers to releases earlier than release 11.2.3.2.0 with writeback flash cache enabled, you need to convert the flash cache to writethrough flash cache before performing the rollback action. Disable the writeback flash cache using the script in My Oracle Support note 1500257.1. Storage servers being rolled back to release 11.2.3.2.0 or later retain the flash cache mode that is currently set.

  2. Check the prerequisites for rollback using the following command:

    [root@dm01 ]# ./patchmgr -cells cell_group -rollback_check_prereq [-rolling]
    
  3. Perform the rollback.

    Example of a non-rolling rollback run as root:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -rollback
    

    Example of a rolling rollback run as a non-root user:

    [oracle@nonExadataHost ]#./patchmgr -cells ~/cell_group -rollback -log_dir auto
    

    Note:

    Firmware updates are not rolled back when rolling back to a previous image. After rolling back, run the following command to apply older firmware versions when needed:

    /etc/init.d/lsidiag stop
    
    /etc/init.d/lsi_mrdsnmpd stop
    
    /opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact
  4. Clean up the Exadata storage servers using the -cleanup option to clean up all the temporary update or rollback files. This option cleans the stale update and rollback states as well as cleaning up to 1.5 GB of disk space on the Exadata storage servers. Use this option before retrying a halted or failed run of the patchmgr utility.

    [root@dm01 ]# ./patchmgr -cells cell_group -cleanup
    

8.9 Updating RoCE Network Fabric Switch Firmware

This topic describes procedures to update and downgrade the firmware on the RoCE Network Fabric switches.

Note the following when updating the RoCE Network Fabric switch firmware:

  • The update of the RoCE Network Fabric switches is executed using patchmgr.
  • Download the appropriate patch ZIP file to any machine with access to the RoCE Network Fabric switches. Refer to My Oracle Support note 888828.1 for the patch information.
  • patchmgr configures ssh passwordless-access to each switch, which requires you to provide the password of the admin user for the switch.
  • You must use a non-root user to perform the patching, and must include the -log_dir option with patchmgr.
  • Switch firmware is always updated in a rolling manner.

Note:

The patchmgr used for updating RoCE Network Fabric switch firmware is not the same as that used for the Oracle Exadata Database Machine database server or storage server software updates.

Caution:

Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. RDMA Network Fabric switch firmware upgrade requires switch reboot, which disrupts some connections on the RDMA Network Fabric network.

8.9.1 Preparing for RoCE Network Fabric Switch Firmware Upgrades or Downgrades

You must follow a specific order when updating the RoCE Network Fabric switches.

  1. Log in to a server that has access to the RoCE Network Fabric switches.
  2. Download the appropriate patch file to the server.

    Starting with Oracle Exadata System Software release 19.3.0, the updates for the switches are in a separate patch. Refer to My Oracle Support note 888828.1 for patch information.

  3. Unzip the patch file.

    The files are unzipped to the patch_switch_release directory.

  4. Change to the directory that contains the patchmgr utility.

    For example:

    # cd patch_switch_19.3.0.0.0.190915
  5. Create a file listing all the RoCE Network Fabric switches that need to be updated, with one switch per line.

    The following is an example of the file listing all the switches to update:

    # cat roceswitches.lst
    myroceswitch-01
    myroceswitch-02
  6. Run the prerequisite check prior to either upgrading or downgrading the firmware.
    # ./patchmgr --roceswitches roceswitches.lst {--upgrade | --downgrade} --roceswitch-precheck 
      [-log_dir {absolute_path_to_log_directory | AUTO}] [--force]

    In the patchmgr command:

    • --roceswitch-precheck instructs patchmgr to perform a firmware upgrade or downgrade simulation on the switch.

    • -log_dir specifies the absolute path to the log directory, or AUTO instructs patchmgr to automatically create the log directory. This option is required when running patchmgr as a non-root user.

    • --force optionally proceeds with the operation even if the switch is already on the target firmware version or the RoCE Network Fabric is experiencing non-critical failures.

    Note:

    Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.

    Note:

    The current user is expected to have SSH equivalency configured prior to running patchmgr. If it is not configured, then patchmgr will give you the option to setup keys and key exchange for SSH equivalency.

    If the output from the command shows overall status is SUCCESS, then proceed with the upgrade. If the output from the command shows overall status is FAIL, then review the error summary in the output to determine which checks failed, and then correct the errors. After correcting all the errors, rerun the prerequisite checks until it is successful.

8.9.2 Updating RoCE Network Fabric Switch Firmware Software

Update the RoCE Network Fabric switches using the patchmgr command.

Before starting this task, complete the steps in Preparing for RoCE Network Fabric Switch Firmware Upgrades or Downgrades.

  1. On the server where you downloaded the patch, use the patchmgr command to update the switches.
    # ./patchmgr --roceswitches roceswitches.lst --upgrade [--verify-config [yes|no]] 
      [-log_dir {absolute_path_to_log_directory | AUTO}] [--force]

    In the patchmgr command:

    • --verify-config specifies whether to verify the switch configuration against the golden configuration. Verification is performed by default if --verify-config is not specified.

    • -log_dir specifies the absolute path to the log directory, or AUTO instructs patchmgr to automatically create the log directory. This option is required when running patchmgr as a non-root user.

    • --force optionally proceeds with the operation even if the switch is already on the target firmware version or the RoCE Network Fabric is experiencing non-critical failures.

    Note:

    Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.
  2. Verify the update.

    Use the show version command to verify the firmware version on the switch:

    For example:

    # show version
    Cisco Nexus Operating System (NX-OS) Software
    TAC support: http://www.cisco.com/tac
    Copyright (C) 2002-2019, Cisco and/or its affiliates.
    All rights reserved.
    ...
    
    Software
      BIOS: version 05.33
      NXOS: version 7.0(3)I8(1)
      BIOS compile time: 09/08/2018
      NXOS image file is: bootflash:///nxos.7.0.3.I8.1.bin
      NXOS compile time: 3/5/2019 13:00:00 [03/05/2019 22:04:55]
    
    Hardware
      cisco Nexus9000 C9336C-FX2 Chassis
      Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 24571632 kB of memory.
      Processor Board ID FDO23040CS1
    
      Device name: dbm01sw-rocea0
      bootflash: 115805356 kB
    Kernel uptime is 17 day(s), 20 hour(s), 50 minute(s), 25 second(s)
    
    Last reset at 188268 usecs after Mon Aug 12 17:14:40 2019
      Reason: Module PowerCycled
      System version:
      Service: HW check by card-client
    
    plugin
      Core Plugin, Ethernet Plugin
    
    Active Package(s):

8.9.3 Downgrading RoCE Network Fabric Switch Firmware

Downgrading firmware means reapplying the older firmware update, which is shipped with the Oracle Exadata Database Machine storage server update.

Note:

The current storage server update determines what release you can downgrade to. This may be different for each release and may not be the firmware you were on before the update. For more information on the older firmware shipped with the release you are updating to, see the patch README file.

Before starting this task, complete the steps in Preparing for RoCE Network Fabric Switch Firmware Upgrades or Downgrades.

  1. Use the patchmgr command to downgrade the firmware on the RoCE Network Fabric switches.
    # ./patchmgr --roceswitches roceswitches.lst --downgrade [--verify-config [yes|no]]
      [-log_dir {absolute_path_to_log_directory | AUTO}] [--force]

    In the patchmgr command:

    • --verify-config specifies whether to verify the switch configuration against the golden configuration. Verification is performed by default if --verify-config is not specified.

    • -log_dir specifies the absolute path to the log directory, or AUTO instructs patchmgr to automatically create the log directory. This option is required when running patchmgr as a non-root user.

    • --force optionally proceeds with the operation even if the switch is already on the target firmware version or the RoCE Network Fabric is experiencing non-critical failures.

    Note:

    Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.
  2. Use the show version command to verify that the switch firmware is downgraded.
    # show version
    Cisco Nexus Operating System (NX-OS) Software
    TAC support: http://www.cisco.com/tac
    Copyright (C) 2002-2019, Cisco and/or its affiliates.
    All rights reserved.
    ...
    
    Software
      BIOS: version 05.33
      NXOS: version 7.0(3)I7(6)
      BIOS compile time: 09/08/2018
      NXOS image file is: bootflash:///nxos.7.0.3.I7.6.bin
      NXOS compile time: 3/5/2019 13:00:00 [03/05/2019 22:04:55]
    
    Hardware
      cisco Nexus9000 C9336C-FX2 Chassis
      Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 24571632 kB of memory.
      Processor Board ID FDO23040CS1
    
      Device name: dbm01sw-rocea0
      bootflash: 115805356 kB
    Kernel uptime is 17 day(s), 20 hour(s), 50 minute(s), 25 second(s)
    
    Last reset at 188268 usecs after Mon Aug 12 17:14:40 2019
      Reason: Module PowerCycled
      System version:
      Service: HW check by card-client
    
    plugin
      Core Plugin, Ethernet Plugin
    
    Active Package(s):

8.10 Updating InfiniBand Network Fabric Switch Firmware

You use the same update utility that is shipped with the storage server update to update and downgrade the InfiniBand Network Fabric switches.

The minimum switch firmware release that can use the update utility is release 1.3.3-2. Switch firmware is always updated in a rolling manner.

Note:

The patchmgr used for updating InfiniBand Network Fabric switch firmware is not the same as that used for the Oracle Exadata Database Machine database server software update.

Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. RDMA Network Fabric switch firmware upgrade requires switch reboot, which disrupts some connections on the RDMA Network Fabric network.

8.10.1 Preparing for InfiniBand Network Fabric Switch Firmware Updates

You must follow a specific order when updating the InfiniBand Network Fabric switches.

  • If a spine switch is present in the rack, it needs to be updated first.
  • If a spine switch is not in the rack, then update the switch running the subnet manager first.
  • If the subnet manager is not running on the switches, then perform the update in any order.

To update the InfiniBand Network Fabric switches, the switch firmware must be at release 1.3.3-2 or later. If the switch firmware is at an earlier release, then update the firmware to release 1.3.3-2 using the instructions in My Oracle Support note 888828.1. :

  1. Log in as the root user to an Oracle Exadata Database Machine database server that has root user SSH access to the switches.
    The database server must be on the same InfiniBand Network Fabric network as the switches.
  2. Use the version command to determine the version of the software the InfiniBand Network Fabric switch is running.
    [root@dbm01-ibs0 ~]# version
    
    SUN DCS 36p version: 2.1.8-1
    Build time: Sep 18 2015 10:26:47
    SP board info:
    Manufacturing Date: 2015.07.01
    Serial Number: "NCDLD0049"
    Hardware Revision: 0x0200
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010
    
  3. Download the appropriate patch file to the database server.
    Refer to My Oracle Support note 888828.1 for patch information.
  4. Uncompress the update.
    The files are uncompressed to the patch_release.date directory.
  5. Create a file listing all the InfiniBand Network Fabric switches that need to be updated, with one switch per line.

    Use the command ibswitches to identify the switches in your rack. Note that switches from Engineered Systems other than Oracle Exadata Database Machine might be visible on the same fabric but should probably not be updated at this time.

    The following is an example of the file constructed after running the ibswitches command:

    [root@dm01 ]# cat ibswitches.lst
    myibswitch-01
    myibswitch-02

    Note:

    If no file name is provided, then the command will be executed on all InfiniBand Network Fabric switches discovered from this host by running ibswitches command.
  6. Change to the patch_release.date directory.
  7. Run the prerequisite check.
    [root@dm01 ]# ./patchmgr --ibswitches ibswitches.lst --upgrade --ibswitch_precheck [--force] [--unkey]
    

    Note:

    Starting with Oracle Exadata System Software release 19.3.0, the patchmgr command uses -- instead of a single hyphen before keywords.

    The --unkey option removes passwordless SSH access to the InfiniBand Network Fabric switches before exiting.

    The --force option overrides failures in the InfiniBand Network Fabric topology and connectivity from the servers to the switches. This does not affect the upgrade of the switch.

If the output from the command shows overall status is SUCCESS, then proceed with the upgrade. If the output from the command shows overall status is FAIL, then review the error summary in the output to determine which checks failed, and then correct the errors. After correcting all the errors, rerun the prerequisite checks until it is successful.

8.10.2 Updating InfiniBand Network Fabric Switch Firmware Software

Update the InfiniBand Network Fabric switches using the patchmgr command.

The minimum switch firmware release that can use the patchmgr utility is release 1.3.3-2. The switch firmware is upgraded in a rolling manner.

  • If a spine switch is present in the rack, then the spine switch is upgraded first.
  • If a spine switch is not in the rack, then upgrade the switch that is running the Subnet Manager first.
  • If the Subnet Manager is not running on the switches, then perform the upgrade in any order.

You should have completed the steps in Preparing for InfiniBand Network Fabric Switch Firmware Updates before starting this task.

  1. Log in as the root user to a database server in the Oracle Exadata Rack that has root user SSH access to the switches.
    The database server must be on the same InfiniBand Network Fabric network as the switches.
  2. Download the appropriate patch file to the database server.
    Refer to My Oracle Support note 888828.1 for patch information.
  3. Uncompress the patch files.
    The files are uncompressed to the patch_release.date directory.
  4. Change to the patch_release.date directory.
  5. Run the prerequisite checks using patchmgr.

    The --unkey option removes passwordless SSH access to the InfiniBand Network Fabric switches before exiting.

    The --force option overrides failures in the InfiniBand Network Fabric topology and connectivity from the servers to the switches. This does not affect the upgrade of the switch.

    # ./patchmgr --ibswitches ibswitches.lst --upgrade --ibswitch_precheck [--force] [--unkey]
    

    Note:

    Starting with Oracle Exadata System Software release 19.3.0, the patchmgr command uses -- instead of a single hyphen before keywords.

    If the output from the command shows overall status is SUCCESS, then proceed with the upgrade.

    If the output from the command shows overall status is FAIL, then review the error summary in the output to determine which checks failed, and then correct the errors.

    After the errors have been corrected, rerun the prerequisite checks until it is successful.

  6. Upgrade the switches using the following command:
    [root@dm01 ]# ./patchmgr --ibswitches ibswitches.lst --upgrade [--force] [--unkey]
  7. Check the output from the command, and verify the upgrade.

    The output should show SUCCESS. If there are errors, then correct the errors and run the upgrade command again.

8.10.3 Downgrading InfiniBand Network Fabric Switch Firmware

Downgrading firmware means reapplying the older firmware update, which is shipped with the Oracle Exadata System Software update.

The current Oracle Exadata System Software update determines what release you can downgrade to. This may be different for each release and may not be the firmware you were on before the update. For more information on the older firmware shipped with the release you are updating to, see the patch README file.

Complete all steps in this task as the root user.

  1. Run the patchmgr command with the --precheck option to verify the switch firmware is ready to be downgraded.

    The ibswitches.lst file is a file that contains the host names of all the InfiniBand Network Fabric switches that need to be updated, with one switch per line.

    # ./patchmgr --ibswitches ibswitches.lst --downgrade --ibswitch_precheck [--force] [--unkey]

    Note:

    Starting with Oracle Exadata System Software release 19.3.0, the patchmgr command uses -- instead of a single hyphen before keywords.

    The --unkey option removes passwordless SSH access to the InfiniBand Network Fabric switches before exiting.

    The --force option overrides failures in the InfiniBand Network Fabric topology and connectivity from the servers to the switches. This does not affect the upgrade of the switch.

  2. Use the patchmgr command to downgrade the firmware on the InfiniBand Network Fabric switches.
    # ./patchmgr --ibswitches ibswitches.lst --downgrade [--force] [--unkey]
  3. Use the version command to verify the firmware on the switch has been downgraded.
    # version
    
    SUN DCS 36p version: 2.1.8-1
    Build time: Sep 18 2015 10:26:47
    SP board info:
    Manufacturing Date: 2015.07.01
    Serial Number: "NCDLD0049"
    Hardware Revision: 0x0200
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010

8.11 Upgrading Oracle Java SE on Oracle Linux

You can upgrade Oracle Java SE (JDK) running Oracle Linux 6 on database servers and storage servers.

Starting with Oracle Exadata System Software release 12.1.2.1.0, the Oracle Exadata Database Machine servers include the Java JDK package. Earlier releases of Oracle Exadata System Software do not use the JDK package. If a server running an earlier release of Oracle Exadata System Software has a package installed, such as java-version-openjdk, the package is not used by Oracle Exadata Database Machine and can be removed. See My Oracle Support note 1405320.1 for details.

To update the JDK package, you must download and update the JDK RPM package, and then reconfigure MS to use the new JDK package. You can update the JDK package by either by configuring YUM to use ULN (currently available only for Oracle Linux 6) or by direct package download (for those not able to use ULN).

8.11.1 Stop the MS Process

Before updating the JDK package, you must stop the Management Server (MS)

Note:

These steps are only applicable to on-premise deployments and management domains (dom0). Guest domains (domUs) do not have MS installed.
  1. Log in as the root user to the server.
  2. Stop MS.
    • For database servers:
      dbmcli -e alter dbserver shutdown services ms
    • For storage servers:
      cellcli -e alter cell shutdown services ms

8.11.2 Download and Update the Java JDK Package

Depending on your current environment, you can use one of three procedures to download and update the JDK package.

Using YUM and Unbreakable Linux Network (ULN) is only supported for Oracle Exadata Database Machine on-premise database servers running Oracle Linux 6. It is not supported for storage servers or Oracle VM environments, either management domains (dom0) or user domains (domU).

Use one of the following methods to download and update the JDK package:

8.11.2.1 Using YUM and ULN to Update the JDK Package on Database Servers

If the database servers for your on-premise Oracle Exadata Database Machine are running Oracle Linux 6, then you can use YUM and Unbreakable Linux Network (ULN) to simplify the update process.

You can either download the JDK package directly, or use YUM and ULN. Using YUM and ULN for updating the JDK package involves the following tasks:

8.11.2.1.1 Configuring YUM to Connect to the Oracle Public Repository

The Oracle Linux YUM server offers a free and convenient way to install the latest Oracle Linux packages. Unbreakable Linux Network (ULN) enables you to update patches and errata for a specific version.

This procedure is only supported for Oracle Exadata Database Machine on-premise database servers running Oracle Linux 6. It is not supported for storage servers or Oracle VM environments, either management domains (dom0) or user domains (domU).

Oracle Linux Yum Server is the Public Repository. For more information on how to obtain updates from the Oracle Linux Yum Server, see http://yum.oracle.com. For more information about ULN, see http://linux.oracle.com.

  1. As the root user on a database server, verify that the RHNS-CA-CERT certificate is not expired.
    # yum list installed

    If the certificate is expired, refer to My Oracle Support Doc ID 2207336.1 to fix the issue.

  2. Switch to the /etc/yum.repos.d directory.
    # cd /etc/yum.repos.d
  3. Download the repository configuration file.

    Use the curl utility to download the repository configuration file that is appropriate for your system. You can alternatively copy the contents of http://yum.oracle.com/public-yum-release.repo into a file, where release corresponds to the Oracle Linux release, for example ol6.

    For example, to get the configuration for the Oracle Linux 6 repository using curl, use the following command:

    # curl -O http://yum.oracle.com/public-yum-ol6.repo
    for more information.
  4. On a database server, install the rhn-setup package to enable uln_register.

    WARNING:

    Do not register your storage servers with ULN or the public yum server.
    # yum install rhn-setup.noarch

WARNING:

If you are not using the latest release of Oracle Linux, you will need to edit the file /etc/yum.repos.d/public-yum-ol6.repo and enable the correct repository to match the system version. Contact Oracle Support Services and refer to My Oracle Support Doc ID 2241729.1.
8.11.2.1.2 Registering an Oracle Linux 6 System with ULN

After you have configured the access to the YUM repository, you must register your Oracle Linux database servers with ULN.

WARNING:

Do not register your storage servers with ULN or the public yum server.
  1. Run uln_register to register your system with ULN.
    # uln_register

    If you do not have a ULN account, you can register at https://linux.oracle.com. Registering for ULN requires a valid customer support identifier (CSI) for Oracle Linux or Oracle VM support.

    When you register your system, if a proxy server is required, then use the ––proxy option to specify the HTTP proxy to use.

    # uln_register --proxy=proxy_hostname:port_number

    If your proxy requires authentication, then use the additional options --proxyUser and --proxyPassword to specify the user name and password.

    # uln_register --proxy=proxy_hostname:port_number 
    --proxyUser=username --proxyPassword=password
  2. When prompted, enter your ULN user name, password, and CSI.
  3. On the Create Profile — Hardware page, enter the required information.
    1. Enter a name for the system that will allow you to identify it on ULN.
    2. Choose whether to upload hardware and software profile data that allows ULN to select the appropriate packages for your system.
8.11.2.1.3 Upgrading JDK on Database Servers Using ULN
Before starting this step, make sure you have completed the steps in Stop the MS Process.

After you have configured the YUM repository and registered your database server with ULN, you can download the RPM from the ULN channel.

For Oracle Exadata System Software versions 12.1.2.1.0 to 12.1.2.2.0, the Oracle Exadata Database Server includes the JDK 7 package installed as an RPM. To update the RPM, use the ULN channel Java SE 7 for Oracle Linux. Make sure you look for JDK 7--do not use JDK 8 or later.

For Oracle Exadata System Software versions 12.1.2.2.1 and later, the Oracle Exadata Database Server includes the JDK 8 package installed as an RPM. To update the RPM, use the ULN channel Java SE 8 for Oracle Linux. Make sure you look for the JDK 8.

WARNING:

Do not register your storage servers with ULN.
  1. With a web browser, log in to https://linux.oracle.com/.
  2. Find the Oracle Exadata Database Server on which you want to install Oracle Java SE and click on the name of that system.

    In the following image, the server names have been blanked out.


    Description of uln_registered_systems.jpg follows
    Description of the illustration uln_registered_systems.jpg
  3. Click in Manage Subscriptions.
  4. Using the arrow buttons, move the desired Java SE channel from the Available channels list to the Subscriber Channels list, then click Save Subscriptions.
  5. When prompted, click Accept to accept the license agreement.
  6. On the database server, from the yum.repos.d directory, upgrade the JDK.
    # yum check-update jdk
    Loaded plugins: rhnplugin, ulninfo
    This system is receiving updates from ULN.
    ol6_x86_64_JavaSE7_public
    By downloading and/or using this software program you agree that your use is 
    subject to the applicable license agreement at https://linux.oracle.com
    /licenses.html.
    
    exadata_dbserver_18.1.4.0.0_x86_64_base                        | 1.2 kB   00:00
    exadata_dbserver_18.1.4.0.0_x86_64_base/primary                | 436 kB   00:00
    exadata_dbserver_18.1.4.0.0_x86_64_base                                     470/470
    ol6_x86_64_JavaSE7_public                                      | 1.2 kB   00:00
    ol6_x86_64_JavaSE7_public/primary                              | 5.9 kB   00:00
    ol6_x86_64_JavaSE7_public                                                     16/16
    ol6_x86_64_ksplice                                             | 1.2 kB   00:00
    ol6_x86_64_ksplice/primary                                     | 160 kB   00:00
    ol6_x86_64_ksplice                                                        1436/1436
    public_ol6_latest                                                       40132/40132
    ...
    Running transaction
      Installing : jdk1.8.0_172-1.8.0_172-fcs.x86_64                                        1/1 
    Unpacking JAR files...
    	rt.jar...
    	jsse.jar...
    	charsets.jar...
    	localedata.jar...
    	jfxrt.jar...
      Verifying  : jdk1.8.0_172-1.8.0_172-fcs.x86_64                                       1/1 
    
    Installed:
      jre.x86_64 0:1.8.0_172-fcs                                                
    
    Complete!
To complete the JDK update, continue with the steps in Reconfigure Management Server (MS).
8.11.2.2 Manually Update the JDK Package on Oracle Exadata System Software versions 12.1.2.1.0 to 12.1.2.2.0

Update the JDK 7 package to the latest release by downloading the latest version of the package and using the rpm utility to install it.

Before starting this step, make sure you have completed the steps in Stop the MS Process.

With Oracle Exadata System Software versions 12.1.2.1.0 to 12.1.2.2.0, the Oracle Exadata Storage Server includes the JDK 7 package installed as an RPM.

Following any upgrades to the server image (using dbnodeupdate or patchmgr), check the JDK version. If the JDK package is reverted to an older version during the upgrade, then use the procedures here to update the JDK package to the latest version.

  1. Download the latest version of JDK 7 using the links found in My Oracle Support Doc ID 1439822.1. Do not download JDK 8 or later.
  2. Extract the contents of the ZIP file.
  3. Locate the JDK RPM.
    The name of the file is similar to jdk-version-linux-x64.rpm, for example jdk-7u91-linux-x64.rpm.
  4. Copy only the RPM file to the target server.
    The file can be placed in a temporary directory, such as /tmp.
  5. As the root user, determine the current version of the installed JDK RPM.
    # rpm -q jdk
  6. If the JDK package is installed and needs to be updated, then use the rpm command to install the update.
    # rpm -Uvh /tmp/jdk-version-linux-x64.rpm
  7. Verify the JDK package was updated.
    # rpm -q jdk
  8. Removed the staged update file.
    # rm -f /tmp/jdk-version-linux-x64.rpm
To complete the JDK update, continue with the steps in Reconfigure Management Server (MS).
8.11.2.3 Manually Update the JDK Package on Oracle Exadata System Software Release 12.1.2.2.1 and Later

Update the JDK 8 package to the latest release by downloading the latest version of the package and using the rpm utility to install it.

Before starting this step, make sure you have completed the steps in Stop the MS Process.

With Oracle Exadata System Software release 12.1.2.2.1 and later, the Oracle Exadata servers includes the JDK 8 package installed as an RPM.

Following any upgrades to the server image (using dbnodeupdate or patchmgr),check the JDK version. If the JDK package is reverted to an older version during the upgrade, then use the procedures here to update the JDK package to the latest version.

  1. Download the latest version of JDK 8 using the links found in My Oracle Support Doc ID 1439822.1. Download only JDK 8 updates.
  2. Extract the contents of the ZIP file.
  3. Locate the JDK RPM.
    The name of the file is similar to jdk-version-linux-x64.rpm, for example jdk-8u172-linux-x64.rpm.
  4. Copy only the RPM file to the target server.
    The file can be placed in a temporary directory, such as /tmp.
  5. As the root user, determine the current version of the installed JDK RPM.
    # rpm -qa|grep jdk
    jdk1.8.0_66-1.8.0_66-fcs.x86_64
  6. If the JDK package is installed and needs to be updated, then use the rpm command to install the update.
    # rpm -Uvh /tmp/jdk-version-linux-x64.rpm
  7. Verify the JDK package was updated.

    With JDK 8, the updated package does not replace the currently installed package, so you will see two version of the JDK package installed.

    # rpm -qa | grep jdk
    jdk1.8.0_66-1.8.0_66-fcs.x86_64
    jdk1.8.0_172-1.8.0_172-fcs.x86_64
  8. Remove the older JDK package from the server.
    If the older version was update 66, then the command would be as follows:
    # rpm -e --nodeps jdk.1.8.0_66-1.8.0_66-fcs.x86_64
  9. Verify only the updated JDK is available on the server.
    # rpm -qa |grep jdk
    jdk1.8.0_172-1.8.0_172-fcs.x86_64
  10. Removed the staged update file.
    # rm -f /tmp/jdk-version-linux-x64.rpm
To complete the JDK update, continue with the steps in Reconfigure Management Server (MS).

8.11.3 Reconfigure Management Server (MS)

After you update the Java JDK package, you must reconfigure the MS processes to use the updated JDK version.

These steps are only applicable on physical deployments and management domains (dom0) on virtual deployments because user domains (domUs) do not have MS installed.

  1. Log in to the database server or storage server as root.
  2. Verify the MS process is still stopped.
    • For database servers:
      dbmcli -e alter dbserver shutdown services ms
    • For storage servers:
      cellcli -e alter cell shutdown services ms
  3. For Oracle Exadata System Software release 12.2.x, version 12.2.1.1.4 or later, or Oracle Exadata System Software release 18c, version 18.1.2 or later, perform the following steps (not needed for other releases):
    1. Go to the UNIX scripts directory.
      • For database servers:
        cd /opt/oracle/dbserver/dbms/deploy/scripts/unix
      • For storage servers:
        cd /opt/oracle/cell/cellserv/deploy/scripts/unix
    2. Redeploy MS.
      • For database servers:
        sh setup_dynamicDeploy DB
      • For storage servers:
        sh setup_dynamicDeploy
  4. Restart the MS.
    • For database servers:
      dbmcli -e alter dbserver startup services ms
    • For storage servers:
      cellcli -e alter cell startup services ms

8.12 Setting up SSH Equivalence

Before updating software on your Oracle Exadata Database Machine, you must configure SSH equivalence.

You can run the Exadata update utilities for Oracle Exadata Database Server, Oracle Exadata Storage Server, and the RDMA Network Fabric switch as either root or as a non-root user from any server running Oracle Linux. The utility can perform precheck, update, and rollback actions on any Exadata server as long as SSH equivalence is set up for the root user for the target Exadata server.

  1. Prepare a file named cell_group or dbs_group that has one storage server or database server host name or IP address per line for each storage server or database server you want to update.
  2. Check for existing SSH equivalence.
    The following command should require no password prompts and no interaction. It should return the list of host names in the cell_group file.
    [oracle@nonExadataHost ]# ./dcli -g cell_group -l root 'hostname -i'
  3. Set up SSH equivalence if not already done so from the launch server.
    Do not do this step if you already have root SSH equivalence.
    Generate SSH keys using the following command:
    [oracle@nonExadataHost ]# ssh-keygen [-t rsa]

    You can use the -t option to specify the key type, such as RSA or DSA. If you do not include the -t option, then RSA is configured by default.

    Accept the defaults so that the SSH keys are created for the root user.

  4. Push the SSH keys to set up SSH equivalence.
    Enter the root password when prompted.
    [oracle@nonExadataHost ]# dcli -g cell_group -l root –k

Note:

Customers in secure environments may have chosen to disable SSH access to Oracle Exadata Storage Server. During normal operations, Oracle Exadata Storage Server does not require SSH access. However, administrative utilities such as the update utility require SSH access. See the subsection "Unlocking a Cell Temporarily" in the topic Disabling SSH on Storage Servers for information on unlocking storage servers.

8.13 Troubleshooting Software Updates on Oracle Exadata Database Machine

Review these topics if you encounter errors or problems when updating the software on Oracle Exadata Database Machine

8.13.1 Troubleshooting Exadata Database Server Updates

You can use the log files generated by the update utility to troubleshoot updates.

The update utility orchestrates updating the Exadata database servers. Updating database nodes with the patchmgr tool is less verbose because it prints only minimal information to the screen. If additional information is required, you can view the patchmgr logs and the dbnodeupdate.sh logs that patchmgr copies over from other servers, if available. The log file (dbnodeupdate.log) and the diag file (dbnodeupdate.<runid>.diag) will eventually exist on two locations:

  • On each updated database server, in the /var/log/cellos directory
  • Consolidated on the node running the update utility.

On the node running the update utility, if the -log_dir flag was set to “auto”, the log files will be stored in the log/<directory based on contents of nodes in list file> directory, relative from the directory where the update utility is started from. For example, if the update utility is located in /u01/dbserver.patch, then the log directory may be /u01/dbserver.patch/dm01db01_dm01db02_e8f1f753.

Important files found in the log directory are:

  • patchmgr.log contains the consolidated screen output from running the remote update commands on the different database servers.
  • <hostname>_dbnodeupdate.<runid>.diag is the diag file for the specific run on a database server.
  • <hostname>_dbnodeupdate.log contains dbnodeupdate.log output appended from /var/log/cellos from the remote database server.

When a prerequisite check, backup, update, or rollback fails, error messages on screen should provide information on which step failed on which node. Consult the log files mentioned above if more information is required. Search the log file for the start of a new run (search for “zzz”).

Check if the time matches your run. If it matches, note the runid for further reference. Then search for ERROR.

If an update action fails before the actual YUM update, you can retry the update after resolving the error. If the update aborted half way, it is recommended that you roll back, resolve the error, and retry.

In rare cases, patchmgr may be unable to determine the status of an update, whether the update was successful or not. In such cases, it displays a message that the update failed. However, it is possible that the update still completed successfully. To determine the actual status of the update:

  • Check the image status of the (database) node. You can do this by running the imageinfo command. The Image status line displays the status.
  • Check the version of the Exadata software. This can also be determined from the imageinfo command.

If the image status is success, and the Exadata version is the new expected version, then the update was successful and you can ignore the update failed message. Then:

  • Run dbnodeupdate.sh -c manually on the particular node to perform the completion steps of the update.
  • Remove the completed node from the (database) node file.
  • Rerun patchmgr to perform the update on the remaining nodes.

Other things to check if the update fails include:

  • The correct syntax for using patchmgr to update database nodes can be found in the patchmgr online help.
  • SSH equivalence must be configured before using patchmgr.
  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.
  • Open a service request with Oracle Support Services to analyze why the patchmgr orchestration failed.

8.13.2 Monitoring, Validating, and Troubleshooting Exadata Storage Server Updates

The recommended way to monitor Exadata storage servers being updated is through e-mail alerts. These alerts can be set up by specifying email addresses in the -smtp_from and -smtp_to flags in the update utility.

If needed for troubleshooting, you can monitor update activity using less -rf patchmgr.stdout from another terminal session or window to see raw log details from the update utility.

You can also monitor activity of the storage server by logging in to the serial console or web-based ILOM console of individual storage servers being updated 5 minutes after the update utility has started. Waiting 5 minutes allows the update utility time to reset the ILOM. Resetting the ILOM disconnects you from the ILOM web console and serial console. You can reconnect once the ILOM has been reset. By waiting 5 minutes, you avoid having to reconnect. You lose the connection during any ILOM update, and need to reconnect. The ILOM does not show any update actions. When needed it is helpful to monitor the activities of the normal cell boot, reboot, and other activities to ensure that the process is proceeding correctly.

Verify the update status after the patchmgr utility completes as follows:

Check image status and history using the imageinfo and imagehistory commands on each cell. A successful update to a system with high capacity or high performance drives shows output similar to the following.

Kernel version: 2.6.39-400.281.1.el6uek.x86_64 #1 SMP Fri Jun 17 20:10:16 PDT 2016 x86_64
Cell version: OSS_12.1.2.3.2_LINUX.X64_160721
Cell rpm version: cell-12.1.2.3.2_LINUX.X64_160721-1.x86_64

Active image version: 12.1.2.3.2.160721
Active image kernel version: 2.6.39-400.281.1.el6uek
Active image activated: 2016-07-21 13:04:34 -0500
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

Cell boot usb partition: /dev/sdac1
Cell boot usb version: 12.1.2.3.2.160721

Inactive image version: 12.1.2.3.1.160411
Inactive image activated: 2016-04-11 19:58:28 -0700
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8

Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
Inactive kernel version for the rollback: 2.6.39-400.128.21.el5uek
Rollback to the inactive partitions: Possible

A successful update to a system with extreme flash drives shows a similar output but has a different path for USB:

Kernel version: 2.6.39-400.281.1.el6uek.x86_64 #1 SMP Fri Jun 17 20:10:16 PDT 2016 x86_64
Cell version: OSS_12.1.2.3.2_LINUX.X64_160721
Cell rpm version: cell-12.1.2.3.2_LINUX.X64_160721-1.x86_64

Active image version: 12.1.2.3.2.160721
Active image kernel version: 2.6.39-400.281.1.el6uek
Active image activated: 2016-07-21 13:04:34 -0500
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

Cell boot usb partition: /dev/sda1
Cell boot usb version: 12.1.2.3.2.160721

Inactive image version: 12.1.2.3.1.160411
Inactive image activated: 2016-04-11 19:58:28 -0700
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8

Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
Inactive kernel version for the rollback: 2.6.39-400.128.21.el5uek
Rollback to the inactive partitions: Possible

8.13.3 Troubleshooting a Failing Prerequisite Check Due to Dependency Problems on Exadata Database Server

When you run a prerequisite check or an update on Exadata database servers that have custom rpms installed, conflicts may cause the prerequisite check or update to fail.

When doing a prerequisite check or an update, the update utility makes two dependency checks: checks against “minimum dependencies” and checks against “exact dependencies”.

To triage a dependency failure, look in the log file. This can be either dbnodeupdate.log on the target Exadata database server or <hostname>_dbnodeupdate.log in the log directory from where the update utility (patchmgr) was launched.

To locate the start of each run in the log file, search for a line that starts with “zzz”. For example:

zzz - /u01/patches/YUM/dbnodeupdate.sh called with arguments -u -l
/u01/patches/YUM/p23564643_121232_Linux-x86-64.zip -v -N at  2016-08-23 23:31:54

The date stamp should match the time of the run you are researching. Each run is identified with a unique runid which can also be found at the start of each run in the same log file:

[1472009516][2016-08-23 23:31:59 -0400][INFO][/u01/patches/YUM/dbnodeupdate.sh][InitLogfile][]  # dbnodeupdate.sh script rel.
 : 5.160809 started at  (runid :230816233155)

You can check for custom rpms, if any, by looking at the diag file of the specific run. The diag file is identified by the runid and can be found in /var/log/cellos on the target Exadata database server or in the directory driving the update utility. The filename for this example would be dbnodeupdate.230816233155.diag. In that file, look for the section with heading RunDetectCustomRpmsSh and rpm -qa --qf "%{n}-%{v}-%{r}.%{arch}\n. You can find details on the (additional) packages installed (if any).

In the log file, search down from the start (“zzz”) for “Exact dependencies” (case sensitive) for the results of the “minimum” and “exact” dependency checks. In this case it says:

Exact dependencies     : Will fail on a next update
Minimum dependencies   : Will fail on a next update

As long as the minimum dependencies check passes, updates or prerequisite checks won’t fail. When both minimum and exact dependencies fail, you need to find out what caused the error.

In order to find the dependency that is causing the error, search for “[ExecUpgrade][] Performing yum package dependency” . This is where the YUM run (typically a dry-run first) is executed. When there is a dependency problem, you should see a YUM message starting with “Error:”. For example:

--> Finished Dependency Resolution
Error: Package: krb5-devel-1.10.3-33.el6.x86_64 (@ol6_latest)
Requires: krb5-libs = 1.10.3-33.el6
Removing: krb5-libs-1.10.3-33.el6.x86_64 (installed)
        krb5-libs = 1.10.3-33.el6
Updated By: krb5-libs-1.10.3-42z1.el6_7.x86_64 (exadata_generated_160616114412)
          krb5-libs = 1.10.3-42z1.el6_7
You could try using --skip-broken to work around the problem

In the example error message above, the krb5-devel-1.10.3-33.el6.x86_64 (krb5-devel) package is installed from a non-Exadata channel (ol6_latest). This krb5-devel package depends on krb5-libs. However in this Exadata update the krb5-libs package is not available. YUM fails the dependency check because updating krb5-libs is not possible without also updating krb5-devel. Because a new version of krb5-devel is not included in the Exadata update the package should be either “pre-updated” or removed. Pre-updating means updating the individual package manually before running the Exadata update. This can be done using the command rpm –Uvh <package-name>.

Removing the custom package is recommended and should be done using the following rpm command:

[root@dm01 ]# rpm –e krb5-devel

After removing the rpm that is causing the error, restart the update or prerequisite check.

8.13.4 Troubleshooting a Multilib Problem on Exadata Database Server

If you have custom packages with different architectures installed on database servers, you may see a similar problem as that described in Troubleshooting a Failing Prerequisite Check Due to Dependency Problems on Exadata Database Server. This typically happens when i686 packages are installed on an Exadata database server.

Note:

While non-64-bit (x86_64) packages are supported, it is recommended to stay away from 32-bit software. When you need specific functionality from a third party, it is recommended to ask for a 64-bit version.

Typically Exadata-branded x86_64 bit rpms are updated in an Exadata update. However, when you have installed similar packages of non-x86_64 bit architecture, the update utility cannot update the 64-bit packages. You would see the following error in the log file:

--> Finished Dependency Resolution
    Error:  Multilib version problems found. This often means that the root
           cause is something else and multilib version checking is just
           pointing out that there is a problem. Eg.:
           
             1. You have an upgrade for libuuid which is missing some
                dependency that another package requires. Yum is trying to
                solve this by installing an older version of libuuid of the
                different architecture. If you exclude the bad architecture
                yum will tell you what the root cause is (which package
                requires what). You can try redoing the upgrade with
                --exclude libuuid.otherarch ... this should give you an error
                message showing the root cause of the problem.
           
             2. You have multiple architectures of libuuid installed, but
                yum can only see an upgrade for one of those arcitectures.
                If you don't want/need both architectures anymore then you
                can remove the one with the missing update and everything
                will work.
           
             3. You have duplicate versions of libuuid installed already.
                You can use "yum check" to get yum show these errors.
           
           ...you can also use --setopt=protected_multilib=false to remove
           this checking, however this is almost never the correct thing to
           do as something else is very likely to go wrong (often causing
           much more problems).
           
           Protected multilib versions: libuuid-2.17.2-12.24.0.1.el6.x86_64 != libuuid-2.17.2-12.18.0.1.el6.i686

The solution for multilib problems is to remove the i686 or i386 package by running “rpm –e <package_name.i686>”.