1. Maintenance Guide
  2. Updating Exadata Software

8 Updating Exadata Software

There are different types of software used on an Oracle Exadata that need be updated regularly.

8.1 About Updating Exadata Software

Exadata software updates apply to three major components:

  • Exadata storage servers
  • Exadata database servers
  • Exadata RDMA Network Fabric switches

Exadata storage server and Exadata database server updates generally contain updates for:

  • Oracle Linux operating system
  • Oracle Exadata System Software
  • Firmware (for example: disk, flash, RAID controller, ILOM, HCA)

The updates do not modify the Oracle Grid Infrastructure home, Oracle Database home (other than relinking during the dbnodeupdate.sh -c step), or customer-installed software.

While it is generally recommended for components to stay in step with the recommended minimum release, you can choose to update different components at different times. For example, you could update RDMA Network Fabric switches at a later time than Exadata storage servers and Exadata database servers. However, you must check My Oracle Support Doc ID 888828.1 for any dependencies.

It is not mandatory to apply each and every Oracle Exadata System Software update that comes out. For example, you can skip two or three releases and update directly to a newer release. Oracle recommends that you update database servers twice a year.

Upgrading is allowed under the following circumstances:

  1. The product version of the target release is higher than the installed software, and
  2. The date code of the target release is higher than the installed software.

For example, consider a system that is currently running image version 12.1.1.1.2.150411, which consists of product version 12.1.1.1.2 and date code 150411.

  • Upgrading to 12.1.2.1.2.150617.1 is allowed because both rules are satisfied.
  • Upgrading to 12.1.2.1.1.150316.2 is not allowed because date code 150411 of the installed software is higher than date code 150316 of the target release.

Related Topics

Parent topic: Updating Exadata Software

8.1.1 About Upgrading to Oracle Linux 8 on Exadata Servers

When you upgrade to Oracle Exadata System Software release 23.1.0, the Oracle Linux version is upgraded to Oracle Linux 8 with Unbreakable Enterprise Kernel (UEK) 6.

Unbreakable Enterprise Kernel (UEK) 6 is required to support the AMD processor architecture introduced in Oracle Exadata X10M models.

Parent topic: About Updating Exadata Software

8.1.2 About Upgrading to Oracle Linux 7 on Exadata Servers

When you upgrade to Oracle Exadata System Software release 19.1.0, the Oracle Linux version is upgraded from Oracle Linux 6 to Oracle Linux 7.

Oracle Exadata System Software release 19.1.0 provides support for Oracle Database 19c, which runs on Oracle Linux 7. When you upgrade to Oracle Exadata System Software release 19.1.0 the operating system is upgraded at the same time.

If you use quorum disks on your Exadata Database Machine, then refer to My Oracle Support note 2453054.1.

Related Topics

Parent topic: About Updating Exadata Software

8.2 Planning for Software Maintenance

Before starting a software update, you should review best practices, determine the version to which you will upgrade, and obtain the proper patching software.

Parent topic: Updating Exadata Software

8.2.1 Understanding Exadata Database Machine Software and Updates

Understanding the different types of software updates required for Exadata Database Machine helps you plan an update schedule.

Parent topic: Planning for Software Maintenance

8.2.1.1 What Software Do You Update on Oracle Exadata?

The software that runs on Oracle Exadata is divided into two main categories.

  • Oracle Exadata System Software

  • Oracle Grid Infrastructure and Oracle Database software

The following table describes the two primary categories of software that run on an Oracle Exadata.

Software Components Installed On Software Content Example Software Version

Oracle Exadata System Software
  • Exadata storage servers
  • Exadata database servers
  • Exadata RDMA Network Fabric switches

 
  • Oracle Linux operating system
  • Oracle VM Server (on database servers for Oracle VM configurations)
  • Oracle Exadata System Software
  • Firmware (for example: disk, flash, RAID controller, ILOM, HCA, RDMA Network Fabric switch firmware)

Oracle Exadata System Software on storage servers and database servers:

  • 12.2.1.1.0
  • 12.1.2.3.4

InfiniBand Network Fabric switch firmware:

  • 2.2.4-3
  • 2.1.8-1

RoCE Network Fabric switch firmware:

  • 7.0(3)I7(6)

Oracle Grid Infrastructure and Oracle Database software

Exadata database servers
  • Oracle Grid Infrastructure (Grid home)
  • Oracle Database (Database home)

12.2.0.1.0

12.1.0.2.160117

11.2.0.4.161018

When an Oracle Exadata is deployed, all of the software described in the table is installed and configured to deliver high performance and availability for Oracle Database. Extensive end-to-end testing ensures all software components supplied with Oracle Exadata work seamlessly together.

My Oracle Support Doc ID 888828.1 is the primary source of information for software that runs on Oracle Exadata. It contains the following information:

  • List of all current and previous releases
  • Minimum requirements for feature usage
  • Compatibility requirements between Oracle Exadata System Software version and Oracle Database software version
  • Compatibility requirements for specific hardware releases
  • Guidelines for related products when used with Oracle Exadata
  • References to other pertinent information sources for Oracle Exadata software maintenance
8.2.1.2 Software Release Types

The following table describes the release types for Oracle Exadata System Software:

Release type Description and Content

Major Version

The major version identifies the major Oracle Linux Operating System (OS) version that underpins Oracle Exadata System Software. For example, ExaOL6X, ExaOL7X, ExaOL8X, and so on.

Software release

A software release contains new features, bug fixes, security fixes, and may include support for new generations of Exadata system hardware and new Oracle Database software releases.

Starting with the Oracle Exadata System Software release 18.1, an Exadata software release is represented using the first two digits in the release number. For example, 18.1, 19.1, 19.2, and so on.

Previously, the first four digits represent an Exadata software release. For example, 12.2.1.1.

A software release may be used to update any prior software release or maintenance release.

Maintenance release

A maintenance release contains bug fixes, security fixes, and may contain feature enhancements.

Starting with the Oracle Exadata System Software release 18.1, a maintenance release is represented using the third digit in the release number. For example, 23.1.2, 23.1.3, 23.1.4, and so on.

Previously, the fifth digit represents a maintenance release. For example, 12.1.2.3.4.

A maintenance release may be used to update any prior software release or maintenance release.

Interim patch

An interim patch is one-off bug fix made available to customers who cannot wait until the fix is included in a maintenance release or software release. Installation of an interim patch is done on an as-needed basis and is not a regularly scheduled planned maintenance event.
8.2.1.3 Software Release Availability

The following table outlines the release cycles for Oracle Exadata System Software, Oracle Grid Infrastructure, and Oracle Database software.

Software Release type Release Frequency Importance of Maintaining Up-to-Date Versions Recommendation for Adopting New Releases
Oracle Exadata System Software

Software Release

6-18 months

Medium

Update to adopt new features and get critical fixes and security updates.
Oracle Exadata System Software

Maintenance Release

Monthly

High

Update regularly to get critical fixes and security updates.

Oracle recommends that you remain within 12 months of the latest maintenance release.

Oracle Grid Infrastructure and Oracle Database software

Long Term Release

24+ months

Medium

Use long term releases for the highest level of stability and the longest length of error correction support.

Oracle Grid Infrastructure and Oracle Database software

Innovation Release

12-24 months

Medium

Use innovation releases to adopt new features as quickly as possible.

Oracle Grid Infrastructure and Oracle Database software

Release Update (RU)

3 months (quarterly)

High

Update quarterly to get critical fixes and security updates.

Oracle recommends that you remain within 12 months of the latest release update.
8.2.1.4 Software Update Frequency

You should plan to update your Exadata software on a regular basis.

These examples show three high-level software maintenance plans with quarterly updates over a four year cycle using typical intervals between releases. The examples illustrate different update strategies designed to meet specific goals.

Note:

Actual release frequencies may vary from these examples.

  • Example 8-1 Production System Software Maintenance Plan

  • Example 8-2 Production System Software Maintenance Plan with Reduced Updates

  • Example 8-3 Development and Test System Software Maintenance Plan

Example 8-1 Production System Software Maintenance Plan

Goal — Minimize risk by regularly applying critical and security fixes, adopting new feature releases only as required, and not adopting a new Exadata software release and new Oracle Database release simultaneously.

Action Y1 Q1 Y1 Q2 Y1 Q3 Y1 Q4 Y2 Q1 Y2 Q2 Y2 Q3 Y2 Q4 Y3 Q1 Y3 Q2 Y3 Q3 Y3 Q4 Y4 Q1 Y4 Q2 Y4 Q3 Y4 Q4 Y5 Q1

Update to latest Oracle Exadata System Software maintenance release

-

X

X

X

X

 -

X

X

X

X

 -

X

X

X

X

 -

X

Update to latest Oracle Exadata System Software release

X

 - - - -

X

 - - - -

X

 - - - -

X

 -

Update to latest Oracle Database Release Update (RU)

X

 -

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Update to latest Oracle Database innovation release

- - - - - - - - - - - - - - - - -

Update to latest Oracle Database long term release

-

X

 - - - - - - - - - - - - - - -

Example 8-2 Production System Software Maintenance Plan with Reduced Updates

Goal — Manage risk and reduce maintenance time by applying less frequent updates. Adopt new Oracle Database feature releases as required. Minimize risk by not adopting a new Exadata software release and new Oracle Database feature release simultaneously.

Action Y1 Q1 Y1 Q2 Y1 Q3 Y1 Q4 Y2 Q1 Y2 Q2 Y2 Q3 Y2 Q4 Y3 Q1 Y3 Q2 Y3 Q3 Y3 Q4 Y4 Q1 Y4 Q2 Y4 Q3 Y4 Q4 Y5 Q1

Update to latest Oracle Exadata System Software maintenance release

- - -

X

 -

X

 - - -

X

 -

X

 - - -

X

 -

Update to latest Oracle Exadata System Software release

-

X

 - - - - -

X

 - - - - -

X

 - - -

Update to latest Oracle Database Release Update (RU)

-

X

 - - -

X

 -

X

 -

X

 -

X

 -

X

 - - -

Update to latest Oracle Database innovation release

- - - - - - - - - - - - - - -

X

 -

Update to latest Oracle Database long term release

- - -

X

 - - - - - - - - - - - - -

Example 8-3 Development and Test System Software Maintenance Plan

Goal — Adopt the latest features and software updates on a quarterly basis.

Action Y1 Q1 Y1 Q2 Y1 Q3 Y1 Q4 Y2 Q1 Y2 Q2 Y2 Q3 Y2 Q4 Y3 Q1 Y3 Q2 Y3 Q3 Y3 Q4 Y4 Q1 Y4 Q2 Y4 Q3 Y4 Q4 Y5 Q1

Update to latest Oracle Exadata System Software maintenance release

-

X

X

X

X

 -

X

X

X

X

 -

X

X

X

X

 -

X

Update to latest Oracle Exadata System Software release

X

 - - - -

X

 - - - -

X

 - - - -

X

 -

Update to latest Oracle Database Release Update (RU)

-

X 

X

X

X

 -

X

X

X

X

 -

X

X

X

X

 -

X

Update to latest Oracle Database innovation release

- - - - -

X

 - - - -

X

 - - - - - -

Update to latest Oracle Database long term release

X

 - - - - - - - - - - - - - -

X

 -
8.2.1.5 Software Update Utilities

When updating software on Oracle Exadata, you use specific utilities depending on the component being updated.

Update Type Utility Components Description

Pre-update readiness check

Oracle EXAchk

Oracle Exadata

Health check tool used to qualify maintenance readiness, identify critical issue exposure, and provide version recommendations for updating software.

Obtain Oracle EXAchk from My Oracle Support Document 1070954.1.

Oracle Exadata System Software

patchmgr
  • Oracle Exadata storage servers
  • Oracle Exadata database servers
  • RDMA Network Fabric switches

Oracle Exadata software update tool that updates all software (Oracle Linux, Oracle Exadata System Software, firmware) on Oracle Exadata database servers, storage servers, or RDMA Network Fabric switches. For example, patchmgr updates Oracle Exadata storage servers and database servers to Oracle Exadata System Software version 12.2.1.1.0, or InfiniBand Network Fabric switches to version 2.2.4-3.

Oracle Grid Infrastructure and Oracle Database software release update

OPatch or opatchauto

Oracle Exadata database servers

Software update tool that applies a release update to an Oracle Grid Infrastructure home or Oracle Database home.

For example, OPatch applies the 12.1.0.2.170117 release update to 12.1.0.2 Oracle Grid Infrastructure and Oracle Database homes.

Obtain OPatch from My Oracle Support Document 274526.1.

Oracle Grid Infrastructure and Oracle Database software patch set, major release, or release update.

Oracle Universal Installer

Oracle Exadata database servers

Software installation tool that installs an Oracle Grid Infrastructure and Oracle Database patch set, major, or release update, and upgrades Oracle Grid Infrastructure.

For example, Oracle Universal Installer installs Oracle Database 12.2.0.1, or installs and upgrades Oracle Grid Infrastructure from 12.1.0.2 to 12.2.0.1.

Use the version of Oracle Universal Installer bundled with the Oracle Grid Infrastructure and Oracle Database release to which you are upgrading.

Upgrade of an Oracle Database

Oracle Database AutoUpgrade

Oracle Exadata database servers

Oracle Database upgrade utility that can upgrade one or many Oracle databases with one command and configuration file.

AutoUpgrade can run pre-upgrade tasks, perform automated corrections where needed, run the database upgrade, and complete post-upgrade tasks.

AutoUpgrade includes built-in scheduling capabilities, automatic retry and fallback, and the ability to set, change or remove initialization parameters during the upgrade.

Upgrade of an Oracle Database

Database Upgrade Assistant (DBUA)

Oracle Exadata database servers

Database tool that upgrades a database to a later patch set, major release, or release update. For example, DBUA upgrades a database from 12.1.0.1 to 12.1.0.2, or from 12.1.0.2 to 12.2.0.1.

Use DBUA bundled with the Oracle Grid Infrastructure and Oracle Database release to which you are upgrading.

Oracle Exadata System Software

Oracle Grid Infrastructure and Oracle Database software

Fleet Patching and Provisioning (previously named Rapid Home Provisioning (RHP))

  • Exadata storage servers
  • Exadata database servers
  • RDMA Network Fabric switches

Software maintenance tool that provisions, patches, and upgrades Oracle Grid Infrastructure and Oracle Database software on one or more clusters from a centralized server.

Use Fleet Patching and Provisioning as a feature of Oracle Clusterware.

Starting with Oracle Grid Infrastructure release 18c, in addition to Oracle Grid Infrastructure and Oracle Database homes, Fleet Patching and Provisioning supports patching the Oracle Exadata components: database nodes, storage cells, and RDMA Network Fabric switches.

Oracle Exadata System Software

Oracle Grid Infrastructure and Oracle Database software

Oracle Enterprise Manager Cloud Control
  • Exadata storage servers
  • Exadata database servers
  • RDMA Network Fabric switches

Oracle Enterprise Manager Cloud Control provides lifecycle management capability to apply software updates to Oracle Exadata.

Note: Facilities for updating Oracle Exadata System Software are deprecated in Oracle Enterprise Manager Cloud Control release 13.4.

8.2.2 Configuration and Operational Best Practices for Software Maintenance

As part of planning your software update, you should review the different methods of performing updates and the best practices for updating the software.

Parent topic: Planning for Software Maintenance

8.2.2.1 Understanding Rolling and Non-Rolling Updates

Software updates can be performed in rolling manner while the database remains online and available, or non-rolling manner where Oracle Clusterware and Oracle Database are shutdown.

The manner an update is performed does not affect how often it should be done, but does determine how long it will take. In general there are two methods:

  • Rolling software updates — A rolling software update is one that is performed to one of a particular component at a time while the others remain online servicing requests. Note the following key points about rolling software updates:

    • Rolling updates have less application downtime compared to non-rolling updates.

    • The overall length of time to complete the update is longer because one component is offline and updated at a time, while all other components remain online and operational.

    • The impact to existing database connections differs depending on the component being updated.

      • Rolling updates for Oracle Exadata storage servers or RDMA Network Fabric switches — All databases remain fully online and available for the duration of the rolling update. There is no disruption to database connections.

      • Rolling updates for Oracle Exadata database servers, Oracle Grid Infrastructure, or Oracle Database — Multi-instance databases using Oracle Real Application Clusters (Oracle RAC) remain available for the duration of the update. However, database connections on the server being updated will be disrupted when the local database instance is shutdown. Use Oracle RAC features for client high availability to minimize application disruption, as described in Table 8-1.

  • Non-Rolling software updates — A non-rolling software update is one that is performed to all of a particular component while those components are offline. Note the following key points about non-rolling software updates:

    • Non-rolling updates are faster than rolling updates for overall maintenance time because multiple components are updated in parallel.
    • Because all of a particular component are offline for the duration of the update, a database (or all databases on a system) will also be offline for the duration of the update, resulting in a complete outage to the application
    • Important applications serviced by the Oracle Exadata being updated in a non-rolling manner are often moved to a standby system in environments using Oracle Data Guard.

  • Combination of rolling and non-rolling software updates — When multiple components are updated in the same maintenance window, it is possible to use a combination of rolling and non-rolling methods to achieve the desired balance of application downtime and maintenance time. One typical combination used in the situation where an application does not handle connection disruption efficiently is to perform Oracle Exadata storage server and RDMA Network Fabric switch updates in a rolling manner, then performing Oracle Grid Infrastructure, Oracle Database, and Oracle Exadata database server updates in a non-rolling manner.

The patchmgr update utility manages the update orchestration in a rolling or non-rolling manner for Oracle Exadata infrastructure components (storage servers, database servers, and RDMA Network Fabric switches).

The following table describes which update method is supported for each component type.

Table 8-1 Rolling versus Non-Rolling Upgrades

Method Component Support Database Availability Impact During Update

Rolling

Oracle Exadata storage servers

No impact.

Oracle RAC and single instance databases remain fully available on all nodes in the cluster.

Rolling

RDMA Network Fabric switches

No impact.

Oracle RAC and single instance databases remain fully available on all nodes in the cluster.

Rolling

Oracle Exadata database servers

Local connections are disconnected and all local instances are shutdown.

Oracle RAC databases remain available through other nodes in the cluster.

Rolling

Oracle Grid Infrastructure

Local connections are disconnected and all local instances are shutdown.

Oracle RAC databases remain available through other nodes in the cluster.

Rolling

Oracle Database - Release Update (RU)

For the database home being updated, local connections are disconnected and local instances are shutdown.

Oracle RAC databases remain available through other nodes in the cluster.

Databases running from other Oracle Database software homes are unaffected.

Non-Rolling

  • Oracle Exadata storage servers

  • Oracle Exadata database servers

  • Oracle Grid Infrastructure

  • Oracle Database - Release Update (RU)

  • Oracle Database - Patch Set

  • Oracle Database - Release

Databases unavailable

Move workload to Oracle Data Guard or Oracle GoldenGate standby systems to minimize impact.

8.2.2.2 Online Updates for Oracle Linux Kernel and Oracle Database Interim Fixes

 Online updates of qualified fixes are supported for Oracle Linux and Oracle Database.

Typically, planned software updates require that the component be restarted after the update. For example, an Exadata database server must be restarted after applying an Exadata software release to update the system firmware and to make active a new Oracle Linux kernel. Similarly, an Oracle database instance must be stopped and restarted to apply a software update to the database home.

Oracle supplies some interim fixes, however, that can be applied online, such that a component does not require a restart to apply the fix and make them active. Online updates of qualified fixes are supported for the following components:

  • Oracle Linux kernel on Exadata database servers using the Ksplice Offline Client.

  • Oracle Database using OPatch Online Patching.

Online updates are typically performed as a temporary measure when a critical fix must be applied to the system before the next scheduled planned software maintenance.

8.2.2.3 Configuration Practices for Optimal Software Maintenance

When configuring an Exadata Database Machine, it is important to adopt features that will lessen the impact and risk of performing software updates.

Configuration Practice Use for Software Maintenance Purpose

Oracle ASM high redundancy disk groups

Rolling updates - Exadata storage servers

When performing storage server updates in a rolling manner with the databases remaining online, it is highly recommended to configure Oracle ASM disk groups with high redundancy. High redundancy disk groups can tolerate the failure of a disk in another storage server during rolling updates.

During rolling, or online, storage server updates, the disks for the storage server being updated are taken offline on one storage server by patchmgr while it is updated. After the update completes the disks are resynchronized by Oracle ASM, and then patchmgr starts to update the next storage server.

While disks are offline the disk group has reduced redundancy. A normal redundancy disk group with reduced redundancy during a rolling update may dismount and have data loss if a disk fails in another storage server.

Oracle ASM disk group redundancy is typically set during initial system configuration. Therefore, consider how you plan to perform storage server updates, rolling or non-rolling, prior to system configuration.

An Oracle Data Guard physical standby system can also provide protection against disk failure during rolling storage server update.

Oracle RAC features for client high availability

Rolling updates - Exadata database servers, Oracle Grid Infrastructure or Oracle Database software

When performing Exadata database server, Oracle Grid Infrastructure, or Oracle Database software updates in a rolling manner on one database server at a time while database services remain running on the other servers in the cluster, the database services must be stopped on the Exadata database server being updated.

To minimize the impact to client applications connected to database instances that will be stopped during maintenance, configure client applications to use database services, Fast Application Notification (FAN), Fast Connection Failover (FCF), and Application Continuity.

Oracle Data Guard physical standby database

Any rolling or non-rolling update (except Oracle Database patch set and release updates)

Oracle Data Guard Standby-First Patch Apply provides support for different Exadata infrastructure, Oracle Grid Infrastructure, and Oracle Database software between a primary database and its physical standby database for the purpose of applying and validating Oracle patches in rolling fashion with minimal risk to the primary database.

Oracle Data Guard transient logical standby database

Any rolling or non-rolling update

Use Oracle Data Guard logical standby database to reduce database upgrade downtime for patch set and release updates. Database upgrade downtime is reduced by allowing the logical standby database to be upgraded to the new version and kept synchronized while the primary database remains online running the current version.

Oracle GoldenGate

Any rolling or non-rolling update

Use Oracle GoldenGate to reduce database upgrade downtime for patch set and release updates. Downtime during a database upgrade is reduced by allowing the target database to be upgraded to the new version and kept synchronized while the source database remains online running the current version.
8.2.2.4 Operational Practices for Optimal Software Maintenance

The following operational practices enable optimal software maintenance:

  • Run Oracle EXAchk regularly — As a general health check tool to ensure an Exadata system continues to meet the current and constantly evolving best practices, run Oracle EXAchk monthly. The exachk report should be utilized as follows:

    • Baseline comparison - Compare the current report against an accepted baseline report using the report comparison feature (-diff option).

    • Critical issue exposure - Review the report for exposure to critical issues. Take prompt action to resolve critical issues reported by Oracle EXAchk.

    • Version recommendation - Review the report for version recommendation. The MAA Scorecard section evaluates current software versions for consistency, compatibility, and whether or not it is current.

  • Qualify maintenance readiness with Oracle EXAchk — Prior to performing software maintenance, run Oracle EXAchk to ensure the system is in a healthy state. Correct any FAIL or WARNING checks before updating any software. After software maintenance is complete, run Oracle EXAchk again to confirm system health.

  • Update software at regular intervals — All software should be updated regularly. Maintaining software at current or recent releases provides the following benefits: better software security, continued compatibility with newer related software, better support and faster resolution of issues, and ability to supply fixes for newly discovered issues.

  • Use the latest versions of the update utilities — Use the latest version of software update utilities Oracle EXAchk, patchmgr, and OPatch.

    Oracle EXAchk is updated regularly to contain new features, fixes, best practice health checks, and version recommendations. Patchmgr for database servers is updated regularly to contain new features, fixes, and workarounds for known database server update issues. OPatch is updated regularly to contain new features and fixes.

  • Perform storage server updates separately from RDMA Network Fabric switch updates — Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. The RDMA Network Fabric switch firmware upgrade requires a switch reboot, which disrupts some connections on the RDMA Network Fabric.

  • Avoid unsupported system changes — Oracle Exadata is an integrated system and engineered to be the best platform for running Oracle Database. Oracle Exadata storage servers and RDMA Network Fabric switches contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. Software updates to Oracle Exadata storage servers and RDMA Network Fabric switches are performed using patchmgr. Configuration or installed software may not be altered manually (without using patchmgr) in any way unless the Oracle Exadata documentation contains steps to perform the desired change. While making an unsupported manual change may have the desired immediate effect, there are many potential negative consequences, such as:

    • Failed future patchmgr software updates
    • Inability to rescue the system
    • Inability to diagnose a software defect efficiently

    Contact Oracle Support Services for further guidance if a desired Oracle Exadata storage server or RDMA Network Fabric switch change is not documented.

  • Minimize database server customization — Oracle Exadata is an integrated system and engineered to be the best platform for running Oracle Database. Oracle Exadata database servers contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. Software updates to Oracle Exadata database servers are performed using patchmgr. However, it may be necessary to manually install additional, site-specific software, such as monitoring agents or backup agents.

    It is supported to manually customize database servers, but note that customizing the operating system by adding or updating packages may require additional actions when applying a future Oracle Exadata System Software update with patchmgr (for example, removing customization prior to updating Oracle Exadata servers and reapplying the customization after the update completes) because the additional software may add new dependencies which are not provided by a future Oracle Exadata System Software update. It is recommended to minimize database server customization. See the topic Updating Oracle Exadata Database Servers for additional details.

  • Test supported configuration changes — Database server site-specific customization must be tested completely, including verifying database servers reboot properly after making configuration changes. Custom configuration changes that prevent successful system reboot will cause future Oracle Exadata System Software updates to fail.

8.2.2.5 Version Compatibility and Mixed Version Support

It is recommended that software for all components be updated regularly to stay in step with the recommended minimum release in order to maintain the most stable and secure system.

However, it is supported to update only a subset of components during a software maintenance window while the remaining components remain at an earlier version. For example, the following scenarios are supported:

  • Update a subset of Oracle Exadata storage servers to a higher version of the Oracle Exadata System Software while the remaining storage servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata storage servers to a higher version of the Oracle Exadata System Software while the database servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata database servers to a higher version of the Oracle Exadata System Software while the storage servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata storage servers to a higher version of the Oracle Exadata System Software while RDMA Network Fabric switches remain at the earlier Oracle Exadata System Software version.

  • Update RDMA Network Fabric switches to a higher version of the Oracle Exadata System Software while Oracle Exadata storage servers remain at the earlier Oracle Exadata System Software version.

  • Update Oracle Exadata storage servers to a higher version of the Oracle Exadata System Software while Oracle Grid Infrastructure or Oracle Database software remains at an earlier release and update level.

While mixed versions are supported (within the same component or across different components), it is highly recommended that this be only a temporary configuration that exists for the purpose and duration of rolling upgrade.

If mixing versions the following rules and considerations must be observed:

  • A specific generation of Oracle Exadata hardware will have a minimum required Oracle Exadata System Software version. For example, Oracle Exadata X6 hardware requires Oracle Exadata System Software release 12.1.2.3.1 or higher.

  • A specific Oracle Database release requires a minimum Oracle Exadata System Software release to fully support Oracle Exadata features. For example, Oracle Database 12c Release 2 (12.2) requires Oracle Exadata System Software release 12.2.1.1.0 or higher on storage servers to support all Oracle Exadata offload features.

  • Features supplied with a new Oracle Exadata System Software release may require a minimum Oracle Grid Infrastructure or Oracle Database software release or release update. For example, the Oracle Exadata feature Smart Scan Offload for Compressed Index Scan requires Oracle Database 12.2 and Oracle Exadata System Software 12.2 on storage servers.

  • Oracle Grid Infrastructure supports mixed versions only during a rolling update (for example, while updating from release 12.1.0.2 to 12.2.0.1, or updating from release 12.1.0.2.161018 to 12.1.0.2.170117). Some Oracle Clusterware functionality is restricted while the cluster is in rolling upgrade mode.

  • Oracle Database supports mixed versions only during a rolling update (for example, while updating from release 12.1.0.2.161018 to 12.1.0.2.170117).

8.2.3 Understanding the Exadata Software Image Version

The Exadata image version number contains the product version, a date code, and a build number.

The Exadata version installed on an Exadata storage server or an Exadata database server is determined with the imageinfo command. The image version of an Exadata release consists of three components. For example, if the command imageinfo -ver returns 12.1.2.1.1.150316.2, then:

  • The product version is 12.1.2.1.1

  • The date code is 150316

  • The Oracle-internal build number is 2. This component is not always used.

In most circumstances it is sufficient to reference a release by the product version only, for example, 12.1.2.1.1. However, when upgrading, you need to consider both the product version and date code of the installed and target releases.

Starting with Oracle Exadata System Software release 18.1, the product version uses a three field format consisting of: Year.Update.Revision, for example 18.1.0.

Parent topic: Planning for Software Maintenance

8.2.4 Rules for Updating to Newer Exadata Version

When upgrading, you need to consider both the product version and date code of the installed and target releases.

In most circumstances it is sufficient to reference a release by the product version only, for example, 12.1.2.1.1.

Upgrading to a specific target version must adhere to the following rules, which are enforced by patchmgr:

  1. The product version of the target release must be higher than the installed software

    and

  2. The date code of the target release must be higher than the installed software.

For example, consider an Exadata Database Machine that is currently running image version 12.1.1.1.2.150411, which consists of product version 12.1.1.1.2 and date code 150411.

  • Upgrading to 12.1.2.1.2.150617.1 is allowed because both rules are satisfied.

  • Upgrading to 12.1.2.1.1.150316.2 is not allowed because date code 150411 of the installed software is higher than date code 150316 of the target release.

Parent topic: Planning for Software Maintenance

8.3 Overview of Performing Exadata Software Updates

For each software update of the Exadata components, there are various actions required to complete the software update.

Parent topic: Updating Exadata Software

8.3.1 Actions to Perform Before Any Software Maintenance

Perform you update any software on your Exadata Database Machine, you should perform these actions.

  • Identify the target release

    The target release can be determined by one of the following:

    • Recommendation in My Oracle Support Document 888828.1.

    • Version recommendations in the MAA Scorecard section of the Oracle ExaCHK report.

  • Run Oracle ExaCHK

    Ensure software maintenance readiness of the system by running the latest version of Oracle ExaCHK (see My Oracle Support Document  1070954.1).  Review the report for the following:

    1. Correct any FAIL or WARNING checks that deviate from your baseline.

    2. Review the MAA Scorecard section for version recommendations. Oracle ExaCHK evaluates current software versions for consistency, compatibility, and whether or not it is current.

    3. Review the MAA Scorecard section for exposure to critical issues.  Target version choice should resolve any critical issue exposure.

    It is recommended to run the latest version of Oracle ExaCHK in the following times:

    • Every month to maintain a system that continues to adhere to Exadata best practices

    • The week before the software update to ensure maintenance readiness

    • The day before the software update to ensure maintenance readiness

    • Immediately after completing the software update

  • Run prerequisite checks

    1. Download the target software release(s) and the latest versions of the software update utilities (patchmgr, OPatch, and so on) from My Oracle Support.

    2. Run the prerequisite checks and correct any issues. The initial prerequisite check run should be performed days or weeks in advance of the maintenance window to allow time to correct issues. Run prerequisite checks again immediately before the update.

  • Review the latest documentation

    1. Review My Oracle Support Document 1270094.1 for recently published critical issues that are not yet automatically checked by Oracle ExaCHK.

    2. Review the Supplemental README in My Oracle Support for the target software version for known issues discovered after the software was released.

Related Topics

Parent topic: Overview of Performing Exadata Software Updates

8.3.2 Overview of Performing Exadata Storage Server Updates

A high level look at the process for updating Exadata storage servers.

Patchmgr manages the update orchestration in a rolling or non-rolling manner for storage servers. When using the -rolling option, Oracle recommends that you use Oracle ASM high redundancy disk groups to tolerate the failure of a disk in another storage server during the update.

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support document 888828.1.

  2. Create a file that contains the list of storage servers to update. This file will be specified as the component_list_file.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all storage servers in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. If storage servers will be updated non-rolling, then stop Oracle Clusterware and all databases accessing the storage servers.

  6. Run patchmgr to update storage servers in a non-rolling (default) or rolling (-rolling option) manner.

  7. If storage servers were updated non-rolling, then restart Oracle Clusterware and all databases.

Related Topics

Parent topic: Overview of Performing Exadata Software Updates

8.3.3 Overview of Performing Exadata Database Servers Updates

When updating Exadata database servers, there are different steps for virtualized environments.

Note:

Patchmgr manages the update orchestration in a rolling or non-rolling manner for Exadata database servers. When using the -rolling option, Oracle recommends that you use Oracle RAC features for client high availability to minimize the impact to client applications connected to database instances that will be stopped during maintenance.

Parent topic: Overview of Performing Exadata Software Updates

8.3.3.1 Steps for Updating Non-Virtualized Configurations

An overview of the steps involved in updating an Exadata database server that does not have Oracle VM configured.

  1. Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  2. Download the target Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support Document 888828.1.

  3. Create the component_list_file containing the list of database servers to update.

  4. Configure SSH equivalence from the user that will run patchmgr to root on all database servers in the component_list_file.

  5. Run patchmgr backup to backup database server root file system and operating system.

  6. Run patchmgr prerequisite check and correct any issues.

  7. Run patchmgr to update database servers in a non-rolling (default) or rolling (-rolling option) manner.

8.3.3.2 Steps for Updating Virtualized Configurations

The management domains (dom0s) and user domains (domUs) are updated independently using separate patchmgr executions.

Updating the Management domain (dom0)

  • Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  • Download the target dom0 Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support Patch Document 888828.1.

  • Create the component_list_file containing the list of database server dom0s to update.

  • Configure SSH equivalence from the user that will run patchmgr to root on all database server dom0s in the component_list_file.

  • Run patchmgr backup to backup database server dom0 root file system and operating system.

  • Run patchmgr prerequisite check and correct any issues.

  • Run patchmgr to update database server dom0s in a non-rolling (default) or rolling (-rolling option) manner.

Updating the User domains (domU)

  1. Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  2. Download the target domU Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support document 888828.1.

  3. Create the component_list_file containing the list of database server domUs to update.

  4. Configure SSH equivalence from the user that will run patchmgr to root on all database server domUs in the component_list_file.

  5. Run patchmgr backup to backup database server domU root file system and operating system.

  6. Run patchmgr prerequisite check and correct any issues.

  7. Run patchmgr to update database server domUs in a non-rolling (default) or rolling (-rolling option) manner.

8.3.4 Overview of Performing RoCE Network Fabric Switch Updates

Patchmgr manages the update orchestration in a rolling manner for RoCE Network Fabric switches

Note:

Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. The RDMA Network Fabric switch firmware upgrade requires a switch reboot, which disrupts some connections on the RDMA Network Fabric.

The general steps for using patchmgr are:

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system. See My Oracle Support document 888828.1.

  2. Create the component_list_file containing the list of RoCE Network Fabric switches to update.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all RoCE Network Fabric switches in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. Run patchmgr to update RoCE Network Fabric switches. RoCE Network Fabric switches are always updated in a rolling manner.

Related Topics

Parent topic: Overview of Performing Exadata Software Updates

8.3.5 Overview of Performing InfiniBand Network Fabric Switch Updates

Patchmgr manages the update orchestration in a rolling manner for InfiniBand Network Fabric switches

Note:

Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. The RDMA Network Fabric switch firmware upgrade requires a switch reboot, which disrupts some connections on the RDMA Network Fabric.

The general steps for using patchmgr are:

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system. See My Oracle Support document 888828.1.

  2. Create the component_list_file containing the list of InfiniBand Network Fabric switches to update.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all InfiniBand Network Fabric switches in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. Run patchmgr to update InfiniBand Network Fabric switches. InfiniBand Network Fabric switches are always updated in a rolling manner.

Related Topics

Parent topic: Overview of Performing Exadata Software Updates

8.3.6 Overview of Performing Oracle Grid Infrastructure and Oracle Database Updates

The steps for updating Oracle Grid Infrastructure or Oracle Database software depend on the type of software update you are performing.

Updating to a Release Update (RU)

Updates to a new Release Update (RU) may be performed in-place or out-of-place. Both in-place and out-of-place methods support rolling and non-rolling updates.

  • In-place

    The update is applied to the current software home using OPatch while the Oracle Grid Infrastructure or Oracle Database software is shutdown on the node being updated.

    This is the default method of applying an RU for Oracle Grid Infrastructure or Oracle Database software, as described in the RU README. The steps to apply the RU must be performed on each node.

  • Out-of-place (recommended)

    A new software home is prepared and updated while the Oracle Grid Infrastructure and Oracle Database software remains running. Once the new homes are prepared, Oracle Grid Infrastructure and Oracle Database are quickly stopped, switched to the new home, and restarted.

    Out-of-place has significant advantage over in-place updates:

    • There is less risk because the new home can be prepared while the Oracle Grid Infrastructure and Oracle Database software remains online.
    • There is less downtime because switching to the new software home is faster than applying an update in-place.
    • Rollback is faster because it is possible to simply switch back to the original software home.

    The recommended way to adopt out-of-place updates is to use Fleet Patching and Provisioning (previously called Rapid Home Provisioning), which provides the following advantages:

    • Distributes software updates to all nodes in the cluster.
    • Orchestrates updates across the cluster in a rolling or non-rolling manner with a single command.
    • Provides control over database service relocation to maintain application availability.

    Out-of-place software updates are also supported by Oracle Enterprise Manager Cloud Control.

    Out-of-place software updates without Fleet Patching and Provisioning or Oracle Enterprise Manager Cloud Control may be accomplished by following My Oracle Support Document 2087150.1.

OJVM Update

The OJVM update is a separate software update for database homes that addresses OJVM security vulnerabilities. It is installed separately from a standard software update. The OJVM update may be installed in a rolling manner under certain situations.

Updating to a New Release

To update to a higher Oracle Grid Infrastructure or Oracle Database release, follow the step-by-step release instructions in My Oracle Support.

Related Topics

Parent topic: Overview of Performing Exadata Software Updates

8.3.7 Using sudo When Performing Software Updates

You can use sudo when running dbnodeupdate.sh and patchmgr (and dbnodeupdate Orchestration).

Parent topic: Overview of Performing Exadata Software Updates

8.3.7.1 Running patchmgr (and dbnodeupdate Orchestration) Using sudo

You can run patchmgr (which is packaged in dbserver.patch.zip) using sudo to perform any of patchmgr's functionalities, such as patching cells, patching RDMA Network Fabric switches, or orchestrating dbnodeupdate.sh execution.

Patchmgr is packaged in dbserver.patch.zip. Perform the following steps to set up the /etc/sudoers file for running patchmgr using sudo:

  1. Log in as the root user and edit /etc/sudoers using visudo.
    # visudo
  2. Add the following entry to the bottom of the sudoers file to allow non-root users, such as the oracle user, to run patchmgr as root.

    Note:

    The first field in the line specifies the non-root user who is granted sudo access for the patchmgr command. The line below uses the oracle user as an example. You can specify a different user if necessary. 
    oracle  ALL=(ALL)    NOPASSWD:SETENV: /u01/stage/patch/dbserverpatch/patchmgr
  3. As root, create the /u01/stage/patch/dbserverpatch directory and unzip dbserver.patch.zip:
    # mkdir -p /u01/stage/patch/dbserverpatch/
# cp dbserver.patch.zip /u01/stage/patch/dbserverpatch/
# cd /u01/stage/patch/dbserverpatch/
# unzip dbserver.patch.zip
  4. Move everything under the /u01/stage/patch/dbserverpatch/dbserver_patch_x.yymmdd directory to /u01/stage/patch/dbserverpatch/.
    # mv /u01/stage/patch/dbserverpatch/dbserver_patch_x.yymmdd/* /u01/stage/patch/dbserverpatch/

Note:

  • Patchmgr expects root SSH equivalence on all database nodes that will be updated, even when run using sudo.
  • The above setup requires that the entire contents of /u01/stage/patch/dbserverpatch be owned by root.
  • If you update dbserver.patch.zip, then you must place the new version in the same location as specified by sudoers.

To verify that the setup is correct, run patchmgr in prereq check mode as the oracle user: 

[oracle]$ cd /u01/stage/patch/dbserverpatch/

[oracle]$ sudo ./patchmgr --dbnodes dbgroup --precheck     \
  --repo http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/23.1.8.0.0/base/x86_64/  \
  --target_version 23.1.8.0.0.231109
8.3.7.2 Running dbnodeupdate.sh Using sudo

Before using dbnodeupdate.sh using sudo, you configure the /etc/sudoers file.

  1. Log in as the root user and edit /etc/sudoers using visudo.
    # visudo
  2. Add the following entry (all on one line) to the bottom of the sudoers file to allow non-root users, such as the oracle user, to run dbnodeupdate.sh as root:

    Note:

    The first field in the line specifies the non-root user who is granted sudo access for the dbnodeupdate.sh command. The line below uses the oracle user as an example. You can specify a different user if necessary. 
    oracle  ALL=(ALL)    NOPASSWD:SETENV: /u01/stage/patch/dbnodeupdate/dbnodeupdate.sh
  3. As root, create the /u01/stage/patch/dbnodeupdate directory and unzip dbnodeupdate.zip:
    # mkdir -p /u01/stage/patch/dbnodeupdate
# cp dbnodeupdate.zip /u01/stage/patch/dbnodeupdate
# cd /u01/stage/patch/dbnodeupdate
# unzip dbnodeupdate.zip

To verify that the setup is correct, run dbnodeupdate.sh in prereq check mode as the oracle user: 

[oracle]$ cd /u01/stage/patch/dbnodeupdate

[oracle]$ sudo ./dbnodeupdate.sh -u -l http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3/base/x86_64/ -v

dbnodeupdate exits if it is run without root privileges.

Note:

  • The above setup requires that everything in /u01/stage/patch/dbnodeupdate be owned by the root user.
  • If you update the dbnodeupdate utility, then you must place the new version in the same location as specified by sudoers.

8.4 Exadata Patchmgr Update Utility

Patchmgr is the utility used to update software for Exadata infrastructure components.

Parent topic: Updating Exadata Software

8.4.1 About the Exadata Patchmgr Update Utility

Patchmgr is designed to simplify the software update process.

Patchmgr has the following capabilities:

  • With a single invocation updates all Oracle Exadata storage servers, database servers, or RDMA Network Fabric switches.

  • Orchestrates the software update across components one at a time when performing a rolling update.

  • Parallelizes the software update to all components at the same time when performing a non-rolling update.

  • Updates all component software, as required, such as firmware, operating system, and Oracle Exadata System Software.

    • Database servers contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. However, it may be necessary to manually install additional, site-specific software, such as monitoring agents or backup agents. It is supported to manually customize database servers, but note that customizing the operating system by adding or updating packages may require additional actions when applying a future Oracle Exadata System Software update with patchmgr.

    • Storage server or RDMA Network Fabric switch configuration or installed software may not be altered manually (update without using patchmgr) in any way unless Oracle Exadata documentation contains steps to perform the desired change.

  • Patchmgr may be run from an Engineered System running Oracle Linux, as the root user or a non-root user.

  • Multiple invocations of patchmgr may be run from the same software directory.

When updating database servers, patchmgr manages the following as needed:

  • Stops and starts databases and clusterware
  • Stops and starts user domains (domUs)
  • Stops and starts Oracle Enterprise Manager Cloud Control agents
  • Unmounts remote network mounts
  • Performs a root file system operating system backup that can be used for rollback
  • Relinks database home and Oracle Grid Infrastructure home binaries
  • Applies updated best practices configuration changes and workarounds for known issues

Parent topic: Exadata Patchmgr Update Utility

8.4.2 Obtaining Patchmgr

You can download the patchmgr utility from My Oracle Support.

Before Oracle Exadata System Software release 19.3.0, the server and RDMA Network Fabric switch updates use the version of patchmgr bundled with the Oracle Exadata System Software release.

Starting with Oracle Exadata System Software release 19.3.0, there are separate patchmgr distributions for the servers and RDMA Network Fabric switches.

By default, the appropriate patchmgr distribution is included with the relevant server or switch update. However, it is recommended to always use the latest patchmgr from My Oracle Support when updating Oracle Exadata servers or switches. The patchmgr utility is updated frequently to address known issues and best practices, and to support new hardware. See My Oracle Support document 888828.1.

Related Topics

Parent topic: Exadata Patchmgr Update Utility

8.4.3 Patchmgr Syntax

Patchmgr is a utility used to update software for Oracle Exadata infrastructure components.

Prerequisites

Patchmgr runs on the "driving system", which is an Oracle Exadata database server or a non-Oracle Exadata system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata systems. If patchmgr runs from an Oracle Exadata database server, then that database server cannot be in the group file supplied to patchmgr.

Syntax

Patchmgr syntax differs depending on the component to be updated, however, the general patchmgr syntax is:

./patchmgr -component component_list_file -action required_arguments [optional_arguments]

Options

Option Description
component Represents the component to update. Valid values are -cells, -dbnodes, -ibswitches, or -roceswitches
component_list_file A text file containing the host names of components to update. For example, if updating Oracle Exadata storage servers (using -cells), then component_list_file will contain the list of all host names of the storage servers.
action The action to perform. In most cases the action is specific to the component being updated.
required_arguments The additional arguments required for certain actions.
optional_arguments Any optional arguments required for certain actions.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.

  • Multiple invocations of patchmgr may be run concurrently from the same software directory by using the -log_dir option. This allows patchmgr to update multiple Oracle Exadata systems concurrently from the same software directory.

  • Patchmgr may be run as the root user or as a non-root user. To run patchmgr as a non-root user, the -log_dir option must be used.

  • The user running patchmgr must have root-level SSH equivalence configured to the servers or switches that patchmgr will update. The SSH equivalence must be bi-directional. That is, the user running patchmgr must be able to use SSH without a password to access the target servers or switches as the root user, and the root user on target servers or switches must be able to use SSH without a password to access the driving system as the user running patchmgr.

  • Prior releases used the dbnodeupdate.sh utility to update database servers. dbnodeupdate.sh has been integrated with and is replaced by patchmgr.

Parent topic: Exadata Patchmgr Update Utility

8.4.3.1 Patchmgr Syntax for Storage Servers

You can use patchmgr to update software for Oracle Exadata storage servers.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata database server or a non-Oracle Exadata system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata systems

Patchmgr Syntax for Storage Servers 

./patchmgr --cells cell_host_file 
{ { --patch_check_prereq | --rollback_check_prereq } [--rolling] [--unkey] [--ignore_alerts] [--ignore_date_validations] | 
  { --patch | --rollback } [--rolling] [--unkey] [--ignore_alerts] | 
  --cleanup [--unkey] }
[ --log_dir { log_directory | auto } ]

Main Arguments

Argument Description
--cells cell_host_file The cell_host_file is a text file containing the host names of the storage servers to update. The file contains one storage server host name or IP address per line. The storage server patching will fail if the list file is not specified.
--patch Apply the patch, including firmware updates, wherever possible (BIOS, Disk Controller and if possible disk drives) to all storage servers in the storage server list file.
--rollback Rolls back the patch.
--cleanup Cleans up all patch files and temporary content on all storage servers. Before cleaning up, collects logs and information for problem diagnostics and analysis. Cleaning up patch files can be done manually if patch fails by removing the directory /root/_cellupd_dpullec_ on each storage server.

Supported Options

The following options are supported for storage server patching and rollback:

Table 8-2 Patchmgr Options for Storage Servers

Option Description
--get property Get information about specific property. Property can be: log_dir - assigned directory for all log files.
--ignore_alerts Ignore any active hardware alerts on the Exadata storage server and proceed with the patching.
--ignore_date_validations Ignore date version validation. This allows upgrades to later releases with an earlier date stamp. For example, 19.1.2.0.0.190306 to 19.2.0.0.0.190225.

--log_dir ( log_directory| auto )

Specifies the absolute path to the log directory, or you can specify auto to use a log directory that is based on the launch directory and the content of the nodes list file.

Specifying the -log_dir option enables you to run multiple patch manager invocations and also to run patch manager as a non-root user.

--patch_check_prereq Runs prerequisite check on all the storage servers to determine if the patch can be applied to the storage servers.
--rollback_check_prereq Runs prerequisite check on all the storage servers to determine if the storage servers can be rolled back for this specific patch.

--rolling

Specifies that the update is to be done in a rolling fashion. If not specified, the update is done in a non-rolling fashion.

Environment variable EXA_PATCH_ACTIVATE_TIMEOUT_SECONDS controls the timeout value waiting for the grid disks to be activated. The default is set to 36000 (10 hours).

Note: Prerequisite checks and cleanups are always done in a non-rolling fashion, even if the -rolling option is specified.

--smtp_from "email_addr"

Specifies the from email address for the patchmgr notification.

--smtp_to "email_addr1 email_addr2 email_addr3 ..."

Specifies the to email addresses for the patchmgr notification.

--smtp_set_envelope_sender

Specifies that the same from address in Return-Path: mail header should be used.

--unkey Removes passwordless SSH access to the cells before exit.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.
  • Multiple invocations of patchmgr may be run concurrently from the same software directory by using the --log_dir option. This allows patchmgr to update multiple Oracle Exadata systems concurrently from the same software directory.

  • Patchmgr may be run as the root user or as a non-root user. The user running patchmgr must have root-level SSH equivalence configured to the servers or switches that patchmgr will update. To run patchmgr as a non-root user, the -log_dir option must be used.

Example 8-4 Run storage server update pre-requisite checks, then update storage servers

./patchmgr -cells cell_group -patch_check_prereq 
./patchmgr -cells cell_group -patch

Example 8-5 Update storage servers in a rolling manner

The following command updates storage servers in a rolling manner, receive email notifications as the update proceeds, and remove passwordless SSH access to cells after the update is complete.

./patchmgr -cells cell_group -patch -rolling -unkey -smtp_from "dbm01@example.com" -smtp_to "admin1@example.com,admin2@example.com"

Example 8-6 Update multiple storage servers at one time

The following commands update storage servers on three Oracle Exadata systems simultaneously from the same software directory; one update using a rolling update, the others updates are non-rolling.

In this example:

  • Each patchmgr command must be run from a separate terminal window.
  • Each patchmgr execution uses a unique logging directory name automatically generated based on the content of the component_list_file. 
(terminal1) ./patchmgr -cells cell_group_exa01 -patch -rolling -log_dir auto 
(terminal2) ./patchmgr -cells cell_group_exa02 -patch -log_dir auto 
(terminal3) ./patchmgr -cells cell_group_exa03 -patch -log_dir auto

To have a subsequent patchmgr execution use an altered component_list_file with different content, yet use the same logging directory as a prior patchmgr execution, use the -get log_dir option to obtain the logging directory. For example:

  1. The logging directory for the initial patchmgr execution is generated automatically.

    ./patchmgr -cells cell_group -patch -log_dir auto

  2. Assume the last cell failed to update and patchmgr will be re-run for the last cell only, using the same logging directory as the initial patchmgr execution. Use the -get log_dir option to obtain the logging directory using the original component_list_file. 

    ./patchmgr -cells cell_group -patch -log_dir auto -get log_dir log_dir=/tmp/patch_12.2.1.1.1.170419/log/dm01cel01_dm01cel02_cacce4ee

  3. Update the cell_group file to contain only the last cell, or use a different file that contains only the last cell. Specify the logging directory from the initial patchmgr execution so all logs for this group of cells are created in the same logging directory. 

    ./patchmgr -cells cell_group -patch -log_dir /tmp/patch_12.2.1.1.1.170419/log/dm01cel01_dm01cel02_cacce4ee

Related Topics

Parent topic: Patchmgr Syntax

8.4.3.2 Patchmgr Syntax for Database Servers

You can use patchmgr to update software for Oracle Exadata database servers.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata database server or a non-Oracle Exadata system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata systems. If patchmgr is run from an Oracle Exadata database server, then that database server cannot be in the file that lists the servers to patch.

Patchmgr Syntax for Database Servers 

./patchmgr --dbnodes database_node_file
{ --backup --repo { base_URL | zipped_iso_file } [--rolling] [--unkey] |
  --precheck --repo { base_URL | zipped_iso_file } --target_version version [--unkey] |
  --upgrade --repo { base_URL | zipped_iso_file } --target_version version [--rolling] [--unkey] | 
  --complete [ --target_version version ] [--unkey] |
  --rollback [--rolling] [--unkey] |
  --cleanup  [--unkey] }
[ --log_dir { log_directory | auto } ]

Main Arguments

Argument Description
--dbnodes database_node_file The database_node_file is a text file containing the host names of the database servers to update. The file contains one database server host name or IP address per line. The database server patching will fail if the list file is not specified.
--backup Perform backup for the database nodes specified in the host list. Runs backup in non-rolling fashion unless the --rolling option is specified.
--precheck Runs the pre-upgrade validation checks in the database nodes specified in the host list in non-rolling fashion.
--upgrade Upgrades the database nodes specified in the host list. Runs upgrade in non-rolling fashion unless the --rolling option is specified.
--complete Runs 'completion-steps' only. In normal cases there is no need to run this separately as this already runs as part of --upgrade. Note: If the database stack or user domains are up, they will be shut down and restarted.
--rollback Rollback the database nodes specified in the host list. Runs rollback in non-rolling fashion unless the --rolling option is specified.
--cleanup Cleans up all temporary content on the database servers specified in the host list in non-rolling fashion.

Supported Options

The following options are supported for storage server patching and rollback:

Table 8-3 Patchmgr Options for Database Servers

Option Description

--allow_active_network_mounts

Allows dbnodeupdate to run with active NFS or SMB mounts.

This is equivalent to the dbnodeupdate.sh -a command.

--allow_non_signed_repo

Allow dbnodeupdate to run with a non-signed repository. (Oracle Exadata System Software release 19.2 and later)

--dbnode_patch_base

User preferred location on target database servers where patch iso image and dbnodeupdate archive files are to be unzipped.

Note: Provided location must be an absolute path of local file system and it should have sufficient amount of free space and inodes.

--force_remove_custom_rpms

Remove any custom RPMs when the database server upgrade includes a major operating system upgrade. For example, from Oracle Linux 7 to Oracle Linux 8.

--ignore_alerts Ignore any active hardware alerts on the Exadata server and proceed with the patching.

--log_dir ( log_directory| auto )

The absolute path to the log directory, or you can specify auto to use a log directory that is based on the directory you started patchmgr from and the content of the nodes list file.

Specifying the --log_dir option enables you to run multiple patch manager invocations and also to run patch manager as a non-root user.

--no_connection_draining

Disables database connection draining for Fleet Patching and Provisioning, formerly known as Rapid Home Provisioning (RHP). The connection draining availability depends on the Oracle Grid Infrastructure release. This option is applicable to only rolling updates.

--nobackup

Do not backup the database servers before an upgrade.

--repo { base_URL | zipped_iso_file }

Specifies the base URL for the Exadata update repository or the path to a zipped ISO file.

This option must be specified for --backup, --precheck, and --upgrade actions.

--rolling

Specifies that the update is to be done in a rolling fashion, one server at a time. If not specified, the update is done in a non-rolling fashion.

Environment variable EXA_PATCH_ACTIVATE_TIMEOUT_SECONDS controls the timeout value waiting for the grid disks to be activated. The default is set to 36000 (10 hours).

Note: Prerequisite checks and cleanups are always done in a non-rolling fashion, even if the -rolling option is specified.

--rolling_backups

Directs patchmgr to backup each node prior to updating that node in a rolling manner. If this option is not specified (default), then patchmgr completes the backups of all nodes in parallel before updating the first node.

This option can only be used in conjunction with the --rolling option. Otherwise, this option is ignored and the default backup method is used.

This option is ignored if --nobackup is included in the command.

--skip_gi_db_validation

Skip certification of Oracle Grid Infrastructure and Oracle Database home compatibility with Oracle Linux 7.

Only for upgrades from Oracle Linux 6 to Oracle Linux 7.

--smtp_from "email_addr"

Specifies the from email address for the patchmgr notification.

--smtp_to "email_addr1 email_addr2 email_addr3 ..."

Specifies the to email addresses for the patchmgr notification.

--smtp_set_envelope_sender

Specifies that the same from address in Return-Path: mail header should be used.

--target_version version

The full patch version as specified in the patch README file. For example: 23.1.8.0.0.231109

This option must be specified for --precheck and --upgrade actions.

--unkey Removes passwordless SSH access to the servers before exit.

Examples

Example 8-7 Backup database servers then perform the update

The following commands backup the Oracle Exadata database servers, run the prerequisite checks for the database server update, and then update the database servers in a rolling manner. 

[root@pmserver ~]# ./patchmgr --dbnodes dbnode_group --backup --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109

[root@pmserver ~]# ./patchmgr --dbnodes dbnode_group --precheck --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109

[root@pmserver ~]# ./patchmgr --dbnodes dbnode_group --upgrade --nobackup --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109 --rolling

Related Topics

Parent topic: Patchmgr Syntax

8.4.3.3 Patchmgr Syntax for RoCE Network Fabric Switches

You can use patchmgr to update software for RoCE Network Fabric switches.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata database server or a non-Oracle Exadata system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata systems.

Note:

Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.

Patchmgr Syntax for RoCE Network Fabric Switches 


./patchmgr --roceswitches [roceswitch_list_file] 
    { --upgrade [--roceswitch-precheck] [--unkey] [--force] |
      --downgrade [--roceswitch-precheck] [--unkey] [--force] |
      --apply-config [--unkey] [--force] |
      --verify-config [ --newswitchlist new_list_file ] [--unkey] }
    [ -log_dir { absolute_path_to_log_directory | AUTO } ]

Main Arguments

Argument Description
--roceswitches [roceswitch_list_file]

Specifies that patchmgr is acting on the RoCE Network Fabric switches.

If specified, the switch list file identifies the RoCE Network Fabric switches.

In its simplest form, the file has one switch host name or IP address on each line. In this case, each switch is assumed to be a leaf switch in a single-rack Exadata environment. For example:

rack1sw-rocea0
rack1sw-roceb0

To specify the configuration type for each switch, append a colon (:) and tag to each switch host name or IP address in the switch list file. The following tags are supported:

  • leaf - Identifies a leaf switch in a single rack system. This configuration type is assumed if no tag is specified.
  • mspine - Identifies a spine switch. Note that one spine switch configuration supports all spine switches on single and multi-rack systems, with and without Exadata Secure RDMA Fabric Isolation.
  • mleaf - Identifies a leaf switch in a multi-rack X8M system.
  • sfleaf - Identifies a leaf switch in a single rack system that is enabled to support Exadata Secure RDMA Fabric Isolation.
  • msfleaf - Identifies a leaf switch in a multi-rack X8M system that is enabled to support Exadata Secure RDMA Fabric Isolation.
  • leaf23 - Identifies a leaf switch in a single rack system that is configured with 23 host ports. This configuration is required only for 8-socket systems (X8M-8 and later) with 3 database servers and 11 storage servers.
  • mleaf23 - Identifies a leaf switch in a multi-rack system that is configured with 23 host ports. This configuration is required only for 8-socket X8M-8 systems with 3 database servers and 11 storage servers.
  • mleaf_u14 - Identifies a leaf switch in a multi-rack system that is configured with 14 inter-switch links. This is the typical multi-rack leaf switch configuration for X9M and later model systems.
  • msfleaf_u14 - Identifies a leaf switch in a multi-rack system that is enabled to support Exadata Secure RDMA Fabric Isolation and is configured with 14 inter-switch links. This configuration is required for X9M and later model systems with Secure Fabric enabled.
  • mleaf23_u13 - Identifies a leaf switch in a multi-rack system that is configured with 23 host ports and 13 inter-switch links. This configuration is required only for 8-socket X9M-8 systems with three database servers and 11 storage servers.

For example:

rack1sw-rocea0:leaf
rack1sw-roceb0:leaf

For multi-rack configurations only, also specify a unique loopback octet for each switch.

The loopback octet is the last octet of the switch loopback address, which uniquely identifies a switch.

To specify the loopback octet for each switch, append a period (.) and numeric loopback octet value to each tagged switch entry in the switch list file.

Caution:

Every switch in a multi-rack configuration must have a unique loopback octet. If multiple switches use the same loopback octet, the RoCE Network Fabric cannot function correctly, resulting in a system outage.

For the leaf switches, start with 101 as the first loopback octet value and increment as follows:

  • 101 - Rack 1 lower leaf switch (rack1sw-rocea0 in the following example)

  • 102 - Rack 1 upper leaf switch (rack1sw-roceb0 in the following example)

  • 103 - Rack 2 lower leaf switch (rack2sw-rocea0 in the following example)

  • 104 - Rack 2 upper leaf switch (rack2sw-roceb0 in the following example)

  • 105 - Rack 3 lower leaf switch

  • 106 - Rack 3 upper leaf switch, and so on.

For the spine switches, start with 201 as the first loopback octet value and increment as follows:

  • 201 - Rack 1 spine switch (rack1sw-roces0 in the following example)

  • 202 - Rack 2 spine switch (rack2sw-roces0 in the following example)

  • 203 - Rack 3 spine switch

  • 204 - Rack 4 spine switch, and so on.

For example, the switch list file for a 2-rack Exadata X9M system might contain: 

rack1sw-rocea0:mleaf_u14.101
rack1sw-roceb0:mleaf_u14.102
rack1sw-roces0:mspine.201
rack2sw-rocea0:mleaf_u14.103
rack2sw-roceb0:mleaf_u14.104
rack2sw-roces0:mspine.202

If no file name is provided, then the command acts on all RoCE Network Fabric switches discovered from the host that is running patchmgr.

--upgrade

Upgrade the firmware on the RoCE Network Fabric switches.

If required, this option also installs or upgrades the client software component that propagates switch alerts to Oracle Auto Service Request (ASR).

Note: Commencing with the August 2022 patchmgr release, patchmgr performs an additional series of checks on the RoCE Network Fabric. The checks occur immediately before any firmware upgrade and also during prerequisite checking using the --roceswitch-precheck option. These checks mitigate the risks of failure associated with unexpected problems in the RoCE Network Fabric. For example, if one of the RoCE Network Fabric ports on a storage server is down, the storage server would become unavailable if the switch connected to the only operational port is taken offline for an upgrade. If any check fails, patchmgr reports the problem and ends immediately. In this case, you must correct the problem with the RoCE Network Fabric before you can perform the upgrade.

--downgrade

Downgrade the firmware on the RoCE Network Fabric switches.

Note: Commencing with the August 2022 patchmgr release, patchmgr performs an additional series of checks on the RoCE Network Fabric. The checks occur immediately before any firmware downgrade and also during prerequisite checking using the --roceswitch-precheck option. These checks mitigate the risks of failure associated with unexpected problems in the RoCE Network Fabric. For example, if one of the RoCE Network Fabric ports on a storage server is down, the storage server would become unavailable if the switch connected to the only operational port is taken offline for an upgrade. If any check fails, patchmgr reports the problem and ends immediately. In this case, you must correct the problem with the RoCE Network Fabric before you can perform the downgrade.

--apply-config

Applies the golden configuration template to each switch.

This option relies on tags in the switch list file, which specify the configuration type for each switch.
--verify-config [ --newswitchlist new_list_file ]

Verify each switch configuration against the golden configuration.

Verification is performed automatically when using --upgrade or --downgrade. You can use this option to perform verification as a standalone operation.

If specified, the --newswitchlist option generates a new switch list file with entries that match the current configuration of each switch.

Supported Options

The following options are supported for RoCE Network Fabric switch configuration and firmware update:

Table 8-4 Patchmgr Options for RoCE Network Fabric Switches

Option Description
--roceswitch-precheck Performs switch firmware upgrade or downgrade simulation on the RoCE Network Fabric switches in the list file but does not perform the actual install. Use this option with --upgrade or --downgrade.
--unkey

In conjunction with --upgrade or --downgrade, this option removes the configuration settings that enable passwordless SSH access to the RoCE Network Fabric switch.

--force

In conjunction with --upgrade or --downgrade, this option proceeds with the upgrade or downgrade even if the switch is already on target firmware version or the RoCE Network Fabric switch is experiencing non-critical failures.

In conjunction with --apply-config, this option bypasses the check that determines if the current switch configuration matches the configuration type specified in the switch list file.

-log_dir ( absolute_path_to_log_directory | AUTO )

When running patchmgr as a non-root user, use -log_dir to specify the absolute path to the log directory or use the keyword AUTO. If you specify AUTO, then patchmgr generates and sets the path to the log directory based on the directory patchmgr is launched from and the content of the nodes list file.

Note: Specifying -log_dir enables multiple patch manager invocations and is required when running patch manager as a non-root user.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.

Example 8-8 Using patchmgr to Upgrade Firmware on RoCE Network Fabric Switches

This example runs the upgrade prerequisite checks on all detected switches, then upgrades the switches.

$ ./patchmgr --roceswitches --upgrade --roceswitch-precheck

$ ./patchmgr --roceswitches --upgrade

Example 8-9 Using patchmgr to Apply Golden Configurations to RoCE Network Fabric Switches

This example applies golden configuration settings to RoCE Network Fabric switches as specified in the switches.lst file. Logs for the operation are written to /tmp/switchlogs. 

$ cat switches.lst
switch456-rocea0:leaf          
switch456-roceb0:leaf          

$ ./patchmgr --roceswitches switches.lst --apply-config –log_dir /tmp/switchlogs

Parent topic: Patchmgr Syntax

8.4.3.4 Patchmgr Syntax for InfiniBand Network Fabric Switches

You can use patchmgr to update software for InfiniBand Network Fabric switches.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata database server or a non-Oracle Exadata system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata systems.

Patchmgr Syntax for InfiniBand Network Fabric Switches 

./patchmgr --ibswitches [ibswitch_list_file]
{ --upgrade | --downgrade } [--ibswitch_precheck] [--unkey] [ --force [ yes | no ]]

Main Arguments

Argument Description
--ibswitches ibswitch_list_file Specifies the name of the InfiniBand Network Fabric switch list file. The file has one switch host name or IP address per line. If no file name is provided, then it runs the command on all InfiniBand Network Fabric switches discovered from this host.
--upgrade Upgrade the InfiniBand Network Fabric switches in the list file to $EXADATA_IMAGE_IBSWITCH_UPGRADE_VERSION.
--downgrade Downgrade the InfiniBand Network Fabric switches in the list file to $EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION.

Supported Options

The following options are supported for InfiniBand Network Fabric switch configuration and firmware update:

Table 8-5 Patchmgr Options for InfiniBand Network Fabric Switches

Option Description
--force [yes | no] Specifies to proceed with the upgrade or downgrade even on non-critical failures. The value yes specifies to proceed by auto-answering prompts with yes. The value no specifies to proceed by auto-answering prompts with no. The default value is yes.
--ibswitch_precheck Runs the pre-update validation checks on the InfiniBand Network Fabric switches in the list file.

--smtp_from "email_addr"

Specifies the from email address for the patchmgr notification.

--smtp_to "email_addr1 email_addr2 email_addr3 ..."

Specifies the to email addresses for the patchmgr notification.

--smtp_set_envelope_sender

Specifies that the same from address in Return-Path: mail header should be used.

--unkey Removes passwordless SSH access to the InfiniBand Network Fabric switches before exit.

Usage Notes

  • Starting with Oracle Exadata System Software release 19.3, the options are prefixed with --. Prior to this release, the options were prefixed with -.

Example 8-10 Using patchmgr for InfiniBand Network Fabric Switches

This example runs the update prerequisite checks on all switches, then upgrades the switches specified in the ib_group file. 

./patchmgr --ibswitches --upgrade --ibswitch_precheck 

./patchmgr --ibswitches ib_group --upgrade

Parent topic: Patchmgr Syntax

8.4.3.5 Patchmgr Syntax for the Management Network Switch

You can use patchmgr to perform firmware upgrades on the 9000 series Management Network Switch found on Oracle Exadata Database Machine X7-2 and later systems.

Prerequisites

Patchmgr is run on the "driving system", which is an Oracle Exadata database server or a non-Oracle Exadata system running Oracle Linux. This allows patchmgr to run from a central server to update multiple Oracle Exadata systems.

Patchmgr Syntax for Management Network Switch

./patchmgr --adminswitches [adminswitch_list_file]
{ --upgrade | --downgrade } [--adminswitch-precheck] [--unkey] [--force]
[ -log_dir { absolute_path_to_log_directory | AUTO } ]

Main Arguments

Argument Description
--adminswitches [adminswitch_list_file]

Specifies that patchmgr is acting on the Management Network Switches.

If specified, the switch list file identifies the Management Network Switches. The file has one switch host name or IP address on each line.

If no file name is provided, then the command acts on all Management Network Switches discovered from the host that is running patchmgr.

--upgrade

Upgrade the firmware on the Management Network Switches.

--downgrade Downgrade the firmware on the Management Network Switches.

Supported Options

The following options are supported for Management Network Switch firmware update:

Table 8-6 Patchmgr Options for Management Network Switches

Option Description
--adminswitch-precheck Performs switch firmware upgrade or downgrade simulation on the Management Network Switches in the list file but does not perform the actual install. Use this option with --upgrade or --downgrade.
--unkey

In conjunction with --upgrade or --downgrade, this option removes the configuration settings that enable passwordless SSH access to the Management Network Switch.

--force

In conjunction with --upgrade or --downgrade, this option proceeds with the upgrade or downgrade even if the switch is already on target firmware version or the Management Network Switch is experiencing non-critical failures.

-log_dir ( absolute_path_to_log_directory | AUTO )

When running patchmgr as a non-root user, use -log_dir to specify the absolute path to the log directory or use the keyword AUTO. If you specify AUTO, then patchmgr generates and sets the path to the log directory based on the directory patchmgr is launched from and the content of the switch list file.

Note: Specifying -log_dir enables multiple patch manager invocations and is required when running patch manager as a non-root user.

Example 8-11 Using patchmgr to Upgrade Firmware on Management Network Switches

This example runs the upgrade prerequisite checks on the switch specified in the switches.lst file, then upgrades the switch. 

$ cat switches.lst
dbm0sw-adm0                  

$ ./patchmgr --adminswitches switches.lst --upgrade --adminswitch-precheck

$ ./patchmgr --adminswitches switches.lst --upgrade

Parent topic: Patchmgr Syntax

8.5 Updating Oracle Exadata Database Servers

Use the following information and procedures when updating database servers within Oracle Exadata.

Parent topic: Updating Exadata Software

8.5.1 Overview of Oracle Exadata Database Server Updates

When updating database servers, there is more than one software that needs to be updated.

Oracle Exadata database server release updates contain updates for the following components within a database server:

  • Oracle Linux operating system
  • Firmware (Disk, RAID controller, ILOM, HCA)
  • Oracle Exadata System Software

The software and firmware components that are updated for a specific release depend on the current Oracle Exadata System Software release the database server is running and the release you are updating to. Oracle Linux operating system packages and Oracle Exadata System Software are always updated while firmware may be updated for only a small selection of the components or not at all.

Updates for Oracle Exadata database servers can be applied independently from the storage servers or RDMA Network Fabric switches unless otherwise specified in My Oracle Support Note 888828.1.

Updating database servers is always performed in-place. This means the active operating system is updated. The actual update is performed using YUM, but the command to do this is wrapped in an Oracle Exadata utility called patchmgr.

The YUM command is wrapped in the update utility to maintain strict ordering of validation and preparation steps during the update process. Also by using a utility Oracle can enforce application of fixes for known issues and best practices.

Oracle Clusterware processes and Oracle Real Application Clusters (Oracle RAC) database instances must not be running on a database server that is being updated. To reduce application-level impact, follow the client failover best practices described in the "Client Failover Best Practices for Highly Available Oracle Databases" technical reference paper.

If you cannot afford cluster-wide downtime, you can update database servers in a rolling fashion. This means updating one database server at a time. If you can afford cluster-wide downtime, you can update all database servers in parallel. Non-rolling updates reduce the overall time required to complete the upgrade at the expense of having a full database outage.

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.2 Installing, Updating, and Managing Non-Oracle Software

Installing and updating non-Oracle branded RPMs on Oracle Exadata database servers are allowed as long as kernel and RDMA Network Fabric packages remain untouched.

Note that customizing the operating system by adding or updating packages may introduce problems when applying an Oracle Exadata System Software update because the non-Oracle software may add new dependencies which will not be provided by the Oracle Exadata System Software update. For this reason it is recommended to stay close to the Oracle Exadata image and customize as little as possible.

Note:

Oracle Exadata does not ship with 32-bit software. This means any customization that introduces 32-bit RPMs on the system will break the Oracle Exadata System Software update and will need to be removed.

In case you have added or updated packages supplied by the Oracle Exadata image, you can remove the exadata-sun-computenode-exact RPM when needed as follows: 

[root@dm01 ]# rpm -e exadata-sun-computenode-exact

If you have installed customized packages, it is recommended that you have scripts to automate the removal (to run before updating Oracle Exadata System Software) and installation (to run after the Oracle Exadata System Software update) of those packages. After the Oracle Exadata System Software update, verify that the customized packages are still compatible and are still needed, before re-installing them.

Note:

Packages should not be forced in using the rpm -Uvh --nodeps command, unless directed by Oracle Support Services.

The same principle applies for customizations other than installation of additional software. Oracle recommends customizing the operating system as little as possible. Be aware that while changes on the database server are allowed within the earlier described boundaries, it is not possible for Oracle to anticipate each and every customization on every configuration item of the operating system.

There are certain common standards that Oracle Exadata utilities depend on. These standards are based on best practices and have been validated over time. By moving away from Oracle Exadata standards, systems might be missing optimal settings, and risk is introduced because it is difficult for Oracle to anticipate and validate software for one-off configurations. This is why customizations can cause unexpected results. As a result of adding software or customizing system configuration, you might require changes or additional steps to the standard Oracle Exadata update process. With or without customizations, it is highly recommended to validate Oracle Exadata System Software updates on test systems before doing them on production systems.

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.3 Customization Levels and Impact

The table below provides some guidance in the risks associated with customizing Exadata database servers. While the prerequisite check checks for most customizations, certain configuration changes might need to be rolled back in order to make updating the database server possible. If still needed, the changes can be restored after the update process.

Table 8-7 Customizations and Their Risks

Item Customization Level Risk

Default provided Engineered System Software Stack

None

None

Setting up interactive shell profiles for system users (root/oracle)

Low

High

Using all free space in VGExaDb

Low

Low

Installing additional (non-Exadata) rpm packages

Medium

Low

Customizing file system with different mount points

Medium

Low

Updating packages shipped with the current Exadata image

Medium

Low

Customizing configuration files, changing basic operating system functionality

High

Medium

Installing additional (non-Exadata) non-rpm packages

High

Medium

Changing LVM layout

High

High

Parent topic: Updating Oracle Exadata Database Servers

8.5.4 Update Utility for Exadata Database Servers

Patchmgr is the update utility for updating Oracle Exadata database servers.

Starting with release 12.2.1.1.0, Exadata software updates for the Exadata database server can only be applied using the patchmgr utility.

Note:

The patchmgr utility for updating Exadata database servers is not the same as the patchmgr script shipped with the Exadata software update.

The patchmgr utility supports all hardware generations and Exadata storage server releases starting with 11.2.3.1.0, Exadata database servers running Oracle Virtual Server (dom0), and Exadata Virtual Machines (domU). The README files for the Oracle Exadata System Software updates specify whether the update itself is applicable for a particular hardware generation or not. The README is not shipped with dbserver.patch.zip but with the Oracle Exadata System Software update zip file.

The utility also supports an Exadata software update that includes a major operating system upgrade. For example, from Oracle Linux 7 to Oracle Linux 8.

The utility takes care of the orchestration. You can perform the update in a rolling or non-rolling fashion across one or multiple Exadata database servers.

The update utility performs the following tasks:

  • Automates all preparation, update, and validation steps, including:
    • Stopping the databases, Grid Infrastructure stack or domU’s
    • Stopping Oracle Enterprise Manager Cloud Control agents,
    • Un-mounting remote network mounts (when required)
  • Uses the built-in dbserver_backup.sh script to perform a backup of the file system hosting the operating system before updating the Exadata database server.
  • Applies Oracle best practices and fixes for the latest known issues.
  • Verifies that the update was successful, relinks the Oracle binaries, and starts the Oracle stack and domU’s

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.5 Update Tool Execution Host

If you are planning to update all Exadata database servers at once, it is a requirement to run the update utility from a Linux node outside the group of Exadata database servers being updated. This is because the update utility cannot update the Exadata database server it is currently running on. If you have no other systems running Oracle Linux or Oracle Solaris you can run the update utility from one of the Exadata database servers. In such cases be sure that the Exadata database server where the update utility is running is not listed in the dbs_group file you specify.

You need to set up ssh equivalence for the root user from the driving node to the root user of all Exadata database servers that will be updated.

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.6 Running the Update Utility as a Non-root User and Running Multiple Invocations Concurrently

By default the update utility assumes you want to run as root. It is however possible to run the update utility as a non-root user from a remote host. It is also possible to run multiple invocations at the same time. This allows you to update multiple logical groups of Exadata Database Servers concurrently. To do this, you run the update utility with the -log_dir flag.

Ensure that ssh equivalence is set up for the current user to the root user of the Exadata database servers to be updated.

The following example shows the option of running as a non-root user and running multiple-invocations.

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/ /p23557378_121223_Linux-x86-64.zip
 -target_version 12.1.2.4.0.160710 -log_dir auto

-log_dir specifies the absolute path to the log directory or the keyword autofor the utility to generate and set a path to the log directory that is based on the launch directory and content of nodes list file. If you want to make sure you are using the same log directory in later invocations where the list of Exadata database servers changes, use the -get log_dirflag to obtain the -log_dir location used on previous sessions. For example, if the following command: 

[oracle@nonExadata ]$ ./patchmgr -dbnodes ~/dbs_group_test  -log_dir auto -get log_dir

This command returns output similar to the following:

log_dir=/u01/test/dbserver_patch_5.160715/log/dbm01_dbm02_e8f1f75

Use the log_dir value in subsequent commands. For example: 

[oracle@nonExadata ]$ ./patchmgr -dbnodes ~/dbs_group_test -precheck 
-log_dir /u01/test/dbserver_patch_5.160715/log/dbm01_dbm02_e8f1f75 
-iso_repo /u01/test/dbserver_patch_5.160715/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.7 Recommended Timeline for Updating Exadata Database Servers

Oracle recommends the following timeline for updating Exadata database servers. By following this approach, you allow yourself time to perform any necessary corrective actions.

Note:

Before making any changes, including prerequisite check, you should make a backup.

Table 8-8 Timeline for Performing Updates

When Tasks

Weeks to days before the update

  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.

  • Download the latest Exachk from My Oracle Support note 1070954.1.

  • Research release-specific My Oracle Support note for known issues.

  • Research Exadata Critical Issues from My Oracle Support note 1270094.1.

  • Run Exachk.

  • Perform a first prerequisite check.

    For details, see Running Prerequisite Checks.

Just before the update

  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.

  • Download the latest Exachk from My Oracle Support note 1070954.1.

  • Research release-specific My Oracle Support note for known issues.

  • Research Exadata Critical Issues from My Oracle Support note 1270094.1.

  • Run Exachk and perform corrective actions as needed.

At update time

  • Perform a “backup only” run using the -backup flag.

    See Backing up Exadata Database Servers Before Planned Maintenance for details.

  • Perform a second prerequisite check.

  • Remove any blocking rpms and re-run the prerequisite check to validate if all changes are complete.

  • Perform the update. Use the -nobackup flag to skip the backup because you already made a “backup only” run.

    See Running the Update for details.

After the update

  • Run Exachk.

  • Reinstall any non-Exadata rpms that you removed before the Exadata update.

Parent topic: Updating Oracle Exadata Database Servers

8.5.8 Downloading and Distributing Exadata Database Server Updates

Each Exadata database server update is packaged as an compressed ISO image file and as a channel on the Oracle Unbreakable Linux Network (ULN). Oracle curates, tests, and certifies the packages in each release across the entire Exadata hardware and software stack. The Exadata ISO and ULN channel include only packages needed to run Exadata System Software and Oracle Database (including Oracle Grid Infrastructure).

You can utilize Exadata database server updates in the following ways:

  • Using a Local YUM Repository Mirror

    This method uses a YUM repository server to mirror the ULN channel for each Exadata database server software update. This approach is recommended when:

    • There is a large number of Exadata database servers to update.

    • You already have a YUM repository server in your network or the infrastructure exists for building a local ULN mirror on a separate Linux server.

    • The YUM repository server is accessible to all the Exadata database servers.

  • Using the ISO File as a YUM Repository

    This method uses a standard Web server to present the ISO file containing an Exadata database server software update as a YUM repository. This approach is recommended when:

    • There is a large number of Exadata database servers to update.

    • You don't already have a YUM repository server in your network and you don't want to build one.

    • You have a Web server on your network that is accessible to all the Exadata database servers.

  • Using the ISO File Directly

    This method uses the Exadata patchmgr utility to directly access the ISO file containing an Exadata database server software update.

    The main benefit of this approach is simplicity, as it only requires a copy of the ISO file on the server running the Exadata patchmgr utility. However, using this approach, the patchmgr utility must propagate the entire ISO file to every Exadata database server that is the target of the update. In contrast, patchmgr only propagates individual packages to the update targets when using other methods. Consequently, this approach requires additional free local storage space on every server being updated.

The following topics describe these methods in greater detail:

Parent topic: Updating Oracle Exadata Database Servers

8.5.8.1 Using a Local YUM Repository Mirror for ULN Oracle Exadata Channels

A local YUM repository server is a separate Linux server that contains a local copy (mirror) of Exadata database server software update packages downloaded from the Oracle Unbreakable Linux Network (ULN). After the repository is configured, all your Exadata database servers can connect to it to retrieve updates.

Note:

Do not use an Exadata database server as the local ULN mirror. Use a separate server to avoid interruptions caused by reboots associated with an Exadata software update. Also, using a separate server avoids dependency conflicts between the packages required to construct the repository and the packages installed or updated to support Exadata system functions.

If no separate server is available, use the direct ISO image method instead. See Using the ISO File Directly.

In general, the procedure to set up the local YUM repository server contains the following steps:

  1. Install and configure the software packages to support a YUM repository and Web server.

  2. Register the system with ULN.

  3. Subscribe to the ULN channels that contain the Exadata database server updates you want to mirror.

The precise instructions depend on the Oracle Linux version used on the local YUM repository server, which is independent from the Oracle Linux version used by Exadata. Consult the appropriate version of Oracle Linux documentation for your situation. As a guide, see Setting Up a Local ULN Mirror for instructions based on the current Oracle Linux release.

The ULN channels containing Exadata database server software use identifiers starting with exadata_dbserver. You can find the available Exadata channels using the ULN Web interface or the uln-channel utility. For example: 

[root@yumrepo ~]# uln-channel --available-channels
...
exadata_dbserver_22.1.18.0.0_x86_64_base
exadata_dbserver_22.1.19.0.0_x86_64_base
exadata_dbserver_22.1.20.0.0_x86_64_base
...
exadata_dbserver_23.1.7.0.0_x86_64_base
exadata_dbserver_23.1.8.0.0_x86_64_base
exadata_dbserver_23.1.9.0.0_x86_64_base
...

When you identify a relevant channel, you can also use the ULN Web interface or the uln-channel utility to subscribe to it and mirror it for use across your Exadata systems. Remember that a single YUM repository server can mirror multiple Exadata database server software channels, and each channel contains software based on a specific Oracle Linux release, such as OL6, OL7, OL8, and so on. For example: 

[root@yumrepo ~]# uln-channel --add --channels exadata_dbserver_22.1.20.0.0_x86_64_base
[root@yumrepo ~]# uln-channel --add --channels exadata_dbserver_23.1.8.0.0_x86_64_base

After creating the ULN mirror, you can reference it using the Exadata patchmgr utility with the --repo option. For example: 

[root@pmserver ~]# patchmgr --dbnodes database_node_file --precheck --repo http://yumrepo/yum/exadata_dbserver_23.1.8.0.0_x86_64_base --target_version 23.1.8.0.0.231109
[root@pmserver ~]# patchmgr --dbnodes database_node_file --upgrade --repo http://yumrepo/yum/exadata_dbserver_23.1.8.0.0_x86_64_base --target_version 23.1.8.0.0.231109 --rolling

Note that the server running patchmgr may be different from the YUM repository server. Also, the precise URL you specify in the --repo option depends on how you set up the ULN mirror and the associated Web server.

8.5.8.2 Using the ISO File as a YUM Repository

Each Exadata database server update is packaged as a compressed ISO image file. The ISO image file is organized as a self-contained YUM repository. You can use a standard HTTP Web server to serve the ISO file as a YUM repository to all your Exadata database servers.

Note:

Do not use an Exadata database server to host the YUM repository. Use a separate server to avoid interruptions caused by reboots associated with an Exadata software update.

If no separate server is available, use the direct ISO image method instead. See Using the ISO File Directly.

The following outlines the general procedure. You must adjust the example commands to suit your situation and server configuration.

  1. Download the Exadata database server update patch from My Oracle Support to your Web server.

    Each Exadata database server update patch contains a compressed ISO image file.

    To find available Exadata database server update patches, see My Oracle Support document 888828.1.

  2. Decompress the patch archive and mount the ISO image file.

    For example:

    [root@webServer /var/stage]# unzip p35869377_231000_Linux-x86-64.zip
[root@webServer /var/stage]# mkdir -p /var/www/html/yum/EXADATA/dbserver/23.1.8/base
[root@webServer /var/stage]# mount -o loop /var/stage/exadata_ol8_base_repo_23.1.8.0.0.231109.iso /var/www/html/yum/EXADATA/dbserver/23.1.8/base

    The example uses the Exadata database server update patch archive (p35869377_231000_Linux-x86-64.zip) for Exadata release 23.1.8, which contains the ISO image file named exadata_ol8_base_repo_23.1.8.0.0.231109.iso. The example assumes that the compressed patch archive is downloaded to /var/stage. The example also assumes that the Web server document root directory is located at /var/www/html, and that the mount point for the ISO image file is /var/www/html/yum/EXADATA/dbserver/23.1.8/base.

  3. Confirm the availability and contents of the Exadata database server update YUM repository.

    Based on the previous example, the YUM repository for the selected Exadata database server update should now be available at http://webServer/yum/EXADATA/dbserver/23.1.8/base/x86_64/.

    You can confirm the availability and contents of the YUM repository by performing a precheck operation with the patchmgr update utility. In the command, specify the YUM repository URL using the --repo option.

    For example:

    [root@pmserver ~]# patchmgr --dbnodes database_node_file --precheck --repo http://webServer/yum/EXADATA/dbserver/23.1.8/base/x86_64/ --target_version 23.1.8.0.0.231109

    Note that the server running patchmgr may be different from the YUM repository server.

  4. Use the YUM repository to perform an update.

    In the patchmgr command, specify the YUM repository URL using the --repo option.

    For example:

    [root@pmserver ~]# patchmgr --dbnodes database_node_file --upgrade --repo http://webServer/yum/EXADATA/dbserver/23.1.8/base/x86_64/ --target_version 23.1.8.0.0.231109 --rolling
8.5.8.3 Using the ISO File Directly

Each Exadata database server update is packaged as a compressed ISO image file. You can use this file directly with the patchmgr update utility.

The main benefit of this approach is simplicity, as it only requires a copy of the zipped ISO file on the server running the Exadata patchmgr utility. However, using this approach, the patchmgr utility must propagate the entire zipped ISO file to every Exadata database server that is the target of the update. In contrast, patchmgr only propagates individual packages to the update targets when using other methods. Consequently, this approach requires additional free local storage space on every server being updated.

The following outlines the general procedure. You must adjust the example commands to suit your situation and server configuration.

  1. Download the Exadata database server update patch from My Oracle Support to your patchmgr server.

    Each Exadata database server update patch contains a compressed ISO image file.

    To find available Exadata database server update patches, see My Oracle Support document 888828.1.

  2. Use the zipped ISO file directly with the patchmgr update utility.

    In the patchmgr command, specify the zipped ISO file using the --repo option.

    For example:

    [root@pmserver ~]# patchmgr --dbnodes database_node_file --precheck --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109
[root@pmserver ~]# patchmgr --dbnodes database_node_file --upgrade --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109 --rolling

    The example uses the Exadata database server update patch archive (p35869377_231000_Linux-x86-64.zip) for Exadata release 23.1.8. The example assumes that the compressed patch archive is downloaded to /var/stage.

8.5.9 Updating Individual Packages

Due to security findings or customization, you may need to update individual (general purpose) Linux packages supplied by the Exadata release. You can do this by removing the exadata-sun-computenode-exact rpm first.

Removing this rpm does not impact any functionality, but it removes a “lock”. By removing this “lock”, you are allowed to update specific individual Linux rpms.

If needed, you can remove the exadata-sun-computenode-exact rpm as follows: 

[root@dm01 ]# rpm -e exadata-sun-computenode-exact

When updating to a new release, the update utility tries to “restore” the “lock” (exadata-sun-computenode-exact). For example, this is possible if newer packages are shipped in the newer release.

If the exadata-sun-computenode-exact rpm cannot be restored, the update utility falls back to exadata-sun-computenode-minimum.

Note:

Do not force in packages using the rpm -Uvh --nodeps command, unless directed by Oracle Support Services.

Parent topic: Updating Oracle Exadata Database Servers

8.5.10 Running Prerequisite Checks

You should always run prerequisite checks before doing the actual update. The prerequisite checks do not require downtime and perform important validations such as:

  • Validation of the Exadata release (minimum is 11.2.2.4.2 running Oracle Linux 5.5)

  • Validation of user input

  • Validation of the installation media (YUM repository, either ISO or HTTP)

  • Validation of disk space and snapshots

  • Validation of YUM settings that are important for the update to finish successfully

  • Checks based on known issues and best practices

The most important validation performed by the update utility is the YUM dependency check. The YUM dependency check is a YUM update dry-run command (introduced in 11.2.3.3.0) that does not do the actual YUM update but does validate dependencies. This is a final test in determining whether or not the update can proceed. It is often due to customizations that prevent successful updates. For example, installation of additional RPMs might require dependent packages that are not in the YUM repository. If this happens, you need to take corrective action to resolve the conflict.

The YUM dependency check (dry-run) is validated against minimum and exact dependencies. These dependencies are enforced by non-functional Exadata RPMs and help administrators stay exactly at (or close to) the original Exadata release when customizing the system. The update utility uses the exadata-sun-computenode-exact and the exadata-sun-computenode-minimum RPMs as follows:

  • The exadata-sun-computenode-exact rpm ensures that only a specific release of Oracle Exadata branded packages is allowed during the update. (release = x)

  • The exadata-sun-computenode-minimum rpm ensures that a specific or later release of Oracle Exadata branded packages is allowed during the update. (release >= x)

With exadata-sun-computenode-exact rpm, the system appears as if it were freshly imaged to the newer release because all the Oracle Exadata packages are exactly the same as on a vanilla installation. The exadata-sun-computenode-minimum rpm, however, sets the minimum dependencies, and enforces that all packages are installed, but it also allows packages to be at a later version. A vanilla installation always starts with both RPMs. To allow customization or updates, you need to remove exadata-sun-computenode-exact.

By default, the update utility attempts to match the exact dependencies when updating to a later Exadata release. When exact dependencies conflict and cannot be enforced, the utility falls back and attempts to apply the exadata-sun-computenode-minimum rpm to enforce minimum dependencies. In such cases the exadata-sun-computenode-exact rpm is not installed.

Missing or not updating with exact dependencies is allowed and not a problem. If a system needs to be updated to the exact dependencies, then the conflict needs to be resolved first. Check the log file to see what packages conflict, remove them cautiously, and then re-run the update utility in prerequisite check mode.

If the prerequisite check fails, you can examine the update utility’s log file for more details and determine which dependencies failed. When both exact and minimum dependencies do not match, the update cannot proceed.

For such cases, check the log file to determine what caused the dependencies to fail. After removing the failed dependencies, re-run the update utility to ensure that at least the minimum dependencies can be enforced.

When dependency errors occur during the prerequisite check or before the update starts, do the following to resolve the problem:

  • Analyze the YUM errors in the log file. Search for Error.

  • Depending on the issue, you may need to de-install, install, or update the rpm packages causing the dependency issue or conflict. The log file lists the failed dependencies.

After the update, you may re-install custom rpm packages that you de-installed, assuming you still require the packages and the packages are compatible with the updated system.

For more options, see the update utility built-in help.

Prerequisite Check Examples

The following command shows an example of a prerequisite check using an ISO repository. The command is run as root.

[root@pmserver ]# ./patchmgr --dbnodes dbs_group --precheck --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109

  • --dbnodes specifies the list of database nodes to be updated.

  • --precheck specifies the prerequisite check action.

  • --repo specifies the location of the compressed ISO file containing the update repository. If you specify a compressed ISO file, the file path must be accessible on the node running the updating utility. Alternatively, you can provide a URL to a YUM repository.

  • --target_version specifies the target release the database servers are being updated to.

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.11 Backing up Exadata Database Servers Before Planned Maintenance

Oracle recommends making a backup of the Exadata database servers before making any changes when updating to a next release.

This means running the update utility in backup-only mode before making any other (manual) changes to make the prerequisite / dependency check pass. The backup-only action backs up the active root and /boot file system only and is sufficient for rolling back (failed) updates.

Note:

The update utility’s built-in backup (dbserver_backup.sh) is insufficient for recovering from non-booting system failures. It is recommend having additional (validated) backup/restore procedures in place to recover the entire database server from failures such as double disk failures.

For regular and virtualized Exadata database servers (domU), when the active system image is running from a file system on /dev/mapper/VGExaDb-LVDbSys1, the backup is made onto /dev/mapper/VGExaDb-LVDbSys2 (and vice versa). For Exadata database servers running dom0 with /dev/mapper/VGExaDb-LVDbSys2 as the active image, the backup goes to /dev/mapper/VGExaDb-LVDbSys3 (and vice versa).

This means that upon rollback of an update the active system image for regular database servers and domU will become /dev/mapper/VGExaDb-LVDbSys2, and for dom0 will become /dev/mapper/VGExaDb-LVDbSys3.

The process of backing up an active system partition requires the default LVM scheme for the Sys* LVM’s and both LVM’s to be the same size. This is because it is impossible to back up a larger LVM partition to a smaller LVM partition. Resizing LVM VGExaDb-LVDbSys1 partition is allowed as long as VGExaDb-LVDbSys2 is resized to the same size.

While the backup is running, an LVM snapshot takes care of a consistent view of the file system. This LVM snapshot is maintained by the backup script and will always claim 1G of free VG space in VGExaDb.

Space for the snapshot is guaranteed by Oracle by using a placeholder LVM called /dev/VGExaDb/LVDoNotRemoveOrUse. When the backup script runs, it removes this placeholder, making sure there is always 1G of free space available for a snapshot. After the backup is completed, the snapshot is removed and the /dev/VGExaDb/LVDoNotRemoveOrUse LVM is re-created.

The time it takes to run the backup depends on how busy the system is and on the size and type of data that is backed up. For example, backing up millions of small files can take significantly longer than backing up a couple of larger files. For this reason it is recommended to make sure directories holding database .aud files are not found on the root file system. Note the following:

  • You can have only one backup. Running a new backup means overwriting the existing backup.

  • Re-running the update utility with the --backup flag (or in default updating mode) will overwrite existing backups.

  • All files in /boot and found on the active Sys LVM’s are backed up. Files in, for example, /u01, are not backed up.

Examples for “Backup Only”

The following example shows running “backup only” as root:

[root@pmserver ]# ./patchmgr --dbnodes dbs_group --backup --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109 --allow_active_network_mounts

  • --dbnodes specifies the list of database nodes to be updated.

  • --backup specifies the “backup only” action.

  • --repo specifies the location of the compressed ISO file containing the update repository. If you specify a compressed ISO file, the file path must be accessible on the node running the updating utility. Alternatively, you can provide a URL to a YUM repository.

  • --target_version specifies the target release for which the backup is run.

  • --allow_active_network_mounts ensures active network mounts remain active while performing the backup operation.

The following example shows a “backup only” action run as a non-root user:

[oracle@pmserver ]$ ./patchmgr --dbnodes ~/dbs_group -backup --repo /var/stage/p35869377_231000_Linux-x86-64.zip --target_version 23.1.8.0.0.231109 --log_dir auto --allow_active_network_mounts --smtp_from "sender@somedomain.com" --smtp_to "recipient@example.com"

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.12 Running the Update

You can perform the actual update of Exadata database servers in a rolling (using the --rolling flag) or non-rolling fashion. The default is non-rolling.

You can also perform the update as root or as a non-root user (using the --log_dir flag), as described in "Running the Update Utility as a Non-root User and Running Multiple Invocations Concurrently".

The update proceeds only if the “minimum dependency check” succeeds. You may need to remove customizations for the update to proceed.

By default the update creates a backup on the inactive system image. If you have already taken a backup and have not changed the image before running the prerequisite check, you can omit the -–nobackup flag when performing the update.

Note:

Use the --nobackup flag only if a backup was already made before running the prerequisite check and the image has not changed.

The update action requires the following mandatory flags:

  • -–upgrade specifies the update action

  • --repo Specifies the base URL for the Exadata update repository or the path to a zipped ISO file.)

  • --target_version specifies the release you want to update to. The patch README always has this information.

You can specify additional flags to allow active remote network mounts during backup and updating (--allow_active_network_mounts) and specify mail recipients for updating status notification (--smtp_from "addr" and --smtp_to "addr1 addr2 addr3 ...")

Example 8-12 Running Update Using ISO Image for YUM Repository

The following example shows an update action run as root and using an ISO image for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[root@pmserver ]# ./patchmgr --dbnodes ~/dbs_group --upgrade --repo /var/stage/p35869377_231000_Linux-x86-64.zip 
--target_version 23.1.8.0.0.231109 --allow_active_network_mounts --smtp_from "sender@somedomain.com" 
--smtp_to "receiver@somedomain.com" --nobackup

The following example shows an update action run as a non-root user from a remote host using an ISO YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[oracle@pmserver ]$ ./patchmgr --dbnodes ~/dbs_group --upgrade --repo /var/stage/p35869377_231000_Linux-x86-64.zip 
--target_version 23.1.8.0.0.231109 --allow_active_network_mounts --log_dir auto --smtp_from "sender@somedomain.com" 
--smtp_to "receiver@somedomain.com" --nobackup

Example 8-13 Running Update Using HTTP Location for YUM Repository

The following example shows an update action run as root using HTTP for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[root@pmserver ]# ./patchmgr --dbnodes ~/dbs_group --upgrade --repo http://yum-repo/yum/EXADATA/dbserver/23.1.8/base/x86_64/ 
--target_version 23.1.8.0.0.231109 --allow_active_network_mounts --smtp_from "sender@somedomain.com" --smtp_to "receiver@somedomain.com"
--nobackup

The following example shows an update action run as a non-root user from a remote host using HTTP for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[oracle@pmserver ]$ ./patchmgr --dbnodes ~/dbs_group --upgrade --repo http://yum-repo/yum/EXADATA/dbserver/23.1.8/base/x86_64/ 
--target_version 23.1.8.0.0.231109 --allow_active_network_mounts --log_dir auto --smtp_from "sender@somedomain.com" 
--smtp_to "receiver@somedomain.com" –nobackup

Related Topics

Parent topic: Updating Oracle Exadata Database Servers

8.5.13 Rolling Back Updates

A backup enables you to roll back updates, regardless of whether the update failed or succeeded. This backup is stored on the inactive system partition, as described in "Backing up Exadata Database Servers Before Planned Maintenance".

When rolling back an update, the update utility performs the following actions:

  • Shuts down the stack and domU’s.
  • Deactivates the active system partition, and activates the inactive system partition.
  • Restores /boot from the inactive partition.
  • Updates the grub boot-loader.

Having only one inactive system partition limits the rollback options to only the previous active image.

Note:

  • For systems being updated to Oracle Linux 6, a backup must be performed before proceeding with the update. The backup is automatic when updating LVM-enabled systems from Oracle Linux 5 to Oracle Linux 6.
  • Database servers running as Oracle VM Server (dom0) switch between LVDbSys2 and LVDbSys3 as the active system partition when rolling back.
  • Database servers running as Oracle VM (domU) have smaller sizes for LVDbSys1 compared to physical hardware deployments.

Example 8-14 Rolling back an update using patchmgr

[root@pmserver ]# ./patchmgr --dbnodes dbs_group --rollback

--dbnodes specifies the list of database nodes to be updated.

--rollback specifies the rollback action.

For more options see the update utility’s built-in help.

Note:

Firmware updates are not rolled back when rolling back to a previous image. The Oracle Exadata System Software releases support later firmware releases. After rolling back, run the following commands to apply older firmware versions when needed: 

/etc/init.d/lsidiag stop

/etc/init.d/lsi_mrdsnmpd stop

/opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact

The last command only applies to releases 11.2.3.3.0 or later.

Parent topic: Updating Oracle Exadata Database Servers

8.6 Updating Software on Oracle Exadata Storage Servers

Use the following information and procedures when updating storage servers within Oracle Exadata.

Parent topic: Updating Exadata Software

8.6.1 Overview of Oracle Exadata Storage Server Updates

When updating storage servers, there is more than one type of software that needs to be updated and different methods of performing the updates.

Oracle Exadata System Software release updates contain updates for the following components within an Oracle Exadata storage server:

  • Oracle Linux operating system
  • Firmware (Flash, Disk, RAID controller, Integrated Lights Out Manager (ILOM), HCA)
  • Oracle Exadata System Software

What software and firmware that is updated depends on the current Oracle Exadata System Software release the storage server is on and the release it is updated to. Oracle Linux operating system packages and Oracle Exadata System Software are always updated, while firmware updates are applied only on a small selection of the components, or not at all.

Updates for Oracle Exadata storage server can be applied independently from the updates to Oracle Exadata database server or RDMA Network Fabric switches unless specified otherwise. It is not mandatory to apply each and every Oracle Exadata System Software update that comes out. For example, you can skip two or three releases and update directly to a newer release.

Updating the Oracle Exadata System Software is always performed “out of place”. This means that a new version of the operating system including the Oracle Exadata System Software is installed on the inactive system partition. The utility to update the Oracle Exadata System Software ships with the update itself.

If you cannot afford cluster-wide downtime, you can update the storage servers in a rolling fashion. Rolling means updating one storage server at a time. If you can afford the cluster-wide downtime, you can update all storage servers in parallel. Non-rolling updates reduce the overall time required.

Starting with Oracle Exadata System Software release 18c (18.1.0), there is a more scalable alternative to using patchmgr for software updates. The storage servers automatically verify preconditions and download the update software from a URL that you specify. Each storage server downloads the software to its active partition, and then loads the software on its passive partition. At a specified time, the storage servers reboot to the new version. The storage servers use the Oracle Automatic Storage Management (Oracle ASM) disk deactivation status to determine when it is safe to deactivate the disks and reboot the storage server to the new software version. These scheduled updates invoke the same scripts that are currently used by the patchmgr process.

Parent topic: Updating Software on Oracle Exadata Storage Servers

8.6.2 Scheduling Automated Updates of Storage Servers

Starting with Oracle Exadata System Software release 18.1.0.0.0, you can schedule software updates for the storage servers.

Perform the following steps from an external server, for example, an Oracle Exadata Database Server.

If you are accessing the Software Update store using the HTTPS protocol, then TLS certificate checks are required by default. If the certificate for the web server that hosts the software update cannot be validated, then the following error is returned:

CELL-00076: An error occurred during download of software update:
source https://example.com:port is not available. 
CELL-00092: The store's TLS certificate cannot be authenticated with 
known CA certificates
  1. Copy the software update ZIP file to a directory which is hosted by a web server.
  2. If you are using Oracle Exadata System Software 18c (18.1.0) or 18c (18.1.1), then the patch file must have a name like 18.1.1.0.0.171018.patch.zip.

    If the downloaded patch has a name like p26875767_181100_Linux-x86-64.zip, then rename the file to 18.1.1.0.0.17018.patch.zip. Rename the ZIP file to use the release number and date string that is used for the directory name within the ZIP file. For example, when you unzip the patch p26875767_181100_Linux-x86-64.zip it extracts the directory patch_18.1.1.0.0.171018.

    If you are using Oracle Exadata System Software release 18.1.2 or higher, then the patch downloaded from My Oracle Support is automatically renamed for you during the validation step.

  3. Determine approximate time to perform the software update of the storage servers.
  4. Create a file on the external server that contains the list of cells to be updated. Name this file cells.
  5. Use dcli to schedule the update of the cells.
    1. Setup passwordless access, if needed.
      $ dcli –g cells –k
    2. Specify the location of the software update ZIP file to use during the update.
      $ dcli –g cells cellcli –e 'alter softwareUpdate store=\"https://host/exa-updates/cell\"'
    3. Specify the time to start the update of the Oracle Exadata System Software on the storage servers.

      If you specify the time before providing the software store location, then the software update download might start before the proper store location has been set.

      $ dcli –g cells cellcli –e 'alter softwareUpdate time=\"1 AM Thursday\"'
  6. Wait for the updates to occur.

    Management Server (MS) will the start file download and run pre-checks up to one week before the scheduled update. MS will generate an alert if any cell does not update as scheduled.

Related Topics

Parent topic: Updating Software on Oracle Exadata Storage Servers

8.6.3 Update Utility for Oracle Exadata Storage Server

You use the patchmgr update utility for updating Oracle Exadata storage servers. For Exadata storage server updates the utility is packaged (and shipped) with the update itself and is available for download from My Oracle Support as storage server update.

Note:

The patchmgr used for updating Exadata storage servers is not the same as the patchmgr used for applying the Exadata database server software update.

Whether or not the utility supports the Exadata hardware you have depends on the Exadata storage server release you are trying to update to. The utility orchestrates the update process across the specified Exadata storage servers. The utility allows running the update in a rolling or non-rolling fashion. You can run the update utility from Exadata database servers or from other servers running Oracle Linux or Oracle Solaris.

The update utility performs the following tasks:

  • Automates the preparation, update, and validation steps

  • Automates rollbacks

The update utility supports multiple sessions: you can run multiple updates concurrently from the same server starting with release 12.1.2.3.2 for Exadata storage servers and starting with release 11.2.3.1.0 for Exadata database servers. This means multiple racks can be updated concurrently from the same server. The update utility can be run as root or as a non-root user. By default the update utility assumes it should run as the root user. If however you want to enable multiple session support or run as a non-root user, then you need to use the -log_dir flag. The -log_dir flag supports two types of arguments: either a location on disk or the keyword auto. If you specify auto, the update utility creates its own log directory based on the storage servers listed in the cell_group file. This behavior causes the update utility to create new directories for each run of updates in the same cluster where one or more clusters were added or removed from the cell_group file. In order to obtain (and reuse) such a directory, the update utility provides the -get flag to determine the log directory for your session. The-get flag scans the working directory for directories in the log directory and returns the directory for your cell_group. For example, the following command: 

[oracle@nonExadataHost ]#./patchmgr -dbnodes ~/cell_group  -log_dir auto -get log_dir

The previous comment might return output similar to the following:

log_dir=/u01/test/patch_12.1.2.4.0.160802/log/dbm02celadm01_dbm02celadm02_9cfbc690

In a subsequent update session, you can re-use the log directory location:

[oracle@nonExadataHost ]# ./patchmgr -cells ~/cell_group -patch_check_prereq -log_dir /u01/test/patch_12.1.2.4.0.160802/log
/dbm02celadm01_dbm02celadm02_9cfbc690

Parent topic: Updating Software on Oracle Exadata Storage Servers

8.6.4 Recommended Timeline for Updating Oracle Exadata Storage Server

Note:

It is highly recommended to validate Oracle Exadata System Software updates on test systems before doing them on production systems.

By following the approach below, you allow yourself time to perform any necessary corrective actions.

Table 8-9 Timeline for Performing Updates

When Tasks

Weeks to days before the update

  • Download the Oracle Exadata System Software update you require from My Oracle Support note 888828.1

  • Download the latest Oracle EXAchk from My Oracle Support note 1070954.1

  • Review the release-specific My Oracle Support note for known issues

  • Run Oracle EXAchk

  • Perform prerequisite check. See Preparing Exadata Storage Servers for Update for details.

  • Correct findings that need attention and re-run the above steps as needed.

At update time

  • Download the Oracle Exadata System Software update from My Oracle Support note 888828.1

  • Download the latest Oracle EXAchk from My Oracle Support note 1070954.1

  • Review the release-specific My Oracle Support note for known issues

  • Run Oracle EXAchk and perform corrective actions as needed

  • Perform prerequisite check and corrective actions as needed.

  • Perform the update. See Running the Update for Exadata Storage Servers for details.

After the update

  • Run Oracle EXAchk.

Parent topic: Updating Software on Oracle Exadata Storage Servers

8.6.5 Preparing Exadata Storage Servers for Update

Perform these preparation steps before updating the Exadata storage servers.

You can perform the update (and rollback) action in a rolling or non-rolling method. You can also perform a prerequisite check in a rolling or non-rolling method. The default is non-rolling.

  1. Set up SSH equivalence for the user that is driving the update utility.
  2. Download and run Oracle ExaCHK. Review and address any open issues. See My Oracle Support note 1070954.1.
  3. Review the release-specific My Oracle Support note for any known issues and workarounds.
  4. Check prerequisites for your method of update or rollback.

    Prerequisites for performing a rolling update:

    1. Verify that your Grid Infrastructure home and Database home software versions and patch levels meet the minimum required for Exadata storage server rolling cell update as documented in My Oracle Support note 888828.1
    2. Verify failgroup_repair_time or disk_repair_time for each Oracle ASM disk group.

      When applying the update in a rolling manner, the update utility updates one server at a time, first taking all grid disks and ASM disks offline, then applying the update to the server, then bringing all ASM disks and grid disks back online. The Oracle ASM repair timeout attributes, disk_repair_time and failgroup_repair_time, need to be set to a value large enough to allow a single storage server update to complete. The default values of 3.6h and 24h, respectively, are the recommended values. Note that during a rolling storage server update disk groups with compatible.asm >= 12.1.0.2.0 will use the value of failgroup_repair_time, and disk groups with compatible.asm < 12.1.0.2.0 will use the value of disk_repair_time.

      Verify ASM repair timeout attributes are set to default or higher values. Use the following command to check repair time attributes for all mounted disk groups in the Oracle ASM instance.

      SQL> col attribute format a30
SQL> col value format a10
SQL> select dg.name as diskgroup, a.name as attribute, a.value
     from v$asm_diskgroup dg, v$asm_attribute a
     where dg.group_number=a.group_number
       and (a.name like '%repair_time' or a.name = 'compatible.asm');

DISKGROUP                      ATTRIBUTE                      VALUE
------------------------------ ------------------------------ ----------
DATA                           disk_repair_time               3.6h
DATA                           failgroup_repair_time          24.0h
DATA                           compatible.asm                 12.1.0.2.0
RECO                           disk_repair_time               3.6h
RECO                           failgroup_repair_time          24.0h
RECO                           compatible.asm                 12.1.0.2.0

      If the Oracle ASM repair timer for any disk group is lower than the default value, then set the repair timer to the default value for the duration of the update. You may set it back to its current value after the update successfully finishes for all storage servers.

    Prerequisites for performing a non-rolling update:

    1. Shut down and stop the Oracle components on each Exadata database server using the following commands, where Grid_home is the directory where the Oracle Grid Infrastructure software is installed:
      [root@dm01 ]# dcli -g dbs_group -l root "Grid_home/bin/crsctl stop crs"

      If Oracle Clusterware was not stopped using the preceding command, then use the following command to force it to stop:

      [root@dm01 ]# crsctl stop crs -f
    2. Use the following command to check Oracle Clusterware status, where Grid_home is the directory where the Oracle Grid Infrastructure software is installed:
      [root@dm01 ]# dcli -g dbs_group -l root "Grid_home/bin/crsctl check crs"

      All Oracle Clusterware components must be offline. If you are performing a non-rolling update in a configuration running Oracle VM, then you must check the Oracle Clusterware state in all VM clusters.

  5. Check if your system meets the following conditions:
    • Oracle Exadata Database Machine hardware uses Sun servers.
    • The installed Oracle Exadata Software release is earlier than release 11.2.2.2.0.
    • The installed Linux ofa package is earlier than 1.5.1-4.0.28.

    If the system meets all of the preceding conditions, then Exadata storage servers running Oracle Linux may encounter a file system corruption that results in the root file system mounted as read-only after reboot. Follow the instructions in My Oracle Support note 1589868.1 before updating the storage servers and database servers.

  6. Unzip the update. It will extract into the patch_release.date_code directory. Change to this patch directory.

  7. Download any patchmgr plug-ins attached to the My Oracle Support note for your target release and install them as documented in the My Oracle Support note. Oracle recommends reviewing the My Oracle Support notes, issues, and workarounds listed in the release note just before starting to actually apply the update.
  8. Clean up any previous update utility runs using the -cleanup flag.

    Note:

    The first time the storage servers are updated the -reset_force flag should be used before running cleanup.

    Example using -reset_force as the root user: 

    [root@dm01 ]# ./patchmgr -cells ~/cell_group –reset_force

    Example using -reset_force as a non-root user: 

    [oracle@nonExadataHost ]# ./patchmgr -cells ~/cell_group -log_dir auto –reset_force

    Example using -cleanup as the root user: 

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -cleanup

    Example using -cleanup as a non-root user: 

    [oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto –cleanup
  9. Run prerequisite check.

    Example running prerequisite check for a rolling update as the root user from an Exadata database server:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -patch_check_prereq -rolling -smtp_from "sender@example.com"  
-smtp_to receiver@example.com

    Example running prerequisite check for a non-rolling update as a non-root user from a non-Exadata database server:

    [oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto -patch_check_prereq -smtp_from "sender@example.com"
-smtp_to "receiver@example.com"

Related Topics

Parent topic: Updating Software on Oracle Exadata Storage Servers

8.6.6 Running the Update for Exadata Storage Servers

After performing the prerequisite steps in Preparing Exadata Storage Servers for Update, you can perform the actual update step.

Note the following when applying the update to the Exadata storage servers:

  • Do not use the serial console or the ILOM web-based console to start the update utility.

    There is a known issue of a system halt on the serial console when a write is attempted to stderr or stdout. If an update is started from the serial console, then it may halt.

    You are using the serial console if the output from the following command is serial.

    [root@dm01 ]# ./echo $consoletype

  • When needed, use the ILOM web-based console to monitor the storage server during the update. You will need to use the ILOM web-based console in case troubleshooting is required.

  • To obtain ILOM and serial console access for the storage servers, use SSH to the ILOM host name or IP address as the root user. Do the following to start the serial console:

    start /SP/console

    To stop it press the Escape key (ESC) followed by (.

  • Start a new login session for each update or rollback procedure. Do not run a rollback procedure from the same login session where an update was applied. Do not run an update from a login session where a rollback procedure was run.

  • Do not interrupt the update process.

  • If you must use a storage server as the patchmgr utility launch system, then do not use /opt/oracle as the staging area for the update. This causes the update to fail and corrupt the storage server. Use the /tmp directory as the staging area, that is, unzip the files for the update in /tmp.

  • Storage servers automatically reboot, as needed, during the update process. Do not reboot or power cycle storage servers while applying updates.

  • Do not edit or open log files in writable mode. You may use any of the following to view a log: view, less, more, or tail. You may cause the update process to be interrupted if you edit the log files during the update.

  • At the end of the patchmgr session, the patchmgr.stdout log file is divided into individual storage server log files with names in the format of cell_name.log. In addition, the /var/log/cellos content from the inactive cell partition is copied to the /var/log/cellos/inactive_partition directory. To locate the inactive partition, use the following command: 

    [root@dm01 ]# ./imageinfo -inactive -sys

Examples running update:

Running the update in a rolling fashion as root from an Exadata database server:

[root@dm01 ]# ./patchmgr -cells ~/cell_group -patch -rolling -smtp_from "sender@somedomain.com" -smtp_to receiver@somedomain.com

Running the update in a non-rolling fashion as a non-root user from a non-Exadata database server:

[oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto -patch -smtp_from "sender@somedomain.com" -smtp_to "receiver@somedomain.com"

After the update is done, clean up the storage servers using the -cleanup option to clean up all the temporary update or rollback files. This option cleans the stale update and rollback states as well as cleaning up to 1.5 GB of disk space on the storage server. Use this option before retrying a halted or failed run of the update utility. See step 8 for details.

Parent topic: Updating Software on Oracle Exadata Storage Servers

8.6.7 Rolling Back Updates for Exadata Storage Servers

You can roll back updated Exadata storage servers only when they are updated successfully. This means the imageinfo command must return success for active image status. Storage servers with incomplete or failed updates cannot be rolled back. Rollbacks can be done in a rolling or non-rolling fashion.

  1. Check the version that the storage servers will be rolled back to and the flashCacheMode setting with the following commands: 

    [root@dm01 ]# dcli -l root -g cell_group imageinfo -ver -inactive

[root@dm01 ]# dcli -l root -g cell_group cellcli -e 'list cell attributes flashCacheMode

    Note:

    If you need to roll back storage servers to releases earlier than release 11.2.3.2.0 with writeback flash cache enabled, you need to convert the flash cache to writethrough flash cache before performing the rollback action. Disable the writeback flash cache using the script in My Oracle Support note 1500257.1. Storage servers being rolled back to release 11.2.3.2.0 or later retain the flash cache mode that is currently set.

  2. Check the prerequisites for rollback using the following command:

    [root@dm01 ]# ./patchmgr -cells cell_group -rollback_check_prereq [-rolling]

  3. Perform the rollback.

    Example of a non-rolling rollback run as root:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -rollback

    Example of a rolling rollback run as a non-root user:

    [oracle@nonExadataHost ]#./patchmgr -cells ~/cell_group -rollback -rolling -log_dir auto

    Note:

    Firmware updates are not rolled back when rolling back to a previous image. After rolling back, run the following command to apply older firmware versions when needed:

    /etc/init.d/lsidiag stop

/etc/init.d/lsi_mrdsnmpd stop

/opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact

  4. Clean up the Exadata storage servers using the -cleanup option to clean up all the temporary update or rollback files. This option cleans the stale update and rollback states as well as cleaning up to 1.5 GB of disk space on the Exadata storage servers. Use this option before retrying a halted or failed run of the patchmgr utility. 

    [root@dm01 ]# ./patchmgr -cells cell_group -cleanup

Parent topic: Updating Software on Oracle Exadata Storage Servers

8.7 Upgrading and Downgrading RoCE Network Fabric Switch Firmware

This topic describes procedures to upgrade and downgrade the firmware on the RoCE Network Fabric switches.

Note the following when updating the RoCE Network Fabric switch firmware:

  • Upgrading and downgrading the RoCE Network Fabric switch firmware is performed using the patchmgr utility.
  • Starting with Oracle Exadata System Software release 19.3.0, there are separate patchmgr distributions for the servers and RDMA Network Fabric switches. Ensure that you use the patchmgr distribution included in the patch ZIP file for the RoCE Network Fabric switches.
  • Download the appropriate patch ZIP file to any machine with access to the RoCE Network Fabric switches. Refer to My Oracle Support note 888828.1 for the patch information.
  • patchmgr configures ssh passwordless-access to each switch, which requires you to provide the password of the admin user for the switch.
  • You must use a non-root user to perform the patching, and must include the -log_dir option with patchmgr.
  • Switch firmware is always upgraded in a rolling manner (one switch at a time).
  • Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. The RDMA Network Fabric switch firmware upgrade requires a switch reboot, which disrupts some connections on the RDMA Network Fabric.
  • Commencing with the August 2022 patchmgr release, patchmgr performs an additional series of checks on the RoCE Network Fabric. The checks occur immediately before any firmware upgrade or downgrade and also during prerequisite checking using the --roceswitch-precheck option. These checks mitigate the risks of failure associated with unexpected problems in the RoCE Network Fabric. For example, if one of the RoCE Network Fabric ports on a storage server is down, the storage server would become unavailable if the switch connected to the only operational port is taken offline for an upgrade. If any check fails, patchmgr reports the problem and ends immediately. In this case, you must correct the problem with the RoCE Network Fabric before you can perform the upgrade or downgrade.

Use the following procedures to upgrade and downgrade the firmware on the RoCE Network Fabric switches:

Parent topic: Updating Exadata Software

8.7.1 Preparing for RoCE Network Fabric Switch Firmware Upgrades or Downgrades

You must follow a specific order when upgrading the RoCE Network Fabric switches.

  1. Log in to a server that has access to the RoCE Network Fabric switches.
  2. Download the appropriate patch file to the server.

    Starting with Oracle Exadata System Software release 19.3.0, the updates for the switches are in a separate patch. Refer to My Oracle Support note 888828.1 for patch information.

  3. Unzip the patch file.

    The files are unzipped to the patch_switch_release directory.

  4. Change to the directory that contains the patchmgr utility.

    For example:

    # cd patch_switch_19.3.0.0.0.190915
  5. Create a switch list file to drive the update of the RoCE Network Fabric switches.
    1. Create a file that contains the host name or IP address of the switches that you want to upgrade. Place each switch on a separate line.
      For example, create a file named switches.lst, which contains the host name of each switch on separate lines. On a single rack system, the file might contain:
      switch123-rocea0
switch123-roceb0
    2. Tag each line to specify the configuration type for each switch.

      To specify the configuration type for each switch, append a colon (:) and tag to each switch host name or IP address in the switch list file. The following tags are supported:

      • leaf - Identifies a leaf switch in a single rack system. This configuration type is assumed if no tag is specified.
      • mspine - Identifies a spine switch. Note that one spine switch configuration supports all spine switches on single and multi-rack systems, with and without Exadata Secure RDMA Fabric Isolation.
      • mleaf - Identifies a leaf switch in a multi-rack X8M system.
      • sfleaf - Identifies a leaf switch in a single rack system that is enabled to support Exadata Secure RDMA Fabric Isolation.
      • msfleaf - Identifies a leaf switch in a multi-rack X8M system that is enabled to support Exadata Secure RDMA Fabric Isolation.
      • leaf23 - Identifies a leaf switch in a single rack system that is configured with 23 host ports. This configuration is required only for 8-socket systems (X8M-8 and later) with 3 database servers and 11 storage servers.
      • mleaf23 - Identifies a leaf switch in a multi-rack system that is configured with 23 host ports. This configuration is required only for 8-socket X8M-8 systems with 3 database servers and 11 storage servers.
      • mleaf_u14 - Identifies a leaf switch in a multi-rack system that is configured with 14 inter-switch links. This is the typical multi-rack leaf switch configuration for X9M and later model systems.
      • msfleaf_u14 - Identifies a leaf switch in a multi-rack system that is enabled to support Exadata Secure RDMA Fabric Isolation and is configured with 14 inter-switch links. This configuration is required for X9M and later model systems with Secure Fabric enabled.
      • mleaf23_u13 - Identifies a leaf switch in a multi-rack system that is configured with 23 host ports and 13 inter-switch links. This configuration is required only for 8-socket X9M-8 systems with three database servers and 11 storage servers.
      For example:
      switch123-rocea0:leaf
switch123-roceb0:leaf
    3. For multi-rack configurations only, specify a unique loopback octet for each switch.

      The loopback octet is the last octet of the switch loopback address, which uniquely identifies a switch.

      To specify the loopback octet for each switch, append a period (.) and numeric loopback octet value to each tagged switch entry in the switch list file. The range of valid loopback octet values is:

      • 101-118 for leaf switches
      • 201-208 for spine switches
      For example, the switch list file for a 2-rack system might contain:
      rack1sw-rocea0:mleaf.101
rack1sw-roceb0:mleaf.102
rack1sw-roces0:mspine.201
rack2sw-rocea0:mleaf.103
rack2sw-roceb0:mleaf.104
rack2sw-roces0:mspine.202
  6. Run the prerequisite check prior to either upgrading or downgrading the firmware.
    # ./patchmgr --roceswitches switches.lst {--upgrade | --downgrade} --roceswitch-precheck [--force]
  [-log_dir {absolute_path_to_log_directory | AUTO}]

    In the patchmgr command:

    • --roceswitch-precheck instructs patchmgr to perform a firmware upgrade or downgrade simulation on the switch.

    • --force optionally proceeds with the operation even if the switch is already on the target firmware version or the RoCE Network Fabric is experiencing non-critical failures.

    • -log_dir specifies the absolute path to the log directory, or AUTO instructs patchmgr to automatically create the log directory. This option is required when running patchmgr as a non-root user.

    Note:

    Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.

    Note:

    The current user is expected to have SSH equivalency configured prior to running patchmgr. If it is not configured, then patchmgr will give you the option to setup keys and key exchange for SSH equivalency.

    If the output from the command shows overall status is SUCCESS, then proceed with the upgrade. If the output from the command shows overall status is FAIL, then review the error summary in the output to determine which checks failed, and then correct the errors. After correcting all the errors, rerun the prerequisite checks until it is successful.

    Note:

    You may see a Config validation failed error because the golden configuration settings have not been applied to the switch. In this case, apply the golden configuration settings from the current patch and then repeat the prerequisite check. See Applying Golden Configuration Settings on Cisco Nexus 9336C-FX2 RoCE Network Fabric Switches.

Related Topics

Parent topic: Upgrading and Downgrading RoCE Network Fabric Switch Firmware

8.7.2 Upgrading RoCE Network Fabric Switch Firmware Software

Upgrade the RoCE Network Fabric switches using the patchmgr command.

Before starting this task, review the notes in Upgrading and Downgrading RoCE Network Fabric Switch Firmware, and complete the steps in Preparing for RoCE Network Fabric Switch Firmware Upgrades or Downgrades.

  1. On the server where you downloaded the patch, use the patchmgr command to upgrade the switches.
    # ./patchmgr --roceswitches switches.lst --upgrade [--force]
  [-log_dir {absolute_path_to_log_directory | AUTO}]

    In the patchmgr command:

    • --force optionally proceeds with the operation even if the switch is already on the target firmware version or the RoCE Network Fabric is experiencing non-critical failures.

    • -log_dir specifies the absolute path to the log directory, or AUTO instructs patchmgr to automatically create the log directory. This option is required when running patchmgr as a non-root user.

    Note:

    Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.
  2. Verify the upgrade.

    Use the show version command to verify the firmware version on the switch:

    For example:

    # show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2019, Cisco and/or its affiliates.
All rights reserved.
...

Software
  BIOS: version 05.33
  NXOS: version 7.0(3)I8(1)
  BIOS compile time: 09/08/2018
  NXOS image file is: bootflash:///nxos.7.0.3.I8.1.bin
  NXOS compile time: 3/5/2019 13:00:00 [03/05/2019 22:04:55]

Hardware
  cisco Nexus9000 C9336C-FX2 Chassis
  Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 24571632 kB of memory.
  Processor Board ID FDO23040CS1

  Device name: dbm01sw-rocea0
  bootflash: 115805356 kB
Kernel uptime is 17 day(s), 20 hour(s), 50 minute(s), 25 second(s)

Last reset at 188268 usecs after Mon Aug 12 17:14:40 2019
  Reason: Module PowerCycled
  System version:
  Service: HW check by card-client

plugin
  Core Plugin, Ethernet Plugin

Active Package(s):

Parent topic: Upgrading and Downgrading RoCE Network Fabric Switch Firmware

8.7.3 Downgrading RoCE Network Fabric Switch Firmware

Downgrading firmware means applying older firmware.

Note:

The current storage server release determines what release you can downgrade to. This may be different for each release and may not be the firmware you were on before the upgrade. For more information on the older firmware shipped with the release you are upgrading to, see the patch README file.

Before starting this task, review the notes in Upgrading and Downgrading RoCE Network Fabric Switch Firmware, and complete the steps in Preparing for RoCE Network Fabric Switch Firmware Upgrades or Downgrades.

  1. Use the patchmgr command to downgrade the firmware on the RoCE Network Fabric switches.
    # ./patchmgr --roceswitches switches.lst --downgrade [--force]
  [-log_dir {absolute_path_to_log_directory | AUTO}]

    In the patchmgr command:

    • --force optionally proceeds with the operation even if the switch is already on the target firmware version or the RoCE Network Fabric is experiencing non-critical failures.

    • -log_dir specifies the absolute path to the log directory, or AUTO instructs patchmgr to automatically create the log directory. This option is required when running patchmgr as a non-root user.

    Note:

    Prior to Oracle Exadata System Software release 19.3.9, you must run patchmgr as a non-root user for patching RoCE Network Fabric switches.
  2. Use the show version command to verify that the switch firmware is downgraded.
    # show version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2019, Cisco and/or its affiliates.
All rights reserved.
...

Software
  BIOS: version 05.33
  NXOS: version 7.0(3)I7(6)
  BIOS compile time: 09/08/2018
  NXOS image file is: bootflash:///nxos.7.0.3.I7.6.bin
  NXOS compile time: 3/5/2019 13:00:00 [03/05/2019 22:04:55]

Hardware
  cisco Nexus9000 C9336C-FX2 Chassis
  Intel(R) Xeon(R) CPU D-1526 @ 1.80GHz with 24571632 kB of memory.
  Processor Board ID FDO23040CS1

  Device name: dbm01sw-rocea0
  bootflash: 115805356 kB
Kernel uptime is 17 day(s), 20 hour(s), 50 minute(s), 25 second(s)

Last reset at 188268 usecs after Mon Aug 12 17:14:40 2019
  Reason: Module PowerCycled
  System version:
  Service: HW check by card-client

plugin
  Core Plugin, Ethernet Plugin

Active Package(s):

Parent topic: Upgrading and Downgrading RoCE Network Fabric Switch Firmware

8.8 Updating InfiniBand Network Fabric Switch Firmware

This topic describes procedures to upgrade and downgrade the firmware on the InfiniBand Network Fabric switches.

Note the following when updating the InfiniBand Network Fabric switch firmware:

  • The upgrade of the InfiniBand Network Fabric switches is performed using patchmgr.

  • The minimum switch firmware release that can use the update utility is release 1.3.3-2. Switch firmware is always updated in a rolling manner.

  • Download the appropriate patch ZIP file to any machine with access to the InfiniBand Network Fabric switches. Refer to My Oracle Support note 888828.1 for the patch information.

  • Switch firmware is always upgraded in a rolling manner.

Note:

Before Oracle Exadata System Software release 19.3.0, the storage server and RDMA Network Fabric switch updates use the version of patchmgr bundled with the Oracle Exadata System Software release. Starting with Oracle Exadata System Software release 19.3.0, there is a separate patchmgr download for the RDMA Network Fabric switches and the storage servers.

Perform storage server updates separately from RDMA Network Fabric switch updates. Do not update storage servers and RDMA Network Fabric switches concurrently. RDMA Network Fabric network connections must be stable during some critical stages of storage server updates. The RDMA Network Fabric switch firmware upgrade requires a switch reboot, which disrupts some connections on the RDMA Network Fabric.

Parent topic: Updating Exadata Software

8.8.1 Preparing for InfiniBand Network Fabric Switch Firmware Updates

You must follow a specific order when updating the InfiniBand Network Fabric switches.

  • If a spine switch is present in the rack, it needs to be updated first.
  • If a spine switch is not in the rack, then update the switch running the subnet manager first.
  • If the subnet manager is not running on the switches, then perform the update in any order.

To update the InfiniBand Network Fabric switches, the switch firmware must be at release 1.3.3-2 or later. If the switch firmware is at an earlier release, then update the firmware to release 1.3.3-2 using the instructions in My Oracle Support note 888828.1. :

  1. Log in as the root user to an Oracle Exadata database server that has root user SSH access to the switches.
    The database server must be on the same InfiniBand Network Fabric network as the switches.
  2. Use the version command to determine the version of the software the InfiniBand Network Fabric switch is running.
    [root@dbm01-ibs0 ~]# version

SUN DCS 36p version: 2.1.8-1
Build time: Sep 18 2015 10:26:47
SP board info:
Manufacturing Date: 2015.07.01
Serial Number: "NCDLD0049"
Hardware Revision: 0x0200
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010
  3. Download the appropriate patch file to the database server.
    Refer to My Oracle Support note 888828.1 for patch information.
  4. Uncompress the update.
    The files are uncompressed to the patch_release.date directory.
  5. Create a file listing all the InfiniBand Network Fabric switches that need to be updated, with one switch per line.

    Use the command ibswitches to identify the switches in your rack. Note that switches from Engineered Systems other than Oracle Exadata might be visible on the same fabric but should probably not be updated at this time.

    The following is an example of the file constructed after running the ibswitches command: 

    [root@dm01 ]# cat ibswitches.lst
myibswitch-01
myibswitch-02

    Note:

    If no file name is provided, then the command runs on all InfiniBand Network Fabric switches discovered from this host by running ibswitches command.
  6. Change to the patch_release.date directory.
  7. Run the prerequisite check.
    [root@dm01 ]# ./patchmgr --ibswitches ibswitches.lst --upgrade --ibswitch_precheck [--force] [--unkey]

    Note:

    Starting with Oracle Exadata System Software release 19.3.0, the patchmgr command uses -- instead of a single hyphen before keywords.

    The --unkey option removes passwordless SSH access to the InfiniBand Network Fabric switches before exiting.

    The --force option overrides failures in the InfiniBand Network Fabric topology and connectivity from the servers to the switches. This does not affect the upgrade of the switch.

If the output from the command shows overall status is SUCCESS, then proceed with the upgrade. If the output from the command shows overall status is FAIL, then review the error summary in the output to determine which checks failed, and then correct the errors. After correcting all the errors, rerun the prerequisite checks until it is successful.

Related Topics

Parent topic: Updating InfiniBand Network Fabric Switch Firmware

8.8.2 Updating InfiniBand Network Fabric Switch Firmware Software

Update the InfiniBand Network Fabric switches using the patchmgr command.

The minimum switch firmware release that can use the patchmgr utility is release 1.3.3-2. The switch firmware is upgraded in a rolling manner.

  • If a spine switch is present in the rack, then the spine switch is upgraded first.
  • If a spine switch is not in the rack, then upgrade the switch that is running the Subnet Manager first.
  • If the Subnet Manager is not running on the switches, then perform the upgrade in any order.

You should have completed the steps in Preparing for InfiniBand Network Fabric Switch Firmware Updates before starting this task.

  1. Log in as the root user to a database server in the Oracle Exadata Rack that has root user SSH access to the switches.
    The database server must be on the same InfiniBand Network Fabric network as the switches.
  2. Download the appropriate patch file to the database server.
    Refer to My Oracle Support note 888828.1 for patch information.
  3. Uncompress the patch files.
    The files are uncompressed to the patch_release.date directory.
  4. Change to the patch_release.date directory.
  5. Run the prerequisite checks using patchmgr.

    The --unkey option removes passwordless SSH access to the InfiniBand Network Fabric switches before exiting.

    The --force option overrides failures in the InfiniBand Network Fabric topology and connectivity from the servers to the switches. This does not affect the upgrade of the switch. 

    # ./patchmgr --ibswitches ibswitches.lst --upgrade --ibswitch_precheck [--force] [--unkey]

    Note:

    Starting with Oracle Exadata System Software release 19.3.0, the patchmgr command uses -- instead of a single hyphen before keywords.

    If the output from the command shows overall status is SUCCESS, then proceed with the upgrade.

    If the output from the command shows overall status is FAIL, then review the error summary in the output to determine which checks failed, and then correct the errors.

    After the errors have been corrected, rerun the prerequisite checks until it is successful.

  6. Upgrade the switches using the following command:
    [root@dm01 ]# ./patchmgr --ibswitches ibswitches.lst --upgrade [--force] [--unkey]
  7. Check the output from the command, and verify the upgrade.

    The output should show SUCCESS. If there are errors, then correct the errors and run the upgrade command again.

Parent topic: Updating InfiniBand Network Fabric Switch Firmware

8.8.3 Downgrading InfiniBand Network Fabric Switch Firmware

Downgrading firmware means reapplying the older firmware update, which is shipped with the Oracle Exadata System Software update.

The current Oracle Exadata System Software update determines what release you can downgrade to. This may be different for each release and may not be the firmware you were on before the update. For more information on the older firmware shipped with the release you are updating to, see the patch README file.

Complete all steps in this task as the root user.

  1. Run the patchmgr command with the --precheck option to verify the switch firmware is ready to be downgraded.

    The ibswitches.lst file is a file that contains the host names of all the InfiniBand Network Fabric switches that need to be updated, with one switch per line. 

    # ./patchmgr --ibswitches ibswitches.lst --downgrade --ibswitch_precheck [--force] [--unkey]

    Note:

    Starting with Oracle Exadata System Software release 19.3.0, the patchmgr command uses -- instead of a single hyphen before keywords.

    The --unkey option removes passwordless SSH access to the InfiniBand Network Fabric switches before exiting.

    The --force option overrides failures in the InfiniBand Network Fabric topology and connectivity from the servers to the switches. This does not affect the upgrade of the switch.

  2. Use the patchmgr command to downgrade the firmware on the InfiniBand Network Fabric switches.
    # ./patchmgr --ibswitches ibswitches.lst --downgrade [--force] [--unkey]
  3. Use the version command to verify the firmware on the switch has been downgraded.
    # version

SUN DCS 36p version: 2.1.8-1
Build time: Sep 18 2015 10:26:47
SP board info:
Manufacturing Date: 2015.07.01
Serial Number: "NCDLD0049"
Hardware Revision: 0x0200
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

Parent topic: Updating InfiniBand Network Fabric Switch Firmware

8.9 Upgrading Oracle Java SE on Oracle Linux

You can upgrade Oracle Java SE (JDK) running Oracle Linux on database servers and storage servers.

Starting with Oracle Exadata System Software release 12.1.2.1.0, the Oracle Exadata servers include the Java JDK package. Earlier releases of Oracle Exadata System Software do not use the JDK package. If a server running an earlier release of Oracle Exadata System Software has a package installed, such as java-version-openjdk, the package is not used by Oracle Exadata and can be removed.

To update the JDK package, you must download and update the JDK RPM package, and then reconfigure MS to use the new JDK package. You can update the JDK package by configuring YUM to use ULN (with Oracle Linux 6 and later) or by direct package download.

Parent topic: Updating Exadata Software

8.9.1 Stop the MS Process

You must stop the Management Server (MS) before updating the JDK package.

  1. Log in to the server as the root user.
  2. Stop MS.

    • On database servers:

      dbmcli -e alter dbserver shutdown services ms

    • On storage servers:

      cellcli -e alter cell shutdown services ms

Parent topic: Upgrading Oracle Java SE on Oracle Linux

8.9.2 Download and Update the Java JDK Package

Depending on your current environment, you can use one of three procedures to download and update the JDK package.

Using YUM and Unbreakable Linux Network (ULN) is only supported for Oracle Exadata on-premise database servers running Oracle Linux. It is not supported for storage servers or Oracle VM environments, either management domains (dom0) or user domains (domU).

Use one of the following methods to download and update the JDK package:

Parent topic: Upgrading Oracle Java SE on Oracle Linux

8.9.2.1 Using YUM and ULN to Update the JDK Package on Database Servers

If the database servers for your on-premise Oracle Exadata are running Oracle Linux 6 or later, then you can use YUM and Unbreakable Linux Network (ULN) to simplify the update process.

You can either download the JDK package directly, or use YUM and ULN. Using YUM and ULN for updating the JDK package involves the following tasks:

Parent topic: Download and Update the Java JDK Package

8.9.2.1.1 Configuring YUM to Connect to the Oracle Public Repository

The Oracle Linux YUM server offers a free and convenient way to install the latest Oracle Linux packages. Unbreakable Linux Network (ULN) enables you to update patches and errata for a specific version.

This procedure is only supported for Oracle Exadata on-premise database servers running Oracle Linux. It is not supported for storage servers or Oracle VM environments, either management domains (dom0) or user domains (domU).

Oracle Linux Yum Server is the Public Repository. For more information on how to obtain updates from the Oracle Linux Yum Server, see http://yum.oracle.com. For more information about ULN, see http://linux.oracle.com.

  1. As the root user on a database server, verify that the RHNS-CA-CERT certificate is not expired.
    # yum list installed

    If the certificate is expired, refer to My Oracle Support Doc ID 2207336.1 to fix the issue.

  2. Switch to the /etc/yum.repos.d directory. 
    # cd /etc/yum.repos.d
  3. Download the repository configuration file.

    Use the curl utility to download the repository configuration file that is appropriate for your system. You can alternatively copy the contents of http://yum.oracle.com/public-yum-release.repo into a file, where release corresponds to the Oracle Linux release, for example ol6 or ol7.

    For example, to get the configuration for the Oracle Linux 6 repository using curl, use the following command: 

    # curl -O http://yum.oracle.com/public-yum-ol6.repo
    for more information.
  4. On a database server, install the rhn-setup package to enable uln_register.

    WARNING:

    Do not register your storage servers with ULN or the public yum server. 
    # yum install rhn-setup.noarch

WARNING:

If you are not using the latest release of Oracle Linux, you will need to edit the file /etc/yum.repos.d/public-yum-ol6.repo and enable the correct repository to match the system version. Contact Oracle Support Services and refer to My Oracle Support Doc ID 2241729.1.
8.9.2.1.2 Registering an Oracle Linux System with ULN

After you have configured the access to the YUM repository, you must register your Oracle Linux database servers with ULN.

WARNING:

Do not register your storage servers with ULN or the public yum server.
  1. Run uln_register to register your system with ULN.
    # uln_register

    If you do not have a ULN account, you can register at https://linux.oracle.com. Registering for ULN requires a valid customer support identifier (CSI) for Oracle Linux or Oracle VM support.

    When you register your system, if a proxy server is required, then use the ––proxy option to specify the HTTP proxy to use. 

    # uln_register --proxy=proxy_hostname:port_number

    If your proxy requires authentication, then use the additional options --proxyUser and --proxyPassword to specify the user name and password. 

    # uln_register --proxy=proxy_hostname:port_number 
--proxyUser=username --proxyPassword=password
  2. When prompted, enter your ULN user name, password, and CSI.
  3. On the Create Profile — Hardware page, enter the required information.
    1. Enter a name for the system that will allow you to identify it on ULN.
    2. Choose whether to upload hardware and software profile data that allows ULN to select the appropriate packages for your system.
8.9.2.1.3 Upgrading JDK on Database Servers Using ULN
Before starting this step, make sure you have completed the steps in Stop the MS Process.

After you have configured the YUM repository and registered your database server with ULN, you can download the RPM from the ULN channel.

For Oracle Exadata System Software versions 12.1.2.1.0 to 12.1.2.2.0, the Oracle Exadata Database Server includes the JDK 7 package installed as an RPM. To update the RPM, use the ULN channel Java SE 7 for Oracle Linux. Make sure you look for JDK 7--do not use JDK 8 or later.

For Oracle Exadata System Software versions 12.1.2.2.1 and later, the Oracle Exadata Database Server includes the JDK 8 package installed as an RPM. To update the RPM, use the ULN channel Java SE 8 for Oracle Linux. Make sure you look for the JDK 8.

WARNING:

Do not register your storage servers with ULN.
  1. With a web browser, log in to https://linux.oracle.com/.
  2. Find the Oracle Exadata Database Server on which you want to install Oracle Java SE and click on the name of that system.

    In the following image, the server names have been blanked out.


    Description of uln_registered_systems.jpg follows
    Description of the illustration uln_registered_systems.jpg
  3. Click in Manage Subscriptions.
  4. Using the arrow buttons, move the desired Java SE channel from the Available channels list to the Subscriber Channels list, then click Save Subscriptions.
  5. When prompted, click Accept to accept the license agreement.
  6. On the database server, from the yum.repos.d directory, upgrade the JDK.
    # yum check-update jdk
Loaded plugins: rhnplugin, ulninfo
This system is receiving updates from ULN.
ol6_x86_64_JavaSE7_public
By downloading and/or using this software program you agree that your use is 
subject to the applicable license agreement at https://linux.oracle.com
/licenses.html.

exadata_dbserver_18.1.4.0.0_x86_64_base                        | 1.2 kB   00:00
exadata_dbserver_18.1.4.0.0_x86_64_base/primary                | 436 kB   00:00
exadata_dbserver_18.1.4.0.0_x86_64_base                                     470/470
ol6_x86_64_JavaSE7_public                                      | 1.2 kB   00:00
ol6_x86_64_JavaSE7_public/primary                              | 5.9 kB   00:00
ol6_x86_64_JavaSE7_public                                                     16/16
ol6_x86_64_ksplice                                             | 1.2 kB   00:00
ol6_x86_64_ksplice/primary                                     | 160 kB   00:00
ol6_x86_64_ksplice                                                        1436/1436
public_ol6_latest                                                       40132/40132
...
Running transaction
  Installing : jdk1.8.0_172-1.8.0_172-fcs.x86_64                                        1/1 
Unpacking JAR files...
	rt.jar...
	jsse.jar...
	charsets.jar...
	localedata.jar...
	jfxrt.jar...
  Verifying  : jdk1.8.0_172-1.8.0_172-fcs.x86_64                                       1/1 

Installed:
  jre.x86_64 0:1.8.0_172-fcs                                                

Complete!
To complete the JDK update, continue with the steps in Reconfigure and Restart Management Server (MS).
8.9.2.2 Manually Update the JDK Package on Oracle Exadata System Software versions 12.1.2.1.0 to 12.1.2.2.0

Update the JDK 7 package to the latest release by downloading the latest version of the package and using the rpm utility to install it.

Before starting this step, make sure you have completed the steps in Stop the MS Process.

With Oracle Exadata System Software versions 12.1.2.1.0 to 12.1.2.2.0, the Oracle Exadata Storage Server includes the JDK 7 package installed as an RPM.

Following any upgrades to the server image (using dbnodeupdate or patchmgr), check the JDK version. If the JDK package is reverted to an older version during the upgrade, then use the procedures here to update the JDK package to the latest version.

  1. Download the latest version of JDK 7 using the links found in My Oracle Support Doc ID 1439822.1. Do not download JDK 8 or later.
  2. Extract the contents of the ZIP file.
  3. Locate the JDK RPM.
    The name of the file is similar to jdk-version-linux-x64.rpm, for example jdk-7u91-linux-x64.rpm.
  4. Copy only the RPM file to the target server.
    The file can be placed in a temporary directory, such as /tmp.
  5. As the root user, determine the current version of the installed JDK RPM.
    # rpm -q jdk
  6. If the JDK package is installed and needs to be updated, then use the rpm command to install the update.
    # rpm -Uvh /tmp/jdk-version-linux-x64.rpm
  7. Verify the JDK package was updated.
    # rpm -q jdk
  8. Removed the staged update file.
    # rm -f /tmp/jdk-version-linux-x64.rpm
To complete the JDK update, continue with the steps in Reconfigure and Restart Management Server (MS).
8.9.2.3 Manually Update the JDK Package on Oracle Exadata System Software Release 12.1.2.2.1 and Later

Update the JDK 8 package to the latest release by downloading the latest version of the package and using the rpm utility to install it.

Before starting this step, make sure you have completed the steps in Stop the MS Process.

With Oracle Exadata System Software release 12.1.2.2.1 and later, the Oracle Exadata servers includes the JDK 8 package installed as an RPM.

Following any upgrades to the server image (using dbnodeupdate or patchmgr),check the JDK version. If the JDK package is reverted to an older version during the upgrade, then use the procedures here to update the JDK package to the latest version.

  1. Download the latest version of JDK 8 using the links found in My Oracle Support Doc ID 1439822.1. Download only JDK 8 updates.
  2. Extract the contents of the ZIP file.
  3. Locate the JDK RPM.
    The name of the file is similar to jdk-version-linux-x64.rpm, for example jdk-8u172-linux-x64.rpm.
  4. Copy only the RPM file to the target server.
    The file can be placed in a temporary directory, such as /tmp.
  5. As the root user, determine the current version of the installed JDK RPM.
    # rpm -qa|grep jdk
jdk1.8.0_66-1.8.0_66-fcs.x86_64
  6. If the JDK package is installed and needs to be updated, then use the rpm command to install the update.
    # rpm -Uvh /tmp/jdk-version-linux-x64.rpm
  7. Verify the JDK package was updated.

    With JDK 8, the updated package does not replace the currently installed package, so you will see two version of the JDK package installed.

    # rpm -qa | grep jdk
jdk1.8.0_66-1.8.0_66-fcs.x86_64
jdk1.8.0_172-1.8.0_172-fcs.x86_64
  8. Remove the older JDK package from the server.
    If the older version was update 66, then the command would be as follows:
    # rpm -e --nodeps jdk.1.8.0_66-1.8.0_66-fcs.x86_64
  9. Verify only the updated JDK is available on the server.
    # rpm -qa |grep jdk
jdk1.8.0_172-1.8.0_172-fcs.x86_64
  10. Removed the staged update file.
    # rm -f /tmp/jdk-version-linux-x64.rpm
To complete the JDK update, continue with the steps in Reconfigure and Restart Management Server (MS).

8.9.3 Reconfigure and Restart Management Server (MS)

After you update the Java JDK package, you must reconfigure and restart MS.

  1. Log in to the database server or storage server as root.
  2. Ensure that MS is stopped.
    • On database servers: 
      dbmcli -e alter dbserver shutdown services ms
    • On storage servers:
      cellcli -e alter cell shutdown services ms
  3. Reconfigure MS.

    Perform the following on systems with:

    • Oracle Exadata System Software release 12.2.x, version 12.2.1.1.4 or later.

    • Oracle Exadata System Software release 18c, version 18.1.2 or later.

    • All Oracle Exadata System Software versions commencing with release 19.1.0.

    1. Go to the UNIX scripts directory.
      • On database servers: 
        cd /opt/oracle/dbserver/dbms/deploy/scripts/unix
      • On storage servers:
        cd /opt/oracle/cell/cellserv/deploy/scripts/unix
    2. Redeploy MS.
      • On database servers: 
        sh setup_dynamicDeploy DB
      • On storage servers:
        sh setup_dynamicDeploy
  4. Restart MS.
    • On database servers: 
      dbmcli -e alter dbserver startup services ms
    • On storage servers: 
      cellcli -e alter cell startup services ms

Parent topic: Upgrading Oracle Java SE on Oracle Linux

8.10 Setting up SSH Equivalence

Before updating software on your Oracle Exadata Database Machine, you must configure SSH equivalence.

You can run the Exadata update utilities for Oracle Exadata Database Server, Oracle Exadata Storage Server, and the RDMA Network Fabric switch as either root or as a non-root user from any server running Oracle Linux. The utility can perform precheck, update, and rollback actions on any Exadata server as long as SSH equivalence is set up for the root user for the target Exadata server.

  1. Prepare a file named cell_group or dbs_group that has one storage server or database server host name or IP address per line for each storage server or database server you want to update.
  2. Check for existing SSH equivalence.
    The following command should require no password prompts and no interaction. It should return the list of host names in the cell_group file.
    [oracle@nonExadataHost ]# ./dcli -g cell_group -l root 'hostname -i'
  3. Set up SSH equivalence if not already done so from the launch server.
    Do not do this step if you already have root SSH equivalence.
    Generate SSH keys using the following command: 
    [oracle@nonExadataHost ]# ssh-keygen [-t rsa]

    You can use the -t option to specify the key type, such as RSA or DSA. If you do not include the -t option, then RSA is configured by default.

    Accept the defaults so that the SSH keys are created for the root user.

  4. Push the SSH keys to set up SSH equivalence.
    Enter the root password when prompted.
    [oracle@nonExadataHost ]# dcli -g cell_group -l root –k

Note:

Customers in secure environments may have chosen to disable SSH access to Oracle Exadata Storage Server. During normal operations, Oracle Exadata Storage Server does not require SSH access. However, administrative utilities such as the update utility require SSH access. See the subsection "Unlocking a Cell Temporarily" in the topic Disabling SSH on Storage Servers for information on unlocking storage servers.

Parent topic: Updating Exadata Software

8.11 Troubleshooting Software Updates on Oracle Exadata

Review these topics if you encounter errors or problems when updating the software on Oracle Exadata

Parent topic: Updating Exadata Software

8.11.1 Troubleshooting Exadata Database Server Updates

You can use the log files generated by the update utility to troubleshoot updates.

The update utility orchestrates updating the Exadata database servers. Updating database nodes with the patchmgr tool is less verbose because it prints only minimal information to the screen. If additional information is required, you can view the patchmgr logs and the dbnodeupdate.sh logs that patchmgr copies over from other servers, if available. The log file (dbnodeupdate.log) and the diag file (dbnodeupdate.<runid>.diag) will eventually exist on two locations:

  • On each updated database server, in the /var/log/cellos directory
  • Consolidated on the node running the update utility.

On the node running the update utility, if the -log_dir flag was set to auto, the log files will be stored in the log/<directory based on contents of nodes in list file> directory, relative from the directory where the update utility is started from. For example, if the update utility is located in /u01/dbserver.patch, then the log directory may be /u01/dbserver.patch/dm01db01_dm01db02_e8f1f753.

Important files found in the log directory are:

  • patchmgr.log contains the consolidated screen output from running the remote update commands on the different database servers.
  • <hostname>_dbnodeupdate.<runid>.diag is the diag file for the specific run on a database server.
  • <hostname>_dbnodeupdate.log contains dbnodeupdate.log output appended from /var/log/cellos from the remote database server.

When a prerequisite check, backup, update, or rollback fails, error messages on screen should provide information on which step failed on which node. Consult the log files mentioned above if more information is required. Search the log file for the start of a new run (search for zzz).

Check if the time matches your run. If it matches, note the runid for further reference. Then search for ERROR.

If an update action fails before the actual YUM update, you can retry the update after resolving the error. If the update failed half-way, it is recommended that you roll back, resolve the error, and retry.

In rare cases, patchmgr may be unable to determine the status of an update, whether the update was successful or not. In such cases, it displays a message that the update failed. However, it is possible that the update still completed successfully. To determine the actual status of the update:

  • Check the image status of the (database) node. You can do this by running the imageinfo command. The Image status line displays the status.
  • Check the version of the Exadata software. This can also be determined from the imageinfo command.

If the image status is success, and the Exadata version is the new expected version, then the update was successful and you can ignore the update failed message. Then:

  • Run dbnodeupdate.sh -c manually on the particular node to perform the completion steps of the update.
  • Remove the completed node from the (database) node file.
  • Rerun patchmgr to perform the update on the remaining nodes.

Other things to check if the update fails include:

  • The correct syntax for using patchmgr to update database nodes can be found in the patchmgr online help.
  • SSH equivalence must be configured before using patchmgr.
  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.
  • Open a service request with Oracle Support Services to analyze why the patchmgr orchestration failed.

Related Topics

Parent topic: Troubleshooting Software Updates on Oracle Exadata

8.11.2 Monitoring, Validating, and Troubleshooting Exadata Storage Server Updates

The recommended way to monitor Exadata storage servers being updated is through e-mail alerts. These alerts can be set up by specifying email addresses in the -smtp_from and -smtp_to flags in the update utility.

If needed for troubleshooting, you can monitor update activity using less -rf patchmgr.stdout from another terminal session or window to see raw log details from the update utility.

You can also monitor activity of the storage server by logging in to the serial console or web-based ILOM console of individual storage servers being updated 5 minutes after the update utility has started. Waiting 5 minutes allows the update utility time to reset the ILOM. Resetting the ILOM disconnects you from the ILOM web console and serial console. You can reconnect once the ILOM has been reset. By waiting 5 minutes, you avoid having to reconnect. You lose the connection during any ILOM update, and need to reconnect. The ILOM does not show any update actions. When needed it is helpful to monitor the activities of the normal cell boot, reboot, and other activities to ensure that the process is proceeding correctly.

Verify the update status after the patchmgr utility completes as follows:

Check image status and history using the imageinfo and imagehistory commands on each cell. A successful update to a system with high capacity or high performance drives shows output similar to the following. 

Kernel version: 2.6.39-400.281.1.el6uek.x86_64 #1 SMP Fri Jun 17 20:10:16 PDT 2016 x86_64
Cell version: OSS_12.1.2.3.2_LINUX.X64_160721
Cell rpm version: cell-12.1.2.3.2_LINUX.X64_160721-1.x86_64

Active image version: 12.1.2.3.2.160721
Active image kernel version: 2.6.39-400.281.1.el6uek
Active image activated: 2016-07-21 13:04:34 -0500
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

Cell boot usb partition: /dev/sdac1
Cell boot usb version: 12.1.2.3.2.160721

Inactive image version: 12.1.2.3.1.160411
Inactive image activated: 2016-04-11 19:58:28 -0700
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8

Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
Inactive kernel version for the rollback: 2.6.39-400.128.21.el5uek
Rollback to the inactive partitions: Possible

A successful update to a system with extreme flash drives shows a similar output but has a different path for USB:

Kernel version: 2.6.39-400.281.1.el6uek.x86_64 #1 SMP Fri Jun 17 20:10:16 PDT 2016 x86_64
Cell version: OSS_12.1.2.3.2_LINUX.X64_160721
Cell rpm version: cell-12.1.2.3.2_LINUX.X64_160721-1.x86_64

Active image version: 12.1.2.3.2.160721
Active image kernel version: 2.6.39-400.281.1.el6uek
Active image activated: 2016-07-21 13:04:34 -0500
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

Cell boot usb partition: /dev/sda1
Cell boot usb version: 12.1.2.3.2.160721

Inactive image version: 12.1.2.3.1.160411
Inactive image activated: 2016-04-11 19:58:28 -0700
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8

Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
Inactive kernel version for the rollback: 2.6.39-400.128.21.el5uek
Rollback to the inactive partitions: Possible

Related Topics

Parent topic: Troubleshooting Software Updates on Oracle Exadata

8.11.3 Troubleshooting a Failing Prerequisite Check Due to Dependency Problems on Exadata Database Server

When you run a prerequisite check or an update on Exadata database servers that have custom rpms installed, conflicts may cause the prerequisite check or update to fail.

When doing a prerequisite check or an update, the update utility makes two dependency checks: checks against "minimum dependencies" and checks against "exact dependencies".

To triage a dependency failure, look in the log file. This can be either dbnodeupdate.log on the target Exadata database server or <hostname>_dbnodeupdate.log in the log directory from where the update utility (patchmgr) was launched.

To locate the start of each run in the log file, search for a line that starts with zzz. For example: 

zzz - /u01/patches/YUM/dbnodeupdate.sh called with arguments -u -l
/u01/patches/YUM/p23564643_121232_Linux-x86-64.zip -v -N at  2016-08-23 23:31:54

The date stamp should match the time of the run you are researching. Each run is identified with a unique runid which can also be found at the start of each run in the same log file: 

[1472009516][2016-08-23 23:31:59 -0400][INFO][/u01/patches/YUM/dbnodeupdate.sh][InitLogfile][]  # dbnodeupdate.sh script rel.
 : 5.160809 started at  (runid :230816233155)

You can check for custom rpms, if any, by looking at the diag file of the specific run. The diag file is identified by the runid and can be found in /var/log/cellos on the target Exadata database server or in the directory driving the update utility. The filename for this example would be dbnodeupdate.230816233155.diag. In that file, look for the section with heading RunDetectCustomRpmsSh and rpm -qa --qf "%{n}-%{v}-%{r}.%{arch}"\n. You can find details on the (additional) packages installed (if any).

In the log file, search down from the start (zzz) for "Exact dependencies" (case sensitive) for the results of the dependency checks. In this case it says: 

Exact dependencies     : Will fail on a next update
Minimum dependencies   : Will fail on a next update

As long as the minimum dependencies check passes, updates or prerequisite checks won’t fail. When both minimum and exact dependencies fail, you need to find out what caused the error.

In order to find the dependency that is causing the error, search for [ExecUpgrade][] Performing yum package dependency. This is where YUM is run (typically a dry-run first). When there is a dependency problem, you should see a YUM message starting with Error:. For example: 

--> Finished Dependency Resolution
Error: Package: krb5-devel-1.10.3-33.el6.x86_64 (@ol6_latest)
Requires: krb5-libs = 1.10.3-33.el6
Removing: krb5-libs-1.10.3-33.el6.x86_64 (installed)
        krb5-libs = 1.10.3-33.el6
Updated By: krb5-libs-1.10.3-42z1.el6_7.x86_64 (exadata_generated_160616114412)
          krb5-libs = 1.10.3-42z1.el6_7
You could try using --skip-broken to work around the problem

In the example error message above, the krb5-devel-1.10.3-33.el6.x86_64 (krb5-devel) package is installed from a non-Exadata channel (ol6_latest). This krb5-devel package depends on krb5-libs. However in this Exadata update the krb5-libs package is not available. YUM fails the dependency check because updating krb5-libs is not possible without also updating krb5-devel. Because a new version of krb5-devel is not included in the Exadata update the package should be either pre-updated or removed. Pre-updating means updating the individual package manually before running the Exadata update. This can be done using the command rpm –Uvh <package-name>.

Removing the custom package is recommended and should be done using the following rpm command:

[root@dm01 ]# rpm –e krb5-devel

After removing the rpm that is causing the error, restart the update or prerequisite check.

Related Topics

Parent topic: Troubleshooting Software Updates on Oracle Exadata

8.11.4 Troubleshooting a Multilib Problem on Exadata Database Server

If you have custom packages with different architectures installed on database servers, you may see a similar problem as that described in Troubleshooting a Failing Prerequisite Check Due to Dependency Problems on Exadata Database Server. This typically happens when i686 packages are installed on an Exadata database server.

Note:

While non-64-bit (x86_64) packages are supported, it is recommended to stay away from 32-bit software. When you need specific functionality from a third party, it is recommended to ask for a 64-bit version.

Typically Exadata-branded x86_64 bit rpms are updated in an Exadata update. However, when you have installed similar packages of non-x86_64 bit architecture, the update utility cannot update the 64-bit packages. You would see the following error in the log file:

--> Finished Dependency Resolution
    Error:  Multilib version problems found. This often means that the root
           cause is something else and multilib version checking is just
           pointing out that there is a problem. Eg.:
           
             1. You have an upgrade for libuuid which is missing some
                dependency that another package requires. Yum is trying to
                solve this by installing an older version of libuuid of the
                different architecture. If you exclude the bad architecture
                yum will tell you what the root cause is (which package
                requires what). You can try redoing the upgrade with
                --exclude libuuid.otherarch ... this should give you an error
                message showing the root cause of the problem.
           
             2. You have multiple architectures of libuuid installed, but
                yum can only see an upgrade for one of those arcitectures.
                If you don't want/need both architectures anymore then you
                can remove the one with the missing update and everything
                will work.
           
             3. You have duplicate versions of libuuid installed already.
                You can use "yum check" to get yum show these errors.
           
           ...you can also use --setopt=protected_multilib=false to remove
           this checking, however this is almost never the correct thing to
           do as something else is very likely to go wrong (often causing
           much more problems).
           
           Protected multilib versions: libuuid-2.17.2-12.24.0.1.el6.x86_64 != libuuid-2.17.2-12.18.0.1.el6.i686

The solution for multilib problems is to remove the i686 or i386 package by running rpm –e <package_name.i686>.

Parent topic: Troubleshooting Software Updates on Oracle Exadata