5 Updating Exadata Software

Exadata software updates apply to three major components:

  • Exadata storage servers

  • Exadata database servers

  • Exadata InfiniBand switches

Exadata storage server and Exadata database server updates generally contain updates for:

  • Oracle Linux operating system

  • Exadata software

  • Firmware (for example: disk, flash, RAID controller, ILOM, HCA)

While it is generally recommended for components to stay in step with the recommended minimum release, you can choose to update different components at different times. For example, you could update Exadata InfiniBand switches at a later time than Exadata storage servers and Exadata database servers. However, you need to check My Oracle Support note 888828.1 for any dependencies.

It is not mandatory to apply each and every Exadata software update that comes out. For example, you can skip two or three releases and update directly to a newer release. Oracle recommends that you update database servers twice a year.

For each of the three major components, this section describes how to prepare, apply, and roll back updates. After discussing a generic approach each section concludes with common update scenarios.

Components of the Image Version

The image version for a release consists of three components. For example, if "imageinfo -ver" returns “12.1.2.1.1.150316.2”, then:

  • The product version is 12.1.2.1.1

  • The date code is 150316

  • The Oracle-internal build number is 2. This component is optional.

In most circumstances it is sufficient to reference a release by the product version only, for example, 12.1.2.1.1. However, when upgrading, you need to consider both the product version and date code of the installed and target releases. Upgrading is allowed under the following circumstances:

  1. The product version of the target release must be higher than the installed software, and

  2. The date code of the target release must be higher than the installed software.

For example, consider a system that is currently running image version 12.1.1.1.2.150411, which consists of product version 12.1.1.1.2 and date code 150411.

  • Upgrading to 12.1.2.1.2.150617.1 is allowed because both rules are satisfied.

  • Upgrading to 12.1.2.1.1.150316.2 is not allowed because date code 150411 of the installed software is higher than date code 150316 of the target release.

5.1 Planning for Software Maintenance

Before starting a software update, you should review best practices, determine the version to which you will upgrade, and obtain the proper patching software.

5.1.1 Understanding Exadata Database Machine Software and Updates

The software that runs on Exadata Database Machine is divided into two categories.

  • Exadata Infrastructure software

  • Oracle Grid Infrastructure and Oracle Database software

The following table describes the two primary categories of software that run on an Exadata Database Machine.

Software Components Installed On Software Content Example Software Version

Exadata Infrastructure software

  • Exadata storage servers

  • Exadata database servers

  • Exadata InfiniBand switches

 

  • Oracle Linux operating system

  • Oracle VM Server (on database servers for Oracle VM configurations)

  • Exadata software

  • Firmware (for example: disk, flash, RAID controller, ILOM, HCA, InfiniBand switch firmware)

Exadata storage servers and database servers:

  • 12.2.1.1.0

  • 12.1.2.3.4

Exadata InfiniBand switches

  • 2.2.4-3

  • 2.1.8-1

Oracle Grid Infrastructure and Oracle Database software

Exadata database servers

  • Oracle Clusterware (Grid home)

  • Oracle Database (Database home)

12.2.0.1.0

12.1.0.2.160117

11.2.0.4.161018

When an Exadata Database Machine is deployed, all of the software described in the table is installed and configured to deliver high performance and availability for Oracle Database. Extensive end-to-end testing ensures all software components supplied with Exadata work seamlessly together.

My Oracle Support Document 888828.1 is the primary source of information for software that runs on Exadata.  It contains the following information:

  • List of all current and previous releases

  • Minimum requirements for feature usage

  • Compatibility requirements between Exadata version and Database version

  • Compatibility requirements for specific hardware releases

  • Guidelines for related products when used with Exadata

  • References to other pertinent information sources for Exadata software maintenance

5.1.1.1 Software Release Types

The primary software categories Exadata Infrastructure software and Grid Infrastructure and Database software are further broken down by the release type.

Software Release type Description and Content

Exadata Infrastructure software

Feature release

The first four digits represent a Exadata feature release, for example, 12.2.1.1.0.

A feature release contains new features, bug fixes, and security fixes. It may be used to update any prior feature or sustaining release.

You can update directly to feature release 12.2.1.1.0 from any prior release from 11.2.3.3.1 to 12.1.2.3.4, inclusive.

Exadata Infrastructure software

Sustaining release

The fifth digit represents a cumulative update to a feature release, for example, 12.1.2.3.4.

A sustaining release contains bug fixes and security fixes. A given feature release will have approximately four sustaining releases. A sustaining release may be used to update any prior feature or sustaining release.

You can update directly to sustaining release 12.1.2.3.4 from any prior release from 11.2.3.3.1 to 12.1.2.3.3, inclusive.

Exadata Infrastructure software

Interim patch

An interim patch is one-off bug fix made available to customers who cannot wait until the fix is included in a sustaining release or feature release. Installation of an interim patch is done on an as-needed basis and is not a regularly scheduled planned maintenance event.

Oracle Grid Infrastructure and Oracle Database software

Major or maintenance release

The first and second digits represent a major and maintenance database release, for example, 12.2 or 12.1.

A major release or maintenance release contains new features, bug fixes, and security fixes.

Oracle Grid Infrastructure and Oracle Database software

Patch set release

The forth digit represents a patch set release, for example, 12.2.0.1 or 12.1.0.2. 

A patch set release contains primarily bug fixes. However, some minor new features and change in functionality may be included in a patch set release.

Oracle Grid Infrastructure and Oracle Database software

Proactive Bundle Patch

The fifth digit represents a cumulative update to a patch set release, for example, 12.1.0.2.170117 or 11.2.0.4.161018.

The proactive bundle patch contains bug fixes and security fixes. It is a superset of the standard Oracle Grid Infrastructure PSU (GIPSU). Only the Proactive Bundle Patch can be installed on Exadata systems, not the standard PSU (with the exception of Oracle Database 12.1.0.1, which uses the standard GIPSU).

For Oracle Database 11.2, the proactive bundle patch is called Database Patch for Exadata.

Oracle Grid Infrastructure and Oracle Database software

Interim patch

An interim patch is one-off bug fix made available to customers who cannot wait until the fix is included in a subsequent patch set release or database patch for Exadata. Installation of an interim patch is done on an as-needed basis and is not a regularly scheduled planned maintenance event.

5.1.1.2 Software Release Availability

Each software release type has a different frequency of availability.

Exadata sustaining releases and quarterly Grid Infrastructure and Database Proactive Bundle Patches are released on regular quarterly cycle as part of the Oracle Critical Patch Update (CPU) program.  Oracle provides these quarterly releases is to address proactive, critical fixes and security vulnerabilities.

Software Release type Release Frequency Importance of Maintaining Up-to-Date Versions Recommendation for Adopting New Releases

Exadata Infrastructure software

Feature release

6-18 months

Medium

Update to adopt new features and get critical fixes and security updates.

Exadata Infrastructure software

Sustaining release

3 months (quarterly)

High

Update quarterly to get critical fixes and security updates. Do not lag more than 12 months.

Oracle Grid Infrastructure and Oracle Database software

Major or maintenance release

24+ months

Low

Update to adopt new features. Update prior to Lifetime Support end date for current release.

Oracle Grid Infrastructure and Oracle Database software

Patch set release

12-24 months

Medium

Update prior to Error Correction Support end date for current patch set, which is typically one year after next patch set release.  Details in My Oracle Support Document 742060.1.

Oracle Grid Infrastructure and Oracle Database software

Proactive Bundle Patch

3 months (quarterly)

High

Update quarterly to get critical fixes and security updates. Do not lag more than 12 months.

5.1.1.3 Software Update Frequency

You should plan to update your Exadata software on a regular basis.

These examples show three high-level quarterly software maintenance plans over a four year cycle using typical intervals between releases.

Note:

Release frequency may vary from what is documented here, particularly for releases that contain new features.

Example 5-1 Production System Software Maintenance Plan

Goal — Minimize risk by applying critical and security fixes as they become available, and adopting new feature releases as late as possible.  Where possible do not adopt a new Exadata feature release and new Database feature release simultaneously.

Action Y1 Q1 Y1 Q2 Y1 Q3 Y1 Q4 Y2 Q1 Y2 Q2 Y2 Q3 Y2 Q4 Y3 Q1 Y3 Q2 Y3 Q3 Y3 Q4 Y4 Q1 Y4 Q2 Y4 Q3 Y4 Q4 Y5 Q1

Update to next Exadata sustaining release

 

 X

 X

 X

 X

 

 X

 X

 X

 X

 

 X

 X

 X

 X

 

 X

Update to next Exadata feature release

 X

       

 X

       

 X

       

 X

 

Update to next Database Proactive Bundle Patch

 X

 

X

X

X

X

X

X

X

X

X

X

X

 

X

X

X

Update to next Database patch set release

                         

X

     

Update to next Database major or maintenance release

 

X

                             

Example 5-2 Production System Software Maintenance Plan with Reduced Updates

Goal — Minimize maintenance time by applying alternating quarterly updates. Where possible do not adopt a new Exadata feature release and new Database feature release simultaneously.

Action Y1 Q1 Y1 Q2 Y1 Q3 Y1 Q4 Y2 Q1 Y2 Q2 Y2 Q3 Y2 Q4 Y3 Q1 Y3 Q2 Y3 Q3 Y3 Q4 Y4 Q1 Y4 Q2 Y4 Q3 Y4 Q4 Y5 Q1

Update to next Exadata feature release

 

X

         

X

         

X

     

Update to next Exadata sustaining release

     

X

 

X

     

X

 

X

     

X

 

Update to next Database major or maintenance release

     

X

                         

Update to next Database patch set release

                             

X

 

Update to next Database Proactive Bundle Patch

 

X

     

X

 

X

     

X

 

X

     

Example 5-3 Development and Test System Software Maintenance Plan

Goal — Adopt the latest features and software updates as soon as possible.

Action Y1 Q1 Y1 Q2 Y1 Q3 Y1 Q4 Y2 Q1 Y2 Q2 Y2 Q3 Y2 Q4 Y3 Q1 Y3 Q2 Y3 Q3 Y3 Q4 Y4 Q1 Y4 Q2 Y4 Q3 Y4 Q4 Y5 Q1

Update to next Exadata feature release

X

       

X

       

X

       

X

 

Update to next Exadata sustaining release

 

X

X

X

X

 

X

X

X

X

 

X

X

X

X

 

X

Update to next Database major or maintenance release

 X

                 

 X

           

Update to next Database patch set release

         

                 

 

Update to next Database Proactive Bundle Patch

 

X

X

X

 

X

X

X

X

 

X

X

X

X

 

X

5.1.1.4 Software Update Utilities

When updating Oracle Exadata Database Machine software, you use specific utilities, depending on the component being updated.

Update Type Utility Components Description

Pre-update readiness check

Oracle ExaCHK

Exadata Database Machine

Exadata healthcheck tool used to qualify maintenance readiness, identify critical issue exposure, and provide version recommendations for updating software.

Obtain Oracle ExaCHK from My Oracle Support Document 1070954.1.

Exadata Infrastructure software

patchmgr

  • Exadata storage servers

  • Exadata database servers

  • Exadata InfiniBand switches

Exadata software update tool that updates all software (Oracle Linux, Exadata infrastructure software, firmware) on Exadata database servers, storage servers, or InfiniBand switches. For example, patchmgr updates Exadata storage servers and database servers to Exadata version 12.2.1.1.0, or Exadata InfiniBand switches to version 2.2.4-3.

Oracle Grid Infrastructure and Oracle Database software Proactive Bundle Patch

OPatch / opatchauto

Exadata database servers

Software update tool that applies a Proactive Bundle Patch to an Oracle Grid Infrastructure home or Oracle Database home. For example, OPatch applies Proactive Bundle Patch 12.1.0.2.170117 to 12.1.0.2 Oracle Grid Infrastructure and Oracle Database homes.

Obtain OPatch from My Oracle Support Document 274526.1.

Oracle Grid Infrastructure and Oracle Database software patch set, major release, or maintenance release.

Oracle Universal Installer (OUI)

Exadata database servers

Software installation tool that installs a Grid Infrastructure and Database patch set, major, or maintenance release, and upgrades Grid Infrastructure. For example, OUI installs Database 12.2.0.1, or installs and upgrades Grid Infrastructure from 12.1.0.2 to 12.2.0.1.

Use the version of OUI bundled with the Oracle Grid Infrastructure and Oracle Database release to which you are upgrading.

Upgrade of an Oracle database

Database Upgrade Assistant (DBUA)

Exadata database servers

Database tool that upgrades a database to a later patch set, major, or maintenance release. For example, DBUA upgrades a database from 12.1.0.1 to 12.1.0.2, or from 12.1.0.2 to 12.2.0.1.

Use DBUA bundled with the Oracle Grid Infrastructure and Oracle Database release to which you are upgrading..

Oracle Grid Infrastructure and Oracle Database software

Rapid Home Provisioning (RHP)

Exadata database servers

Software maintenance tool that provisions, patches, and upgrades Oracle Grid Infrastructure and Oracle Database software on one or more clusters from a centralized server.

Use RHP as a feature of Oracle Clusterware.

Exadata Infrastructure software

Oracle Grid Infrastructure and Oracle Database software

Enterprise Manager Cloud Control

  • Exadata storage servers

  • Exadata database servers

  • Exadata InfiniBand switches

Enterprise Manager Cloud Control provides lifecycle management capability to apply software updates to Exadata systems.

5.1.2 Configuration and Operational Best Practices for Software Maintenance

As part of planning your software update, you should review the different methods of performing udpates and the best practices for updating the software.

5.1.2.1 Understanding Rolling and Non-Rolling Updates

Software updates can be performed in rolling manner while the database remains online and available, or non-rolling manner where Oracle Clusterware and Oracle Database are shutdown.

The manner an update is performed does not affect how often it should be done, but does determine how long it will take.  In general there are two methods:

  • Rolling software updates — A rolling software update is one that is performed to one of a particular component at a time while the others remain online servicing requests.  Note the following key points about rolling software updates:

    • Rolling updates have less application downtime compared to non-rolling updates.

    • The overall length of time to complete the update is longer because one component is offline and updated at a time, while all other components remain online and operational.

    • The impact to existing database connections differs depending on the component being updated.

      • Rolling updates for Exadata storage servers or Exadata InfiniBand switches — All databases remain fully online and available for the duration of the rolling update.  There is no disruption to database connections.

      • Rolling updates for Exadata database servers, Oracle Grid Infrastructure, or Oracle Database — Multi-instance databases using Oracle Real Application Clusters (RAC) remain available for the duration of the update.  However, database connections on the server being updated will be disrupted when the local database instance is shutdown.  Use Oracle RAC features for client high availability to minimize application disruption, as described in Table X.

  • Non-Rolling software updates — A non-rolling software update is one that is performed to all of a particular component while those components are offline.  Note the following key points about non-rolling software updates:

    • Non-rolling updates are faster than rolling updates for overall maintenance time because multiple components are updated in parallel.

    • Because all of a particular component are offline for the duration of the update, a database (or all databases on a system) will also be offline for the duration of the update, resulting in a complete outage to the application

    • Important applications serviced by the Exadata Database Machine being updated in a non-rolling manner are often moved to a standby system in environments using Oracle Data Guard.

  • Combination of rolling and non-rolling software updates — When multiple components are updated in the same maintenance window, it is possible to use a combination of rolling and non-rolling methods to achieve the desired balance of application downtime and maintenance time. One typical combination used in the situation where an application does not handle connection disruption efficiently is to perform Exadata storage server and InfiniBand switch updates in a rolling manner, then performing Oracle Grid Infrastructure, Oracle Database, and Exadata database server updates in a non-rolling manner.

The Exadata patchmgr update utility manages the update orchestration in a rolling or non-rolling manner for Exadata infrastructure components (storage servers, database servers, and InfiniBand switches).

The following table describes which update method is supported for each component type. 

Method Component Support Database Availability Impact During Update

Rolling

Exadata storage servers

No impact.

Oracle RAC and single instance databases remain fully available on all nodes in the cluster.

Rolling

Exadata InfiniBand switches

No impact.

Oracle RAC and single instance databases remain fully available on all nodes in the cluster.

Rolling

Exadata database servers

Local connections are disconnected and all local instances are shutdown .

Oracle RAC databases remain available through other nodes in the cluster.

Rolling

Grid Infrastructure

Local connections are disconnected and all local instances are shutdown .

Oracle RAC databases remain available through other nodes in the cluster.

Rolling

Database - Proactive Bundle Patch

For the database home being updated, local connections are disconnected and local instances are shutdown.

Oracle RAC databases remain available through other nodes in the cluster.

Databases running from other Oracle Database software homes are unaffected.

Non-Rolling

  • Exadata storage servers

  • Exadata database servers

  • Grid Infrastructure

  • Database - Proactive Bundle Patch

  • Database - Patch Set

  • Database - Release

Databases unavailable

Move workload to Oracle Data Guard or Oracle Golden Gate standby system to minimize impact.

5.1.2.2 Online Updates for Oracle Linux Kernel and Oracle Database Interim Fixes

 Online updates of qualified fixes are supported for Oracle Linux and Oracle Database.

Typically, planned software updates require that the component be restarted after the update. For example, an Exadata database server must be restarted after applying an Exadata software release to update the system firmware and to make active a new Oracle Linux kernel. Similarly, an Oracle database instance must be stopped and restarted to apply a software update to the database home.

Oracle supplies some interim fixes, however, that can be applied online, such that a component does not require a restart to apply the fix and make them active. Online updates of qualified fixes are supported for the following components:

  • Oracle Linux kernel on Exadata database servers using the Ksplice Offline Client.

  • Oracle Database using OPatch Online Patching.

Online updates are typically performed as a temporary measure when a critical fix must be applied to the system before the next scheduled planned software maintenance.

5.1.2.3 Configuration Practices for Optimal Software Maintenance

When configuring an Exadata Database Machine, it is important to adopt features that will lessen the impact and risk of performing software updates.

Configuration Practice Use for Software Maintenance Purpose

Oracle ASM high redundancy disk groups

Rolling updates - Exadata storage servers

When performing storage server updates in a rolling manner with the databases remaining online, it is highly recommended to configure Oracle ASM disk groups with high redundancy. High redundancy disk groups can tolerate the failure of a disk in another storage server during rolling updates.

During rolling, or online, storage server updates, the disks for the storage server being updated are taken offline on one storage server by patchmgr while it is updated. After the update completes the disks are resynchronized by Oracle ASM, and then patchmgr starts to update the next storage server.

While disks are offline the disk group has reduced redundancy. A normal redundancy disk group with reduced redundancy during a rolling update may dismount and have data loss if a disk fails in another storage server.

Oracle ASM disk group redundancy is typically set during initial system configuration. Therefore, consider how you plan to perform storage server updates, rolling or non-rolling, prior to system configuration.

An Oracle Data Guard physical standby system can also provide protection against disk failure during rolling storage server update.

Oracle RAC features for client high availability

Rolling updates - Exadata database servers, Oracle Grid Infrastructure or Oracle Database software

When performing Exadata database server, Oracle Grid Infrastructure, or Oracle Database software updates in a rolling manner on one database server at a time while database services remain running on the other servers in the cluster, the database services must be stopped on the Exadata database server being updated.

To minimize the impact to client applications connected to database instances that will be stopped during maintenance, configure client applications to use database services, Fast Application Notification (FAN), Fast Connection Failover (FCF), and Application Continuity.

Oracle Data Guard physical standby database

Any rolling or non-rolling update (except Oracle Database patch set and release updates)

Oracle Data Guard Standby-First Patch Apply provides support for different Exadata infrastructure, Oracle Grid Infrastructure, and Oracle Database software between a primary database and its physical standby database for the purpose of applying and validating Oracle patches in rolling fashion with minimal risk to the primary database.

Oracle Data Guard transient logical standby database

Any rolling or non-rolling update

Use Oracle Data Guard logical standby database to reduce database upgrade downtime for patch set and release updates. Database upgrade downtime is reduced by allowing the logical standby database to be upgraded to the new version and kept synchronized while the primary database remains online running the current version.

Oracle GoldenGate

Any rolling or non-rolling update

Use Oracle GoldenGate to reduce database upgrade downtime for patch set and release updates. Downtime during a database upgrade is reduced by allowing the target database to be upgraded to the new version and kept synchronized while the source database remains online running the current version.

5.1.2.4 Operational Practices for Optimal Software Maintenance

The following operational practices enable optimal software maintenance:

  • Run Oracle ExaCHK regularly — As a general health check tool to ensure an Exadata system continues to meet the current and constantly evolving best practices, run Oracle ExaCHK monthly. The exachk report should be utilized as follows:

    • Baseline comparison - Compare the current report against an accepted baseline report using the report comparison feature (-diff option).

    • Critical issue exposure - Review the report for exposure to critical issues. Take prompt action to resolve critical issues reported by Oracle ExaCHK.

    • Version recommendation - Review the report for version recommendation. The MAA Scorecard section evaluates current software versions for consistency, compatibility, and whether or not it is current.

  • Qualify maintenance readiness with Oracle ExaCHK — Prior to performing software maintenance, run Oracle ExaCHK to ensure the system is in a healthy state. Correct any FAIL or WARNING checks before updating any software. After software maintenance is complete, run Oracle ExaCHK again to confirm system health.

  • Update software at regular intervals — All software should be updated regularly. Maintaining software at current or recent releases provides the following benefits: better software security, more stable sustaining releases, continued compatibility with newer related software, better support and faster resolution of issues, and ability to supply fixes for newly discovered issues.

  • Use the latest versions of the update utilities — Use the latest version of software update utilities Oracle ExaCHK, patchmgr, and OPatch.

    Oracle ExaCHK is updated regularly to contain new features, fixes, best practice health checks, and version recommendations. Patchmgr for database servers is updated regularly to contain new features, fixes, and workarounds for known database server update issues. OPatch is updated regularly to contain new features and fixes.

  • Avoid unsupported system changes — Exadata Database Machine is an integrated system and engineered to be the best platform for running Oracle Database. Exadata storage servers and InfiniBand switches contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. Software updates to Exadata storage servers and InfiniBand switches are performed using patchmgr. Configuration or installed software may not be altered manually (without using patchmgr) in any way unless Exadata documentation contains steps to perform the desired change. While making an unsupported manual change may have the desired immediate effect, there are many potential negative consequences, such as:

    • Failed future patchmgr software updates

    • Inability to rescue the system

    • Inability to diagnose a software defect efficiently

    Contact Oracle Support for further guidance if a desired Exadata storage server or InfiniBand switch change is not documented.

  • Minimize database server customization — Exadata Database Machine is an integrated system and engineered to be the best platform for running Oracle Database. Exadata database servers contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. Software updates to Exadata database servers are performed using patchmgr. However, it may be necessary to manually install additional, site-specific software, such as monitoring agents or backup agents.

    It is supported to manually customize database servers, but note that customizing the operating system by adding or updating packages may require additional actions when applying a future Exadata software update with patchmgr (for example, removing customization prior to updating Exadata servers and reapplying the customization after the Exadata update completes) because the additional software may add new dependencies which are not provided by a future Exadata update. It is recommended to minimize database server customization. See the section "Updating Exadata Database Servers" for additional details.

  • Test supported configuration changes — Database server site-specific customization must be tested completely, including verifying database servers reboot properly after making configuration changes. Custom configuration changes that prevent successful system reboot will cause future Exadata updates to fail.

5.1.2.5 Version Compatibility and Mixed Version Support

It is recommended that software for all components be updated regularly to stay in step with the recommended minimum release in order to maintain the most stable and secure system.

However, it is supported to update only a subset of components during a software maintenance window while the remaining components remain at an earlier version.  For example, the following scenarios are supported:

  • Update a subset of Exadata storage servers to a higher version of the Exadata software while the remaining Exadata storage servers remain at the earlier Exadata software version.

  • Update Exadata storage servers to a higher version of the Exadata software while the Exadata database servers remain at the earlier Exadata software version.

  • Update Exadata database servers to a higher version of the Exadata software while the Exadata storage servers remain at the earlier Exadata software version.

  • Update Exadata storage servers to a higher version of the Exadata software while InfiniBand switches remain at the earlier Exadata software version.

  • Update InfiniBand switches to a higher version of the Exadata software while Exadata storage servers remain at the earlier Exadata software version.

  • Update Exadata storage servers to a higher version of the Exadata software while Oracle Grid Infrastructure or Oracle Database software remains at an earlier release, patch set, and quarterly patch level.

While mixed versions are supported (within the same component or across different components), it is highly recommended that this be only a temporary configuration that exists for the purpose and duration of rolling upgrade.

If mixing versions the following rules and considerations must be observed:

  • A specific generation of Exadata hardware will have a minimum required Exadata software version.  For example, Exadata X6 hardware requires Exadata software 12.1.2.3.1 or higher.  

  • A specific Oracle Database release requires a minimum Exadata software release to fully support Exadata features.  For example, Oracle Database 12c Release 2 (12.2) requires Exadata software release 12.2.1.1.0 or higher on Exadata storage servers to support all Exadata offload features.

  • Features supplied with a new Exadata release may require a minimum Oracle Grid Infrastructure or Oracle Database software release, patch set, or patch.  For example, the Exadata feature Smart Scan Offload for Compressed Index Scan requires Oracle Database 12.2 and Exadata software 12.2 on storage servers.

  • Grid Infrastructure supports mixed version for limited duration for the purposes of rolling update only (for example, updating from 12.1.0.2 to 12.2.0.1, or updating from 12.1.0.2.161018 to 12.1.0.2.170117).  Some Oracle Clusterware functionality is restricted while the cluster is in rolling upgrade mode.

  • Database supports mixed version for Proactive Bundle Patch for the purposes of rolling update only (for example, updating from 12.1.0.2.161018 to 12.1.0.2.170117).

5.1.2.6 Ordering of Updates

 In general, when performing software updates, the updates may be applied to components in any order based on business needs and maintenance window requirements.

The recommended order for applying software updates is:

  1. Oracle Grid Infrastructure and Oracle Database software

  2. Exadata database server software

  3. Exadata storage server software

  4. Exadata InfiniBand switch software

There are exceptions where specific update ordering is required, which are documented in the Exadata software supplemental README for the version being updated to.   Some examples of required update ordering are:

  • When updating a system with Oracle ASM Cluster File System (Oracle ACFS) configured, the Grid Infrastructure home must contain the fix for bug 22810422 before updating the database server (non-Oracle VM or Oracle VM) to Exadata software release 12.2.

  • When updating an Oracle VM configuration from Exadata software release 12.1 to Exadata software release 12.2, all domUs must be updated to Exadata software release 12.2 before updating dom0 to Exadata software release 12.2.

  • When updating an Oracle VM configuration, the Grid Infrastructure home in domU must be running the Proactive Bundle Patch release 12.1.0.2.161018 (October 2016) or later before updating a domU to Exadata software release 12.2.

In general, software update ordering requirements occur when updating one or more components to a later version that contains new features, for example, when updating Exadata software from release 12.1 to release 12.2. Refer to the Exadata software supplemental README for the software version being updated to for additional details.

5.1.3 Understanding the Exadata Software Version

The Exadata version installed on an Exadata storage server or Exadata database server is determined with the imageinfo command.

The image version of an Exadata release consists of three components. For example, if imageinfo -verreturns 12.1.2.1.1.150316.2, then:

  • The product version is 12.1.2.1.1

  • The date code is 150316

  • The Oracle-internal build number is 2. This component is not always used.

5.1.3.1 Rules for Updating to Newer Exadata Version

When upgrading, you need to consider both the product version and date code of the installed and target releases.

In most circumstances it is sufficient to reference a release by the product version only, for example, 12.1.2.1.1.

Upgrading to a specific target version must adhere to the following rules, which are enforced by patchmgr:

  1. The product version of the target release must be higher than the installed software

    and

  2. The date code of the target release must be higher than the installed software.

For example, consider an Exadata Database Machine that is currently running image version 12.1.1.1.2.150411, which consists of product version 12.1.1.1.2 and date code 150411.

  • Upgrading to 12.1.2.1.2.150617.1 is allowed because both rules are satisfied.

  • Upgrading to 12.1.2.1.1.150316.2 is not allowed because date code 150411 of the installed software is higher than date code 150316 of the target release.

5.1.4 Exadata Patchmgr Update Utility

Patchmgr is the utility used to update software for Exadata infrastructure components.

Patchmgr has the following capabilities:

  • With a single invocation updates all Exadata storage servers, Exadata database servers, or InfiniBand switches.

  • Orchestrates the software update across components one at a time when performing a rolling update.

  • Parallelizes the software update to all components at the same time when performing a non-rolling update.

  • Updates all component software, as required, such as firmware, operating system, and Exadata software.

    • Database servers contain all software necessary to run Oracle Database and are configured to run Oracle Database optimally. However, it may be necessary to manually install additional, site-specific software, such as monitoring agents or backup agents.  It is supported to manually customize database servers, but note that customizing the operating system by adding or updating packages may require additional actions when applying a future Exadata software update with patchmgr.

    • Storage server or InfiniBand switch configuration or installed software may not be altered manually (update without using patchmgr) in any way unless Exadata documentation contains steps to perform the desired change.

  • Patchmgr may be run from an Exadata system or a non-Exadata system running Oracle Linux, as the root user or a non-root user.

  • Multiple invocations of patchmgr may be run from the same software directory.

When updating database servers, patchmgr manages the following as needed:

  • Stops and starts databases and clusterware

  • Stops and starts user domains (domUs)

  • Stops and starts Oracle Enterprise Manager Cloud Control agents

  • Unmounts remote network mounts

  • Performs a root file system operating system backup that can be used for rollback

  • Relinks database home and Oracle Grid Infrastructure home binaries

  • Applies updated best practices configuration changes and workarounds for known issues

5.1.4.1 Obtaining Patchmgr

Storage server and InfiniBand switch updates use the version of patchmgr that is bundled with the Exadata software release to which you are updating.

Database server updates use patchmgr from My Oracle Support patch 21634633.  It is recommended to always use the latest patchmgr from My Oracle Support patch 21634633 when updating Exadata database servers.

5.1.4.2 Patchmgr Syntax

Patchmgr is a utility used to update software for Exadata infrastructure components.

Prerequisites

Patchmgr is run on the "driving system", which is an Exadata database server or a non-Exadata system running Oracle Linux.  This allows patchmgr to run from a central server to update multiple Exadata systems, or to update all database servers of a single Exadata system with a single patchmgr invocation.  If patchmgr is run from an Exadata database server, then that database server cannot be in the group file supplied to patchmgr.

Syntax

Patchmgr syntax differs depending on the component to be updated, however, the general patchmgr syntax is:

./patchmgr -component component_list_file -action required_arguments [optional_arguments]

Options

Option Description
component Represents the component to update. Valid values are -cells, -dbnodes, or
-ibswitches
component_list_file A text file containing the hostnames of components to update.  For example, if updating Exadata storage servers (-cells), component_list_file will contain the list of all hostnames of Exadata storage servers.
action The action to perform.  In most cases the action is specific to the component being updated.
required_arguments The additional arguments required for certain actions.
optional_arguments Any optional arguments required for certain actions.

Usage Notes

  • Multiple invocations of patchmgr may be run concurrently from the same software directory by using the -log_dir option.  This allows patchmgr to update multiple Exadata systems concurrently from the same software directory.

  • Patchmgr may be run as the root user or as a non-root user.  The user running patchmgr must have root-level SSH equivalence configured to the servers or switches that patchmgr will update.  To run patchmgr as a non-root user, the -log_dir option must be used.

  • Prior releases used the dbnodeupdate.sh utility to update database servers.  dbnodeupdate.sh has been integrated with and is replaced by patchmgr.

Examples

Example 5-4 Run storage server update pre-requisite checks, then update storage servers

./patchmgr -cells cell_group -patch_check_prereq 
./patchmgr -cells cell_group -patch

Example 5-5 Update storage servers in a rolling manner

The following command updates storage servers in a rolling manner, receive email notifications as the update proceeds, and remove passwordless SSH access to cells after the update is complete.

./patchmgr -cells cell_group -patch -rolling -unkey -smtp_from "dbm01@example.com" -smtp_to "admin1@example.com,admin2@example.com"

Example 5-6 Update muitiple storage servers at one time

The following commands update storage servers on three Exadata systems simultaneously from the same software directory; one update using a rolling update, the others updates are non-rolling.

In this example:

  • Each patchmgr command must be run from a separate terminal window. 

  • Each patchmgr execution uses a unique logging directory name automatically generated based on the content of the component_list_file.

(terminal1) ./patchmgr -cells cell_group_exa01 -patch -rolling -log_dir auto 
(terminal2) ./patchmgr -cells cell_group_exa02 -patch -log_dir auto 
(terminal3) ./patchmgr -cells cell_group_exa03 -patch -log_dir auto

To have a subsequent patchmgr execution use an altered component_list_file with different content, yet use the same logging directory as a prior patchmgr execution, use the -get log_dir option to obtain the logging directory.  For example:

  1. The logging directory for the initial patchmgr execution is generated automatically.

    ./patchmgr -cells cell_group -patch -log_dir auto
    
  2. Assume the last cell failed to update and patchmgr will be re-run for the last cell only, using the same logging directory as the initial patchmgr execution.  Use the -get log_dir option to obtain the logging directory using the original component_list_file.

    ./patchmgr -cells cell_group -patch -log_dir auto -get log_dir log_dir=/tmp/patch_12.2.1.1.1.170419/log/dm01cel01_dm01cel02_cacce4ee
    
  3. Update the cell_group file to contain only the last cell, or use a different file that contains only the last cell.  Specify the logging directory from the initial patchmgr execution so all logs for this group of cells are created in the same logging directory.

    ./patchmgr -cells cell_group -patch -log_dir /tmp/patch_12.2.1.1.1.170419/log/dm01cel01_dm01cel02_cacce4ee
    

Example 5-7 Backup database servers then perform the update

The following commands backup the Exadata database servers, run the pre-requisite checks for the Exadata database server update, and then update the Exadata database servers in a non-rolling manner.

./patchmgr -dbnodes dbnode_group -backup -iso_repo /tmp/p25640941_122111_Linux-x86-64.zip -target_version 12.2.1.1.1.170419 
./patchmgr -dbnodes dbnode_group -precheck -iso_repo /tmp/p25640941_122111_Linux-x86-64.zip -target_version 12.2.1.1.1.170419 
./patchmgr -dbnodes dbnode_group -upgrade -nobackup -iso_repo /tmp/p25640941_122111_Linux-x86-64.zip -target_version 12.2.1.1.1.170419

5.2 Overview of Performing Exadata Software Updates

This section describes the high-level actions required to perform software updates to Exadata components.

5.2.1 Actions to Perform Before Any Software Maintenance

Perform you update any software on your Exadata Database Machine, you should perform these actions.

  • Identify the target release — The target release can be determined by one of the following:

    • Recommendation in My Oracle Support Document 888828.1.

    • Version recommendations in the MAA Scorecard section of the Oracle ExaCHK report.

  • Run Oracle ExaCHK — Ensure software maintenance readiness of the system by running the latest version of Oracle ExaCHK (see My Oracle Support Document  1070954.1).  Review the report for the following:

    1. Correct any FAIL or WARNING checks that deviate from your baseline.

    2. Review the MAA Scorecard section for version recommendations. Oracle ExaCHK evaluates current software versions for consistency, compatibility, and whether or not it is current.

    3. Review the MAA Scorecard section for exposure to critical issues.  Target version choice should resolve any critical issue exposure.

    It is recommended to run the latest version of Oracle ExaCHK in the following times:

    • Every month to maintain a system that continues to adhere to Exadata best practices

    • The week before the software update to ensure maintenance readiness

    • The day before the software update to ensure maintenance readiness

    • Immediately after completing the software update

  • Run prerequisite checks

    1. Download the target software release(s) and the latest versions of the software update utilities (patchmgr, OPatch, and so on) from My Oracle Support.

    2. Run the prerequisite checks and correct any issues. The initial prerequisite check run should be performed days or weeks in advance of the maintenance window to allow time to correct issues. Run prerequisite checks again immediately before the update.

      Note:

      Special consideration must be given to running database server prerequisite checks and the -nomodify_at_prereq flag.  See Updating Exadata Database Servers for details.
  • Review the latest documentation

    1. Review My Oracle Support Document 1270094.1 for recently published critical issues that are not yet automatically checked by Oracle ExaCHK.

    2. Review the Supplemental README in My Oracle Support for the target software version for known issues discovered after the software was released.

5.2.2 Overview of Performing Exadata Storage Server Updates

A high level look at the process for updating Exadata storage servers.

Patchmgr manages the update orchestration in a rolling or non-rolling manner for storage servers. When using the -rolling option, Oracle recommends that you use Oracle ASM high redundancy disk groups to tolerate the failure of a disk in another storage server during the update.

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support document 888828.1.

  2. Create a file that contains the list of storage servers to update. This file will be specified as the component_list_file.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all storage servers in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. If storage servers will be updated non-rolling, then stop Oracle Clusterware and all databases accessing the storage servers.

  6. Run patchmgr to update storage servers in a non-rolling (default) or rolling (-rolling option) manner.

  7. If storage servers were updated non-rolling, then restart Oracle Clusterware and all databases.

5.2.3 Overview of Performing Exadata Database Servers Updates

When updating Exadata database servers, there are different steps for virtualized environments.

Note:

Patchmgr manages the update orchestration in a rolling or non-rolling manner for Exadata database servers. When using the -rolling option, Oracle recommends that you use Oracle RAC features for client high availability to minimize the impact to client applications connected to database instances that will be stopped during maintenance.

5.2.3.1 Steps for Updating Non-Virtualized Configurations

An overview of the steps involved in updating an Exadata database server that does not have Oracle VM configured.

  1. Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  2. Download the target Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support Document 888828.1.

  3. Create the component_list_file containing the list of database servers to update.

  4. Configure SSH equivalence from the user that will run patchmgr to root on all database servers in the component_list_file.

  5. Run patchmgr backup to backup database server root file system and operating system.

  6. Run patchmgr prerequisite check and correct any issues.

  7. Run patchmgr to update database servers in a non-rolling (default) or rolling (-rolling option) manner.

5.2.3.2 Steps for Updating Virtualized Configurations

The management domains (dom0s) and user domains (domUs) are updated independently using separate patchmgr executions.

Updating the Management domain (dom0)

  • Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  • Download the target dom0 Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support Patch Document 888828.1.

  • Create the component_list_file containing the list of database server dom0s to update.

  • Configure SSH equivalence from the user that will run patchmgr to root on all database server dom0s in the component_list_file.

  • Run patchmgr backup to backup database server dom0 root file system and operating system.

  • Run patchmgr prerequisite check and correct any issues.

  • Run patchmgr to update database server dom0s in a non-rolling (default) or rolling (-rolling option) manner.

Updating the User domains (domU)

  1. Download the latest patchmgr from My Oracle Support Patch 21634633 and stage it on the driving system.

  2. Download the target domU Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support document 888828.1.

  3. Create the component_list_file containing the list of database server domUs to update.

  4. Configure SSH equivalence from the user that will run patchmgr to root on all database server domUs in the component_list_file.

  5. Run patchmgr backup to backup database server domU root file system and operating system.

  6. Run patchmgr prerequisite check and correct any issues.

  7. Run patchmgr to update database server domUs in a non-rolling (default) or rolling (-rolling option) manner.

5.2.4 Overview of Performing Exadata InfiniBand Switch Updates

Patchmgr manages the update orchestration in a rolling manner for InfiniBand switches

  1. Download the target Exadata software from My Oracle Support and stage it on the driving system.  See My Oracle Support document 888828.1.

  2. Create the component_list_file containing the list of InfiniBand switches to update.

  3. Configure SSH equivalence from the user that will run patchmgr to root on all InfiniBand switches in the component_list_file.

  4. Run patchmgr prerequisite check and correct any issues.

  5. Run patchmgr to update InfiniBand switches. InfiniBand switches are always updated in a rolling manner.

5.2.5 Overview of Performing Grid Infrastructure and Database Updates

The steps for updating Oracle Grid Infrastructure or Oracle Database software depend on the type of software update you are performing.

Updating to a New Proactive Bundle Patch

Updates to a new Proactive Bundle Patch may be performed in-place or out-of-place. Both in-place and out-of-place methods support rolling and non-rolling updates.

  • In-place

    The update is applied to the current software home using OPatch while the Oracle Grid Infrastructure or Oracle Database software is shutdown on the node being updated.

    This is the default method of applying a Proactive Bundle Patch update for Oracle Grid Infrastructure or Oracle Database software, as described in the Proactive Bundle Patch README. The steps to apply the Proactive Bundle Patch must be performed on each node.

  • Out-of-place (recommended)

    A new software home is prepared and updated while the Grid Infrastructure and/or Database software remains running. Once the new home(s) is prepared, Grid Infrastructure and/or Database are quickly stopped, switched to the new home, and restarted.

    Out-of-place has significant advantage over in-place updates:

    • There is less risk because the new home can be prepared while the Grid Infrastructure and/or Database software remains online.

    • There is less downtime because switching to the new software home is faster than applying an update in-place.

    • Rollback is faster because it is possible to simply switch back to the original software home.

    The recommended way to adopt out-of-place updates is to use Oracle Rapid Home Provisioning (RHP). RHP provides the following advantages:

    • Distributes software updates to all nodes in the cluster.

    • Orchestrates updates across the cluster in a rolling or non-rolling manner with a single command.

    • Provides control over database service relocation to maintain application availability.

    Out-of-place software updates are also supported by Enterprise Manager Cloud Control.

    Out-of-place software updates without RHP or EM may be accomplished by following My Oracle Support Document 2087150.1.

OJVM Patch Set Update

The OJVM patch set update (PSU) is a separate software update for database homes that addresses OJVM security vulnerabilities. It is installed separately from the standard Proactive Bundle Patch. The OJVM PSU patch may be installed in an Oracle RAC Rolling manner under certain situations.

Updating to a New Patch Set Release, Major Release, or Maintenance Release

To update to a higher patch set, maintenance, or major release for grid infrastructure or database, follow the step-by-step instructions in My Oracle Support.

  • Update to 12.2.0.1 - 12.2.0.1 Grid Infrastructure and Database Upgrade for Exadata Database Machine (My Oracle Support Document 2111010.1)

  • Update to 12.1.0.2 - 12.1.0.2 Grid Infrastructure and Database Upgrade for Exadata Database Machine (My Oracle Support Document 1681467.1)

  • Update to 11.2.0.4 - 11.2.0.4 Grid Infrastructure and Database Upgrade for Exadata Database Machine (My Oracle Support Document 1565291.1 and My Oracle Support Document 1555036.1)

5.3 Updating Exadata Database Servers

Exadata database server release updates contain updates for the following components within a database server:

  • Oracle Linux operating system

  • Firmware (Disk, RAID controller, ILOM, HCA)

  • Exadata software

The software and firmware components that are updated for a specific release depend on the current Exadata software release the database server is running and the release you are updating to. Linux operating system packages and Exadata software are always updated while firmware may be updated for only a small selection of the components or not at all.

Updates for Exadata database servers can be applied independently from the Exadata storage servers or InfiniBand switches unless otherwise specified in My Oracle Support Note 888828.1.

Updating Exadata database servers is always be performed in place”. This means the active operating system is updated. The actual update is performed using YUM, but the command to do this is wrapped in an Exadata utility called the “update utility”.

The YUM command is wrapped in the update utility to maintain strict ordering of validation and preparation steps during the update process. Also by using a utility Oracle can enforce application of fixes for known issues and best practices.

Clusterware processes and Oracle RAC database instances must not be running on a database server that is being updated. To reduce application-level impact, follow the client failover best practices described in Oracle Database High Availability Best Practices.

If you cannot afford clusterwide downtime, you can update database servers in a rolling fashion. This means updating one Exadata database server at a time. If you can afford clusterwide downtime, you can update all Exadata database servers in parallel. Non-rolling updates reduce the overall time required at the expense of having a full database outage.

5.3.1 Update Paths for Exadata Database Servers

To update Exadata database servers running Oracle Linux 5.5 or later and Oracle Exadata software release 11.2.2.4.2 or later, you must use the “update utility” for applying Exadata software updates.

To update Exadata database servers running Exadata software releases older than 11.2.2.4.2, you should first update the servers to Oracle Linux 5.5 and Exadata Software release 11.2.2.4.2 or later before you can use the update utility. See the table below for details.

Table 5-1 Update Paths

Currently Installed Oracle Exadata Software Release Currently Installed Oracle Linux Release Recommended Action

Release 11.2.2.4.2 or later

Release 5.5. or later

Update to the target release using the update utility.

Releases earlier than 11.2.2.4.2

Release 5.5. or later

  1. Use patch 13513611 to update the database servers to release 11.2.2.4.2. Refer to the steps in section 8 of the patch 13513611 README.

  2. Update to a later release using the update utility.

Releases earlier than 11.2.2.4.2

Releases earlier than 5.5

  1. Update to release 11.2.3.2.0. Refer to My Oracle Support note 1284070.1 for details.

  2. Update to a later release using the Update utility.

5.3.2 Installing, Updating, and Managing Non-Exadata Software

Installing and updating non-Exadata branded rpms on Exadata database servers are allowed as long as kernel and InfiniBand-specific packages remain untouched.

Note that customizing the operating system by adding or updating packages may introduce problems when applying an Exadata software update because the additional software may add new dependencies which will not be provided by the Exadata update. For this reason it is recommended to stay close to the Exadata image and customize as little as possible.

Note:

Exadata does not ship with 32-bit software. This means any customization that introduces 32-bit rpms on the system will break the Exadata software update and will need to be removed.

In case you have added or updated packages supplied by the Exadata image, you can remove the exadata-sun-computenode-exact rpm when needed as follows:

[root@dm01 ]# rpm -e exadata-sun-computenode-exact

If you have installed customized packages, it is recommended that you have scripts to automate the removal (to run before updating Exadata software) and installation (to run after the Exadata update) of those packages. After the Exadata update, verify that the customized packages are still compatible and are still needed, before re-installing them.

Note:

Packages should not be forced in using the “rpm -Uvh --nodeps” command, unless directed by Oracle Support Services.

The same principle applies for customizations other than installation of additional software. Oracle recommends customizing the operating system as little as possible. Be aware that while changes on the database server are allowed within the earlier described boundaries, it is not possible for Oracle to anticipate each and every customization on every configuration item of the operating system.

There are certain common standards that Exadata utilities depend on. These standards are based on best practices and have been validated over time. By moving away from Exadata standards, systems might be missing optimal settings, and risk is introduced because it is difficult for Oracle to anticipate and validate software for one-off configurations. This is why customizations can cause unexpected results. As a result of adding software or customizing system configuration, you might require changes or additional steps to the standard Exadata update process. With or without customizations, it is highly recommended to validate Exadata updates on test systems before doing them on production systems.

See Also:

See Running Prerequisite Checksfor more information on exadata-sun-computenode-exact.

5.3.3 Customization Levels and Impact

The table below provides some guidance in the risks associated with customizing Exadata database servers. While the prerequisite check checks for most customizations, certain configuration changes might need to be rolled back in order to make updating the database server possible. If still needed, the changes can be restored after the update process.

Table 5-2 Customizations and Their Risks

Item Customization Level Risk

Default provided Engineered System Software Stack

None

None

Setting up interactive shell profiles for system users (root/oracle)

Low

High

Using all free space in VGExaDb

Low

Low

Installing additional (non-Exadata) rpm packages

Medium

Low

Customizing file system with different mount points

Medium

Low

Updating packages shipped with the current Exadata image

Medium

Low

Customizing configuration files, changing basic operating system functionality

High

Medium

Installing additional (non-Exadata) non-rpm packages

High

Medium

Changing LVM layout

High

High

5.3.4 Update Utility for Exadata Database Servers

Patchmgr is the update utility for updating Oracle Exadata database servers.

Starting with release 12.2.1.1.0 (for Exadata x86_64) and 12.1.2.4.0 (for Exadata SPARC), Exadata software updates for the Exadata database server can be applied only through the patchmgr utility.

Note:

The patchmgr utility for updating Exadata database servers is not the same as the patchmgr script shipped with the Exadata software update.

Patchmgr for database server is packaged in dbserver.patch.zip and is available as a separate download from My Oracle Support note 1553103.1. Always make sure that you use the latest release as the utility is updated frequently to address known issues and best practices, and to support new hardware.

The utility supports all hardware generations and Exadata storage server releases starting with 11.2.3.1.0, Oracle Exadata database servers running Oracle Virtual Server (dom0) and Exadata Virtual Machines (domU). The README files for the Oracle Exadata software updates specify whether the update itself is applicable for a particular hardware generation or not. The README is not shipped with dbserver.patch.zip but with the Oracle Exadata software update zip file.

The utility also supports Exadata software updates from Oracle Linux 5 to Oracle Linux 6.

The utility takes care of the orchestration. You can perform the update in a rolling or non-rolling fashion across one or multiple Exadata database servers.

The update utility performs the following tasks:

  • Automates all preparation, update, and validation steps, including:

    • stopping the databases, Grid Infrastructure stack or domU’s

    • stopping Oracle Enterprise Manager Cloud Control agents,

    • un-mounting remote network mounts (when required)

  • Uses the built-in dbserver_backup.sh script to perform a backup of the file system hosting the operating system before updating the Exadata database server.

  • Applies Oracle best practices and fixes for the latest known issues.

  • Verifies that the update was successful, relinks the Oracle binaries, and starts the Oracle stack and domU’s

5.3.5 Update Tool Execution Host

If you are planning to update all Exadata database servers at once, it is a requirement to run the update utility from a Linux node outside the group of Exadata database servers being updated. This is because the update utility cannot update the Exadata database server it is currently running on. If you have no other systems running Oracle Linux or Oracle Solaris you can run the update utility from one of the Exadata database servers. In such cases be sure that the Exadata database server where the update utility is running is not listed in the dbs_group file you specify.

You need to set up ssh equivalence for the root user from the driving node to the root user of all Exadata database servers that will be updated.

5.3.6 Running the Update Utility as a Non-root User and Running Multiple Invocations Concurrently

By default the update utility assumes you want to run as root. It is however possible to run the update utility as a non-root user from a remote host. It is also possible to run multiple invocations at the same time. This allows you to update multiple logical groups of Exadata Database Servers concurrently. To do this, you run the update utility with the -log_dir flag.

Ensure that ssh equivalence is set up for the current user to the root user of the Exadata database servers to be updated.

The following example shows the option of running as a non-root user and running multiple-invocations.

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/ /p23557378_121223_Linux-x86-64.zip
 -target_version 12.1.2.4.0.160710 -log_dir auto

-log_dir specifies the absolute path to the log directory or the keyword autofor the utility to generate and set a path to the log directory that is based on the launch directory and content of nodes list file. If you want to make sure you are using the same log directory in later invocations where the list of Exadata database servers changes, use the -get log_dirflag to obtain the -log_dir location used on previous sessions. For example, if the following command:

[oracle@nonExadata ]$ ./patchmgr -dbnodes ~/dbs_group_test  -log_dir auto -get log_dir

This command returns output similar to the following:

log_dir=/u01/test/dbserver_patch_5.160715/log/dbm01_dbm02_e8f1f75

Use the log_dir value in subsequent commands. For example:

[oracle@nonExadata ]$ ./patchmgr -dbnodes ~/dbs_group_test -precheck 
-log_dir /u01/test/dbserver_patch_5.160715/log/dbm01_dbm02_e8f1f75 
-iso_repo /u01/test/dbserver_patch_5.160715/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts

5.3.7 Recommended Timeline for Updating Exadata Database Servers

Oracle recommends the following timeline for updating Exadata database servers. By following this approach, you allow yourself time to perform any necessary corrective actions.

Note:

Before making any changes, including prerequisite check, you should make a backup.

Table 5-3 Timeline for Performing Updates

When Tasks

Weeks to days before the update

  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.

  • Download the latest Exachk from My Oracle Support note 1070954.1.

  • Research release-specific My Oracle Support note for known issues.

  • Research Exadata Critical Issues from My Oracle Support note 1270094.1.

  • Run Exachk.

  • Perform a first prerequisite check.

    Note: Sometimes the update utility needs to make minimal changes to the Exadata database server to make YUM dependency checks work. If you wish to ensure no modifications are made, run the prerequisite check with the -nomodify_at_prereq flag.

    For details, see Running Prerequisite Checks.

  • Correct findings that need attention. You can ignore any dependency problems if you are unable to remove rpms at this point. Because -nomodify_at_prereq was specified, broken dependencies can be expected.

Just before the update

  • Download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.

  • Download the latest Exachk from My Oracle Support note 1070954.1.

  • Research release-specific My Oracle Support note for known issues.

  • Research Exadata Critical Issues from My Oracle Support note 1270094.1.

  • Run Exachk and perform corrective actions as needed.

At update time

  • Perform a “backup only” run using the -backup flag.

    See Backing up Exadata Database Servers Before Planned Maintenance for details.

  • Perform a second prerequisite check. This time, omit the -nomodify_at_prereq flag to allow the update utility to make the required changes to get YUM dependencies to work. See Running Prerequisite Checks for details.

  • Remove any blocking rpms and re-run the prerequisite check to validate if all changes are complete.

  • Perform the update. Use the -nobackup flag to skip the backup because you already made a “backup only” run.

    See Running the Update for details.

After the update

  • Run Exachk.

  • Reinstall any non-Exadata rpms that you removed before the Exadata update.

5.3.8 Preparing and Populating the YUM Repository with the Oracle Exadata Channel Content

The Exadata software update procedure for Exadata database servers uses the Unbreakable Linux Network (ULN) for distribution. The updates are distributed as a set of packages (rpms) grouped in a channel. You need to ensure that these packages are available locally before performing the prerequisite check or update.

A channel can be subscribed to by a local non-Exadata database server. Channel content is then downloaded to the local server and the repository is made available as a YUM repository which is used to update Exadata database servers.

If no Internet connection is available from the data center or if ULN synchronization is not possible, then a ready-to-use YUM repository can be downloaded as an ISO image.

Note:

Each Exadata Storage Server Update comes with its own channel/ISO image. This ISO image and ULN channel are not intended to be a complete replacement of a generic Linux ULN channel. The Exadata ISO and ULN channel contain only the Oracle Exadata channel content. If your environment uses other Linux packages for which updates are not contained in the Oracle Exadata ULN channel or ISO image, you should update those packages using Linux ULN channels as needed.

Use the following methods to build a YUM repository for hosting the Exadata software update. The update utility uses the local YUM repository to update the Exadata database servers.

5.3.8.1 Downloading the ISO Image as a YUM Repository from My Oracle Support and Passing the Location of the Compressed Image to the Update Utility

The Oracle Exadata channel holding the Oracle Exadata software update is available as a downloadable image (a compressed ISO file) from My Oracle Support and can be used in two ways:

  • Use the file location of the compressed file as an argument for the update utility using the -iso_repo flag

  • Publish using HTTP and use the URL for the update utility using the -yum_repo flag.

The file location of the compressed file can be passed directly to the update utility which is recommended for the following conditions:

  • There is no separate server available to be used as a repository YUM HTTP server.

  • The Exadata database servers do not have customized software that requires updates from additional ULN channels.

  • The simplest method is preferred.

Note:

The compressed ISO image should only exist on the node running the update utility. The distribution of the compressed ISO image is handled by the update utility

You can perform a prerequisite check (-precheck) to verify the repository is usable.

5.3.8.2 Setting Up a Local Mirror as a YUM HTTP Repository from the ULN Oracle Exadata Channel on a Separate Server

This method is recommended for the following conditions:

  • There is a large number of Exadata database servers to update.

  • The single repository server is accessible to all the Exadata database servers.

  • An infrastructure exists for building a local ULN mirror on a separate Linux server.

  • Exadata database servers have customized software that requires updates from other ULN channels.

The mirror server is a separate Linux server that holds the downloaded Exadata software update for Exadata database servers as downloaded from ULN. The Exadata database servers connect to this local mirror (YUM HTTP repository) to retrieve updates.

Note:

Do not use an Exadata database server as the local ULN mirror. Doing so may lead to dependency conflicts between the packages required by ULN to construct the repository, and the packages installed or updated on the Exadata database server by the Exadata software update. If no separate server is available, then use the ISO image method instead. See "Downloading the ISO Image as a YUM Repository from My Oracle Support and Passing the Location of the Compressed Image to the Update Utility".

When setting up a local repository for the first time, you may need additional Linux subscriptions, depending on the Linux release being used for the server that will be hosting the YUM repository. The server should subscribe to the following additional ULN channels in order for the repositories to be built:

  • For 64-bit systems: Enterprise Linux 5 systems: el5_x86_64_addons are required.

  • For 64-bit systems: Oracle Linux 5 systems: el5_x86_64_addons are required.

  • For 64-bit systems: Oracle Linux 6 systems: ol6_x86_64_addons are required.

Note:

After registration, depending on the operating system running your local ULN mirror, the system server automatically subscribes to el5_x86_64_latest, ol5_x86_64_latest, or ol6_x86_64_latest.

Review the prerequisites and server setup procedure as described in "How to create a local Unbreakable Linux Network mirror" at http://www.oracle.com/technetwork/articles/servers-storage-admin/yum-repo-setup-1659167.html to create and maintain a local ULN mirror as YUM repository. For step 5 in the above instructions, in addition to the listed channels, perform the following step for Exadata:

  • Add the channels listed in the Oracle Exadata patch README, such as exadata_dbserver_12.1.2.3.2_x86_64_base.

    The Oracle Exadata channel to subscribe to depends on the Oracle Exadata software release you want to update to. See the patch README for the release for details.

The YUM mirror creation procedure mentioned in the link above provides a script (uln-yum-mirror) which you can use to populate the local mirror. Running the script as root starts the rpm download from ULN to a local directory holding the repository.

You need to install a web server and configure its “document Root” directory to point to this repository location on disk so that the Oracle Exadata channel content is made available to the Oracle Exadata Linux database servers using the HTTP web server. This is described in step 7 from the mentioned instructions.

When you run the update utility, you specify the URL to the repository location. This URL should always be at the level of the “repodata” directory. You can run a prerequisite check (-precheck) to verify that the repository works. See the prerequisite check example using YUM repository in "Running Prerequisite Checks."

Note:

Exadata database servers must be running Exadata release 12.1.2.2.0 or later to access YUM repositories listening on IPv6 addresses only.

5.3.8.3 Downloading the ISO Image as a YUM Repository from My Oracle Support and Making It Available as a YUM HTTP Repository on a Web Server

You can download, uncompress, and place the ISO image channel holding the Oracle Exadata software update on a web server that has HTTP connectivity to every Oracle Exadata database server. This method is recommended for the following conditions:

  • There is a large number of database servers to update.

  • The single repository server is accessible from all the database servers.

    Note:

    If you run the update utility on a non-Exadata system against this ISO image, this is taken care of locally on each database server. In such cases there is no need to set up a local mirror.

  • The database servers do not have customized software that requires updates from additional ULN channels.

    Note:

    If the database servers have custom software installed it is better to use an HTTP YUM repository and not the ISOs, because this improves flexibility in adding back/installing rpms.

Use the following procedure to make the ISO image of the Oracle Exadata channel content available on a Linux-based server running an HTTP server. You can adapt the instructions for other operating systems. In this procedure, the ISO image is copied to the /u01/app/oracle/stage directory. The “Document Root” entry in the web server document is /var/www/html. The procedure uses release 12.1.1.1.0 as an example.

Note:

Oracle recommends that a separate server (non Exadata database server) be used as the web server.

  1. Create the ISO image mount point as the root user using the following commands:

    [root@nonExadataHost ]# mkdir -p /var/www/html/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base
    
    [root@nonExadataHost ]# mkdir –p /u01/app/oracle/stage
    
  2. Copy the compressed ISO image to the web server into /u01/app/oracle/stage.

  3. Uncompress and mount the ISO image as the root user using the following commands:

    [root@nonExadataHost ]# unzip p17997668_121110_Linux-x86-64.zip
    
    [root@nonExadataHost ]# mount -o loop /u01/app/oracle/stage /121110_base_repo.iso /var/www/html/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base
    
  4. Start the httpd service as the root user. This assumes httpd is installed.

    [root@nonExadataHost ]# service httpd start
    
  5. Identify and test the YUM HTTP repository by connecting to it using a web browser. The following is an example of the repository URL.

    http://yum-repo/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base/x86_64/
    
  6. Run the prerequisite check as the root user to verify that the repository is set up properly. See Running Prerequisite Checks.

5.3.9 Managing Exadata Obsoleted Packages

If you are updating to release 11.2.3.3.0 or later, some packages on the Exadata database server become obsolete. While updating an Exadata database server, the update utility prints the exclude rpm list and obsolete rpm list in the log file.

The following example shows the exclusion and obsolete lists from the log file. In this example, an exclusion list has not yet been created by the user.

RPM exclusion list  : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list   : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
                    : RPM obsolete list is extracted from exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64.rpm

To see which packages will become obsolete, review the contents of the obsolete.lst file. This file lists the packages defined to be obsolete by Exadata; these packages will be removed during the update when no action is taken. Packages manually added to this list are ignored. The following is a small sample of the obsolete.lst file:

[root@dm01 ]# cat /etc/exadata/yum/obsolete.lst
# Generated by dbnodeupdate.sh runid: 021213024645
at.x86_64
java-*-openjdk
rhino.noarch
jline.noarch
jpackage-utils.noarch

To prevent a package listed in the obsolete.lst file from being removed, create the /etc/exadata/yum/exclusion.lst file, and put in the rpm name (wildcards are allowed) for the packages you want to keep. Place the /etc/exadata/yum/exclusion.lst file on all Exadata database servers where you want to use it.

The following example shows a package added to the exclusion list:

[root@dm01 ]# cat /etc/exadata/yum/exclusion.lst
java-*-openjdk

After you have added an entry to the exclusion.lst file and re-run the update utility, the utility detects the exclusion list. The rpm packages on the exclusion list are still shown in the obsolete.lst file, but the listed packages in the exclusion.lst file will not be removed during the update.

5.3.10 Updating Individual Packages

Due to security findings or customization, you may need to update individual (general purpose) Linux packages supplied by the Exadata release. You can do this by removing the exadata-sun-computenode-exact rpm first.

Removing this rpm does not impact any functionality, but it removes a “lock”. By removing this “lock”, you are allowed to update specific individual Linux rpms.

If needed, you can remove the exadata-sun-computenode-exact rpm as follows:

[root@dm01 ]# rpm -e exadata-sun-computenode-exact

When updating to a new release, the update utility tries to “restore” the “lock” (exadata-sun-computenode-exact). For example, this is possible if newer packages are shipped in the newer release.

If the exadata-sun-computenode-exact rpm cannot be restored, the update utility falls back to exadata-sun-computenode-minimum.

Note:

Do not force in packages using the rpm -Uvh --nodeps command, unless directed by Oracle Support Services.

5.3.11 Running Prerequisite Checks

You should always run prerequisite checks before doing the actual update. The prerequisite checks do not require downtime and execute important validations such as:

  • Validation of the Exadata release (minimum is 11.2.2.4.2 running Oracle Linux 5.5)

  • Validation of user input

  • Validation of the installation media (YUM repository, either ISO or HTTP)

  • Validation of disk space and snapshots

  • Validation of YUM settings that are important for the update to finish successfully

  • Known issues / best practices

The most important validation executed by the update utility is the YUM dependency check. The YUM dependency check is a YUM update dry-run command (introduced in 11.2.3.3.0) that does not do the actual YUM update but does validate dependencies. This is a final test in determining whether or not the update can proceed. It is often due to customizations that prevent successful updates. For example, installation of additional RPMs might require dependent packages that are not in the YUM repository. If this happens, you need to take corrective action to resolve the conflict.

The YUM dependency check (dry-run) is validated against minimum and exact dependencies. These dependencies are enforced by non-functional Exadata RPMs and help administrators stay exactly at (or close to) the original Exadata release when customizing the system. The update utility uses the exadata-sun-computenode-exact and the exadata-sun-computenode-minimum RPMs as follows:

  • The exadata-sun-computenode-exact rpm ensures that only a specific release of Oracle Exadata branded packages is allowed during the update. (release = x)

  • The exadata-sun-computenode-minimum rpm ensures that a specific or later release of Oracle Exadata branded packages is allowed during the update. (release >= x)

With exadata-sun-computenode-exact rpm, the system appears as if it were freshly imaged to the newer release because all the Oracle Exadata packages are exactly the same as on a vanilla installation. The exadata-sun-computenode-minimum rpm, however, sets the minimum dependencies, and enforces that all packages are installed, but it also allows packages to be at a later version. A vanilla installation always starts with both RPMs. To allow customization or updates, you need to remove exadata-sun-computenode-exact.

By default, the update utility attempts to match the exact dependencies when updating to a later Exadata release. When exact dependencies conflict and cannot be enforced, the utility falls back and attempts to apply the exadata-sun-computenode-minimum rpm to enforce minimum dependencies. In such cases the exadata-sun-computenode-exact rpm is not installed.

Missing or not updating with exact dependencies is allowed and not a problem. If a system needs to be updated to the exact dependencies, then the conflict needs to be resolved first. Check the log file to see what packages conflict, remove them cautiously, and then re-run the update utility in prerequisite check mode.

If the prerequisite check fails due to dependency issues, you can view the errors on-screen. The update utility’s log file has more details and shows which dependencies failed. When both exact and minimum dependencies do not match, the update cannot proceed.

For such cases, check the log file to determine what caused the dependencies to fail. After removing the failed dependencies, re-run the update utility to ensure that at least the minimum dependencies can be enforced.

Note:

The update utility may need to remove certain packages during prerequisite check in order to make sure the YUM prerequisite check works as some Exadata storage server updates have known dependencies that need to be resolved. While the removal of such RPMs during the prerequisite check should be rare and should never impact Exadata functionality it is understood that not all systems allow a prerequisite check to make modifications. For such systems it is recommended to use the -nomodify_at_prereq flag. When this flag is used, the update utility will not remove packages from your system, but it will generate a list of packages that would have been removed (installed or not). When this flag is used, the dependency check may fail. For this reason it is recommended that you re-run the dependency check without the -nomodify_at_prereq flag after you have performed a backup-only run in your planned maintenance window.

When dependency errors occur during the prerequisite check or before the update starts, do the following to resolve the problem:

  • Analyze the YUM errors in the log file. Search for Error.

  • Depending on the issue, you may need to de-install, install, or update the rpm packages causing the dependency issue or conflict. The log file lists the failed dependencies.

After the update, you may re-install custom rpm packages that you de-installed, assuming you still require the packages and the packages are compatible with the updated system.

By default, prerequisite check warnings such as active NFS mounts result in a failed check. If you want to allow active NFS mounts, then starting in release 12.1.2.1.1 you can use the -allow_active_network_mounts flag.

For more options, see the update utility built-in help.

Prerequisite Check Examples

The following command shows an example of a prerequisite check that does not remove any RPMs using ISO. The command is run as root.

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip 
-target_version 12.1.2.3.0.160207.3 -nomodify_at_prereq

-dbnodes specifies the list of database nodes to be updated.

-precheck specifies the prerequisite check action.

-iso_repo specifies the location of the ISO (YUM) repository on the node running the updating utility. This can be replaced by -yum_repo <location>.

-target_version specifies the target release the database servers are being updated to.

-nomodify_at_prereq specifies no RPMs will be removed at prerequisite check time.

The following command shows an example of a prerequisite check that allows the removal of necessary RPMs using ISO. The command is run as a non-root user.

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes dbs_group -precheck -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip 
-target_version 12.1.2.3.0.160207.3 -log_dir auto

5.3.12 Backing up Exadata Database Servers Before Planned Maintenance

Oracle recommends making a backup of the Exadata database servers before making any changes when updating to a next release.

This means running the update utility in backup-only mode before running the prerequisite check without the -nomodify_at_prereq flag or before making any other (manual) changes to make the prerequisite / dependency check pass. The backup-only action backs up the active root and /boot file system only and is sufficient for rolling back (failed) updates.

Note:

The update utility’s built-in backup (dbserver_backup.sh) is insufficient for recovering from non-booting system failures. It is recommend having additional (validated) backup/restore procedures in place to recover the entire database server from failures such as double disk failures.

For regular and virtualized Exadata database servers (domU), when the active system image is running from a file system on /dev/mapper/VGExaDb-LVDbSys1, the backup is made onto /dev/mapper/VGExaDb-LVDbSys2 (and vice versa). For Exadata database servers running dom0 with /dev/mapper/VGExaDb-LVDbSys2 as the active image, the backup goes to /dev/mapper/VGExaDb-LVDbSys3 (and vice versa).

This means that upon rollback of an update the active system image for regular database servers and domU will become /dev/mapper/VGExaDb-LVDbSys2, and for dom0 will become /dev/mapper/VGExaDb-LVDbSys3.

The process of backing up an active system partition requires the default LVM scheme for the Sys* LVM’s and both LVM’s to be the same size. This is because it is impossible to back up a larger LVM partition to a smaller LVM partition. Resizing LVM VGExaDb-LVDbSys1 partition is allowed as long as VGExaDb-LVDbSys2 is resized to the same size.

While the backup is running, an LVM snapshot takes care of a consistent view of the file system. This LVM snapshot is maintained by the backup script and will always claim 1G of free VG space in VGExaDb.

Space for the snapshot is guaranteed by Oracle by using a placeholder LVM called /dev/VGExaDb/LVDoNotRemoveOrUse. When the backup script runs, it removes this placeholder, making sure there is always 1G of free space available for a snapshot. After the backup is completed, the snapshot is removed and the /dev/VGExaDb/LVDoNotRemoveOrUse LVM is re-created.

You can perform the backup while all system services are up and running. Starting with release 12.1.2.1.1, the utility supports active network mounts (NFS) with the -allow_active_network_mounts flag.

The time it takes to run the backup depends on how busy the system is and on the size and type of data that is backed up. For example, backing up millions of small files can take significantly longer than backing up a couple of larger files. For this reason it is recommended to make sure directories holding database .aud files are not found on the root file system. Note the following:

  • You can have only one backup. Running a new backup means overwriting the existing backup.

  • Re-running the update utility with the -backup flag (or in default updating mode) will overwrite existing backups.

  • All files in /boot and found on the active Sys LVM’s are backed up. Files in, for example, /u01, are not backed up.

Examples for “Backup Only”

The following example shows running “backup only” as root:

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -backup -iso_repo /u01/exa/p22750145_121230_Linux-x86-64.zip 
-target_version 12.1.2.3.0.160207.3 -allow_active_network_mounts
  • -dbnodes specifies the list of database nodes to be updated.

  • - backup specifies the “backup only” action.

  • -iso_repo specifies the location of the ISO (YUM) repository on the node running the update utility. Alternatively, you can use the -yum_repo flag, which specifies the HTTP location of the YUM repository.

    -target_version specifies the target release for which the backup is run.

  • -allow_active_network_mounts ensures active network mounts remain active while performing the backup.

The following example shows a “backup only” action run as a non-root user:

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group_scab -backup -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -smtp_from "sender@somedomain.com" 
-smtp_to "recipient@example.com"

5.3.13 Running the Update

You can perform the actual update of Exadata database servers in a rolling (using the -rolling flag) or non-rolling fashion. The default is non-rolling.

You can also perform the update as root or as a non-root user (using the -log_dir flag), as described in "Running the Update Utility as a Non-root User and Running Multiple Invocations Concurrently".

The update proceeds only if the “minimum dependency check” succeeds. You may need to remove customizations for the update to proceed.

By default the update creates a backup on the inactive system image. If you have already taken a backup before running the prerequisite check without using the -nomodify_at_prereq flag, you can skip the backup using the –nobackup flag when performing the update.

Note:

Use the -nobackup flag only if a backup was already made before running the prerequisite check without the -nomodify_at_prereq flag.

The update action requires the following mandatory flags:

  • –upgrade to specify the update action

  • -iso_repo (for ISO image) or -yum_repo (for HTTP locations) to point to the YUM repository (see Example 5-9)

  • -target_version to specify the release you want to update to. The patch README always has this information.

You can specify additional flags to allow active remote network mounts during backup and updating (-allow_active_network_mounts) and specify mail recipients for updating status notification (-smtp_from "addr" and -smtp_to "addr1 addr2 addr3 ...")

Example 5-8 Running Update Using ISO Image for YUM Repository

The following example shows an update action run as root and using an ISO image for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[root@dm01 ]# ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -smtp_from "sender@somedomain.com" 
-smtp_to "receiver@somedomain.com" -nobackup

The following example shows an update action run as a non-root user from a remote host using an ISO YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -iso_repo /u01/iso/p23557378_121223_Linux-x86-64.zip 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -log_dir auto -smtp_from "sender@somedomain.com" 
-smtp_to "receiver@somedomain.com" -nobackup

Example 5-9 Running Update Using HTTP Location for YUM Repository

The following example shows an update action run as root using HTTP for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[root@dm01 ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -yum_repo http://yum-repo/yum/ol6/EXADATA/dbserver/12.1.2.2.3/base/x86_64/ 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -smtp_from "sender@somedomain.com" -smtp_to "receiver@somedomain.com"
-nobackup

The following example shows an update action run as a non-root user from a remote host using HTTP for the YUM repository. Active network mounts are allowed, and mail information is specified for status notification:

[oracle@nonExadataHost ]$ ./patchmgr -dbnodes ~/dbs_group -upgrade -yum_repo http://yum-repo/yum/ol6/EXADATA/dbserver/12.1.2.2.3/base/x86_64/ 
-target_version 12.1.2.2.3.160720 -allow_active_network_mounts -log_dir auto -smtp_from "sender@somedomain.com" 
-smtp_to "receiver@somedomain.com" –nobackup

5.3.14 Rolling Back Updates

A backup enables you to roll back updates, regardless of whether the update failed or succeeded. This backup is stored on the inactive system partition, as described in "Backing up Exadata Database Servers Before Planned Maintenance".

When rolling back an update, the update utility performs the following actions:

  • Shuts down the stack and domU’s.

  • Deactivates the active partition, and activates the inactive partition.

  • Restores /boot from the inactive partition.

  • Updates the grub boot-loader.

Having only one inactive system partition limits the rollback options to only the previous active image.

Example 5-10 Running Rollback

[root@dm01 ]# ./patchmgr -dbnodes dbs_group -rollback -target_version 12.1.2.3.0.160207

-dbnodes specifies the list of database nodes to be updated.

-rollback specifies the rollback action.

-target_version specifies the target release to roll back to.

For more options see the update utility’s built-in help.

Note:

Firmware updates are not rolled back when rolling back to a previous image. After rolling back, run the following commands to apply older firmware versions when needed:

/etc/init.d/lsidiag stop

/etc/init.d/lsi_mrdsnmpd stop

/opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact

The last command only applies to releases 11.2.3.3.0 or later.

5.3.15 Troubleshooting Exadata Database Server Updates

You can use the log files generated by the update utility to troubleshoot updates.

The update utility orchestrates updating the Exadata database servers. The log file (dbnodeupdate.log) and the diag file (dbnodeupdate.<runid>.diag) will eventually exist on two locations:

  • On each updated database server, in the /var/log/cellos directory

  • Consolidated on the node running the update utility.

On the node running the update utility, if the -log_dir flag was set to “auto”, the log files will be stored in the log/<directory based on contents of nodes in list file> directory, relative from the directory where the update utility is started from. For example, if the update utility is located in /u01/dbserver.patch, then the log directory may be /u01/dbserver.patch/dm01db01_dm01db02_e8f1f753.

Important files found in the log directory are:

  • patchmgr.log contains the consolidated screen output from running the remote update commands on the different database servers.

  • <hostname>_dbnodeupdate.<runid>.diag is the diag file for the specific run on a database server.

  • <hostname>_dbnodeupdate.log contains dbnodeupdate.log output appended from /var/log/cellos from the remote database server.

When a prerequisite check, backup, update, or rollback fails, error messages on screen should provide information on which step failed on which node. Consult the log files mentioned above if more information is required. Search the log file for the start of a new run (search for “zzz”).

Check if the time matches your run. If it matches, note the runid for further reference. Then search for ERROR.

If an update action fails before the actual YUM update, you can retry the update after resolving the error. If the update aborted half way, it is recommended that you roll back, resolve the error, and retry.

5.4 Updating Exadata Storage Servers

Exadata storage server release updates contain updates for the following components within an Exadata storage server:

  • Oracle Linux operating system

  • Firmware (Flash, Disk, RAID controller, ILOM, HCA)

  • Exadata software

What software and firmware that is updated depends on the current Exadata software release the storage server is on and the release it is updated to. Linux operating system packages and Exadata software are always updated, while firmware updates are applied only on a small selection of the components or not at all.

Updates for Exadata storage servers can be applied independently from the updates to Exadata database servers or InfiniBand switches unless specified otherwise. It is not mandatory to apply each and every Exadata software update that comes out. For example, you can skip two or three releases and update directly to a newer release.

Updating the Exadata storage server is always performed “out of place”. This means that a new version of the operating system including the Exadata storage server software is installed on the inactive system partition. The utility to update the Exadata software ships with the update itself.

If you cannot afford cluster-wide downtime, you can update the storage servers in a rolling fashion. Rolling means updating one Exadata storage server at a time. If you can afford the cluster-wide downtime, you can update all Exadata storage servers in parallel. Non-rolling updates reduce the overall time required.

Starting with Exadata Storage Server Software release 18.1.0.0.0, there is a more scalable alternative to using patchmgr for software updates. The storage servers automatically verify preconditions and download the update software from a URL that you specify. Each storage server downloads the software to its active partition, and then loads the software on its passive partition. At a specified time, the cells reboot to the new version. The storage servers use the Oracle ASM disk deactivation status to determine when it is safe to deactivate the disks and reboot the storage server to the new software version. These scheduled updates invoke the same scripts that are currently used by the patchmgr process.

5.4.1 Scheduling Automated Updates of Storage Servers

Starting with Exadata Storage Server Software release 18.1.0.0.0, you can schedule software updates for the storage servers.

Perform the following steps from an external server, for example, a compute node.

If you are accessing the Software Update store using the HTTPS protocol, then TLS certificate checks are required by default. If the certificate for the web server that hosts the software update cannot be validated, then the following error is returned:

CELL-00076: An error occurred during download of software update:
source https://example.com:port is not available. CELL-00092:
The store's TLS certificate cannot be authenticated with known CA certificates
  1. Copy the software update ZIP file to a directory which is hosted by a web server.
  2. If you are using Oracle Exadata System Software 18c (18.1.0) or 18c (18.1.1), then the patch file must have a name like 18.1.1.0.0.171018.patch.zip.
    If the downloaded patch has a name like p26875767_181100_Linux-x86-64.zip, then rename the file to 18.1.1.0.0.17018.patch.zip. Rename the ZIP file to use the release number and date string that is used for the directory name within the ZIP file. For example, when you unzip the patch p26875767_181100_Linux-x86-64.zip it extracts the directory patch_18.1.1.0.0.171018.
  3. Determine approximate time to perform the software update of the storage servers.
  4. Create a file on the external server that contains the list of cells to be updated. Name this file cells.
  5. Use dcli to schedule the update of the cells.
    1. Setup password-less access, if needed.
      $ dcli –g cells –k 
      
    2. Specify the location of the software update ZIP file to use during the update.
      $ dcli –g cells cellcli –e 'alter softwareUpdate store=\"https://host/exa-updates/cell\"'
      
    3. Specify the time to update the Exadata Storage Server software on the cells.

      If you specify the time before providing the software store location, then the software update download might start before the proper store location has been set.

      $ dcli –g cells cellcli –e 'alter softwareUpdate time=\"1 AM Thursday\"'
      
  6. Wait for the updates to occur.

    Management Server (MS) will the start file download and run pre-checks up to one week before the scheduled update. MS will generate an alert if any cell does not update as scheduled.

Related Topics

5.4.2 Update Utility for Oracle Exadata Storage Server

You use the patchmgr update utility for updating Oracle Exadata storage servers. For Exadata storage server updates the utility is packaged (and shipped) with the update itself and is available for download from My Oracle Support as storage server update.

Note:

The patchmgr used for updating Exadata storage servers is not the same as the patchmgr used for applying the Exadata database server software update.

Whether or not the utility supports the Exadata hardware you have depends on the Exadata storage server release you are trying to update to. The utility orchestrates the update process across the specified Exadata storage servers. The utility allows running the update in a rolling or non-rolling fashion. You can run the update utility from Exadata database servers or from other servers running Oracle Linux or Oracle Solaris.

The update utility performs the following tasks:

  • Automates the preparation, update, and validation steps

  • Automates rollbacks

The update utility supports multiple sessions: you can run multiple updates concurrently from the same server starting with release 12.1.2.3.2 for Exadata storage servers and starting with release 11.2.3.1.0 for Exadata database servers. This means multiple racks can be updated concurrently from the same server. The update utility can be run as root or as a non-root user. By default the update utility assumes it should run as the root user. If however you want to enable multiple session support or run as a non-root user, then you need to use the -log_dir flag. The -log_dir flag supports two types of arguments: either a location on disk or the keyword auto. If you specify auto, the update utility creates its own log directory based on the storage servers listed in the cell_group file. This behavior causes the update utility to create new directories for each run of updates in the same cluster where one or more clusters were added or removed from the cell_group file. In order to obtain (and reuse) such a directory, the update utility provides the -get flag to determine the log directory for your session. The-get flag scans the working directory for directories in the log directory and returns the directory for your cell_group. For example, the following command:

[oracle@nonExadataHost ]#./patchmgr -dbnodes ~/cell_group  -log_dir auto -get log_dir

The previous comment might return output similar to the following:

log_dir=/u01/test/patch_12.1.2.4.0.160802/log/dbm02celadm01_dbm02celadm02_9cfbc690

In a subsequent update session, you can re-use the log directory location:

[oracle@nonExadataHost ]# ./patchmgr -cells ~/cell_group -patch_check_prereq -log_dir /u01/test/patch_12.1.2.4.0.160802/log
/dbm02celadm01_dbm02celadm02_9cfbc690

5.4.3 Recommended Timeline for Updating Exadata Storage Servers

Note:

It is highly recommended to validate Exadata updates on test systems before doing them on production systems.

By following the approach below, you allow yourself time to perform any necessary corrective actions.

Table 5-4 Timeline for Performing Updates

When Tasks

Weeks to days before the update

  • Download the Exadata storage server update you require from My Oracle Support note 888828.1

  • Download the latest Exachk from My Oracle Support note 1070954.1

  • Review the release-specific My Oracle Support note for known issues

  • Run Exachk

  • Perform prerequisite check. See Preparing Exadata Storage Servers for Update for details.

  • Correct findings that need attention and re-run the above steps as needed.

At update time

  • Download the Exadata storage server update from My Oracle Support note 888828.1

  • Download the latest Exachk from My Oracle Support note 1070954.1

  • Review the release-specific My Oracle Support note for known issues

  • Run Exachk and perform corrective actions as needed

  • Perform prerequisite check and corrective actions as needed.

  • Perform the update. See Running the Update for Exadata Storage Servers for details.

After the update

  • Run Exachk.

5.4.4 Preparing Exadata Storage Servers for Update

Perform these preparation steps before updating the Exadata storage servers.

You can perform the update (and rollback) action in a rolling or non-rolling method. You can also perform a prerequisite check in a rolling or non-rolling method. The default is non-rolling.

  1. Set up SSH equivalence for the user that is driving the update utility.

  2. Download and run Oracle ExaCHK. Review and address any open issues. See My Oracle Support note 1070954.1.

  3. Review the release-specific My Oracle Support note for any known issues and workarounds.

  4. Check prerequisites for your method of update or rollback.

    Prerequisites for performing a rolling update:

    1. Verify that your Grid Infrastructure home and Database home software versions and patch levels meet the minimum required for Exadata storage server rolling cell update as documented in My Oracle Support note 888828.1

    2. Verify failgroup_repair_time or disk_repair_time for each Oracle ASM disk group.

      When applying the update in a rolling manner, the update utility updates one server at a time, first taking all grid disks and ASM disks offline, then applying the update to the server, then bringing all ASM disks and grid disks back online. The Oracle ASM repair timeout attributes, disk_repair_time and failgroup_repair_time, need to be set to a value large enough to allow a single storage server update to complete. The default values of 3.6h and 24h, respectively, are the recommended values. Note that during a rolling storage server update disk groups with compatible.asm >= 12.1.0.2.0 will use the value of failgroup_repair_time, and disk groups with compatible.asm < 12.1.0.2.0 will use the value of disk_repair_time.

      Verify ASM repair timeout attributes are set to default or higher values. Use the following command to check repair time attributes for all mounted disk groups in the Oracle ASM instance.

      SQL> col attribute format a30
      SQL> col value format a10
      SQL> select dg.name as diskgroup, a.name as attribute, a.value
           from v$asm_diskgroup dg, v$asm_attribute a
           where dg.group_number=a.group_number
             and (a.name like '%repair_time' or a.name = 'compatible.asm');
      
      DISKGROUP                      ATTRIBUTE                      VALUE
      ------------------------------ ------------------------------ ----------
      DATA                           disk_repair_time               3.6h
      DATA                           failgroup_repair_time          24.0h
      DATA                           compatible.asm                 12.1.0.2.0
      RECO                           disk_repair_time               3.6h
      RECO                           failgroup_repair_time          24.0h
      RECO                           compatible.asm                 12.1.0.2.0
      
      

      If the Oracle ASM repair timer for any disk group is lower than the default value, then set the repair timer to the default value for the duration of the update. You may set it back to its current value after the update successfully finishes for all storage servers.

    Prerequisites for performing a non-rolling update:

    1. Shut down and stop the Oracle components on each Exadata database server using the following commands, where Grid_home is the directory where the Oracle Grid Infrastructure software is installed:

      [root@dm01 ]# dcli -g dbs_group -l root "Grid_home/bin/crsctl stop crs"
      

      If Oracle Clusterware was not stopped using the preceding command, then use the following command to force it to stop:

      [root@dm01 ]# crsctl stop crs -f
      

      Use the following command to check Oracle Clusterware status, where Grid_home is the directory where the Oracle Grid Infrastructure software is installed:

      [root@dm01 ]# dcli -g dbs_group -l root "Grid_home/bin/crsctl check crs"
      

      All Oracle Clusterware components must be offline. If you are performing a non-rolling update in a configuration running Oracle VM, then you must check the Oracle Clusterware state in all VM clusters.

    2. Shut down all cell services on all storage servers to be updated. This may be done by the root user on each cell by running cellcli -e 'alter cell shutdown services all' or by running the following dcli command to do all cells at the same time:

      [root@dm01 ]# dcli -g cell_group -l root "cellcli -e alter cell shutdown services all"
      
  5. Check if your system meets the following conditions:

    • Oracle Exadata Database Machine hardware uses Sun servers.

    • The installed Oracle Exadata Software release is earlier than release 11.2.2.2.0.

    • The installed Linux ofa package is earlier than 1.5.1-4.0.28.

    If the system meets all of the preceding conditions, then Exadata storage servers running Oracle Linux may encounter a file system corruption that results in the root file system mounted as read-only after reboot. Follow the instructions in My Oracle Support note 1589868.1 before updating the storage servers and database servers.

  6. Unzip the update. It will extract into the patch_release.date_code directory. Change to this patch directory.

  7. Download any patchmgr plug-ins attached to the My Oracle Support note for your target release and install them as documented in the My Oracle Support note. Oracle recommends reviewing the My Oracle Support notes, issues, and workarounds listed in the release note just before starting to actually apply the update.

  8. Clean up any previous update utility runs using the -cleanup flag.

    Note:

    The first time the storage servers are updated the -reset_force flag should be used before running cleanup.

    Example using -reset_force as the root user:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group –reset_force
    

    Example using -reset_force as a non-root user:

    [oracle@nonExadataHost ]# ./patchmgr -cells ~/cell_group -log_dir auto –reset_force
    

    Example using -cleanup as the root user:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -cleanup
    

    Example using -cleanup as a non-root user:

    [oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto –cleanup
    
  9. Run prerequisite check.

    Example running prerequisite check for a rolling update as the root user from an Exadata database server:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -patch_check_prereq -rolling -smtp_from "sender@example.com"  
    -smtp_to receiver@example.com
    

    Example running prerequisite check for a non-rolling update as a non-root user from a non-Exadata database server:

    [oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto -patch_check_prereq -smtp_from "sender@example.com"
    -smtp_to "receiver@example.com"
    

5.4.5 Running the Update for Exadata Storage Servers

After performing the prerequisite steps in Preparing Exadata Storage Servers for Update, you can perform the actual update step.

Note the following when applying the update to the Exadata storage servers:

  • Do not use the serial console or the ILOM web-based console to start the update utility.

    There is a known issue of a system halt on the serial console when a write is attempted to stderr or stdout. If an update is started from the serial console, then it may halt.

    You are using the serial console if the output from the following command is serial.

    [root@dm01 ]# ./echo $consoletype
    
  • When needed, use the ILOM web-based console to monitor the storage server during the update. You will need to use the ILOM web-based console in case troubleshooting is required.

  • To obtain ILOM and serial console access for the storage servers, use SSH to the ILOM host name or IP address as the root user. Do the following to start the serial console:

    start /SP/console
    

    To stop it press the Escape key (ESC) followed by (.

  • Start a new login session for each update or rollback procedure. Do not run a rollback procedure from the same login session where an update was applied. Do not run an update from a login session where a rollback procedure was run.

  • Do not interrupt the update process.

  • If you must use a storage server as the patchmgr utility launch system, then do not use /opt/oracle as the staging area for the update. This causes the update to fail and corrupt the storage server. Use the /tmp directory as the staging area, that is, unzip the files for the update in /tmp.

  • Storage servers automatically reboot, as needed, during the update process. Do not reboot or power cycle storage servers while applying updates.

  • Do not edit or open log files in writable mode. You may use any of the following to view a log: view, less, more, or tail. You may cause the update process to be interrupted if you edit the log files during the update.

  • At the end of the patchmgr session, the patchmgr.stdout log file is divided into individual storage server log files with names in the format of cell_name.log. In addition, the /var/log/cellos content from the inactive cell partition is copied to the /var/log/cellos/inactive_partition directory. To locate the inactive partition, use the following command:

    [root@dm01 ]# ./imageinfo -inactive -sys
    

Examples running update:

Running the update in a rolling fashion as root from an Exadata database server:

[root@dm01 ]# ./patchmgr -cells ~/cell_group -patch -rolling -smtp_from "sender@somedomain.com" -smtp_to receiver@somedomain.com

Running the update in a non-rolling fashion as a non-root user from a non-Exadata database server:

[oracle@nonExadataHost ]$ ./patchmgr -cells ~/cell_group -log_dir auto -patch -smtp_from "sender@somedomain.com" -smtp_to "receiver@somedomain.com"

After the update is done, clean up the storage servers using the -cleanup option to clean up all the temporary update or rollback files. This option cleans the stale update and rollback states as well as cleaning up to 1.5 GB of disk space on the storage server. Use this option before retrying a halted or failed run of the update utility. See step 8 for details.

5.4.6 Rolling Back Updates for Exadata Storage Servers

You can roll back updated Exadata storage servers only when they are updated successfully. This means the imageinfo command must return success for active image status. Storage servers with incomplete or failed updates cannot be rolled back. Rollbacks can be done in a rolling or non-rolling fashion.

  1. Check the version that the storage servers will be rolled back to and the flashCacheMode setting with the following commands:

    [root@dm01 ]# dcli -l root -g cell_group imageinfo -ver -inactive
    
    [root@dm01 ]# dcli -l root -g cell_group cellcli -e 'list cell attributes flashCacheMode
    

    Note:

    If you need to roll back storage servers to releases earlier than release 11.2.3.2.0 with writeback flash cache enabled, you need to convert the flash cache to writethrough flash cache before performing the rollback action. Disable the writeback flash cache using the script in My Oracle Support note 1500257.1. Storage servers being rolled back to release 11.2.3.2.0 or later retain the flash cache mode that is currently set.

  2. Check the prerequisites for rollback using the following command:

    [root@dm01 ]# ./patchmgr -cells cell_group -rollback_check_prereq [-rolling]
    
  3. Perform the rollback.

    Example of a non-rolling rollback run as root:

    [root@dm01 ]# ./patchmgr -cells ~/cell_group -rollback
    

    Example of a rolling rollback run as a non-root user:

    [oracle@nonExadataHost ]#./patchmgr -cells ~/cell_group -rollback -log_dir auto
    

    Note:

    Firmware updates are not rolled back when rolling back to a previous image. After rolling back, run the following command to apply older firmware versions when needed:

    /etc/init.d/lsidiag stop
    
    /etc/init.d/lsi_mrdsnmpd stop
    
    /opt/oracle.cellos/CheckHWnFWProfile -action updatefw -mode exact
    
  4. Clean up the Exadata storage servers using the -cleanup option to clean up all the temporary update or rollback files. This option cleans the stale update and rollback states as well as cleaning up to 1.5 GB of disk space on the Exadata storage servers. Use this option before retrying a halted or failed run of the patchmgr utility.

    [root@dm01 ]# ./patchmgr -cells cell_group -cleanup
    

5.4.7 Monitoring, Validating, and Troubleshooting Exadata Storage Server Updates

The recommended way to monitor Exadata storage servers being updated is through e-mail alerts. These alerts can be set up by specifying email addresses in the -smtp_from and -smtp_to flags in the update utility.

If needed for troubleshooting, you can monitor update activity using less -rf patchmgr.stdout from another terminal session or window to see raw log details from the update utility.

You can also monitor activity of the storage server by logging in to the serial console or web-based ILOM console of individual storage servers being updated 5 minutes after the update utility has started. Waiting 5 minutes allows the update utility time to reset the ILOM. Resetting the ILOM disconnects you from the ILOM web console and serial console. You can reconnect once the ILOM has been reset. By waiting 5 minutes, you avoid having to reconnect. You lose the connection during any ILOM update, and need to reconnect. The ILOM does not show any update actions. When needed it is helpful to monitor the activities of the normal cell boot, reboot, and other activities to ensure that the process is proceeding correctly.

Verify the update status after the patchmgr utility completes as follows:

Check image status and history using the imageinfo and imagehistory commands on each cell. A successful update to a system with high capacity or high performance drives shows output similar to the following.

Kernel version: 2.6.39-400.281.1.el6uek.x86_64 #1 SMP Fri Jun 17 20:10:16 PDT 2016 x86_64
Cell version: OSS_12.1.2.3.2_LINUX.X64_160721
Cell rpm version: cell-12.1.2.3.2_LINUX.X64_160721-1.x86_64

Active image version: 12.1.2.3.2.160721
Active image kernel version: 2.6.39-400.281.1.el6uek
Active image activated: 2016-07-21 13:04:34 -0500
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

Cell boot usb partition: /dev/sdac1
Cell boot usb version: 12.1.2.3.2.160721

Inactive image version: 12.1.2.3.1.160411
Inactive image activated: 2016-04-11 19:58:28 -0700
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8

Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
Inactive kernel version for the rollback: 2.6.39-400.128.21.el5uek
Rollback to the inactive partitions: Possible

A successful update to a system with extreme flash drives shows a similar output but has a different path for USB:

Kernel version: 2.6.39-400.281.1.el6uek.x86_64 #1 SMP Fri Jun 17 20:10:16 PDT 2016 x86_64
Cell version: OSS_12.1.2.3.2_LINUX.X64_160721
Cell rpm version: cell-12.1.2.3.2_LINUX.X64_160721-1.x86_64

Active image version: 12.1.2.3.2.160721
Active image kernel version: 2.6.39-400.281.1.el6uek
Active image activated: 2016-07-21 13:04:34 -0500
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

Cell boot usb partition: /dev/sda1
Cell boot usb version: 12.1.2.3.2.160721

Inactive image version: 12.1.2.3.1.160411
Inactive image activated: 2016-04-11 19:58:28 -0700
Inactive image status: success
Inactive system partition on device: /dev/md6
Inactive software partition on device: /dev/md8

Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
Inactive kernel version for the rollback: 2.6.39-400.128.21.el5uek
Rollback to the inactive partitions: Possible

5.5 Updating Exadata InfiniBand Switch Firmware

You use the same update utility that is shipped with the storage server update to update and downgrade the InfiniBand switches. The minimum switch firmware release that can use the update utility is release 1.3.3-2. Switch firmware is always updated in a rolling manner.

Note:

The patchmgr used for updating InfiniBand switch firmware is not the same as that used for the Exadata database server software update.

5.5.1 Preparing for InfiniBand Switch Firmware Updates

If a spine switch is present in the rack, it needs to be updated first. If a spine switch is not in the rack, then update the switch running the subnet manager first. If the subnet manager is not running on the switches, then perform the update in any order.

To update the InfiniBand switches, the switch firmware must be at release 1.3.3-2 or later. If the switch firmware is at an earlier release, then update the firmware to release 1.3.3-2 using the instructions in My Oracle Support note 888828.1. Use the version command to determine the version of the software the InfiniBand switch is running:

  1. Log in as the root user to an Exadata database server that has root user SSH access to the switches. The database server must be on the same InfiniBand network as the switches.

  2. Determine the version of your switches:

    [root@dbm01-ibs0 ~]# version
    
    SUN DCS 36p version: 2.1.8-1
    Build time: Sep 18 2015 10:26:47
    SP board info:
    Manufacturing Date: 2015.07.01
    Serial Number: "NCDLD0049"
    Hardware Revision: 0x0200
    Firmware Revision: 0x0000
    BIOS version: SUN0R100
    BIOS date: 06/22/2010
    
  3. Download the appropriate patch file to the database server. Refer to My Oracle Support note 888828.1 for patch information.

  4. Uncompress the update. The files are uncompressed to the <patch_release.date> directory.

  5. Create a file listing all the InfiniBand switches that need to be updated, with one switch per line. Use the command ibswitches to identify the switches in your rack. Note that switches from other non-Exadata engineered systems might be visible on the same fabric but should probably not be updated at this time. The following is an example of the file constructed after running ibswitches:

    [root@dm01 ]# cat ibswitches.lst
    myibswitch-01
    myibswitch-02
    

    Note:

    If no filename is provided, the command will be executed on all InfiniBand switches discovered from this host by running ibswitches command.

  6. Change to the <patch_release.date> directory.

  7. Run the prerequisite check:

    [root@dm01 ]# ./patchmgr -ibswitches ibswitches.lst -upgrade -ibswitch_precheck [-force] [-unkey]
    

    Notes:

    • The -unkey option removes passwordless SSH access to the InfiniBand switches before exiting.

    • The -force option overrides failures in the InfiniBand topology and connectivity from the servers to the switches. This does not affect the upgrade of the switch.

    If the output from the command shows overall status is SUCCESS, then proceed with the upgrade. If the output from the command shows overall status is FAIL, then review the error summary in the output to determine which checks failed, and then correct the errors. After correcting all the errors, rerun the prerequisite checks until it is successful.

5.5.2 Updating InfiniBand Switch Firmware Software

Update the switches using the following command:

[root@dm01 ]# ./patchmgr -ibswitches ibswitches.lst -upgrade [-force] [-unkey]

5.5.3 Downgrading InfiniBand Switch Firmware Software

Downgrading firmware means reapplying the older firmware update, which is shipped with the Exadata storage server update. This means the storage server update defines what release you can downgrade to. This may be different for each release and may not be the firmware you were on before the update. For more information on the older firmware shipped with the release you are updating to, see the patch README.

Example command for downgrading the InfiniBand switch firmware:

[root@dm01 ]# ./patchmgr -ibswitches ibswitches.lst -downgrade -ibswitch_precheck [-force] [-unkey]

[root@dm01 ]# ./patchmgr -ibswitches ibswitches.lst -downgrade [-force] [-unkey]

5.6 Setting up SSH Equivalence

Before updating software on your Exadata Database Machine, you must configure SSH equivalence.

You can run the Exadata update utilities for Exadata database server and storage server as either root or as a non-root user from any server running Oracle Linux or Oracle Solaris. The utility can perform precheck, update, and rollback actions on any Exadata server as long as SSH equivalence is set up for the root user for the target Exadata server.

  1. Prepare a file named cell_group or dbs_group that has one storage server or database server host name or IP address per line for each storage server or database server you want to update.
  2. Check for existing SSH equivalence.
    The following command should require no password prompts and no interaction. It should return the list of host names in the cell_group file.
    [oracle@nonExadataHost ]# ./dcli -g cell_group -l root 'hostname -i'
    
  3. Set up SSH equivalence if not already done so from the launch server.
    Do not do this step if you already have root SSH equivalence.
    Generate SSH keys using the following command:
    [oracle@nonExadataHost ]# ssh-keygen -t dsa
    

    Accept the defaults so that the SSH keys are created for the root user.

  4. Push the SSH keys to set up SSH equivalence.
    Enter the root password when prompted.
    [oracle@nonExadataHost ]# dcli -g cell_group -l root –k
    

Note:

Customers in secure environments may have chosen to disable SSH access to Exadata storage servers. During normal operations, Exadata storage server does not require SSH access. However, administrative utilities such as the update utility require SSH access. See the subsection "Unlocking a Cell Temporarily" in the section "Disabling SSH on Storage Servers" in the Oracle Exadata Storage Server Software User's Guide for information on unlocking storage servers.

5.7 Troubleshooting a Failing Prerequisite Check Due to Dependency Problems on Exadata Database Server

When you run a prerequisite check or an update on Exadata database servers that have custom rpms installed, conflicts may cause the prerequisite check or update to fail.

When doing a prerequisite check or an update, the update utility makes two dependency checks: checks against “minimum dependencies” and checks against “exact dependencies”.

To triage a dependency failure, look in the log file. This can be either dbnodeupdate.log on the target Exadata database server or <hostname>_dbnodeupdate.log in the log directory from where the update utility (patchmgr) was launched.

To locate the start of each run in the log file, search for a line that starts with “zzz”. For example:

zzz - /u01/patches/YUM/dbnodeupdate.sh called with arguments -u -l
/u01/patches/YUM/p23564643_121232_Linux-x86-64.zip -v -N at  2016-08-23 23:31:54

The date stamp should match the time of the run you are researching. Each run is identified with a unique runid which can also be found at the start of each run in the same log file:

[1472009516][2016-08-23 23:31:59 -0400][INFO][/u01/patches/YUM/dbnodeupdate.sh][InitLogfile][]  # dbnodeupdate.sh script rel.
 : 5.160809 started at  (runid :230816233155)

You can check for custom rpms, if any, by looking at the diag file of the specific run. The diag file is identified by the runid and can be found in /var/log/cellos on the target Exadata database server or in the directory driving the update utility. The filename for this example would be dbnodeupdate.230816233155.diag. In that file, look for the section with heading RunDetectCustomRpmsSh and rpm -qa --qf "%{n}-%{v}-%{r}.%{arch}\n. You can find details on the (additional) packages installed (if any).

In the log file, search down from the start (“zzz”) for “Exact dependencies” (case sensitive) for the results of the “minimum” and “exact” dependency checks. In this case it says:

Exact dependencies     : Will fail on a next update
Minimum dependencies   : Will fail on a next update

As long as the minimum dependencies check passes, updates or prerequisite checks won’t fail. When both minimum and exact dependencies fail, you need to find out what caused the error.

In order to find the dependency that is causing the error, search for “[ExecUpgrade][] Performing yum package dependency” . This is where the YUM run (typically a dry-run first) is executed. When there is a dependency problem, you should see a YUM message starting with “Error:”. For example:

--> Finished Dependency Resolution
Error: Package: krb5-devel-1.10.3-33.el6.x86_64 (@ol6_latest)
Requires: krb5-libs = 1.10.3-33.el6
Removing: krb5-libs-1.10.3-33.el6.x86_64 (installed)
        krb5-libs = 1.10.3-33.el6
Updated By: krb5-libs-1.10.3-42z1.el6_7.x86_64 (exadata_generated_160616114412)
          krb5-libs = 1.10.3-42z1.el6_7
You could try using --skip-broken to work around the problem

In the example error message above, the krb5-devel-1.10.3-33.el6.x86_64 (krb5-devel) package is installed from a non-Exadata channel (ol6_latest). This krb5-devel package depends on krb5-libs. However in this Exadata update the krb5-libs package is not available. YUM fails the dependency check because updating krb5-libs is not possible without also updating krb5-devel. Because a new version of krb5-devel is not included in the Exadata update the package should be either “pre-updated” or removed. Pre-updating means updating the individual package manually before running the Exadata update. This can be done using the command rpm –Uvh <package-name>.

Removing the custom package is recommended and should be done using the following rpm command:

[root@dm01 ]# rpm –e krb5-devel

After removing the rpm that is causing the error, restart the update or prerequisite check.

5.8 Troubleshooting a Multilib Problem on Exadata Database Server

If you have custom packages with different architectures installed on database servers, you may see a similar problem as that described in Troubleshooting a Failing Prerequisite Check Due to Dependency Problems on Exadata Database Server. This typically happens when i686 packages are installed on an Exadata database server.

Note:

While non-64-bit (x86_64) packages are supported, it is recommended to stay away from 32-bit software. When you need specific functionality from a third party, it is recommended to ask for a 64-bit version.

Typically Exadata-branded x86_64 bit rpms are updated in an Exadata update. However, when you have installed similar packages of non-x86_64 bit architecture, the update utility cannot update the 64-bit packages. You would see the following error in the log file:

--> Finished Dependency Resolution
    Error:  Multilib version problems found. This often means that the root
           cause is something else and multilib version checking is just
           pointing out that there is a problem. Eg.:
           
             1. You have an upgrade for libuuid which is missing some
                dependency that another package requires. Yum is trying to
                solve this by installing an older version of libuuid of the
                different architecture. If you exclude the bad architecture
                yum will tell you what the root cause is (which package
                requires what). You can try redoing the upgrade with
                --exclude libuuid.otherarch ... this should give you an error
                message showing the root cause of the problem.
           
             2. You have multiple architectures of libuuid installed, but
                yum can only see an upgrade for one of those arcitectures.
                If you don't want/need both architectures anymore then you
                can remove the one with the missing update and everything
                will work.
           
             3. You have duplicate versions of libuuid installed already.
                You can use "yum check" to get yum show these errors.
           
           ...you can also use --setopt=protected_multilib=false to remove
           this checking, however this is almost never the correct thing to
           do as something else is very likely to go wrong (often causing
           much more problems).
           
           Protected multilib versions: libuuid-2.17.2-12.24.0.1.el6.x86_64 != libuuid-2.17.2-12.18.0.1.el6.i686

The solution for multilib problems is to remove the i686 or i386 package by running “rpm –e <package_name.i686>”.