2 Introduction to Oracle Clusterware and Oracle RAC Administration and Deployment

This chapter provides an overview of administering Oracle Clusterware and Oracle Real Application Clusters (Oracle RAC) environments. This chapter includes the following topics:

Oracle Real Application Clusters Documentation Overview

This section describes the Oracle RAC documentation set. This book, the Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide, provides Oracle RAC-specific administration and application deployment information.

This book describes how to administer the Oracle Clusterware components such as the voting disks and the Oracle Cluster Registry (OCR). This book also explains how to administer storage and how to use Oracle RAC scalability features to add and delete instances and nodes. This book also discusses how to use Recovery Manager (RMAN), and how to perform backup and recovery in Oracle RAC.

The Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide describes Oracle RAC deployment topics by explaining how to deploy automatic workload management and take advantage of high availability by using services. This book describes how the Automatic Workload Repository (AWR) tracks and reports service levels and how you can use service level thresholds and alerts to improve high availability in your Oracle RAC environment. This book also explains how to make your applications highly available by using Oracle Clusterware.

The Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide explains how to monitor and tune performance in Oracle RAC environments by using Oracle Enterprise Manager and by using information in AWR and Oracle performance views. This book also highlights application-specific deployment techniques for online transaction processing and data warehousing environments.

The appendices in this book describe:

  • Troubleshooting tips such as debugging and log file use and general information about installing and using the Cluster Verification Utility (CVU)

  • The Oracle Clusterware command-line reference

  • The Oracle Cluster Registry command syntax

  • How to use the Server Control (SRVCTL) utility to start and stop the database and instances, to manage configuration information, and to delete or move instances and services

  • Oracle RAC tools error messages

In addition to this book, refer to the platform-specific Oracle Real Application Clusters installation and configuration guide on your product installation media as described under the following heading.

Platform-Specific Oracle Real Application Clusters Installation and Configuration Guides

Each platform-specific Oracle Database 10g installation media contains a copy of an Oracle Real Application Clusters platform-specific installation and configuration guide in HTML and PDF formats. These Oracle RAC installation books contain the pre-installation, installation, and post-installation information for the various UNIX- and Windows-based platforms on which Oracle RAC operates.


Additional information for this release may be available in the Oracle Database 10g README or Release Notes. If these documents are available for this release, then they are on your Oracle product installation media.

Introduction to Administering Oracle Real Application Clusters

Install Oracle Clusterware and your Oracle Database 10g software with the Oracle Universal Installer (OUI) and create your database with the Database Configuration Assistant (DBCA). This ensures that your Oracle RAC environment has the optimal network configuration, database structure, and parameter settings for the environment that you selected. As a DBA, after installation your tasks are to administer your Oracle RAC environment at three levels:

  • Instance Administration

  • Database Administration

  • Cluster Administration

Administering Oracle Real Application Clusters

Use the following tools to perform administrative tasks in Oracle RAC:

  • Cluster Verification Utility (CVU): Install and use CVU before you install Oracle RAC to ensure that your configuration meets the minimum Oracle RAC installation requirements. Also use the CVU for on-going administrative tasks, such as node addition and node deletion.

    See Also:

    Your platform-specific Oracle Clusterware and Oracle RAC installation guide for information about how to manually install CVU
  • Enterprise Manager: Oracle recommends that you use Enterprise Manager to perform administrative tasks whenever feasible

  • Task-specific GUIs such as the Database Configuration Assistant (DBCA) and the Virtual Internet Protocol Configuration Assistant (VIPCA)

  • Command-line tools such as SQL*Plus, Server Control (SRVCTL), the Oracle Clusterware command-line interface, and the Oracle Interface Configuration tool (OIFCFG)

Voting Disk and Oracle Cluster Registry Device Administration

You can use multiple disks for both the voting disk and the Oracle Cluster Registry (OCR) disk to prevent them from becoming single points of failure. Administer these components as described in Chapter 3, " Administering Oracle Clusterware Components".

Database Instance Management and Database Administration in Oracle RAC

Use Enterprise Manager, SQL*Plus, or SRVCTL to administer database instances and Oracle RAC databases as described in Chapter 5, " Administering Database Instances and Cluster Databases".

Storage Management in Oracle Real Application Clusters

When you create your database, you can create Automatic Storage Management (ASM) disk groups and configure mirroring for ASM disk groups using DBCA. After your Oracle RAC database is operational, you can administer ASM disk groups with Enterprise Manager or the SRVCTL utility as described in Chapter 4, " Administering Storage".

You can configure ASM in a separate standalone ASM-only home. This enables instances for single-instance Oracle and Oracle RAC databases to share a single ASM instance on a node. You also have the option to upgrade ASM independently of your database upgrades.

The Oracle tools that you use to manage ASM, including DBCA, Database Upgrade Assistant (DBUA), Enterprise Manager, and the silent mode install and upgrade commands, include options to manage ASM instances and disk groups. For example, you can run DBCA to create a new ASM instance or ASM disk group independently of creating a database.

When you perform ASM-related options during installs, upgrades, or other operations, the tool you are using may automatically extend ASM to other nodes in your cluster. This can include installing ASM software into the same home as on the current node and starting the ASM instance. For example, if you use DBCA to create a database using a new Oracle home, then DBCA will attempt to extend ASM to the new Oracle home on all of the nodes you select.

In some cases, your current configuration may not be compatible with an ASM activity that you are trying to perform, either explicitly or with an automatic ASM extension to other nodes. Should you be using DBCA, as described earlier, to build a database using a new Oracle home, and if the ASM version is from an earlier release of the Oracle software but does not exist on all of the nodes you selected for the database, then ASM cannot be extended. Instead, the DBCA session displays an error, prompting you either to run the add node script or to upgrade ASM using the Database Upgrade Assistant (DBUA).

Oracle Clusterware for Oracle Real Application Clusters

When you create an Oracle RAC database, you can also use DBCA to create services and assign them to instances. After your Oracle RAC database is operational, you can use Enterprise Manager, DBCA, or SRVCTL to create and administer services and high availability components as described in Chapter 6, "Introduction to Workload Management".

Other high availability components include node resources such as the Virtual Internet Protocol (VIP) address, the Global Services Daemon, the Oracle Notification Service, and the Oracle Net Listeners. These resources are automatically started when Oracle Clusterware starts the node and then automatically restarts them if they stop. The application level resources are the instances and the Oracle Clusterware background processes that run on each instance.

You can use the VIPCA to administer VIP addresses and you can use SRVCTL to administer other node resources. The information that describes the configuration of these components is stored in the Oracle Cluster Registry (OCR) that you can administer as described in Chapter 3, " Administering Oracle Clusterware Components".

Additional Oracle Real Application Clusters Administrative Topics

This book contains the following additional Oracle RAC administrative topics:

Overview of Using Enterprise Manager with Oracle Real Application Clusters

Enterprise Manager is a Web-based tool with Oracle RAC-specific administration and performance-related features. If you create your Oracle RAC database with the Database Configuration Assistant (DBCA), then the Enterprise Manager Database Control tool is automatically configured for your Oracle RAC environment. This means that all instances that were part of your installation have an Enterprise Manager Agent running on them. Enterprise Manager Database Control enables you to manage a single Oracle RAC database with its instance targets, Oracle Net Services Listener targets, host targets, and a cluster target.

You can configure Enterprise Manager Grid Control on additional hosts, which were not part of your initial Oracle RAC installation, either inside or outside your cluster environment. Enterprise Manager Grid Control enables you to manage multiple cluster databases, cluster database instances, and the hosts on which cluster database components operate.

Grid Control enables you to monitor and administer your entire computing environment from one network location. Use Grid Control to manage all of your enterprise services, including hosts, databases, Listeners, application servers, HTTP Servers, and Web applications, as one cohesive unit. Enterprise Manager Grid Control only requires one Agent on one host in your cluster environment to perform cluster database and instance discovery. Install Enterprise Manager Grid Control from a your Oracle Database 10g installation media.

See Also:

Oracle Enterprise Manager Concepts for more information about using Enterprise Manager

You can also use both Enterprise Manager Database Control and the Enterprise Manager Grid Control to:

  • Administer database services: Create, configure, start, stop, relocate, obtain status, and so on

  • Create and assign resource plans: Assign resource plans to cluster database instances

  • Administer storage: Assign undo tablespaces and re-assign them from one instance to another, administer redo log assignments among cluster database instances, and switch archive log modes

  • Administer Automatic Storage Management: Administer ASM instances and ASM disk groups if the database uses ASM

  • Perform general database activities: Start up and shut down Oracle RAC databases and instances, perform backup and recovery operations, edit server parameter file (spfile) settings for instances or for entire cluster databases, and so on

  • Display host configurations: Memory, CPU, device I/O, network interfaces, the operating system and installed patches

The following sections introduce Oracle Real Application Clusters (Oracle RAC) application deployment and performance by explaining the main points that you need to remember when you deploy applications on Oracle RAC:

Overview of Deploying Applications on Oracle Real Application Clusters

To optimally deploy applications on Oracle RAC, remember the following points:

  • Storage for Oracle RAC datafiles must be shared storage: When you install Oracle RAC, use Automatic Storage Management (ASM) or a Cluster File System for datafile storage when available.

  • Create your database with the Database Configuration Assistant (DBCA).

  • Define services for your environment with DBCA or Enterprise Manager and administer them with Oracle Enterprise Manager or the Server Control (SRVCTL) Utility.

  • Use the Server Parameter File (SPFILE): The SPFILE should be located on either a cluster file system file or on a shared raw device.

  • Use Automatic Undo Management.

  • Use Automatic Segment-Space Management.

  • Use the Automatic Database Diagnostic Monitor (ADDM) to reduce the effort required to tune Oracle systems.

See Also:

Your platform-specific Oracle Real Application Clusters installation guide for more information about configuring these features for Oracle Database 10g Oracle Real Application Clusters

Code Changes are Not Required for Applications

Applications that perform well on single-instance Oracle databases do not require code changes to perform well on Oracle RAC databases. The same compatibility guidelines that apply to Oracle single-instance databases also apply to Oracle RAC. For example, if an application scales well in an SMP environment, then the application will scale well with Oracle RAC.

The primary feature of Oracle RAC that enables applications to scale across multiple instances is Cache Fusion. Cache Fusion enable each instance to locate the most useful version of a block of data for its needs, whether that block resides on disk or in the memory of another instance. The cluster interconnect provides a fast pathway for Cache Fusion to transfer required blocks of data between instances when necessary.

Implementing Oracle Features with Oracle Real Application Clusters

The Oracle features described in this section enhance the performance and simplify the administration of your Oracle RAC environment. The features discussed in this section are:

Automatic Storage Management

Automatic Storage Management (ASM) simplifies database administration by eliminating the need for you to manage Oracle database files. Instead, ASM enables you to create disk groups that comprise disks and the files that reside on them.

Cluster File Systems in Oracle Real Application Clusters

Depending on your hardware platform, you can store Oracle homes and Oracle datafiles on a cluster file system. Cluster file systems are simpler to configure and manage than raw device storage. Cluster file systems also offer scalable, low latency, highly resilient storage that significantly reduces costs.

Storage Management Features and Oracle Real Application Clusters

The advanced storage features of Oracle Automatic Storage Management greatly enhance manageability for Oracle RAC just as with single instance Oracle. Other storage features include Oracle-managed files, automatic segment-space management, and automatic undo management. Refer to the Oracle database documentation for more information about using storage management features.

Services in Oracle Database 10g

With Oracle Database 10g, you can define application workloads as services so that you can individually manage and control them. You can create a service for each application or for major components within a complex application. Once created, you can define where and when a service runs. Your entire database workload can be separated into a few services, each of which can be managed independently, reducing the need to manage individual users or sessions for many tasks.

In an Oracle RAC database, you can use services to maximize the use of your cluster's processing resources. You can assign each service to one or more instances for normal startup (preferred), depending on its processing requirements. Additionally, you can define one or more alternate (available) instances that a service can use if one of the service's assigned (preferred) instances becomes unavailable.

In both cluster and non-cluster environments, you can track performance metrics by service using the Automatic Workload Repository (AWR). You can set thresholds on performance metrics so that your system automatically generates alerts if the thresholds are exceeded. You can also map services to Resource Manager consumer groups to provide more fine-grained resource allocation controls such as placing limits on CPU consumption. Other Oracle tools and facilities such as Oracle Scheduler, Parallel Query, and Oracle Streams Advanced Queuing can also use services to manage their workloads.

The settings for Resource Manager are instance specific, that is, you can have different Resource Manager settings on different instances. Therefore, ensure that your Resource Manager settings are appropriate for the instances on which you are using it. The Automatic Workload Management features of Oracle Database 10g are integrated with the resource manager. The only difference in the way you use Resource Manager with Oracle RAC is that Resource Manager has instance specific settings.


An example that shows how to manage a complex environment with database services is available on the Oracle Technology Network at the URL http://www.oracle.com/index.html. The example includes entries from the related files, such as listener, database, and Net Services parameter files, as well as the commands to create and use the services for application management.

Runtime Connection Load Balancing is used when selecting connections from a connection pool. For connection pools that support services at one instance only, the first available connection in the pool is used. When connection pools support services that span multiple instances using a policy such as service metrics, Runtime Connection Load Balancing distributes work requests across instances that are adequately serving a service. This avoids sending work to slow, hung, failed or restricted instances.

For connection load balancing, the method for balancing connections across a service uses four metrics: session count by instance, run queue length of the node, goodness by service, and weighted session count by service. The metrics used depend on what goals have been defined for the service and how Oracle Net Services has been configured.Fast Connection Failover is used to prevent connections being directed to failed nodes or instances. Integrated connection pools will clean up connections when a failure occurs or add additional connections when new instances become available. This allows the application to be immediately aware of cluster configuration changes and react to them without any programming or configuration changes at the application tier.

See Also:

Chapter 6, "Introduction to Workload Management" for more detailed information about these features

The Oracle Clusterware and High Availability in Oracle Real Application Clusters

This section introduces the following high availability features:

High Availability with Oracle Clusterware

Oracle Real Application Clusters 10g introduces a complete, integrated clusterware management solution on all Oracle Database 10g platforms. This clusterware functionality provides all of the features required to manage your cluster database including node membership, group services, global resource management, and high availability functions. You install Oracle Clusterware as a separate install that you can complete independently or as a pre-requisite to the Oracle RAC installation process. Oracle database features such as Oracle 10g services use the underlying Oracle Clusterware mechanisms to provide their capabilities. Oracle also continues to support select third-party clusterware products on specified platforms.

See Also:

Chapter 14, "Making Applications Highly Available Using Oracle Clusterware" for more information about Oracle Clusterware

The Oracle Clusterware and High Availability

High availability configurations have redundant hardware and software that maintain operations by avoiding single points-of-failure. During outages, Oracle Clusterware relocates the processing performed by the inoperative component to a backup component. The Oracle recovery processes quickly re-master resources, recover partial or failed transactions, and rapidly restore the system.

You can combine many Oracle products and features to create highly reliable computing environments. Doing this requires capacity and redundancy planning. In addition, consider your overall system costs and your return on investment. There are also other practical considerations such as selecting the appropriate hardware and deciding whether to use idle machines that are part of your high availability configuration.

Additional Oracle High Availability Features and Solutions

This section describes the following additional high availability solutions:

Connection Load Balancing with Oracle Real Application Clusters

The connection load balancing feature of Oracle Net Services automatically distributes connections among active instances. Connection load balancing does this based on the workload of each node and instance in a cluster.

Recovery Manager (RMAN) in Oracle Real Application Clusters

Recovery Manager (RMAN) is an Oracle tool that you can use to backup, copy, restore, and recover datafiles, control files, SPFILEs, and archived redo logs. You can invoke RMAN as a command-line utility or use Oracle Enterprise Manager.

A best practice is to configure RMAN so that all instances can access all of the archive log threads throughout your cluster database. In the event of media recovery, the recovering instance requires access to all of the archived redo log threads. You can simplify media recovery administration by ensuring that a recovering instance can access a local copy of the archive log threads from all of the instances in your cluster database.

See Also:

Chapter 7, "Configuring Recovery Manager and Archiving" for details about configuring RMAN for use with Oracle RAC and Oracle Database Backup and Recovery Advanced User's Guide for detailed information about RMAN

Data Guard

Oracle Data Guard works with standby databases to protect your data against errors, failures, and corruptions that might otherwise destroy your database. Data Guard protects critical data by automating the creation, management, and monitoring aspects of standby database environments. Oracle Data Guard automates the otherwise manual process of maintaining a transactional consistent copy of an Oracle database to recover from the loss of or damage to the production database.

Primary/Secondary Instance Configurations in Earlier Releases

If you are upgrading from a pre-Oracle 10g Primary/Secondary configuration, then the Database Upgrade Assistant (DBUA) creates a service on your database with one preferred instance and one available instance.