This chapter includes the following topics:
A cluster comprises multiple interconnected computers or servers that appear as if they are one server to end users and applications. Oracle RAC enables you to cluster Oracle databases. Oracle RAC uses Oracle Clusterware for the infrastructure to bind multiple servers so they operate as a single system.
Oracle Clusterware is a portable cluster management solution that is integrated with Oracle Database. Oracle Clusterware is also a required component for using Oracle RAC. In addition, Oracle Clusterware enables both single-instance Oracle databases and Oracle RAC databases to use the Oracle high-availability infrastructure. Oracle Clusterware enables you to create a clustered pool of storage to be used by any combination of single-instance and Oracle RAC databases.
Oracle Clusterware is the only clusterware that you need for most platforms on which Oracle RAC operates. You can also use clusterware from other vendors if the clusterware is certified for Oracle RAC.
See Also:Oracle Clusterware Administration and Deployment Guide and the Oracle Clusterware install guides for more details
Single-instance Oracle databases have a one-to-one relationship between the Oracle database and the instance. Oracle RAC environments, however, have a one-to-many relationship between the database and instances. An Oracle RAC database can have up to 100 instances,Foot 1 all of which access one database. All database instances must use the same interconnect, which can also be used by Oracle Clusterware.
Oracle RAC databases differ architecturally from single-instance Oracle databases in that each Oracle RAC database instance also has:
At least one additional thread of redo for each instance
An instance-specific undo tablespace
The combined processing power of the multiple servers can provide greater throughput and Oracle RAC scalability than is available from a single server.
Figure 1-1 shows how Oracle RAC is the Oracle Database option that provides a single system image for multiple servers to access one Oracle database. In Oracle RAC, each Oracle instance usually runs on a separate server.
Traditionally, an Oracle RAC environment is located in one data center. However, you can configure Oracle RAC on an extended distance cluster, which is an architecture that provides extremely fast recovery from a site failure and allows for all nodes, at all sites, to actively process transactions as part of single database cluster. In an extended distance cluster, the nodes in the cluster are located in two buildings that are separated by greater distances (anywhere from across the street, to across a campus, or across a city). For availability reasons, the data needs to be located at both sites, thus requiring the need to implement disk mirroring technology for storage.
If you choose to implement this architecture, you must assess whether this architecture is a good solution for your business, especially with regard to distance, latency, and the degree of protection it provides. Oracle RAC on extended distance clusters provides higher availability than is possible with a local Oracle RAC configurations, but an extended distance cluster may not fulfill all of the disaster-recovery requirements of your organization. A feasible separation provides great protection for some disasters (for example, local power outage, airplane crash, server room flooding) but it cannot provide protection against all types of outages. For comprehensive protection against disasters—including protection against corruptions and regional disasters—Oracle recommends the use of Oracle Data Guard with Oracle RAC, as described in the Oracle Database High Availability Overview and on the Maximum Availability Architecture (MAA) Web site at
Oracle RAC is a unique technology that provides high availability and scalability for all application types. The Oracle RAC infrastructure is also a key component for implementing the Oracle enterprise grid computing architecture. Having multiple instances access a single database prevents the server from being a single point of failure. Oracle RAC enables you to combine smaller commodity servers into a cluster to create scalable environments that support mission critical business applications. Applications that you deploy on Oracle RAC databases can operate without code changes.
Oracle Clusterware provides a complete, integrated clusterware management solution on all Oracle Database platforms. This clusterware functionality provides all of the features required to manage your cluster database including node membership, group services, global resource management, and high availability functions. You can install Oracle Clusterware independently or as a prerequisite to the Oracle RAC installation process. Oracle Database features such as services use the underlying Oracle Clusterware mechanisms to provide their capabilities. Oracle Database also continues to support select third-party clusterware products on specified platforms.
Oracle Clusterware is designed for, and tightly integrated with, Oracle RAC. When you create an Oracle RAC database using any of the management tools, the database is registered with and managed by Oracle Clusterware, along with the other Oracle Database processes such as Virtual Internet Protocol (VIP) address, Global Services Daemon (GSD), the Oracle Notification Service (ONS), and the Oracle Net listeners. These resources are automatically started when Oracle Clusterware starts the node and automatically restarted if they fail. The Oracle Clusterware daemons run on each node.
You can use Oracle Clusterware to manage high-availability operations in a cluster. Anything that Oracle Clusterware manages is known as a CRS resource, which could be a database, an instance, a service, a listener, a VIP address, an application process, and so on. Oracle Clusterware manages CRS resources based on the resource's configuration information that is stored in the Oracle Cluster Registry (OCR). Oracle Clusterware stores the information that describes the configuration of these components in the OCR that you can administer as described in the Oracle Clusterware Administration and Deployment Guide.
Oracle RAC requires Oracle Clusterware to provide the cluster infrastructure that allows multiple servers to work together. Oracle Clusterware provides group membership, communications infrastructure, event monitoring, and a high availability framework.
The following sections describe these concepts in more detail:
An Oracle RAC database is a shared everything database. All data files, control files, SPFILEs,Foot 2 and redo log files in Oracle RAC environments must reside on cluster-aware shared disks so that all of the cluster database instances can access these storage components. All database instances must use the same interconnect, which can also be used by Oracle Clusterware. Because Oracle RAC databases use a shared everything architecture, Oracle RAC requires cluster-aware storage for all database files.
In Oracle RAC, the Oracle Database software manages disk access and is certified for use on a variety of storage architectures. It is your choice as to how to configure your disk, but you must use a supported cluster-aware storage solution. Oracle Database provides the following file storage options for Oracle RAC:
This is the recommended solution to manage your disk.
OCFS2 is available for Linux and OCFS is available for Windows platforms. However you may optionally use a third-party cluster file system or cluster-aware volume manager that is certified for Oracle RAC.
All nodes in an Oracle RAC environment must connect to a Local Area Network (LAN) to enable users and applications to access the database. Applications should use the Oracle Database services feature to connect to an Oracle database. Services enable you to define rules and characteristics to control how users and applications connect to database instances. These characteristics include a unique name, workload balancing and failover options, and high availability characteristics. Oracle Net Services enable the load balancing of application connections across all of the instances in an Oracle RAC database.
Users can access an Oracle RAC database using a client/server configuration or through one or more middle tiers, with or without connection pooling. Users can be database administrators, developers, application users, power users, such as data miners who create their own searches, and so on.
Most public networks typically use TCP/IP, but you can use any supported hardware and software combination. Oracle RAC database instances can be accessed through a database's default IP address and through VIP addresses.
The interconnect network is a private network that connects all of the servers in the cluster. The interconnect network uses a switch (or multiple switches) that only the nodes in the cluster can access. Configure User Datagram Protocol (UDP) on a Gigabit Ethernet for your cluster interconnect. On Linux and Unix systems, you can configure Oracle Clusterware to use either the UDP or Reliable Data Socket (RDS) protocols. Windows clusters use the TCP protocol. Crossover cables are not supported for use with Oracle Clusterware interconnects.
Note:Do not use the interconnect for the private network for user communication, because Cache Fusion uses the private interconnect for interinstance communication.
In addition to the node's host name and IP address, you must also assign a virtual host name and an IP address to each node. You should use the virtual host name or VIP address to connect to the database instance. For example, you might enter the virtual host name
CRM in the address list of the
A virtual IP address is an alternate public address that client connections use instead of the standard public IP address. To configure VIP addresses, you need to reserve a spare IP address for each node, and the IP addresses must use the same subnet as the public network.
If a node fails, then the node's VIP address fails over to another node on which the VIP address can accept TCP connections but it cannot accept Oracle connections. Generally, VIP addresses fail over when:
The node on which a VIP address runs fails
All interfaces for the VIP address fail
All interfaces for the VIP address are disconnected from the network
Clients that attempt to connect to the VIP address receive a rapid
connection refused error instead of waiting for TCP connect timeout messages. You configure VIP addresses in the address list for your database connection definition to enable connectivity.
If you use Network Attached Storage (NAS), then you are required to configure a second private network. Access to this network is typically controlled by the vendor's software. The private network uses static IP addresses.
Oracle RAC databases have two or more database instances that each contain memory structures and background processes. An Oracle RAC database has the same processes and memory structures as a single-instance Oracle database as well as additional process and memory structures that are specific to Oracle RAC. Any one instance's database view is nearly identical to any other instance's view in the same Oracle RAC database; the view is a single system image of the environment.
Each instance has a buffer cache in its System Global Area (SGA). Using Cache Fusion, Oracle RAC environments logically combine each instance's buffer cache to enable the instances to process data as if the data resided on a logically combined, single cache.
Note:The SGA size requirements for Oracle RAC are greater than the SGA requirements for single-instance Oracle databases due to Cache Fusion.
To ensure that each Oracle RAC database instance obtains the block that it needs to satisfy a query or transaction as fast as possible, and to ensure data integrity, Oracle RAC instances use two processes, the Global Cache Service (GCS) and the Global Enqueue Service (GES). The GCS and GES maintain records of the statuses of each data file and each cached block using a Global Resource Directory (GRD). The GRD contents are distributed across all of the active instances, which increases the size of the SGA for an Oracle RAC instance.
After one instance caches data, any other instance within the same cluster database can acquire a block image from another instance in the same database faster than by reading the block from disk. Therefore, Cache Fusion moves current blocks between instances rather than re-reading the blocks from disk. When a consistent block is needed or a changed block is required on another instance, Cache Fusion transfers the block image directly between the affected instances. Oracle RAC uses the private interconnect for interinstance communication and block transfers. The GES Monitor and the Instance Enqueue Process manages access to Cache Fusion resources and enqueue recovery processing.
The Oracle RAC processes and their identifiers are as follows:
In an Oracle RAC environment, the atomic controlfile to memory service (ACMS) per-instance process is an agent that contributes to ensuring a distributed SGA memory update is either globally committed on success or globally aborted in the event of a failure.
The GTX0-j process provides transparent support for XA global transactions in a RAC environment. The database autotunes the number of these processes based on the workload of XA global transactions.
The LMON process monitors global enqueues and resources across the cluster and performs global enqueue recovery operations.
The LMD process manages incoming remote resource requests within each instance.
The LMS process maintains records of the datafile statuses and each cached block by recording information in a Global Resource Directory (GRD). The LMS process also controls the flow of messages to remote instances and manages global data block access and transmits block images between the buffer caches of different instances. This processing is part of the Cache Fusion feature.
The LCK0 process manages non-Cache Fusion resource requests such as library and row cache requests.
The RMSn processes perform manageability tasks for Oracle RAC. Tasks accomplished by an RMSn process include creation of resources related Oracle RAC when new instances are added to the clusters.
RSMN—Remote Slave Monitor manages background slave process creation and communication on remote instances. These background slave processes perform tasks on behalf of a coordinating process running in another instance.
Note:Many of the Oracle Database components that this section describes are in addition to the components that are described for single-instance Oracle databases in Oracle Database Concepts.
Automatic workload management enables you to manage the distribution of workloads to provide optimal performance for users and applications. This includes providing the highest availability for database connections, rapid failure recovery, and balancing workloads optimally across the active configuration. Oracle Database with Oracle RAC includes many features that can enhance workload management, such as connection load balancing, fast connection failover, the load balancing advisory, and runtime connection load balancing. Automatic workload management provides the greatest benefits to Oracle RAC environments. You can, however, take advantage of some of the features of automatic workload management by using Oracle Database services in single-instance Oracle databases, especially those that use Oracle Data Guard or Oracle Streams. Automatic workload management comprises the following components:
High Availability Framework: The Oracle RAC high availability framework enables Oracle Clusterware to maintain components in a running state at all times. Oracle high availability implies that Oracle Clusterware monitors and restarts critical components if they stop, unless you override the restart processing. The high availability framework also provides alerts to clients when configurations change. This enables clients to immediately react to the changes, enabling application developers to hide outages and reconfigurations from end users. The scope of Oracle RAC high availability spans from the restarting of stopped Oracle Database processes, including some instance background processes, to failing over the processing of an entire instance to other available instances through the relocation of services.
Load Balancing Advisory: This is the ability of the database to provide information to applications about the current service levels being provided by the database and its instances. Applications can take advantage of this information to direct connection requests to the instance that will provide the application request with the best service quality to complete the application's processing. Oracle Database has integrated its Java Database Connectivity (JDBC) and Oracle Data Provider for .NET (ODP.NET) connection pools, and the OCI Session pool to work with the load balancing information. Applications can use the integrated connection pools without programmatic changes. All applications achieve connection load balancing through the integration of the Oracle listener with the load balancing advisory.
Services: Cluster-managed services provide a powerful automatic workload management facility that enables the enterprise grid vision. Services are entities that you can define in Oracle RAC databases. In a cluster, services enable you to group database workloads and route the work to the optimal instances that are assigned to process the service. Furthermore, you can use services to define the resources that Oracle Database assigns to process workloads and to monitor workload resources. Applications that you assign to services transparently acquire the defined automatic workload management characteristics, including high availability and load balancing rules. Many Oracle Database features are integrated with services, such as Resource Manager (which enables you to restrict the resources that a service can use within an instance), Oracle Streams, Advanced Queuing (to achieve queue location transparency), and Oracle Scheduler (to map services to specific job classes).
In Oracle RAC databases, the service performance rules that you configure control the amount of work that Oracle Database allocates to each available instance for that service.
Connection Load Balancing: Oracle Net Services provides connection load balancing for database connections. Connection load balancing occurs when the connection is created. Connections for a given service are balanced across all of the running instances that offer the service using information from the load balancing advisory. You should define how you want connections to be balanced in the service definition. However, you must still configure Oracle Net Services.
Install the grid infrastructure (which includes Oracle Clusterware and ASM) and Oracle Database software using Oracle Universal Installer, and create your database with Database Configuration Assistant (DBCA). This ensures that your Oracle RAC environment has the optimal network configuration, database structure, and parameter settings for the environment that you selected. As a database administrator, after installation your tasks are to administer your Oracle RAC environment at three levels:
Cluster Administration (database administrators may or may not have privileges for cluster administration
This section introduces the installation processes for Oracle RAC under the following topics:
Note:You must first install Oracle Clusterware before installing Oracle RAC. See Oracle Clusterware Administration and Deployment Guide for more information.
If you plan on running multiple databases in your cluster, it is important to understand the software compatibility requirements, especially if you are using different releases of Oracle Database.
For Oracle RAC nodes running the Oracle9i database, you must install an Oracle9i cluster:
For UNIX, the cluster software can be HACMP, Serviceguard, Sun Cluster, or Veritas SF
For Windows and Linux, the cluster software is Oracle Cluster Manager
If you want to install Oracle RAC running Oracle Database 10g or later releases, you must also install Oracle Clusterware. Oracle Clusterware must be the same release as the latest version of Oracle Database, or later. See your platform-specific Oracle Clusterware Installation guide and the Oracle Clusterware Administration and Deployment Guide for more information.
If you are running Oracle RAC 10g and Oracle RAC 11g in the same cluster, you must be running Oracle Clusterware 11g (only)
Oracle requires that you install the Oracle9i RAC software first if you are going to run it in a cluster with Oracle RAC 10g or Oracle RAC 11g.
Note:If you are adding Oracle RAC to servers that are part of a cluster, either migrate to Oracle Clusterware (and remove the third-party clusterware) or ensure that the clusterware you are running is supported to run with the version of Oracle RAC you are installing. Before installing Oracle Clusterware with third-party clusterware, ensure that you install the correct options for the two to work together. Oracle strongly recommends that you do not run different cluster software on the same servers unless they are certified to work together and that you have install them in an integrated manner.
Once you install the grid infrastructure and Oracle Clusterware is operational, run Oracle Universal Installer to install the Oracle database software with Oracle RAC components.
During the installation, Oracle Universal Installer runs DBCA to create your Oracle RAC database according to the options that you select. DBCA also runs the Net Configuration Assistant (NETCA) to configure the network for your Oracle RAC environment.
See Also:Oracle Database Net Services Administrator's Guide for more information about NETCA
Oracle RAC software is distributed as part of the Oracle Database installation media. By default, the standard Oracle Database software installation process installs the Oracle RAC option when it recognizes that you are performing the installation on a cluster. The OUI installs Oracle RAC into a directory structure, which can be referred to as the Oracle home, which is separate from other Oracle software running on the system. Because OUI is cluster aware, it installs Oracle RAC software on all of the nodes that you defined to be part of the cluster.
Oracle recommends that you select ASM during the installation to simplify storage management; ASM automatically manages the storage of all database files within disk groups. If you are using Oracle Database Standard Edition, then you must use ASM to store all of the database files. You can also configure services during installation, depending on your processing requirements.
By default, Oracle Database creates one service for your environment and the service is for the database. (The default database service is typically identified using the combination of the
DB_DOMAIN initialization parameters:
db_domain.) The default service is available on all instances in an Oracle RAC environment, unless the database is in restricted mode. Oracle recommends that you create at least one service, in addition to the default database server, and use that service so that you have full control to manage your workload.
Note:Avoid changing host names after you complete the Oracle Clusterware installation, including adding or deleting domain qualifications. Nodes with changed host names must be deleted from the cluster and added back with the new name.
You can extend Oracle ASM and Oracle RAC in grid environments to additional nodes by copying cloned images of the ASM and Oracle RAC database homes to other nodes in the cluster. Oracle cloning copies images of the software to other nodes that have similar hardware and software. Cloning is best suited to a scenario where you need to quickly extend your Oracle RAC environment to several nodes of the same configuration.
Oracle provides the following methods of extending Oracle Clusterware environments:
Oracle cloning procedure using cloning scripts
Oracle Enterprise Manager cloning
addNode.sh script and OUI cloning
Note:Oracle cloning is not a replacement for Oracle Enterprise Manager cloning that is part of the Provisioning Pack. During Oracle Enterprise Manager cloning, the provisioning process includes a series of steps where details about the home you want to capture, the location you want to deploy to, and various other parameters are collected.
For new installations or if you have to install only one Oracle RAC database, you should use the traditional automated and interactive installation methods, such as OUI, or the Provisioning Pack feature of Oracle Enterprise Manager. If your goal is to add or delete Oracle RAC from nodes in the cluster, you can use the
The cloning process assumes that you successfully installed a grid infrastructure home and an Oracle home with Oracle RAC on at least one node. In addition, all root scripts must have run successfully on the node from which you are extending your cluster database.
At a high level, Oracle cloning involves the following main tasks:
Clone the grid infrastructure home following the instructions in Oracle Clusterware Administration and Deployment Guide.
The process for cloning the Oracle home onto new nodes is similar to the process for cloning the Oracle Clusterware home.
Run NETCA on each new node to create a listener.
If you have not already created a database, then run the DBCA to create one.
Follow the post-cloning procedures to complete the extension of your Oracle RAC environment onto the new nodes.
Chapter 8, " Adding and Deleting Oracle RAC from Nodes on Linux and UNIX Systems" for information about adding and deleting nodes and instances on Linux and UNIX Systems
Oracle Enterprise Manager online Help system for more information about the Provisioning Pack
Oracle Database Net Services Administrator's Guide for more information about NETCA
This section describes the following Oracle RAC environment management topics:
Any enterprise that is designing and implementing a high availability strategy with Oracle RAC must begin by performing a thorough analysis of the business drivers that require high availability. An analysis of business requirements for high availability combined with an understanding of the level of investment required to implement different high availability solutions enables the development of a high availability architecture that will achieve both business and technical objectives. See the following resources for help choosing and implementing the architecture that best fits your availability requirements:
Chapter 11, "Design and Deployment Techniques" provides a high-level overview you can use to evaluate the high availability requirements of your business.
Oracle Database High Availability Overview describes how to select the most suitable architecture for your organization, describes several high availability architectures, and provides guidelines for choosing the one that best meets your requirements.
Oracle enables you to administer a cluster database as a single system image through Oracle Enterprise Manager, SQL*Plus, or through Oracle RAC command-line interfaces such as Server Control (
Oracle Enterprise Manager—Oracle Enterprise Manager has both the Database Control and Grid Control GUI interfaces for managing both single-instance database and Oracle RAC database environments. Oracle recommends that you use Oracle Enterprise Manager to perform administrative tasks whenever feasible.
Server Control (
SRVCTL is a command-line interface that you can use to manage an Oracle RAC database from a single point. You can use
SRVCTL to start and stop the database and instances and to delete or move instances and services. You can also use
SRVCTL to manage configuration information, Oracle Clusterware, and ASM.
SQL*Plus—SQL*Plus commands operate on the current instance. The current instance can be either the local default instance on which you initiated your SQL*Plus session, or it can be a remote instance to which you connect with Oracle Net Services.
Cluster Verification Utility (CVU)—CVU is a command-line tool that you can use to verify a range of cluster and Oracle RAC components such as shared storage devices, networking configurations, system requirements, and Oracle Clusterware, as well as operating system groups and users. You can use CVU for preinstallation checks and for post-installation checks of your cluster environment. CVU is especially useful during preinstallation and during installation of Oracle Clusterware and Oracle RAC components. Oracle Universal Installer runs CVU after installing Oracle Clusterware and Oracle Database to verify your environment.
Install and use CVU before you install Oracle RAC to ensure that your configuration meets the minimum Oracle RAC installation requirements. Also use the CVU for ongoing administrative tasks, such as node addition and node deletion.
DBCA—DBCA is the recommended method for creating and initially configuring Oracle RAC databases.
NETCA—Configures the network for your Oracle RAC environment.
Chapter 3, " Administering Database Instances and Cluster Databases" for an introduction to Oracle RAC administration using Oracle Enterprise Manager, SQL*Plus, and the
Appendix A, "Server Control Utility Reference" for
SRVCTL reference information
Oracle Clusterware Administration and Deployment Guide for information about Oracle Clusterware tools such as the
OIFCFG tool for allocating and deallocating network interfaces, the
OCRCONFIG command-line tool for managing the Oracle Cluster Registry, and the Cluster Verification Utility (CVU)
Oracle Database Net Services Administrator's Guide for more information about NETCA
Web-based Oracle Enterprise Manager Database Control and Grid Control enable you to monitor an Oracle RAC database. The Oracle Enterprise Manager Console is a central point of control for the Oracle environment that you access by way of a graphical user interface (GUI). See Monitoring Oracle Real Application Clusters and Oracle Clusterware and the Oracle Database 2 Day + Real Application Clusters Guide, Oracle Enterprise Manager Concepts, for detailed information about using Oracle Enterprise Manager to monitor Oracle RAC environments.
Also, note the following recommendations about monitoring Oracle RAC environments:
Use the Oracle Enterprise Manager Console to initiate cluster database management tasks.
Use Oracle Enterprise Manager Grid Control to administer multiple Oracle RAC databases.
Use the global views, or
GV$ views, which are based on
V$ views. The
catclustdb.sql script creates the
GV$ views. Run this script if you do not create your database with DBCA. Otherwise, DBCA runs this script for you.
Use the sophisticated management and monitoring features of the Oracle Database Diagnostic and Tuning packs that include the Automatic Database Diagnostic Monitor (ADDM) and AWR.
See Also:The Oracle Database Performance Tuning Guide describes the Oracle Database automatic features for performance diagnosing and tuning, including ADDM.
You do not need to modify your application for Oracle RAC; Oracle RAC scales without application modifications. If your application scaled well on a single-instance Oracle database, then it will scale in an Oracle RAC environment. Many of the tuning tasks that you would perform on a single-instance Oracle database can also improve Oracle RAC database performance. This is especially true if your environment required scalability across a greater number of CPUs.
Some of the performance features specific to Oracle RAC include:
Oracle Database dynamically allocates Cache Fusion resources as needed
The dynamic mastering of resources improves performance by keeping resources local to data blocks
You do not have to tune any parameters for Cache Fusion
No application-level tuning is necessary
You can use a bottom-up tuning approach with virtually no effect on your existing applications
More Detailed Performance Statistics
More views for Oracle RAC performance monitoring
Oracle Enterprise Manager Database Control and Grid Control are Integrated with Oracle RAC
Footnote LegendFootnote 1: For configurations running Oracle Database 10g Release 2 (10.2) and later releases, Oracle Clusterware supports 100 nodes in a cluster, and Oracle RAC supports 100 instances in an Oracle RAC database.