Overview of Oracle RAC Architecture and Processing

Installing Oracle RAC requires software, a network, and a storage configuration.

Oracle RAC requires several components:

Oracle Clusterware for concurrent access to the same storage and the same set of data files from all of the nodes in a cluster
A communications protocol for enabling interprocess communication (IPC) across all of the nodes in a cluster
Multiple database instances that process data as if the data resided on a logically combined, single cache
A mechanism for monitoring and communicating the statuses of the nodes in a cluster

Understanding Cluster-Aware Storage Solutions

Learn about the Oracle Real Application Clusters shared everything architecture and what shared everything means for your storage options.

An Oracle RAC database is a shared everything database. All data files, control files, SPFILEs, and redo log files in Oracle RAC environments must reside on cluster-aware shared disks. This enables all of the cluster database instances to access these storage components. Because Oracle RAC databases use a shared everything architecture, Oracle RAC requires cluster-aware storage for all of the database files.

In Oracle RAC, Oracle AI Database manages the disk access and is certified for use on a variety of storage architectures. It is your choice how to configure your storage, but you must use a supported cluster-aware storage solution. Oracle AI Database provides the following Oracle RAC storage options:

Oracle Automatic Storage Management (Oracle ASM)
Oracle recommends this solution to manage your storage.
A certified cluster file system
Oracle recommends Oracle Advanced Cluster File System (Oracle ACFS).
A third-party cluster file system on a cluster-aware volume manager that is certified for Oracle RAC. For example:
Oracle OCFS2 (Linux only)
IBM GPFS (IBM AIX only)
Certified network file system (NFS) solution

Oracle RAC and Network Connectivity

All nodes in Oracle RAC environments must connect to at least one Local Area Network (LAN). This network is commonly referred to as the public network, and it enables users and applications to access the database.

In addition to the public network, Oracle RAC requires private network connectivity that is used exclusively for communication among the nodes and database instances that are running on those nodes. This network is commonly referred to as the interconnect.

The interconnect network is a private network that connects all of the servers in a cluster. The interconnect network must use at least one switch and a Gigabit Ethernet adapter.

Note:

Oracle supports interfaces with higher bandwidth but does not support using crossover cables with the interconnect.

Do not use the interconnect, the private network, for user communication, because Cache Fusion uses the interconnect for interinstance communication.

You can configure Oracle RAC to use either the User Datagram Protocol (UDP) or Reliable Data Socket (RDS) protocols for inter-instance communication on the interconnect. Oracle Clusterware uses the same interconnect using the UDP protocol, but cannot be configured to use RDS.

An additional network connectivity is required when using Network Attached Storage (NAS). Network attached storage can be typical NAS devices, such as NFS filers, or it can be storage that is connected using Fibre Channel over IP, for example. This additional network communication channel should be independent of the other communication channels that Oracle RAC uses (the public and private network communication). If the storage network communication must be converged with one of the other network communication channels, then ensure that storage-related communication gets first priority.

Overview of Using Dynamic Database Services to Connect to Oracle AI Database

Applications should use the Dynamic Database Services feature to connect to an Oracle AI Database over the public network.

Dynamic Database Services enable you to define rules and characteristics to control how users and applications connect to database instances. These characteristics include a unique name, workload balancing and failover options, and high availability characteristics.

Users can access an Oracle RAC database using a client/server configuration or through one or more middle tiers, with or without connection pooling. By default, a user connection to an Oracle RAC database is established using the TCP/IP protocol but Oracle supports other protocols. Oracle RAC database instances must be accessed through the SCAN for the cluster.

Overview of Virtual IP Addresses

Node VIPs are virtual IP (VIP) addresses that clients use to connect to Oracle RAC databases.

Oracle Clusterware hosts the node VIP addresses on a public network. The process for a typical connection attempt from a database client to an Oracle RAC database instance can be summarized as follows:

The database client connects to SCAN (which includes a SCAN VIP on a public network), providing the SCAN listener with a valid service name.
The SCAN listener then determines which database instance hosts this service and routes the client to the local or node listener on the respective node.
The node listener, listening on a node VIP and a given port, retrieves the connection request and connects the client to an instance on the local node.

If multiple public networks are used on the cluster to support client connectivity through multiple subnets, then the preceding operation is performed within a given subnet.

If a node fails, then the VIP address fails over to another node on which the VIP address can accept TCP connections, but it does not accept connections to the Oracle AI Database. Clients that attempt to connect to a VIP address that does not reside on its home node receive a rapid connection refused error instead of waiting for TCP connect timeout messages. When the network on which the VIP is configured comes back online, Oracle Clusterware fails back the VIP to its home node, where connections are accepted. Generally, VIP addresses fail over when:

The node on which a VIP address runs fails
All interfaces for the VIP address fail
All interfaces for the VIP address are disconnected from the network

Oracle RAC supports multiple public networks to enable access to the cluster through different subnets. Each network resource represents its own subnet and each database service uses a particular network to access the Oracle RAC database. Each network resource is a resource that is managed by Oracle Clusterware, which enables the VIP behavior previously described.

SCAN is a single network name that is defined either in your organization's Domain Name Server (DNS) or in the Grid Naming Service (GNS) that operates in a round robin order among three IP addresses. Oracle recommends that all of the connections to the Oracle RAC database use the SCAN in their client connection string. Incoming connections are load balanced across the active instances providing the requested service through the three SCAN listeners. With SCAN, you do not have to change the client connection even if the configuration of the cluster changes (nodes added or removed). SCAN fully supports multiple subnets. This means that you can create one SCAN for each subnet in which you want your cluster to operate.

Restricted Service Registration in Oracle RAC

The valid node checking feature provides the ability to configure and dynamically update a set of IP addresses or subnets from which registration requests are allowed by the listener.

Database instance registration with a listener succeeds only when the request originates from a valid node. The network administrator can specify a list of valid nodes, excluded nodes, or disable valid node checking altogether. The list of valid nodes explicitly lists the nodes and subnets that can register with the database. The list of excluded nodes explicitly lists the nodes that cannot register with the database. The control of dynamic registration results in increased manageability and security of Oracle RAC deployments.

By default, the SCAN listener agent sets REMOTE_ADDRESS_REGISTRATION_listener_name to a private IP endpoint. The SCAN listener accepts registration requests only from the private network. Remote nodes that are not accessible to the private network of the SCAN listener must be included in the list of valid nodes by using the registration_invited_nodes_alias parameter in the listener.ora file, or by modifying the SCAN listener using the command-line interface, SRVCTL.

Note:

Starting with Oracle Grid Infrastructure 12c, for a SCAN listener, if the VALID_NODE_CHECKING_REGISTRATION_listener_name and REGISTRATION_INVITED_NODES_listener_name parameters are set in the listener.ora file, then the listener agent overwrites these parameters.

If you use the SRVCTL utility to set the invitednodes and invitedsubnets values, then the listener agent automatically sets VALID_NODE_CHECKING_REGISTRATION_listener_name to SUBNET and sets REGISTRATION_INVITED_NODES_listener_name to the specified list in the listener.ora file.

For other listeners managed by CRS, the listener agent sets VALID_NODE_CHECKING_REGISTRATION_listener_name to be SUBNET in the listener.ora file only if it is not already set in the listener.ora file. The SRVCTL utility does not support setting the invitednodes and invitedsubnets values for a non-SCAN listener. The listener agent does not update REGISTRATION_INVITED_NODES_listener_name in the listener.ora file for a non SCAN listener.

About Oracle RAC Software Components

In an Oracle RAC environment, Oracle RAC Cache Fusion automatically synchronizes each Oracle instance's buffer cache.

Oracle RAC database automatically manages the block transfer between the instances without any manual intervention from the Database Administrators. The Global Cache Service (GCS) and the Global Enqueue Service (GES) perform the bulk of the work. The GCS and GES coordinate buffer cache blocks and enqueues in the Global Resource Directory (GRD). The GRD contents are distributed across all the active instances, resulting in a slight increase in the SGA size.

Note:

GRD and Oracle RAC Cache Fusion components use approximately 10 percent of the total SGA. When you move from a single-instance database to an Oracle RAC database, plan to increase your SGA accordingly.
Starting with Oracle AI Database 26ai, Release Update 23.7:
The default value of the number of lock manager tickets has increased, resulting in improved performance in very busy concurrent workloads. This also results in a slight increase in the memory usage by the GCS process.
Improved monitoring capabilities by GCR* processes may result in a slight increase in memory usage. The V_GCR_METRICS view provides details on the new monitoring capabilities.

Once the cache is warm, the blocks can be transferred to any other instance needing the block via the private network. This is often faster than reading the block from a disk. Even in cases where a consistent block is requested from another instance, Cache Fusion transfers the block image directly.

In addition, Cache Fusion monitors the latency on the private networks and the service time on the disks, and automatically chooses the best path.

About Oracle RAC Background Processes

The global cache service (GCS) and the global enqueue service (GES) processes, along with the global resource directory (GRD) collaborate to enable Cache Fusion.

The Oracle RAC processes and their identifiers are as follows:

ACMS: Atomic Controlfile to Memory Service ( ACMS )
In Oracle RAC environments, the ACMS process on each instance is an agent that contributes to ensuring a distributed SGA memory update is either globally committed on success or globally terminated if a failure occurs.
BG00: Background Process
The ora_bg is a standard background process in Oracle RAC environments that performs various maintenance and monitoring tasks. It is responsible for buffer cache management, log writer ( LGWR ) process support, checkpoint coordination, background process monitoring, and statistics gathering.
GCW: Global Conflict Resolution (GCR) Monitor Process (LMHB)
The ora_gcw is an infrastructure helper processes for LMHB. It monitors the LMHB process for stall and restarts, if necessary.
GTX0-j: Global Transaction Process
The GTX0-j process provides transparent support for XA global transactions in Oracle RAC environments. The database autotunes the number of these processes based on the workload of XA global transactions.
LMON: Global Enqueue Service Monitor
The LMON process monitors global enqueues and resources across the cluster and performs global enqueue recovery operations.
LMD: Global Enqueue Service Daemon
The LMD process manages incoming remote resource requests within each instance.
LMS: Global Cache Service Process
The LMS process maintains records of the data file statuses and each cached block by recording information in the global resource directory (GRD). The LMS process also controls the flow of messages to remote instances and manages global data block access and transmits block images between the buffer caches of different instances. This processing is part of Cache Fusion.
LCK0: Instance Enqueue Process
The LCK0 process manages non-Cache Fusion resource requests such as library and row cache requests.
RMSn : Oracle RAC Management Processes ( RMSn )
The RMSn processes perform manageability tasks for Oracle RAC. Tasks that are accomplished by an RMSn process include the creation of resources that are related to Oracle RAC when new instances are added to the clusters.
RSMN: The RSMN process manages background secondary process creation and communication on remote instances. These background secondary processes perform tasks on behalf of a coordinating process running in another instance.
SCMN: Statistics Collection and Management
SCMN is the main thread for a multithreaded process in a threaded Oracle RAC architecture. All other threads are spawned within the processes through SCMN based on the requests in the instance. It is an idle main thread, which waits for any requests, especially spawn threads, and takes care of them along with some periodic maintenance operations.
VOSD: Virtual Operating System Daemon
The VOSD process executes time bound Oracle AI Database service actions. VOSD is spawned at instance startup and it is responsible for executing system service actions critical for the database.

Note:

Many of the Oracle AI Database components that this section describes are in addition to the components that are described for single-instance Oracle AI Databases in Oracle AI Database Concepts.