1
Overview of Oracle8i Parallel Server

This chapter provides a conceptual and component overview of Oracle Parallel Server. This information helps you prepare and plan your Oracle Parallel Server installation on Windows NT. 

Note: OPS-specific implementation, installation and configuration issues are covered in Oracle8i Parallel Server Setup and Configuration Guide.

Specific topics discussed are:

• Introduction
• Oracle Parallel Server Software Components
• Oracle Parallel Server Instances
• Oracle Parallel Server Features
• New Features In This Release

 

 

Introduction

Oracle Parallel Server is an architecture that allows multiple instances to access a shared database. Oracle Parallel Server offers the following (terms will be described later in this chapter):

• Superior scalability through distribution of workload across nodes
• High-availability through multiple nodes accessing the database; if one node fails, the database is still accessible through surviving nodes

An Oracle Parallel Server can handle node or disk failure in a clustered environment with no or minimal downtime. The Oracle Parallel Server architecture provides the following features:

• Separate Oracle instances running on different nodes operate against a set of common database files that reside on shared disks physically accessible by all nodes that make up the cluster.
• All instances can execute transactions concurrently against the same database, and each instance can have multiple users executing transactions concurrently.

Coordination of each node accessing the shared database provides the following:

• Data consistency and integrity
• Data and lock recovery support

 

 

Oracle Parallel Server Software Components

The following components make up an Oracle Parallel Server:

Component	Description
Oracle8i Enterprise Edition	Provides the applications and files to manage a database. All other Oracle Parallel Server components are layered on top of Oracle Enterprise Edition.
Oracle Parallel Server Option	Provides the necessary Oracle Parallel Server scripts initialization files, and data files to create and support an Oracle Parallel Server.
Operating System Dependent (OSD) layer	Consists of several software components developed by vendors. The OSD layer maps the key OS/cluster-ware services required for proper operation of Oracle Parallel Server. See "Operating System Dependent Layer".

Operating System Dependent Layer

A vendor-supplied Operating System Dependent (OSD) layer that passed certification must be installed after Oracle Parallel Server Option is installed. The OSD layer consists of several software components developed by vendors. The OSD layer maps the key OS/cluster-ware services required for proper operation of Oracle Parallel Server.

OSD Components

The OSD layer consists of:

Component	Description
Cluster Manager (CM)	Discovers the state of the cluster and instance membership.
Inter-Process Communication (IPC)	Provides reliable transfer of messages between instances on different nodes.
Input/Output (IO)	Provides I/O to access shared disks.
Startup (START)	Provides one-time configuration to startup functionality.

 

Note: The Performance and Management (P&M) component available as an OSD component prior to this release has now integrated with the Oracle Parallel Server Option.

These components provide key services required for proper operation of the Oracle Parallel Server Option and are used by various clients, such as Integrated Distributed Lock Manager (IDLM). Each OSD module interacts with the Oracle Parallel Server runtime environment as a single Dynamic Link Library (DLL). These components are more fully described later in this chapter.

Figure 1-1 illustrates the OSD components in a cluster with two nodes:

Figure 1-1 OSD Components

Cluster Manager

The Cluster Manager (CM) monitors process health, specifically of the Oracle8 release 8.1 database instance. LMON, a background process that monitors the health of the Integrated Distributed Lock Manager (IDLM), registers and de-registers from CM.

It is critical that all Oracle Parallel Server instances receive the same membership information when events occur. Notification changes cause relevant Oracle Parallel Server recovery operations to be initiated. If any node or instance is determined to be dead or otherwise not a properly functioning part of the system, CM terminates all processes on that node or instance. Thus, any process or thread running Oracle code can safely assume its node or instance is an active member of the system.

If there is a failure, recovery is transparent to user applications. CM automatically reconfigures the system to isolate the failed node and instance and notify IDLM of the status. IDLM subsequently recovers any of the locks from the failed node or instance. Oracle Parallel Server can then recover the database to a valid state.

The IDLM relies on the Cluster Manager for timely and correct information. If the IDLM cannot get the information it needs, it will shut down the instance.

CM consists of the following components:

• Node Monitor Component
• Cluster Management Component

 

 

Node Monitor Component

The Node Monitor component of CM manages access to shared disks and monitors the status of various cluster resources (including nodes, networks and the Cluster Management component).

The Node Monitor component:

• Provides the basic node management interface modules needed by Oracle Parallel Server in a cluster environment.
• Allows the Oracle Parallel Server environment to discover and track the membership state of nodes by providing a common view of cluster membership across the cluster.
• Runs on all nodes and monitors the topology of the cluster by querying all nodes for their current membership.
• Detects changes in the state of active nodes signaling those events, diagnosing the changes, and coordinating a new common and consistent state among all nodes.
• Notifies the Cluster Management component and subsequently Oracle Parallel Server of the actual changes to membership.

 

 

Cluster Management Component

The Cluster Management component of CM replaces the release 8.0 Group Membership Service (PGMS) function of determining what groups are up and managing instance members. Each instance registers with their database specific group managed by the Cluster Management component. Cluster Management then interacts with the Node Monitor component of CM.

Figure 1-2 depicts how the Node Monitor and Cluster Management components coordinate.

Figure 1-2 Cluster Manager

 

Note: Some vendors may not support both Node Monitor and Cluster Management components.

Inter-Process Communication (Node-to-Node)

Oracle Parallel Server derives most of its functional benefits from its ability to run on multiple interconnected machines. Oracle Parallel Server relies heavily on the underlying Inter-Process Communication (IPC) component to facilitate this.

IPC defines the protocols and interfaces required for the Oracle Parallel Server environment to transfer reliable messages between instances. Messages are the fundamental logical units of communication in this interface. The core IPC functionality is built around an asynchronous, queued messaging model. IPC is designed to send/receive discrete messages as fast as the hardware allows. With an optimized communication layer, various services can be implemented above it. This is how the IDLM carries out all of its communication.

Input/Output

The Input/Output (IO) component provides interprocess capabilities that a cluster implementation must support to enable proper operation of the Oracle Parallel Server environment.

The Oracle Parallel Server environment is extremely dependent on the ability of the underlying OS/cluster implementation to support simultaneous disk sharing across all nodes that run coordinated Oracle Parallel Server instances. Unlike switch-over based technologies, all Oracle Parallel Server instances are active and can operate on any database entity in the shared physical database simultaneously. It is this capability that gives Oracle Parallel Server a large portion of its parallel scalability. It is the role of the IDLM to coordinate the simultaneous access to shared databases in a way that maintains consistency and data integrity.

At a high level, the Oracle Parallel Server shared I/O model can be described as a distributed disk cache implemented across all nodes that define the Oracle Parallel Server cluster. The core of Oracle Parallel Server can be viewed as a major client of the cache. Disk blocks from the shared devices are read into a particular node instance cache only after mediation by the IDLM. The other node instance may read the same blocks into its cache and operate on them simultaneously. Updates to those blocks are carefully coordinated. In general, all shared disk based I/O operations are mediated by the IDLM. The set of distributed IDLMs on each node can be thought of as managing the distributed aspects of the cache.

Disk update operations must be carefully coordinated so that all nodes see the same data in a consistent way. Any Oracle Parallel Server instance intending to update a cached data block must enter into a dialog with the IDLM to ensure it has exclusive right to update the block. Once it does this, the instance is free to update the block until its rights have been revoked by the IDLM. When the exclusive update right is revoked, the instance with block updates must write the block to disk so that the other node can see the changes. Given this rather high-level view of the IDLM I/O consistency model, it is clear that disk blocks can migrate around to each instance's block cache and all updates are flushed to disk when an instance other than the owner desires access to the block. It is this property that directly determines the reliance of Oracle Parallel Server on shared disk implementations.

Startup

The Startup (START) component initiates the Oracle Parallel Server components in a specific order during instance startup. It is up to the vendor to determine this startup sequence.

Oracle Parallel Server Instances

Each Oracle Parallel Server instance, a running instance of Oracle8i software, is comprised of a System Global Area (SGA) and Oracle background processes. The SGA is a shared memory region that contains data and control information for an instance. Users can connect to any instance to access the information that resides within the shared database.

Oracle Parallel Server instances coordinate with the following components:

Component	Description
node	A server where an instance resides.
cluster	A set of physically interconnected nodes, and a shared disk storage subsystem.
database	The set of all instances with access to the same data files. A database is limited to a set of Oracle Parallel Server instances that run on only the nodes defined within a cluster.  The database name is defined by global database name, a name comprised of the database name and domain, entered during installation.

All instances share:

• the same set of data files
• the same control files

 

 

An instance contains:

• at least the following background processes:
• PMON
• SMON
• DBWR
• LGWR

 

 

• additional background processes:
• LCKN to manage the locks used by an instance and to coordinate requests for those locks by other instances. Additional lock processes, LCK1 through
• LCK9, are available for systems that require exceptionally high throughput of instance lock requests. The single lock process per instance, LCK0, is usually sufficient for most systems.
• LMON to monitor the health of the Integrated Distributed Lock Manager.
• LMD0 to process lock requests
• BSP0 to manage out-going messages to requesting nodes for the Consistent Read (CR) Server. (The LCK process manages in-coming lock requests for instances holding locks needed by other instances.)

 

 

• the Integrated Distributed Lock Manager (IDLM) component in the SGA, an area for global locks and resources. This area was formerly part of the external Distributed Lock Manager (DLM).

 

 

See Oracle8i Concepts for information about Oracle8i database processes and memory structures.

The data files are located on disk drives that are shared between the multiple nodes. If one node fails, client applications (written to do so) can re-route users to another node. One of the surviving nodes automatically performs recovery by rolling back any incomplete transactions that the other node was attempting. This ensures the logical consistency of the database. 

Note: An instance does not include data files, you can start up an instance without mounting the data files.

Unique Identification of Instances

Following the OFA naming convention, each node's instance has its own INITSID.ORA file that uniquely defines the instance with instance-related parameters. This file calls INITDB_NAME.ORA, which lists common parameters shared from node-to-node. Both files are created by the Oracle Database Configuration Assistant after installation.

The INITSID.ORA file is impacted by a number of elements, including:

Element	Description
Oracle System Identifier (SID)	Uniquely identifies a node's instance.  The database name, specified by the DB_NAME parameter in the INITDB_NAME.ORA file, and unique thread ID make up each node's SID. The thread ID starts at 1 for the first instance in the cluster, and is incremented by 1 for the next instance, and so on.  The SID is defined as an ORACLE_SID environment variable on UNIX platforms and in the ORACLE_SID registry entry on Windows NT.
Instance Name	Represents the name of the instance and is used to uniquely identify a specific instance when multiple instances share common services names.  The instance name is the same as the SID. It is depicted by the INSTANCE_NAME parameter in the INITSID.ORA file.
Rollback Segments	Specifies at least two rollback segments to store "undo" information. If the ROLLBACK_SEGMENTS parameter is set in the INITSID.ORA file, an instance acquires all of the rollback segments named in this parameter, even if the number of rollback segments exceeds the minimum number required by the instance. By default, the Oracle Database Configuration Assistant creates rollback segment identified by RBSthread_id_rollback_number. For example, on node with a sequence number of 1, the rollback segments are RBS1-1 and RBS1-2.
Thread ID	Specifies the redo thread that is to be used by the instance. Any available redo thread number can be used, but an instance cannot use the same thread number as another instance. Also, an instance cannot start when its redo thread is disabled. An instance cannot mount a database if the thread is used by another instance or if the thread is disabled.  The thread starts at 1 node for the first instance in the cluster, and is incremented by 1 for the next instance, and so on.Redo thread numbers are used in the naming of redo log files, which record changes made to data. When redo log files are generated, they include the thread, allowing you to easily identify a particular node's redo log files.  Threads are depicted by the THREAD parameter in the INITSID.ORA file.

Examples of Relationships Between Elements

If the DB_NAME is OP and the thread IDs are 1, 2, and 3, then the SID for each node is:

Thread ID	SID
1	OP1
2	OP2
3	OP3

If the DB_NAME is OP and the node numbers are 0, 1, 4, and 6, then the instance elements are effected in the following manner:

Computer	Thread ID	SID	INSTANCE_NAME	ROLLBACK_SEGMENT name
Computer A	1	OP1	OP1	RBS1-1, RBS1-2
Computer B	2	OP2	OP2	RBS2-1, RBS2-2
Computer C	3	OP3	OP3	RBS3-1, RBS3-2
Computer D	4	OP4	OP4	RBS4-1, RBS4-2

Database Files

A database is logically divided into tablespaces that contain all data stored in the database. Tablespaces, in turn, are made up of one or more data files.

With Oracle Parallel Server, all participating instances access the same database files.

Figure 1-3 shows the relationship between two Oracle instances and the shared disks on which the database files are stored:

Figure 1-3 Oracle Parallel Server Instance Architecture

Oracle Parallel Server Features

Oracle Parallel Server has the following features:

Feature	Description
Integrated Distributed Lock Manager (IDLM)	Maintains a list of system resources and provides locking mechanisms to control allocation and modification of Oracle resources. Every process interested in the database resource protected by the IDLM must open a lock on the resource. See "Integrated Distributed Lock Manager".
Parallel Cache Management (PCM)	Provides instance locks (with minimal use of the IDLM) that cover one or more data blocks of any class: data block, index blocks, undo blocks, segment headers, and so on. Oracle Parallel Server uses these instance locks to coordinate access to shared resources. The IDLM maintains the status of the instance locks See "Parallel Cache Management".
Oracle Parallel Query (OPQ)	Ships queries between nodes so that multiple nodes can execute on behalf of a single query. See "Oracle Parallel Query".

Integrated Distributed Lock Manager

The Integrated Distributed Lock Manager (IDLM) maintains a list of system resources and provides locking mechanisms to control allocation and modification of Oracle resources. Resources are structures of data. The IDLM does not control access to tables or anything in the database itself. Every process interested in the database resource protected by the IDLM must open a lock on the resource.

Oracle Parallel Server uses the IDLM facility to coordinate concurrent access to resources, such as data blocks and rollback segments, across multiple instances. The Integrated Distributed Lock Manager facility has replaced the external Distributed Lock Manager which was used in earlier releases of Oracle Server.

The IDLM uses the LMON and LMDN processes. LMON manages instance and processes deaths and associated recovery for the IDLM. In particular, LMON handles the part of recovery associated with global locks. The LMDN process handles remote lock requests (those which originate from another instance).

The IDLM:

• Provides a distributed architecture. In the distributed architecture each node in the cluster (or each instance of an Oracle database) participates in global lock management and manages a piece of the global lock database. The lock database is distributed among all the participants.
• Provides continual service and maintains the integrity of the lock database in the event of multiple node and instance failures.
• Maintains information about the locks on all nodes that are interested in a given resource. In this situation, the IDLM usually nominates one node to manage all relevant information about the resource and its locks.
• Performs deadlock detection.
• Synchronizes modifications made to the Oracle database files so that no changes are lost.
• Synchronizes disk block access between multiple nodes.

 

 

The IDLM is a resource manager and, thus, does not control access to the database.

Example

A node in a cluster needs to modify block number n in the database file. At the same time, another node needs to update the same block n to complete a transaction.

Without the IDLM, both nodes update the same block at the same time. With the IDLM, only one node is allowed to update the block. The other node must wait. The IDLM ensures that only one instance has the right to update a block at any one time. This provides data integrity by ensuring that all changes made are saved in a consistent manner.

Interaction with CM

The IDLM uses CM to determine which instances are active. When the instance is started, the LMON and LMDN processes are started and the IDLM registers with CM. The IDLM deregisters with CM when the database is shutdown.

Parallel Cache Management

Parallel Cache Management (PCM) provides instance locks (with minimal use of the IDLM) that cover one or more data blocks of any class: data block, index blocks, undo blocks, segment headers, and so on. Oracle Parallel Server uses these instance locks to coordinate access to shared resources. The IDLM maintains the status of the instance locks.

PCM locks ensure cache coherency by forcing instances to acquire a lock before modifying or reading any database block. PCM locks allow only one instance at a time to modify a block. If a block is modified by an instance, the block must first be written to disk before another instance can acquire the PCM lock, read the block, and modify it.

Example

If node 1 needs access to data that is currently in node 2's buffer cache, node 1 can submit a request to the IDLM. Node 2 then writes the needed blocks to disk. Only then is Node 1 notified by the IDLM to read updated and consistent data from the disk.

PCM Lock Implementation

You use the initialization parameter GC_FILES_TO_LOCKS to specify the number of PCM locks which cover the data blocks in a data file or set of data files. The smallest granularity is one PCM lock per data block; this is the default. PCM locks usually account for the greatest proportion of instance locks in a parallel server.

PCM locks are implemented in four ways:

PCM Lock	Description
hashed locks	Hashed locks are pre-allocated and statically assigned to one or more blocks at startup time. Hashed locks offer the advantage of no overhead for lock allocation when a data block is accessed. The first instance which starts up creates an IDLM resource and an IDLM lock (in null mode) on the IDLM resource for each hashed PCM lock. The first instance initializes each lock. The instance then proceeds to convert IDLM locks to other modes as required. When a second instance requires a particular IDLM lock, it waits until the lock is available and then converts the lock to the mode required. The total number of locks that can be allocated is limited by system resources. This usually means that multiple blocks have to be covered by the same lock. In other words, there is a low lock granularity. This might result in false pinging. The startup of the instance also requires more time, since all the lock resources have to be allocated at startup time. Typically, hashed locks are never released; each will stay in the mode in which it was last requested. You can, however, specify releasable hashed locks by using the R option with the GC_FILES_TO_LOCKS parameter. Releasable hashed PCM locks are taken from the pool of GC_RELEASABLE_LOCKS.
fine-grain locks	This is the default implementation.  With fine-grain locking, locks are dynamically allocated at block-access time. The resources for the lock are only allocated during the time the lock is needed and are released when the lock is released. This makes it possible to achieve very high-lock granularity. If resource minimization is the goal, fine-grain locks can also cover multiple blocks, but are still allocated dynamically. Since locks are allocated only as required, the instance can start up much faster than with hashed locks. An IDLM resource is created and an IDLM lock is obtained only when a user actually requests a block. Once a fine-grain lock has been created, it can be converted to various modes as required by various instances.  Typically, fine-grain locks are releasable: An instance can give up all references to the resource name during normal operation. You can, however, allocate fixed locks in a fine grained manner with the GC_FILES_TO_LOCKS parameter. Creating a 1 to 1 ratio of locks to blocks creates DBA locking.

 

Additional Information: See Chapter 9, Parallel Cache Management Instance Locks" and Chapter 15, "Allocating PCM Instance Locks", of the Oracle8i Parallel Server Concepts and Administration guide. It is possible to have both fine-grain locking and hashed locking enabled at the same time.

Below is a comparison of both PCM locks:

Hash PCM Locks	Fine-Grain PCM Locks
Allocated at instance startup, resulting in a slower startup.	Allocated when user requests a block, resulting in faster instance startup.
Released only at instance shutdown.	Dynamically re-used by other blocks, requiring less memory.
Statically hashed to blocks at startup time, requiring more memory.

Choosing a Lock

Use the table below to choose a PCM lock:

When to use hashed locks...	When to use fine-grain locks...
Data is mostly read-only.	Small amount of data is updated by many instances.
Data can be partitioned.	There is not enough memory for the configuration of hash locking.
Large set of data is modified by a relatively small set of instances.

Oracle Parallel Query

With the Oracle Parallel Query (OPQ), Oracle can divide the work of processing certain types of SQL statements among multiple query server processes.

Architectural Overview

When parallel execution is not being used, a single server thread performs all necessary processing for the sequential execution of a SQL statement. For example, to perform a full table scan (such as SELECT * FROM EMP), one thread performs the entire operation.

OPQ performs the operations in parallel using multiple parallel processes. One process, known as the parallel coordinator, dispatches the execution of a statement to several parallel server processes and coordinates the results from all the server processes to send the results back to the user.

The parallel coordinator breaks down execution functions into parallel pieces and then integrates the partial results produced by the parallel server processes. The number of parallel server processes assigned to a single operation is the degree of parallelism for an operation. Multiple operations within the same SQL statement all have the same degree of parallelism.

OPQ on Oracle Parallel Server

Oracle Parallel Server provides the framework for the Parallel Query Option to work between nodes. The OPQ behaves the same way in Oracle with or without the Parallel Server Option. The only difference is that Oracle Parallel Server enables OPQ to ship queries between nodes so that multiple nodes can execute on behalf of a single query. Here, the server breaks the query up it into smaller operations that run against a common database which resides on shared disks. Because it is performed by the server, this parallelism can occur at a low level of server operation, rather than at an external SQL level.

In some applications, an individual query often consumes a great deal of CPU resource and disk I/O (unlike most online insert or update transactions). To take advantage of multi-processing systems, the data server must parallelize individual queries into units of work which can be processed simultaneously.

If the query were not processed in parallel, disks would be read serially with a single I/O. A single CPU would have to scan all rows in a table. With the query parallelized, disks are read in parallel, with multiple I/Os.

Several CPUs can each scan a part of the table in parallel, and aggregate the results. Parallel query benefits not only from multiple CPUs but also from greater I/O bandwidth availability.

OPQ can run with or without the Oracle Parallel Server. Without the Oracle Parallel Server option, OPQ cannot perform multi-node parallelism. Oracle Parallel Server optimizes Oracle8 Enterprise Edition running on clustered hardware, using a parallel cache architecture to avoid shared memory bottlenecks in OLTP and decision support applications.

Figure 1-4 Parallel Query Operation

OPQ Statement

OPQ within Oracle Parallel Server performs parallelism within a node and among nodes via the parallel query slave processes on each node.

A sample SQL statement is shown below:

After you have run a query, you can use the information derived from V$PQ_SYSSTAT to view the number of slave processes used, and other information for the system.
 

Additional Information: See Chapter 28, "Parallel Execution", in the Oracle8 Concepts guide.

New Features In This Release

These tables describe the new products and features of Oracle Parallel Server release 8.1.5 specific to Windows NT. See Getting to Know Oracle8i and Oracle8i Parallel Server Concepts and Administration for the list of new Oracle Parallel Server features common to all operating systems.

New Feature	Description	For More Information, See...
Optimal Flexible Architecture (OFA)	Provides a structured method for installing Oracle8 databases and applications in a manner which helps promote ease of maintenance, improved performance, and reliability.	See Appendix A, "Directory Structure" in this book and Oracle8i Enterprise Edition Getting Started for Windows NT for more information on OFA. Note: OFA has caused a number of changes to the Oracle directory tree, file names, directory names, and the definition of Oracle home since release 8.0.x.
Oracle Universal Installer	The new installer is based on a Java engine that provides an extensible environment able to handle advanced software installation requirements. Oracle Universal Installer provides users with a consistent interface across all platforms on which Oracle is available.  Note: Versions of the Installer previous to release 8.1 are referred to as "Oracle Installer."	Oracle8i Enterprise Edition Installation for Windows NT
Cluster-aware software	The Oracle Universal Installer and the Oracle Database Configuration Assistant are cluster-aware in this release. Installation and configuration can be performed from one node in the cluster rather than each individual node. Components are "pushed" out to the other nodes.	Oracle8i Parallel Server Setup and Configuration Guide
Easy configuration	Configuration was a manual process prior to this release. While this manual process is still available for this release, the Oracle Database Configuration Assistant can be run to create an Oracle Parallel Server database. The Oracle Database Configuration Assistant can be run automatically after installation or at a later time.	Chapter 3, "Installation", and Chapter 4, "Configuring Oracle Parallel Server", in Oracle8i Parallel Server Setup and Configuration Guide.
OSD Performance and Management (P&M) component implemented in Oracle Parallel Server Manager Option	The P&M component was a vendor-supplied component installed with the OSD layer. This component interacted with the Oracle Enterprise Manager tools. Because P&M must be implemented in a precise manner, this component is now installed with the Oracle Parallel Server Option rather than as an OSD module.	Chapter 3 of this guide.
initialization file name changes	The initialization file that contains common parameters shared from node-to-node was called INIT_COM.ORA in prior releases. This file name has been replaced by INITDB_NAME.ORA. Each instance still has it's own initialization file, INITSID.ORA.	"Unique Identification of Instances" Chapter 6, "Administering Oracle Parallel Server in Oracle8i Parallel Server Setup and Configuration Guide.