|Oracle8i Parallel Server Concepts
Release 2 (8.1.6)
Part Number A76968-01
This chapter provides a detailed discussion of the inter-instance coordination activities that take place in a cluster. As mentioned in Chapter 3, Oracle uses locks within a cluster to coordinate lock resources, data, and inter-instance data requests. This chapter describes the details about how Oracle coordinates these resources.
Topics in this chapter include:
Coordination of concurrent tasks within a cluster is called synchronization. Resources such as data blocks and locks must be synchronized as nodes within a cluster acquire and release ownership of them. The synchronization provided by Oracle Parallel Server maintains cluster-wide concurrency of the resources and in turn ensures the integrity of the shared data.
The key to successful parallel processing is to divide the tasks that require resources among the nodes so that very little synchronization is necessary. The less synchronization that is necessary, the better your system's speedup and scaleup. The overhead of synchronization can be very expensive if excessive inter-node communication is necessary.
The synchronization effort to achieve parallel processing among nodes ideally uses a high-speed interconnect linking the parallel processors. For parallel processing within a node, messaging is not necessary; shared memory is used instead. As mentioned in the previous chapter, messaging and locking between nodes is handled by the Distributed Lock Manager.
The amount of synchronization depends on the amount of resources and the number of users and tasks working on the resources. Little synchronization may be needed to coordinate a small number of concurrent tasks, but many concurrent tasks can require significant synchronization.
Each instance of Oracle Parallel Server has a dictionary cache, or row cache, containing data dictionary information in its System Global Area. The data dictionary structure is the same for Oracle instances in Oracle Parallel Server as for instances in exclusive mode. Parallel Server uses global locks to coordinate data dictionary activity among multiple instances.
Parallel Cache Management uses several types of locks to control access to data and resources within a cluster. The way you configure the locks and the degree of granularity you use affect the performance of applications running on Oracle Parallel Server. The degree of granularity refers to the number of locks per instance.
You can most positively influence each type of locking by properly designing your applications. Second, setting initialization parameters and properly administering your cluster can also positively affect your system's overhead. On the other hand, improper lock use can cause your system to spend so much time synchronizing shared resources that you cannot achieve any speedup or scaleup.
Oracle Parallel Server uses two primary groups of locks as described in the following section:
There are two types of local locks, latches and enqueues. Latches are not Parallel Server-specific and are synchronized only within each instance. Latches thus do not have an effect on the global operations of clustered environments. Enqueues, however, can be both local to an instance and global to a cluster.
The following provides a brief description of these two types of locks:
Latches are simple, low level serialization mechanisms that protect in-memory data structures in the System Global Area. Latches do not protect data files, are automatic, and are held for a very short time in exclusive mode. As mentioned, because latches are synchronized within a node, they do not facilitate internode synchronization.
Enqueues are shared memory structures that serialize access to database resources. Enqueues are local to one instance if you do not enable Parallel Server. Or when you enable Parallel Server, enqueues can be global to a database. Enqueues are associated with a session or transaction and Oracle can use them in any of the following modes:
Enqueues are held longer than latches, have more granularity and more modes than latches, and protect more database resources. For example, if you request a table lock, or a DML lock, your request is assigned an "enqueue" lock. Enqueues are managed by the Distributed Lock Manager. When Parallel Server is enabled, most local enqueues become global enqueues.
Oracle Parallel Server synchronizes global locks among all active instances in a cluster. Global locks include two main types:
The Distributed Lock Manager tracks the status of all Oracle locking mechanisms. Oracle only creates global locks if you start an Oracle instance with Parallel Server enabled. The exception to this is the mount lock. The Distributed Lock Manager synchronizes global locks by communicating the status of a lock resource to all instances within an Oracle Parallel Server cluster.
Global locks are held by background processes within instances rather than by transactions. An instance owns a global lock that protects a resource, such as a data block or data dictionary entry, when the resource enters the instance's System Global Area. The Distributed Lock Manager manages locking only for resources accessed by more than one instance.
The following sections in this chapter are:
Non-parallel Cache Management includes coordination for resources other than data blocks. The Parallel Cache Management features of Oracle Parallel Server are described later in this chapter.
There are many different types of non-Parallel Cache Management locks. These control access to data files and control files. They also control library and dictionary caches, and perform various types of communication between instances. These locks do not protect data file blocks. Examples of these are Data Modification Language enqueues (table locks), transaction enqueues, and Data Definition Language locks or dictionary locks. The System Change Number (SCN), and the mount lock are global locks, not enqueues.
The context of Oracle Parallel Server causes most local enqueues to become global; they can still be seen in the fixed tables and views that show enqueues, such as V$LOCK. The V$LOCK table does not, however, show global locks, such as SCN locks, mount locks, and Parallel Cache Management locks.
This section describes some of the most common non-PCM global locks. It covers the following information:
This section explains how Oracle uses non-PCM locks to manage locks for transactions, tables, and other entities within an Oracle environment. Figure 4-1 highlights non-PCM locks in relation to other locks used in Oracle.
Whereas PCM locks are static (you allocate them when you design your application), non-PCM locks are very dynamic. Their number and corresponding space requirements will change as your system's initialization parameter values change.
Oracle8i Reference for descriptions of all non-PCM locks.
Row locks are locks that protect selected rows. A transaction acquires a global enqueue and an exclusive lock for each individual row modified by one of the following statements:
These locks are stored in the block, and each lock refers to the global transaction enqueue.
A transaction lock is acquired in exclusive mode when a transaction initiates its first change. It is held until the transaction does a COMMIT or ROLLBACK. SMON also acquires it in exclusive mode when recovering (undoing) a transaction. Transaction locks are used as a queuing mechanism for processes awaiting the release of an object locked by a transaction in progress.
Table locks are DML locks that protect entire tables. A transaction acquires a table lock when a table is modified by one of the following statements: INSERT, UPDATE, DELETE, SELECT with the FOR UPDATE clause, and LOCK TABLE. A table lock can be held in any of several modes: null (N), row share (RS), row exclusive (RX), share lock (S), share row exclusive (SRX), and exclusive (X).
When a database object (table, view, procedure, function, package, package body, trigger, index, cluster, synonym) is referenced during parsing or compiling of a SQL (DML/DDL), PL/SQL, or Java statement, the process parsing or compiling the statement acquires the library cache lock in the correct mode. In Oracle8 the lock is held only until the parse or compilation completes (for the duration of the parse call).
The data dictionary cache contains information from the data dictionary, the meta-data store. This cache provides efficient access to the data dictionary.
Creating a new table, for example, causes the meta-data of that table to be cached in the data dictionary. If a table is dropped, the meta-data needs to be removed from the data dictionary cache. To synchronize access to the data dictionary cache, latches are used in exclusive mode and in single shared mode. Global locks are used in multiple shared (parallel) mode.
In Oracle Parallel Server, the data dictionary cache on all nodes may contain the meta-data of a table that gets dropped on one instance. The meta-data for this table needs to be flushed from the data dictionary cache of every instance. This is performed and synchronized by global locks.
The mount lock shows whether an instance has mounted a particular database. This lock is only used with Oracle Parallel Server. It is the only multi-instance lock used by Parallel Server in exclusive mode, where it prevents another instance from mounting the database in shared mode.
In Oracle Parallel Server single shared mode, this lock is held in shared mode. Another instance can successfully mount the same database in shared mode. In Parallel Server exclusive mode, however, another instance will not able to obtain the lock.
Understanding how Oracle Parallel Server synchronizes caches across instances can help you understand the overhead affecting system performance. Consider a five-node parallel server where a user drops a table on one of the nodes. Since each of the five dictionary caches has a copy of the definition of the dropped table, the node dropping the table from its cache must also cause the other four dictionary caches to drop their copies of the dropped table. Oracle Parallel Server handles this automatically through the Distributed Lock Manager. Users on the other nodes are notified of the change in lock status.
There are significant advantages to having each node store library and table information. Occasionally, the DROP TABLE statement forces other caches to be flushed, but the brief effect this has on performance does not necessarily diminish the advantage of having multiple caches.
The processing within Oracle that requires synchronization includes:
In Oracle Parallel Server exclusive mode, all synchronization is done within the instance. In shared mode, synchronization is accomplished with the help of the Distributed Lock Manager component that maintains the status of the global locks.
The most often required database resources are data blocks. Parallel Cache Management provides global locks to cover one or more types of data blocks. The types covered are:
Parallel Cache Management locks ensure cache coherency by requiring that instances acquire a lock before modifying or reading a database block. Thus, Parallel Cache Management locks allow only one instance at a time to modify a block.
Parallel Cache Management of the buffer caches located on separate nodes provides Parallel Server cache coherency. The set of global constant (GC_*) initialization parameters associated with Parallel Cache Management buffer cache locks are not the same locks as those used with the dictionary cache, library cache, and so on.
Parallel Cache Management ensures that a master copy data block, also known as the "consistent read block" (the version of the block holding all the changes) is held in at least one of the System Global Areas in the cluster if it is to be changed. If an instance needs to read it, the current version of the block may reside in many buffer caches under shared locks. Thus, the most recent copy of the block in all System Global Areas contains all changes made to that block by all instances, regardless of whether any transactions on those instances have committed.
If a data block is modified in one buffer cache, then copies in other buffer caches are no longer current. New copies can be obtained after the modification operation completes.
Oracle only performs Parallel Cache Management lock operations for cache coherency when the current version of a data block is in one instance's buffer cache and another instance requests that block for update.
Multiple transactions running on a single Oracle Parallel Server instance can share access to a set of data blocks for reading without additional global lock operations. In this case, there is no contention or conflict. This remains true as long as the blocks are not needed for writing by transactions running on other instances.
Instances use global locks to indicate ownership of a resource master copy. When an instance becomes a database resource master or "owner", it also becomes owner of the global lock covering the resource with fixed locking. However, releasable locks are, of course, released.
A master copy indicates it is an updatable copy of the resource. The instance only downgrades the global lock when another instance requests the resource for update. Once another instance owns the master copy of the resource, it becomes the owner of the global lock.
Consider the following example and the illustrations in Figure 4-2. This example assumes one Parallel Cache Management lock covers one block, although many blocks could be covered.
Parallel Cache Management locks and row locks operate independently. An instance can disown a Parallel Cache Management lock without affecting row locks held in the set of blocks covered by the Parallel Cache Management lock. A row lock is acquired during a transaction. A database resource such as a data block acquires a Parallel Cache Management lock when it is read for update by an instance. During a transaction, a Parallel Cache Management lock can therefore be disowned and owned many times if the blocks are needed by other instances.
In contrast, transactions do not release row locks until changes to the rows are either committed or rolled back. Oracle uses internal mechanisms for concurrency control to isolate transactions so modifications to data made by one transaction are not visible to other transactions until the transaction modifying the data commits. The row lock concurrency control mechanisms are independent of parallel cache management: concurrency control does not require Parallel Cache Management locks, and Parallel Cache Management lock operations do not depend on individual transactions committing or rolling back.
An instance can acquire the global lock that covers a set of data blocks in either shared or exclusive mode, depending on the access type required.
If one instance needs to update a data block and a second instance already owns the global lock covering the block, the first instance uses the Distributed Lock Manager to request that the second instance disown the global lock, writing the block(s) to disk if necessary.
Multiple instances can own a global lock in shared mode as long as they only intend to read, not modify, blocks covered by that global lock. Thus, all instances can be sure that their memory-resident copies of the block are current, or that they can read a current copy from disk without any global lock operations to request the block from another instance. This means instances do not have to disown global locks for the portion of a database accessed for read-only use, which may be a substantial portion of the time in many applications.
This mode is used so that locks need not be continually obtained and released. Locks are simply converted from one mode to another.
Block access between instances is done on a per-block level. When an instance locks a block in exclusive mode, other instances cannot access the block. Every time Oracle tries to read a block from the database it must obtain a global lock. Ownership of the lock is thus assigned to the instance.
Since Oracle Parallel Server runs in environments with multiple memories, there can be multiple copies of the same data block in each instance's memory. Internode synchronization using the Distributed Lock Manager ensures the validity of all copies of the block: these block-level locks are the buffer cache locks.
Block level locking occurs only when Parallel Server is enabled. It is transparent to the user and to the application. Row level locking operates whether Parallel Server is enabled or disabled.
Parallel Cache Management locks are typically more numerous than non-Parallel Cache Management locks. However, there are still enough non-Parallel Cache Management locks for which you must carefully plan adequate Distributed Lock Manager capacity. Typically 5% to 10% of locks are non-Parallel Cache Management. Non-Parallel Cache Management locks do not grow in volume the way Parallel Cache Management locks do.
You can control Parallel Cache Management locks in detail by setting initialization parameters to allocate the number desired. However, you have almost no control over non-Parallel Cache Management locks. You can attempt to eliminate the need for table locks by setting DML_LOCKS = 0 or by using the ALTER TABLE ENABLE/DISABLE TABLE LOCK command, but other non-Parallel Cache Management locks will still persist.
Oracle8i Parallel Server Administration, Deployment, and Performance for information about allocating resources for Distributed Lock Manager locks.