Chapter 15 Installing and Configuring Parallel Database Systems

This chapter provides instructions for setting up and administering Parallel Database Systems on your Sun Cluster servers, and includes the following sections:

• "General Information for Parallel Database Systems"

• "Installing OPS"

• "Installing Informix-Online XPS"

General Information for Parallel Database Systems

Shared Disk Architecture

The shared disk configuration of Sun Cluster is used by OPS. In this configuration, a single database is shared among multiple instances of OPS, all of which access the database concurrently. Conflicting access to the same data is controlled by means of a distributed lock manager (the Oracle UNIX Distributed Lock Manager (DLM)). If a process or a node crashes, the DLM is reconfigured to recover from such failure.

Shared Nothing Architecture

The shared nothing disk configuration of Sun Cluster is used by Informix-Online XPS. The database server instance(s) on each node has sole access to its own database partition.

A database query from a client is analyzed by the servers for its table partitions and forwarded across the private network to the appropriate servers. The results are merged across the private network and returned to the client.

Shared Memory Issues

Some applications (OPS, for example) sometimes require modification of the /etc/system file so that the minimum amount of shared memory that may be requested is unusually high. For example, if the field shmsys:shminfo_shmmin in the /etc/system file is set to a value greater than 200 bytes, the sm_config(1M) command will not be able to acquire shared memory, as it ends up requesting a smaller number of bytes than the minimum the system can allocate. As a result, the shmget(2) system call made by the sm_config(1M) command fails, thus aborting sm_config(1M).

To work around this problem, edit the /etc/system file and set the value of shmsys:shminfo_shmmin to 200. The value of shmsys:shminfo_shmmax should be greater than 200. Then reboot the machine for the new values to take effect.

If you encounter semsys warnings and core dumps, it could mean that the semaphore values contained in the semsys:seminfo_* fields in the /etc/system file do not match the actual physical limits of the machine.

OPS and IP Failover

In the event of a node failure in an OPS environment, Oracle SQL*Net or Net8 clients may be configured to reconnect to the surviving server without the use of IP failover.

In an OPS environment, multiple Oracle instances cooperate to provide access to the same shared database. The Oracle clients can access the database using any of the instances. Thus, if one or more instances have failed, clients can continue to access the database by connecting to a surviving instance.

There are many ways to reconnect to a surviving instance so that the reconnection is transparent to the end user:

• Design the application such that if the Oracle client loses the connection to the Oracle instance, it reconnects to an alternate instance. This implies the client is aware that it is operating in a multi-instance (OPS) environment.

However, such a solution is seldom used. Instead, most implementations use middle ware, such as the Tuxedo transaction monitor (TM), to implement the reconnection logic. The Oracle client connects to the TM, which in turn connects to one of the many database instances. The TM hides the failure of a particular database instance from the clients by reconnecting to alternate instances. The advantage of the TM approach is that existing Oracle client applications need not be rewritten to take advantage of the multiple instances in an OPS environment. The disadvantage is the cost of integrating with a TM.

• Design the application (Oracle client) such that when it loses the connection to the database instance, it retries the connection to the same server. Thus, Oracle client applications designed for a non-parallel environment can be moved into an OPS environment without redesign. The infrastructure is then designed to ensure the connection is routed to the surviving server.

High Availability Features of the Oracle SQL*Net

From the Oracle client perspective the model is simple, when the server crashes the client sees a broken connection. The client reconnects to the server, and resubmits the transaction. The Oracle SQL*Net provides features and capabilities to incorporate multiple instances running on different hosts under the same service. Hence, when the client reconnects, it is automatically connected through to the surviving instance. The reconnection is not automatic. The client typically incorporates the code to reconnect broken connections (to the same service as before).

With a node or instance failure, the surviving instance(s) must first recover the failed instances state. During this recovery time clients will see a lack of response from the instance. This recovery has nothing do with the Sun Cluster framework. Recovery is totally dependent on Oracle, the transaction volume, and recovery mechanism for OPS.

Installing OPS

These are the high-level steps to install OPS in a Sun Cluster configuration. If you are installing Oracle7 Parallel Server, refer to the Oracle7 for Sun SPARC Solaris 2.x Installation and Configuration Guide, Release 7.x. If you are installing Oracle8 Parallel Server, refer to the Oracle8 Parallel Server Concepts and Administration, Release 8.0 Guide. If you are installing Oracle8i Parallel Server, refer to your Oracle8i installation documentation.

1. Configure the UNIX kernel Interprocess Communication (IPC) parameters to accommodate the Shared Global Area (SGA) structure of Oracle8i. Reboot all nodes.

2. Set up the Oracle user environment on all nodes, using your Oracle documentation.

3. Use the scinstall(1M) command to install Sun Cluster 2.2. Specify OPS when prompted to select a data service, and use VxVM with the cluster feature enabled as the volume manager.

4. Reboot all nodes.

5. Install the ORCLudlm package on all nodes. This package is located at on the Oracle product CD-ROM, in the ops_patch directory.

6. On all nodes, install VERITAS Volume Manager (VxVM) with the cluster feature enabled.

7. Reboot all nodes.

8. Create the VERITAS root disk group (rootdg). See your VERITAS documentation for detailed instructions about creating the rootdg.

9. Start the cluster on one node only.

10. Create Oracle disk groups and raw volumes for the OPS database. See your VERITAS and Oracle8i installation documentation for details.

11. Stop the cluster.

12. Restart the cluster on all nodes.

13. Install Oracle with the OPS option on all nodes. See your Oracle/OPS documentation for detailed instructions about installing OPS.

14. For OPS versions 8.0.5 and earlier, configure the Oracle GMS daemon. See "Starting the Oracle GMS Daemon" for details. This step is not necessary for OPS 8.1.6 or later versions.

Configuring Oracle SQL*Net and Net8

The methods used to configure Oracle SQL*Net or Net8 for OPS are identical to the methods used for Oracle, with one exception: with OPS instances, Net8 can be configured for load balancing and failover. The configuration methods differ for different versions of Oracle and OPS; consult your Oracle documentation for detailed procedures to configure the connections between the Oracle client and Oracle server.

Starting the Oracle GMS Daemon

In Sun Cluster 2.2, Oracle's Group Membership Services (GMS) daemon is not started by scogms(1M) as part of the Sun Cluster framework installation. This means that for Oracle Parallel Server databases, the GMS daemon must be started manually using the binary ogmsctl that is provided by the Oracle Parallel Server Option installation on a cluster node. The ogmsctl daemon must be up and running even when the instance on the node is started in exclusive mode. Therefore, this daemon should be running when the database is created.

The GMS binaries ogms and ogmsctl are located in the /bin directory under $ORACLE_HOME. The default home directory for the GMS daemon is /tmp/.ogms. This directory contains trace files and the gms0000.dat file.

The GMS daemon must be started from the oracle user login. The following commands start the GMS daemon.

# su - oracle
$ ogmsctl start

For the daemon to start successfully, the node upon which it is run must be a cluster member.

The following command stops the GMS daemon.

$ ogmsctl stop

The ogmsctl command has the following options.

Installing Informix-Online XPS

For information about installing Informix-Online XPS, refer to the Installation Notes INFORMIX-OnLine XPS document. Any changes made in the /etc/system file to support Informix-Online XPS must be based on the information contained in your Informix documentation.

The Informix-Online XPS documentation always supersedes the information in this section.

(Network Security) Informix requires the Private Interconnect IP address of the cluster nodes to be available in the /etc/hosts file during installation. If any of the nodes of the cluster are configured to run as NIS or DNS name servers, this requirement may present a security problem because the name servers may make the private addresses available to unauthorized hosts. In the interest of security, if Informix is installed on your cluster, you may not want to configure the nodes as NIS or DNS name servers.

How to Install Informix-Online XPS

In a Sun Cluster configuration, perform the following steps to install Informix-Online XPS.

1. Edit the /etc/hosts file using your favorite text editor.

   For example:

   # vi /etc/hosts

2. Add lines similar to the following to the /etc/hosts file.

   204.152.65.1 ssa28-scid0 204.152.65.17 ssa28-scid1 204.152.65.2 ssa28a-scid0 204.152.65.18 ssa28a-scid1

   In this example, 204.152.65.1 and 204.152.65.17 are the IP addresses assigned by the Sun Cluster system to the SCI cards on the primary node. IP addresses 204.152.65.2 and 204.152.65.18 are the addresses assigned to the SCI cards on the secondary node.

   From node 1, you can use ssa28a-scid0 or ssa28a-scid1 to communicate with node 2 in the cluster. This choice enables you to select the connection (SCI 0 or SCI 1) that is used to carry the message.

   The Informix onconfig configuration file uses these names to set up communication between the two nodes.