test

Sun GlassFish Enterprise Server 2.1 High Availability Administration Guide

Chapter 2 Installing and Setting Up High Availability Database

Note –

The HADB software is supplied with the Enterprise Server standalone distribution of Sun GlassFish Enterprise Server. For information about available distributions of Sun GlassFish Enterprise Server, see Distribution Types and Their Components in Sun GlassFish Enterprise Server 2.1 Installation Guide. HADB features are available only in the enterprise profile. For information about profiles, see Usage Profiles in Sun GlassFish Enterprise Server 2.1 Administration Guide.

This chapter covers the following topics:

Preparing for HADB Setup

This section discusses the following topics:

After performing these tasks, see Chapter 3, Administering High Availability Database.

For the latest information on HADB, see Sun GlassFish Enterprise Server 2.1 Release Notes.

Prerequisites and Restrictions

Before setting up and configuring HADB, make sure your network and hardware environment meets the requirements described in the Sun GlassFish Enterprise Server 2.1 Release Notes. Additionally, there are restrictions with certain file systems; for example, with Veritas. For more information, see the Release Notes.

HADB uses Intimate Shared Memory (SHM_SHARE_MMU flag) when it creates and attaches to its shared memory segments. The use of this flag essentially locks the shared memory segments into physical memory and prevents them from being paged out. Therefore, HADB's shared memory is locked into physical memory, which can easily impact installations on low-end machines. Ensure you have the recommended amount of memory when co-locating Application Server and HADB.

Configuring Network Redundancy

Configuring a redundant network will enable HADB to remain available, even if there is a single network failure. You can configure a redundant network in two ways:

Setting Up Network Multipathing

Before setting up network multipathing, refer to the Administering Network Multipathing section in the IP Network Multipathing Administration Guide on docs.sun.com.

ProcedureTo Configure HADB Host Machines that Already Use IP Multipathing

  1. Set network interface failure detection time.

    For HADB to properly support multipathing failover, the network interface failure detection time must not exceed one second (1000 milliseconds), as specified by the FAILURE_DETECTION_TIME parameter in /etc/default/mpathd. Edit the file and change the value of this parameter to 1000 if the original value is higher:


    FAILURE_DETECTION_TIME=1000

    To put the change into effect, use this command:


    pkill -HUP in.mpathd

  2. Set up IP addresses to use with HADB.

    As described in the IP Network Multipathing Administration Guide, multipathing involves grouping physical network interfaces into multipath interface groups. Each physical interface in such a group has two IP addresses associated with it:

    • a physical interface address used for transmitting data.

    • a test address for Solaris internal use only.

    Specify only one physical interface address from the multipath group when you use hadbm create --hosts.

Example 2–1 Setting up Multipathing

Suppose there are two host machines named host1 and host2. If they each have two physical network interfaces, then set up the two interfaces as a multipath group. Run ifconfig -a on each host.

The output on host1 is:


bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>
mtu 1500 index 5 inet 129.159.115.10 netmask ffffff00 broadcast 129.159.115.255 
groupname mp0

bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 5 inet 129.159.115.11 netmask ffffff00 broadcast 129.159.115.255

bge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> 
mtu 1500 index 6 inet 129.159.115.12 netmask ffffff00 broadcast 129.159.115.255 
groupname mp0

bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> 
mtu 1500 index 6 inet 129.159.115.13 netmask ff000000 broadcast 129.159.115.255

The output on host2 is:


bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> 
mtu 1500 index 3 inet 129.159.115.20 netmask ffffff00 broadcast 129.159.115.255 
groupname mp0

bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> 
mtu 1500 index 3 inet 129.159.115.21 netmask ff000000 broadcast 129.159.115.255

bge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> 
mtu 1500 index 4 inet 129.159.115.22 netmask ffffff00 broadcast 129.159.115.255 
groupname mp0

bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> 
mtu 1500 index 4 inet 129.159.115.23 netmask ff000000 broadcast 129.159.115.255

In this example, the physical network interfaces on both hosts are listed after bge0 and bge1. Those listed after bge0:1 and bge1:1 are multipath test interfaces (marked DEPRECATED in the ifconfig output), as described in the IP Network Multipathing Administration Guide.

To set up HADB in this environment, select one physical interface address from each host. In this example, HADB uses IP address 129.159.115.10 from host1 and 129.159.115.20 from host2. To create a database with one database node per host, use the command hadbm create --hosts. For example


hadbm create --hosts 129.159.115.10,129.159.115.20

To create a database with two database nodes on each host, use the command:


hadbm create --hosts 129.159.115.10,129.159.115.20,
129.159.115.10,129.159.115.20

In both cases, you must configure the agents on host1 and host2 with separate parameters to specify which interface on the machines the agents should use. So, on host1 use:


ma.server.mainternal.interfaces=129.159.115.10

And on host2 use:


ma.server.mainternal.interfaces=129.159.115.20

For information on the ma.server.mainternal.interfaces variable, see Configuration File.

Configuring Double Networks

To enable HADB to tolerate single network failures, use IP multipathing if the operating system (for example, Solaris) supports it. Do not configure HADB with double networks on Windows Server 2003—the operating system does not work properly with double networks.

If your operating system is not configured for IP multipathing, and HADB hosts are equipped with two NICs, you can configure HADB to use double networks. For every host, the IP addresses of each of the network interface card (NIC) must be on separate IP subnets.

Within a database, all nodes must be connected to a single network, or all nodes must be connected to two networks.

Note –

Routers between the subnets must be configured to forward UDP multicast messages between subnets.

When creating an HADB database, use the –hosts option to specify two IP addresses or host names for each node: one for each NIC IP address. For each node, the first IP address is on net-0 and the second on net-1. The syntax is as follows, with host names for the same node separated by a plus sign (+):

--hosts=node0net0name+node0net1name
,node1net0name+node1net1name
,node2net0name+node2net1name
, ...

For example, the following argument creates two nodes, each with two network interfaces. The following host option is used to create these nodes:

--hosts 10.10.116.61+10.10.124.61,10.10.116.62+10.10.124.62

Thus, the network addresses

Notice that 10.10.116.61 and 10.10.116.62 are on the same subnet, and 10.10.124.61 and 10.10.124.62 are on the same subnet.

In this example, the management agents must use the same subnet. Thus, the configuration variable ma.server.mainternal.interfaces must be set to, for example, 10.10.116.0/24. This setting can be used on both agents in this example.

Configuring Shared Memory and Semaphores

You must configure shared memory and semaphores before installing HADB. The procedure depends on your operating system.

If you run other applications than HADB on the hosts, calculate these applications' use of shared memory and semaphores, and add them to the values required by HADB. The values recommended in this section are sufficient for running up to six HADB nodes on each host. You need only increase the values if you either run more than six HADB nodes, or the hosts are running applications that require additional shared memory and semaphores.

If the number of semaphores is too low, HADB can fail and display this error message: No space left on device. This can occur either while starting the database, or during run time.

ProcedureTo configure shared memory and semaphores on Solaris

Since the semaphores are a global operating system resource, the configuration depends on all processes running on the host, and not HADB alone. On Solaris, configure the semaphore settings by editing the /etc/system file.

  1. Log in as root.

  2. Configure shared memory.

    • Set shminfo_shmmax, which specifies the maximum size of a single shared memory segment on the host. Set this value to the total amount of RAM installed on the HADB host machine, expressed as a hexadecimal value, but no more than 2 GB.

      For example, for 2 GB of RAM, set the value as follows in the /etc/system file:


      set shmsys:shminfo_shmmax=0x80000000
      Note –

      To determine a host machine’s memory, use this command:


      prtconf | grep Memory

    • On Solaris 8 or earlier, set shminfo_shmseg, the maximum number of shared memory segments to which one process can attach. Set the value to six times the number of nodes per host. For up to six nodes per host, add the following to the /etc/system file: 


      set shmsys:shminfo_shmseg=36

      On Solaris 9 and later, shmsys:shminfo_shmseg is obsolete.

    • Set shminfo_shmmni, the maximum number of shared memory segments in entire system. Since each HADB node allocates six shared memory segments, the value required by HADB must be at least six times the number of nodes per host. On Solaris 9, for up to six nodes per host, there is no need to change the default value.

  3. Configure semaphores.

    Check the /etc/system file for the following semaphore configuration entries, for example:


    set semsys:seminfo_semmni=10
set semsys:seminfo_semmns=60
set semsys:seminfo_semmnu=30

    If the entries are present, increment the values as indicated below.

    If the /etc/system file does not these entries, add them at the end of the file:

    • Set seminfo_semmni, the maximum number of semaphore identifiers. Each HADB node needs one semaphore identifier. On Solaris 9, for up to six nodes per host, there is no need to change the default value. For example:


      set semsys:seminfo_semmni=10

    • Set seminfo_semmns, the maximum number of semaphores in the entire system. Each HADB node needs eight semaphores. On Solaris 9, or up to six nodes per host, there is no need to change the default value. For example:


      set semsys:seminfo_semmns=60

    • Set seminfo_semmnu, the maximum number of undo structures in the system. One undo structure is needed for each connection (configuration variable NumberOfSessions, default value 100). For up to six nodes per host, set it to 600:


      set semsys:seminfo_semmnu=600

  4. Reboot the machine.

ProcedureTo configure shared memory on Linux

On Linux, you must configure shared memory settings. You do not need to adjust the default semaphore settings.

  1. Log in as root.

  2. Edit the file /etc/sysctl.conf.

    With Redhat Linux, you can also modify sysctl.conf to set the kernel parameters.

  3. Set the values of kernel.shmax and kernel.shmall, as follows:


    echo MemSize > /proc/sys/shmmax
echo MemSize > /proc/sys/shmall

    where MemSize is the number of bytes.

    The kernel.shmax parameter defines the maximum size in bytes for a shared memory segment. The kernel.shmall parameter sets the total amount of shared memory in pages that can be used at one time on the system. Set the value of both of these parameters to the amount physical memory on the machine. Specify the value as a decimal number of bytes.

    For example, to set both values to 2GB, use the following:


    echo 2147483648 > /proc/sys/kernel/shmmax
echo 2147483648 > /proc/sys/kernel/shmall

  4. Reboot the machine using this command:

    sync; sync; reboot

Procedure for Windows

Windows does not require any special system settings. However, if you want to use an existing J2SE installation, set the JAVA_HOME environment variable to the location where the J2SE is installed.

Synchronizing System Clocks

You must synchronize clocks on HADB hosts, because HADB uses time stamps based on the system clock. HADB uses the system clock to manage timeouts and to time stamp events logged to history files. For troubleshooting, you must analyze all the history files together, since HADB is a distributed system. So, it is important that all the hosts’ clocks be synchronized

Do not adjust system clocks on a running HADB system. Doing so can cause problems in the operating system or other software components that can in turn cause problems such as hangs or restarts of HADB nodes. Adjusting the clock backward can cause some HADB server processes to hang as the clock is adjusted.

To synchronize clocks:

If HADB detects a clock adjustment of more than one second, it logs it to the node history file, for example:

NSUP INF 2003-08-26 17:46:47.975 Clock adjusted.
 Leap is +195.075046 seconds.

Installation

In general, you can install HADB on the same system as Enterprise Server (co-located topology) or on separate hosts (separate tier topology).

You must install the HADB management client to be able to set up high availability with the asadmin configure-ha-cluster command. When using the Java Enterprise System installer, you must install an entire HADB instance to install the management client, even if the nodes are to be installed on a separate tier.

HADB Installation

On a single or dual CPU system, you can install both HADB and Enterprise Server if the system has at least two Gbytes of memory. If not, install HADB on a separate system or use additional hardware. To use the asadmin configure-ha-cluster command , you must install both HADB and Enterprise Server.

Each HADB node requires 512 Mbytes of memory, so a machine needs one Gbyte of memory to run two HADB nodes. If the machine has less memory, set up each node on a different machine. For example, you can install two nodes on:

Default Installation Directories

Throughout this manual, HADB_install_dir represents the directory in which HADB is installed. The default installation directory depends on whether you install HADB as part of the Java Enterprise System. For Java Enterprise System, the default installation directory is /opt/SUNWhadb/4. For the standalone Application Server installer, it is /opt/SUNWappserver/hadb/4 .

Node Supervisor Processes Privileges

The node supervisor processes (NSUP) ensure the availability of HADB by exchanging “I’m alive” messages with each other. The NSUP executable files must have root privileges so they can respond as quickly as possible. The clu_nsup_srv process does not consume significant CPU resources, has a small footprint, and so running it with real-time priority does not affect performance.

Note –

The Java Enterprise System installer automatically sets the NSUP privileges properly, so you do not need to take any further action. However, with the standalone Enterprise Server (non-root) installer, you must set the privileges manually before creating a database.

Symptoms of Insufficient Privileges

If NSUPs do not have the proper privileges, you might notice symptoms of resource starvation such as:

Restrictions

If NSUP cannot set the real-time priority errno is set to EPERM on Solaris and Linux. On Windows it issues the warning “Could not set real-time priority”. The error is written to the ma.log file, and the process continues without real-time priority.

Setting real-time priorities is not possible when:

ProcedureTo Give Node Supervisor Processes Root Privileges

  1. Log in as root.

  2. Change your working directory to HADB_install_dir/lib/server.

    The NSUP executable file is clu_nsup_srv .

  3. Set the file’s suid bit with this command:

    chmod u+s clu_nsup_srv

  4. Set the file’s ownership to root with this command:

    chown root clu_nsup_srv

    This starts the clu_nsup_srv process as root, and enables the process to give itself realtime priority.

    To avoid any security impact, the real-time priority is set immediately after the process is started and the process falls back to the effective UID once the priority has been changed. Other HADB processes run with normal priority.

Setting up High Availability

This section provides the steps for creating a highly available cluster, and testing HTTP session persistence.

This section discusses the following topics:

ProcedureTo Prepare the System for High Availability

  1. Install Application Server instances and the Load Balancer Plug-in.

    For more information, see the Java Enterprise System Installation Guide (if you are using Java ES) or Sun GlassFish Enterprise Server 2.1 Installation Guide (if you are using the standalone Enterprise Server installer).

  2. Create Enterprise Server domains and clusters.

    For information on how to create a domain , see Creating a Domain in Sun GlassFish Enterprise Server 2.1 Administration Guide. For information on how to create a cluster, see To Create a Cluster.

  3. Install and configure your web server software.

  4. Setup and configure load balancing.

    For more information, see Setting Up HTTP Load Balancing.

Starting the HADB Management Agent

The management agent, ma, executes management commands on HADB hosts and ensures availability of the HADB node supervisor processes by restarting them if they fail.

You can start the management agent two ways:

Configuring a Cluster for High Availability

Before starting this section, you must have created one or more Enterprise Server clusters. For information on how to create a cluster, see To Create a Cluster.

From the machine on which the Domain Administration Server is running, configure the cluster to use HADB using this command:

asadmin configure-ha-cluster --user admin --hosts hadb_hostname1,hadb_hostname2 [,...] --devicesize 256 clusterName

Replace hadb_hostname1, hadb_hostname2, and so forth, with the host name of each machine where HADB is running, and clusterName with the name of the cluster. For example:

asadmin configure-ha-cluster --user admin --hosts host1,host2,host1,host2 --devicesize 256 cluster1

This example creates two nodes on each machine, which are highly available even in case of HADB failover. Note that the order of the host names following the –hosts option is significant, so the previous example would be different than --hosts host1,host1,host2,host2.

If you are using just one machine, you must provide the host name twice. In production settings, using more than one machine is recommended.

Configuring an Application for High Availability

In Admin Console, select the application under Applications > Enterprise Applications. Set Availability Enabled and then click Save.

Restarting a Cluster

To restart a cluster in Admin Console, choose Clusters > cluster-name. Click Stop Instances. Once the instances have stopped, click “Start Instances.”

Alternatively, use these asadmin commands:

asadmin stop-cluster --user admin cluster-name
asadmin start-cluster --user admin cluster-name

For more information on these commands, see stop-cluster(1) and start-cluster(1).

Restarting the Web Server

To restart the Web Server, type this Web Server command:

web_server_root/https-hostname/reconfig

Replace web_server_root with your Web Server root directory and hostname with the name of your host machine.

ProcedureTo Clean Up the Web Server Instance Acting as Load Balancer

  1. Delete the Load Balancer configuration:

    asadmin delete-http-lb-ref --user admin --config MyLbConfig FirstCluster

    asadmin delete-http-lb-config --user admin MyLbConfig

  2. If you created a new Web Server instance you can delete it by:

    1. Log on to the Web Server’s Administration Console.

    2. Stop the instance.

      Delete the instance.

Upgrading HADB

HADB is designed to provide “always on” service that is uninterrupted by upgrading the software. This section describes how to upgrade to a new version of HADB without taking the database offline or incurring any loss of availability. This is known as an online upgrade.

The following sections describe how to upgrade your HADB installation:

ProcedureTo upgrade HADB to a newer version

  1. Install new version of HADB.

  2. Register the new HADB version, as described in Registering HADB Packages

    Registering the HADB package in the HADB management domain makes it easy to upgrade or change HADB packages. The management agent keeps track of where the software packages are located, as well as the version information for the hosts in the domain. The default package name is a string starting with V and containing the version number of the hadbm program.

  3. Change the package the database uses.

    Enter the following command:


    hadbm set PackageName=package

    where package is the version number of the new HADB package.

  4. Un-register your existing HADB installation as described in Un-registering HADB Packages

  5. If necessary, replace the management agent startup script.

    For more information, see Replacing the Management Agent Startup Script

  6. Verify the results, as described in Verifying HADB Upgrade.

  7. (Optional) Remove the binary files for the old HADB version.

    After verifying that HADB has been upgraded properly, you can delete the old HADB packages.

Registering HADB Packages

Use the hadbm registerpackage command to register the HADB packages that are installed on the hosts in the management domain. HADB packages can also be registered when creating a database with hadbm create.

Before using the hadm registerpackage command, ensure that all management agents are configured and running on all the hosts in the hostlist, the management agent’s repository is available for updates, and no software package is already registered with the same package name.

The command syntax is:

hadbm registerpackage --packagepath=path [--hosts=hostlist] [--adminpassword=password | --adminpasswordfile=file] [--agent=maurl] [[package-name]]

The package-name operand is the name of the package.

The following table describes the special hadbm registerpackage command option. See Security Options and General Options for a description of other command options.

Table 2–1 hadbm registerpackage Options

Option 

Description 

--hosts=hostlist

-H 

List of hosts, either comma-separated or enclosed in double quotes and space separated. 

--packagepath=path

-L 

Path to the HADB software package. 

For example, the following command registers software package v4 on hosts host1, host2, and host3:

hadbm registerpackage 
--packagepath=hadb_install_dir/SUNWHadb/4.4 
--hosts=host1,host2,host3 v4

The response is:

Package successfully registered.

If you omit the --hosts option, the command registers the package on all enabled hosts in the domain.

Un-registering HADB Packages

Use the hadbm unregisterpackage command to remove HADB packages that are registered with the management domain.

Before using the hadbm unregisterpackage command, ensure that:

The command syntax is:

hadbm unregisterpackage  
--hosts=hostlist  
[--adminpassword=password | --adminpasswordfile= file]  
[--agent= maurl]  
[package-name ]

The package-name operand is the name of the package.

See Registering HADB Packages for a description of the --hosts option. If you omit the --hosts option, the hostlist defaults to the enabled hosts where the package is registered. See Security Options and General Options for a description of other command options.

Example 2–2 Example of un-registering HADB

To un-register software package v4 from specific hosts in the domain:

hadbm unregisterpackage --hosts=host1,host2,host3 v4

The response is:

Package successfully unregistered.

Replacing the Management Agent Startup Script

When you install a new version of HADB, you may need to replace the management agent startup script in /etc/init.d/ma-initd. Check the contents of the file, HADB_install_dir/lib/ma-initd. If it is different from the old ma-initd file, replace the old file with the new file.

ProcedureVerifying HADB Upgrade

Follow this procedure to verify that HADB has been properly upgraded:

  1. Confirm the version of the running HADB processes.

    Enter the following commands on all HADB nodes to display the HADB version:

    new-path/bin/ma -v

    new-path/bin/hadbm -v

    where new-path is the path to the new HADB installation.

    The results should show the new HADB version number.

  2. Confirm the database is running.

    Enter this command:

    new-path/bin/hadbm status -n

    If the upgrade is successful, the results will show all the HADB nodes in running state.

  3. Ensure that products using HADB have changed their configuration settings to the new HADB path.

  4. Run any upgrade tests for the products using HADB.