The HADB software is supplied with the Enterprise Server standalone distribution of Sun GlassFish Enterprise Server. For information about available distributions of Sun GlassFish Enterprise Server, see Distribution Types and Their Components in Sun GlassFish Enterprise Server v2.1.1 Installation Guide. HADB features are available only in the enterprise profile. For information about profiles, see Usage Profiles in Sun GlassFish Enterprise Server v2.1.1 Administration Guide.
This chapter covers the following topics:
After performing these tasks, see Chapter 3, Administering High Availability Database.
For the latest information on HADB, see Sun GlassFish Enterprise Server v2.1.1 Release Notes.
Before setting up and configuring HADB, make sure your network and hardware environment meets the requirements described in the Sun GlassFish Enterprise Server v2.1.1 Release Notes. Additionally, there are restrictions with certain file systems; for example, with Veritas. For more information, see the Release Notes.
HADB uses Intimate Shared Memory (SHM_SHARE_MMU flag) when it creates and attaches to its shared memory segments. The use of this flag essentially locks the shared memory segments into physical memory and prevents them from being paged out. Therefore, HADB's shared memory is locked into physical memory, which can easily impact installations on low-end machines. Ensure you have the recommended amount of memory when co-locating Application Server and HADB.
Configuring a redundant network will enable HADB to remain available, even if there is a single network failure. You can configure a redundant network in two ways:
On Solaris 9, by setting up network multipathing.
On all platforms except Windows Server 2003, by configuring a double network.
Before setting up network multipathing, refer to the Administering Network Multipathing section in the IP Network Multipathing Administration Guide on docs.sun.com.
Set network interface failure detection time.
For HADB to properly support multipathing failover, the network interface failure detection time must not exceed one second (1000 milliseconds), as specified by the FAILURE_DETECTION_TIME parameter in /etc/default/mpathd. Edit the file and change the value of this parameter to 1000 if the original value is higher:
To put the change into effect, use this command:
pkill -HUP in.mpathd
Set up IP addresses to use with HADB.
As described in the IP Network Multipathing Administration Guide, multipathing involves grouping physical network interfaces into multipath interface groups. Each physical interface in such a group has two IP addresses associated with it:
a physical interface address used for transmitting data.
a test address for Solaris internal use only.
Specify only one physical interface address from the multipath group when you use hadbm create --hosts.
Suppose there are two host machines named host1 and host2. If they each have two physical network interfaces, then set up the two interfaces as a multipath group. Run ifconfig -a on each host.
The output on host1 is:
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 5 inet 126.96.36.199 netmask ffffff00 broadcast 188.8.131.52 groupname mp0 bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5 inet 184.108.40.206 netmask ffffff00 broadcast 220.127.116.11 bge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6 inet 18.104.22.168 netmask ffffff00 broadcast 22.214.171.124 groupname mp0 bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 6 inet 126.96.36.199 netmask ff000000 broadcast 188.8.131.52
The output on host2 is:
bge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 184.108.40.206 netmask ffffff00 broadcast 220.127.116.11 groupname mp0 bge0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3 inet 18.104.22.168 netmask ff000000 broadcast 22.214.171.124 bge1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 inet 126.96.36.199 netmask ffffff00 broadcast 188.8.131.52 groupname mp0 bge1:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4 inet 184.108.40.206 netmask ff000000 broadcast 220.127.116.11
In this example, the physical network interfaces on both hosts are listed after bge0 and bge1. Those listed after bge0:1 and bge1:1 are multipath test interfaces (marked DEPRECATED in the ifconfig output), as described in the IP Network Multipathing Administration Guide.
To set up HADB in this environment, select one physical interface address from each host. In this example, HADB uses IP address 18.104.22.168 from host1 and 22.214.171.124 from host2. To create a database with one database node per host, use the command hadbm create --hosts. For example
hadbm create --hosts 126.96.36.199,188.8.131.52
To create a database with two database nodes on each host, use the command:
hadbm create --hosts 184.108.40.206,220.127.116.11, 18.104.22.168,22.214.171.124
In both cases, you must configure the agents on host1 and host2 with separate parameters to specify which interface on the machines the agents should use. So, on host1 use:
And on host2 use:
For information on the ma.server.mainternal.interfaces variable, see Configuration File.
To enable HADB to tolerate single network failures, use IP multipathing if the operating system (for example, Solaris) supports it. Do not configure HADB with double networks on Windows Server 2003—the operating system does not work properly with double networks.
If your operating system is not configured for IP multipathing, and HADB hosts are equipped with two NICs, you can configure HADB to use double networks. For every host, the IP addresses of each of the network interface card (NIC) must be on separate IP subnets.
Within a database, all nodes must be connected to a single network, or all nodes must be connected to two networks.
Routers between the subnets must be configured to forward UDP multicast messages between subnets.
When creating an HADB database, use the –hosts option to specify two IP addresses or host names for each node: one for each NIC IP address. For each node, the first IP address is on net-0 and the second on net-1. The syntax is as follows, with host names for the same node separated by a plus sign (+):
--hosts=node0net0name+node0net1name ,node1net0name+node1net1name ,node2net0name+node2net1name , ...
For example, the following argument creates two nodes, each with two network interfaces. The following host option is used to create these nodes:
Thus, the network addresses
For node0 are 10.10.116.61 and 10.10.124.61
For node1 are 10.10.116.62 and 10.10.124.62
Notice that 10.10.116.61 and 10.10.116.62 are on the same subnet, and 10.10.124.61 and 10.10.124.62 are on the same subnet.
In this example, the management agents must use the same subnet. Thus, the configuration variable ma.server.mainternal.interfaces must be set to, for example, 10.10.116.0/24. This setting can be used on both agents in this example.
You must configure shared memory and semaphores before installing HADB. The procedure depends on your operating system.
If you run other applications than HADB on the hosts, calculate these applications' use of shared memory and semaphores, and add them to the values required by HADB. The values recommended in this section are sufficient for running up to six HADB nodes on each host. You need only increase the values if you either run more than six HADB nodes, or the hosts are running applications that require additional shared memory and semaphores.
Since the semaphores are a global operating system resource, the configuration depends on all processes running on the host, and not HADB alone. On Solaris, configure the semaphore settings by editing the /etc/system file.
Log in as root.
Configure shared memory.
Set shminfo_shmmax, which specifies the maximum size of a single shared memory segment on the host. Set this value to the total amount of RAM installed on the HADB host machine, expressed as a hexadecimal value, but no more than 2 GB.
For example, for 2 GB of RAM, set the value as follows in the /etc/system file:
To determine a host machine’s memory, use this command:
prtconf | grep Memory
On Solaris 8 or earlier, set shminfo_shmseg, the maximum number of shared memory segments to which one process can attach. Set the value to six times the number of nodes per host. For up to six nodes per host, add the following to the /etc/system file:
On Solaris 9 and later, shmsys:shminfo_shmseg is obsolete.
Set shminfo_shmmni, the maximum number of shared memory segments in entire system. Since each HADB node allocates six shared memory segments, the value required by HADB must be at least six times the number of nodes per host. On Solaris 9, for up to six nodes per host, there is no need to change the default value.
Check the /etc/system file for the following semaphore configuration entries, for example:
set semsys:seminfo_semmni=10 set semsys:seminfo_semmns=60 set semsys:seminfo_semmnu=30
If the entries are present, increment the values as indicated below.
If the /etc/system file does not these entries, add them at the end of the file:
Set seminfo_semmni, the maximum number of semaphore identifiers. Each HADB node needs one semaphore identifier. On Solaris 9, for up to six nodes per host, there is no need to change the default value. For example:
Set seminfo_semmns, the maximum number of semaphores in the entire system. Each HADB node needs eight semaphores. On Solaris 9, or up to six nodes per host, there is no need to change the default value. For example:
Set seminfo_semmnu, the maximum number of undo structures in the system. One undo structure is needed for each connection (configuration variable NumberOfSessions, default value 100). For up to six nodes per host, set it to 600:
Reboot the machine.
On Linux, you must configure shared memory settings. You do not need to adjust the default semaphore settings.
Log in as root.
Edit the file /etc/sysctl.conf.
With Redhat Linux, you can also modify sysctl.conf to set the kernel parameters.
Set the values of kernel.shmax and kernel.shmall, as follows:
echo MemSize > /proc/sys/shmmax echo MemSize > /proc/sys/shmall
where MemSize is the number of bytes.
The kernel.shmax parameter defines the maximum size in bytes for a shared memory segment. The kernel.shmall parameter sets the total amount of shared memory in pages that can be used at one time on the system. Set the value of both of these parameters to the amount physical memory on the machine. Specify the value as a decimal number of bytes.
For example, to set both values to 2GB, use the following:
echo 2147483648 > /proc/sys/kernel/shmmax echo 2147483648 > /proc/sys/kernel/shmall
Reboot the machine using this command:
sync; sync; reboot
Windows does not require any special system settings. However, if you want to use an existing J2SE installation, set the JAVA_HOME environment variable to the location where the J2SE is installed.
You must synchronize clocks on HADB hosts, because HADB uses time stamps based on the system clock. HADB uses the system clock to manage timeouts and to time stamp events logged to history files. For troubleshooting, you must analyze all the history files together, since HADB is a distributed system. So, it is important that all the hosts’ clocks be synchronized
Do not adjust system clocks on a running HADB system. Doing so can cause problems in the operating system or other software components that can in turn cause problems such as hangs or restarts of HADB nodes. Adjusting the clock backward can cause some HADB server processes to hang as the clock is adjusted.
To synchronize clocks:
On Solaris, use xntpd (network time protocol daemon).
On Linux, use ntpd.
On Windows, use NTPTime on Windows
If HADB detects a clock adjustment of more than one second, it logs it to the node history file, for example:
NSUP INF 2003-08-26 17:46:47.975 Clock adjusted. Leap is +195.075046 seconds.
In general, you can install HADB on the same system as Enterprise Server (co-located topology) or on separate hosts (separate tier topology).
You must install the HADB management client to be able to set up high availability with the asadmin configure-ha-cluster command. When using the Java Enterprise System installer, you must install an entire HADB instance to install the management client, even if the nodes are to be installed on a separate tier.
On a single or dual CPU system, you can install both HADB and Enterprise Server if the system has at least two Gbytes of memory. If not, install HADB on a separate system or use additional hardware. To use the asadmin configure-ha-cluster command , you must install both HADB and Enterprise Server.
Each HADB node requires 512 Mbytes of memory, so a machine needs one Gbyte of memory to run two HADB nodes. If the machine has less memory, set up each node on a different machine. For example, you can install two nodes on:
Two single-CPU systems, each with 512 Mbytes to one Gbyte of memory
A single or dual CPU system with one Gbyte to two Gbytes of memory
You can install HADB with either the Java Enterprise System installer or the Enterprise Server standalone installer. In either installer, choose the option to install HADB (called High Availability Session Store in Java ES) in the Component Selection page. Complete the installation on your hosts. If you are using the Enterprise Server standalone installer, and choose two separate machines to run HADB, you must choose an identical installation directory on both machines.
Throughout this manual, HADB_install_dir represents the directory in which HADB is installed. The default installation directory depends on whether you install HADB as part of the Java Enterprise System. For Java Enterprise System, the default installation directory is /opt/SUNWhadb/4. For the standalone Application Server installer, it is /opt/SUNWappserver/hadb/4 .
The node supervisor processes (NSUP) ensure the availability of HADB by exchanging “I’m alive” messages with each other. The NSUP executable files must have root privileges so they can respond as quickly as possible. The clu_nsup_srv process does not consume significant CPU resources, has a small footprint, and so running it with real-time priority does not affect performance.
The Java Enterprise System installer automatically sets the NSUP privileges properly, so you do not need to take any further action. However, with the standalone Enterprise Server (non-root) installer, you must set the privileges manually before creating a database.
If NSUPs do not have the proper privileges, you might notice symptoms of resource starvation such as:
False network partitioning and node restarts, preceded by a warning “Process blocked for n seconds” in HADB history files.
Aborted transactions and other exceptions.
If NSUP cannot set the real-time priority errno is set to EPERM on Solaris and Linux. On Windows it issues the warning “Could not set real-time priority”. The error is written to the ma.log file, and the process continues without real-time priority.
Setting real-time priorities is not possible when:
HADB is installed in Solaris 10 non-global zones
PRIV_PROC_LOCK_MEMORY (allow a process to lock pages in physical memory) and/or PRIV_PROC_PRIOCNTL privileges are revoked in Solaris 10
Users turn off setuid permission
Users install the software as tar files (the non-root installation option for the Application Server)
Log in as root.
Change your working directory to HADB_install_dir/lib/server.
The NSUP executable file is clu_nsup_srv .
Set the file’s suid bit with this command:
chmod u+s clu_nsup_srv
Set the file’s ownership to root with this command:
chown root clu_nsup_srv
This starts the clu_nsup_srv process as root, and enables the process to give itself realtime priority.
To avoid any security impact, the real-time priority is set immediately after the process is started and the process falls back to the effective UID once the priority has been changed. Other HADB processes run with normal priority.
This section provides the steps for creating a highly available cluster, and testing HTTP session persistence.
This section discusses the following topics:
Install Application Server instances and the Load Balancer Plug-in.
For more information, see the Java Enterprise System Installation Guide (if you are using Java ES) or Sun GlassFish Enterprise Server v2.1.1 Installation Guide (if you are using the standalone Enterprise Server installer).
Create Enterprise Server domains and clusters.
For information on how to create a domain , see Creating a Domain in Sun GlassFish Enterprise Server v2.1.1 Administration Guide. For information on how to create a cluster, see To Create a Cluster.
Install and configure your web server software.
Setup and configure load balancing.
For more information, see Setting Up HTTP Load Balancing.
The management agent, ma, executes management commands on HADB hosts and ensures availability of the HADB node supervisor processes by restarting them if they fail.
You can start the management agent two ways:
As a service, for production use. See Starting the Management Agent as a Service. To ensure availability of the management agent, make sure it is restarted automatically when the system reboots. See Ensuring Automatic Restart of the Management Agent.
As a regular process (in console mode), for evaluation, testing, or development. See Starting the Management Agent in Console Mode.
In each case, the procedures are different depending on whether you are using Java Enterprise System or the standalone Application Server.
Before starting this section, you must have created one or more Enterprise Server clusters. For information on how to create a cluster, see To Create a Cluster.
From the machine on which the Domain Administration Server is running, configure the cluster to use HADB using this command:
asadmin configure-ha-cluster --user admin --hosts hadb_hostname1,hadb_hostname2 [,...] --devicesize 256 clusterName
Replace hadb_hostname1, hadb_hostname2, and so forth, with the host name of each machine where HADB is running, and clusterName with the name of the cluster. For example:
asadmin configure-ha-cluster --user admin --hosts host1,host2,host1,host2 --devicesize 256 cluster1
This example creates two nodes on each machine, which are highly available even in case of HADB failover. Note that the order of the host names following the –hosts option is significant, so the previous example would be different than --hosts host1,host1,host2,host2.
If you are using just one machine, you must provide the host name twice. In production settings, using more than one machine is recommended.
In Admin Console, select the application under Applications > Enterprise Applications. Set Availability Enabled and then click Save.
To restart a cluster in Admin Console, choose Clusters > cluster-name. Click Stop Instances. Once the instances have stopped, click “Start Instances.”
Alternatively, use these asadmin commands:
asadmin stop-cluster --user admin cluster-name asadmin start-cluster --user admin cluster-name
To restart the Web Server, type this Web Server command:
Replace web_server_root with your Web Server root directory and hostname with the name of your host machine.
Delete the Load Balancer configuration:
asadmin delete-http-lb-ref --user admin --config MyLbConfig FirstCluster
asadmin delete-http-lb-config --user admin MyLbConfig
If you created a new Web Server instance you can delete it by:
HADB is designed to provide “always on” service that is uninterrupted by upgrading the software. This section describes how to upgrade to a new version of HADB without taking the database offline or incurring any loss of availability. This is known as an online upgrade.
The following sections describe how to upgrade your HADB installation:
Install new version of HADB.
Register the new HADB version, as described in Registering HADB Packages
Registering the HADB package in the HADB management domain makes it easy to upgrade or change HADB packages. The management agent keeps track of where the software packages are located, as well as the version information for the hosts in the domain. The default package name is a string starting with V and containing the version number of the hadbm program.
Change the package the database uses.
Enter the following command:
hadbm set PackageName=package
where package is the version number of the new HADB package.
Un-register your existing HADB installation as described in Un-registering HADB Packages
If necessary, replace the management agent startup script.
For more information, see Replacing the Management Agent Startup Script
Verify the results, as described in Verifying HADB Upgrade.
(Optional) Remove the binary files for the old HADB version.
After verifying that HADB has been upgraded properly, you can delete the old HADB packages.
Use the hadbm registerpackage command to register the HADB packages that are installed on the hosts in the management domain. HADB packages can also be registered when creating a database with hadbm create.
Before using the hadm registerpackage command, ensure that all management agents are configured and running on all the hosts in the hostlist, the management agent’s repository is available for updates, and no software package is already registered with the same package name.
The command syntax is:hadbm registerpackage --packagepath=path [--hosts=hostlist] [--adminpassword=password | --adminpasswordfile=file] [--agent=maurl] [[package-name]]
The package-name operand is the name of the package.Table 2–1 hadbm registerpackage Options
List of hosts, either comma-separated or enclosed in double quotes and space separated.
Path to the HADB software package.
For example, the following command registers software package v4 on hosts host1, host2, and host3:
hadbm registerpackage --packagepath=hadb_install_dir/SUNWHadb/4.4 --hosts=host1,host2,host3 v4
The response is:
Package successfully registered.
If you omit the --hosts option, the command registers the package on all enabled hosts in the domain.
Use the hadbm unregisterpackage command to remove HADB packages that are registered with the management domain.
Before using the hadbm unregisterpackage command, ensure that:
All management agents are configured and running on all the hosts in the hostlist.
The management agent’s repository is available for updates.
The new HADB package is registered in the management domain
No existing databases are configured to run on the package about to be unregistered.
The command syntax is:
hadbm unregisterpackage --hosts=hostlist [--adminpassword=password | --adminpasswordfile= file] [--agent= maurl] [package-name ]
The package-name operand is the name of the package.
See Registering HADB Packages for a description of the --hosts option. If you omit the --hosts option, the hostlist defaults to the enabled hosts where the package is registered. See Security Options and General Options for a description of other command options.
To un-register software package v4 from specific hosts in the domain:
hadbm unregisterpackage --hosts=host1,host2,host3 v4
The response is:
Package successfully unregistered.
When you install a new version of HADB, you may need to replace the management agent startup script in /etc/init.d/ma-initd. Check the contents of the file, HADB_install_dir/lib/ma-initd. If it is different from the old ma-initd file, replace the old file with the new file.
Follow this procedure to verify that HADB has been properly upgraded:
Confirm the version of the running HADB processes.
Enter the following commands on all HADB nodes to display the HADB version:
where new-path is the path to the new HADB installation.
The results should show the new HADB version number.
Confirm the database is running.
Enter this command:
new-path/bin/hadbm status -n
If the upgrade is successful, the results will show all the HADB nodes in running state.
Ensure that products using HADB have changed their configuration settings to the new HADB path.
Run any upgrade tests for the products using HADB.