High Availability

This section describes known high availability database (HADB) issues and associated solutions.

HADB Configuration with Double Networks. (No ID)

HADB configured with double networks on two subnets works properly on Solaris SPARC. However, due to problems in the operating system or network drivers on some hardware platforms, it has been observed that Solaris x86 and Linux platforms do not always handle double networks properly. This causes the following problems with HADB:

• On Linux, some of the HADB processes are blocked when sending messages. This causes HADB node restarts and network partitioning.

• On Solaris x86, some problems may arise after a network failure that prevent switching to the other network interface. This does not happen all the time, so it is still better to have two networks than one. These problems are partially solved in Solaris 10.

• Trunking is not supported.

• HADB does not support double networks on Windows 2003 (ID 5103186).

HADB Database Creation Fails. (No ID)

Creating a new database may fail with the following error, stating that too few shared memory segments are available:

Description

HADB-E-21054: System resource is unavailable: HADB-S-05512: Attaching shared memory segment with key "xxxxx" failed, OS status=24 OS error message: Too many open files.

Solution

Verify that shared memory is configured and the configuration is working. In particular, on Solaris 8, inspect the file /etc/system, and check that the value of the variable shmsys:shminfo_shmseg is at least six times the number of nodes per host.

Shared memory segments locked and cannot be paged out. (ID 5052548)

Description

HADB 4.3-0.16 and later is configured to use Intimate Shared Memory when it creates and attaches to its shared memory segments (uses the SHM_SHARE_MMU flag). The use of this flag essentially locks the shared memory segments into physical memory and prevents them from being paged out. This can easily cause problems with installations on low end machines.

Therefore if a developer has a machine with 512MB of memory and plenty of swap space available when using Application Server7.0 EE, and then installed 7.1 EE or later, he or she will encounter problems configuring the default clsetup cluster (which creates two HADB nodes, each with a devicesize of 512, which results in there not being enough physical RAM to support the shared memory that both nodes require.

Solution

Make sure you have the recommended amount of memory when co-locating Application Server and HADB. See HADB Requirements and Supported Platforms for more information.

hadbm set does not check resource availability (disk and memory space). (ID 5091280)

Description

When increasing device or buffer sizes using hadbm set, the management system checks resource availability when creating databases or adding nodes, but does not check if there are sufficient resources available when device or main-memory buffer sizes are changed.

Solution

Verify that there is enough free disk/memory space on all hosts before increasing any of the devicesize or buffersize configuration attributes.

Heterogeneous paths for packagepath not supported. (ID 5091349)

Description

It is not possible to register the same software package with the same name with different locations at different hosts; for example:

hadbm registerpackage test --packagepath=/var/install1 --hosts europa11
Package successfully registered.
hadbm registerpackage test --packagepath=/var/install2 --hosts europa12
hadbm:Error 22171: A software package has already been registered with 
the package name test.

Solution

HADB does not support heterogeneous paths across nodes in a database cluster. Make sure that the HADB server installation directory (--packagepath) is the same across all participating hosts.

createdomain may fail. (IDs 6173886, 6253132)

Description

If running the management agent on a host with multiple network interfaces, the create domain command may fail if not all network interfaces are on the same subnet:

hadbm:Error 22020: The management agents could not establish a 
domain, please check that the hosts can communicate with UDP multicast.

The management agents will (if not configured otherwise) use the “first” interface for UDP multicasts (“first” as defined by the result from java.net.NetworkInterface.getNetworkInterfaces()).

Solution

The best solution is to tell the management agent which subnet to use (set ma.server.mainternal.interfaces in the configuration file, e.g., ma.server.mainternal.interfaces=10.11.100.0). Alternatively one may configure the router between the subnets to route multicast packets (the management agent uses multicast address 228.8.8.8).

Before retrying with a new configuration of the management agents, you may have to clean up the management agent repository. Stop all agents in the domain, and delete all files and directories in the repository directory (identified by repository.dr.path in the management agent configuration file). This must be done on all hosts before restarting the agents with a new configuration file.

Directories need to be cleaned up after deleting an HADB instance. (ID 6190878)

Description

After deleting an HADB instance, subsequent attempts to create new instances with the configure-ha-cluster command fail. The problem is that old directories are left from the original HADB instance in ha_install_dir/rep/* and ha_install_dir/config/hadb/instance_name.

Solution

Be sure to manually delete these directories after deleting an HADB instance.

Starting, stopping, and reconfiguring HADB may fail or hang. (IDs 6230792, 6230415)

Description

On Solaris 10 Opteron, starting, stopping or reconfiguring HADB using the hadbm command may fail or hang with one of the following errors:

hadbm:Error 22009: The command issued had no progress in the last 
300 seconds.
HADB-E-21070: The operation did not complete within the time limit, 
but has not been cancelled and may complete at a later time.

This may happen if there are inconsistencies reading/writing to a file (nomandevice) which the clu_noman_srv process uses. This problem can be detected by looking for the following messages in the HADB history files:

n:3 NSUP INF 2005-02-11 18:00:33.844 p:731 Child process noman3 733 
does not respond.
n:3 NSUP INF 2005-02-11 18:00:33.844 p:731 Have not heard from it in 
104.537454 sec.
n:3 NSUP INF 2005-02-11 18:00:33.844 p:731 Child process noman3 733 
did not start.

Solution

The following workaround is unverified, as the problem has not been reproduced manually. However, running this command for the affected node should solve the problem.

hadbm restartnode --level=clear nodeno dbname

Note that all devices for the node will be reinitialized. You may have to stop the node before reinitializing it.

The management agent terminates with the exception "IPV6_MULTICAST_IF failed". (ID 6232140)

Description

When starting on a host running Solaris 8 with several NIC cards installed, if there is a mixture of cards with IPv6 and IPv4 enabled, the management agent may terminate with the exception "IPV6_MULTICAST_IF failed."

Solution

Set the environment variable JAVA_OPTIONS to -Djava.net.preferIPv4Stack=true; for example:

export JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true"

Alternatively, use Solaris 9 or later, which do not exhibit this problem.

clu_trans_srv cannot be interrupted. (ID 6249685)

Description

There is a bug in the 64-bit version of Red Hat Enterprise Linux 3.0 that makes the clu_trans_srv process end up in an uninterruptible mode when performing asynchronous I/O. This means that kill -9 does not work and the operating system must be rebooted.

Solution

Use a 32-bit version of Red Hat Enterprise Linux 3.0.

hadbm does not support passwords containing capital letters. (ID 6262824)

Description

Capital letters in passwords are converted to lowercase when the password is stored in hadb.

Solution

Do not use passwords containing capital letters.

Downgrading from HADB Version 4.4.2.5 to HADB Version 4.4.1.7 causes ma to fail with different error codes. (ID 6265419)

Description

When downgrading to a previous HADB version, the management agent may fail with different error codes.

Solution

It is possible to downgrade the HADB database, however the management agent cannot be downgraded if there changes have been made in the repository objects. After a downgrade, you must keep use the management agent from the latest HADB version.

Install/removal and symlink preservation. (ID 6271063)

Description

Regarding install/removal of HADB c package (Solaris: SUNWhadbc, Linux: sun-hadb-c) version <m.n.u-p>, the symlink /opt/SUNWhadb/<m> is never touched once it exists. Thus, it is possible that an orphaned symlink will exist.

Solution

Delete the symlink before install or after uninstall unless in use.

Management agents in global and local zones may interfere. (ID 6273681)

Description

On Solaris 10, stopping a management agent by using the ma-initd script in a global zone stops the management agent in the local zone as well.

Solution

Do not install the management agent both in the global and local zone.

hadbm/ma should give a better error message when a session object has timed out and deleted at MA. (ID 6275103)

Description

Sometimes, a resource contention problem on the server may cause a management client to become disconnected. When reconnecting, a misleading error message "hadbm:Error 22184: A password is required to connect to the management agent" may be returned.

Solution

Check if there is a resource problem on the server, take proper action (e.g., add more resources), and retry the operation.

Non-root users cannot manage HADB. (ID 6275319)

Description

Installing with Java Enterprise System (as root) does not permit non-root users to manage HADB.

Solution

Always login as root to manage HADB.

The Management Agent should not use special-use interfaces. (ID 6293912)

Description

Special use interfaces with IP addresses like 0.0.0.0 should not be registered as valid interfaces to be used for HADB nodes in the Management Agent. Registering such interfaces may cause problems if HADB nodes are set up on these interfaces by means of a user issuing a hadbm create command using host names instead of IP addresses. The nodes will then be unable to communicate, causing the create command to hang.

Solution

When using hadbm create on hosts with multiple interfaces, always specify the IP addresses explicitly using DDN notation.

Reassembly failures on Windows. (ID 6291562)

Description

On the Windows platform, with certain configurations and loads, there may be a large number of reassembly failures in the operating system. The problem has been seen with configurations of more than twenty nodes when running several table scans (select *) in parallel. The symptoms may be that transactions abort frequently, repair or recovery may take a long time to complete, and there may be frequent timeouts in various parts of the system.

Solution

To fix the problem, the Windows registry variable HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters can be set to a value higher than the default 100. It is recommended that you increase this value to 0x1000 (4096). For more information, see. article 811003 from the Microsoft support pages.

When running hadbm start <db_name>, part of the inputted password is displayed without being masked. (ID 6303581, 6346059, 6307497)

Description

It is possible when a machine is under load that the masking mechanism fails and some characters from the password being entered are exposed. This poses a minor security risk, and the password should always be masked.

Solution

Put the passwords in their own password files (the method normally recommended since Application Server 8.1) and refer to these with either the --adminpassword or --dbpasswordfile options.

JES5 HADB installed in Global Zone not accessible from Sparse Local Zones (ID 6460979)

Description

When the Application Server is installed in a Solaris Global Zone to /usr/SUNWappserver, the HADB component installed with that Application Server instance will not be available in Sparse Local Zones.

The problem is that HADB is installed to /opt/SUNWhadb in the Global Zone, but this directory is not readable from Sparse Local Zones. Unfortunately, the HADB bundle in JES5 is not relocateable.

Solution

Because the Application Server HADB component is not relocatable, the HADB component must be installed separately in each Sparse Local Zone from which you want to access HADB.