This section describes known high availability database (HADB) issues and associated solutions.
Load balancer plug-in healthcheck generates a large number of connection/disconnection at the background (load). For health check purposes, a runDaemonMonitor thread performs connect/disconnect for every Application Server listener. This can lead to connection saturation on Enterprise Server.
A new attribute, monitor-interval-in-seconds, has been developed for the loadbalancer.xml file. This attribute can be used to insert a pause between connect/disconnect events in the case where hundreds of listeners are configure for the load balancer plug-in. Default pause value is 0.
HADB configured with double networks on two subnets works properly on Solaris SPARC. However, due to problems in the operating system or network drivers on some hardware platforms, it has been observed that Solaris x86 and Linux platforms do not always handle double networks properly. This causes the following problems with HADB:
On Linux, some of the HADB processes are blocked when sending messages. This causes HADB node restarts and network partitioning.
On Solaris x86, some problems may arise after a network failure that prevent switching to the other network interface. This does not happen all the time, so it is still better to have two networks than one. These problems are partially solved in Solaris 10.
Trunking is not supported.
HADB does not support double networks on Windows 2003 (ID 5103186).
Creating a new database may fail with the following error, stating that too few shared memory segments are available:
HADB-E-21054: System resource is unavailable: HADB-S-05512: Attaching shared memory segment with key "xxxxx" failed, OS status=24 OS error message: Too many open files.
Verify that shared memory is configured and the configuration is working. In particular, on Solaris 8, inspect the file /etc/system, and check that the value of the variable shmsys:shminfo_shmseg is at least six times the number of nodes per host.
When increasing device or buffer sizes using hadbm set, the management system checks resource availability when creating databases or adding nodes, but does not check if there are sufficient resources available when device or main-memory buffer sizes are changed.
Verify that there is enough free disk/memory space on all hosts before increasing any of the devicesize or buffersize configuration attributes.
It is not possible to register the same software package with the same name with different locations at different hosts; for example:
hadbm registerpackage test --packagepath=/var/install1 --hosts europa11 Package successfully registered. hadbm registerpackage test --packagepath=/var/install2 --hosts europa12 hadbm:Error 22171: A software package has already been registered with the package name test. |
HADB does not support heterogeneous paths across nodes in a database cluster. Make sure that the HADB server installation directory (--packagepath) is the same across all participating hosts.
If running the management agent on a host with multiple network interfaces, the createdomain command may fail if not all network interfaces are on the same subnet:
hadbm:Error 22020: The management agents could not establish a domain, please check that the hosts can communicate with UDP multicast. |
The management agents will (if not configured otherwise) use the "first" interface for UDP multicasts ("first" as defined by the result from java.net.NetworkInterface.getNetworkInterfaces()).
The best solution is to tell the management agent which subnet to use (set ma.server.mainternal.interfaces in the configuration file, e.g., ma.server.mainternal.interfaces=10.11.100.0). Alternatively one may configure the router between the subnets to route multicast packets (the management agent uses multicast address 228.8.8.8).
Before retrying with a new configuration of the management agents, you may have to clean up the management agent repository. Stop all agents in the domain, and delete all files and directories in the repository directory (identified by repository.dr.path in the management agent configuration file). This must be done on all hosts before restarting the agents with a new configuration file.
On Solaris 10 Opteron, starting, stopping or reconfiguring HADB using the hadbm command may fail or hang with one of the following errors:
hadbm:Error 22009: The command issued had no progress in the last 300 seconds. HADB-E-21070: The operation did not complete within the time limit, but has not been cancelled and may complete at a later time. |
This may happen if there are inconsistencies reading/writing to a file (nomandevice) which the clu_noman_srv process uses. This problem can be detected by looking for the following messages in the HADB history files:
n:3 NSUP INF 2005-02-11 18:00:33.844 p:731 Child process noman3 733 does not respond. n:3 NSUP INF 2005-02-11 18:00:33.844 p:731 Have not heard from it in 104.537454 sec. n:3 NSUP INF 2005-02-11 18:00:33.844 p:731 Child process noman3 733 did not start. |
The following workaround is unverified, as the problem has not been reproduced manually. However, running this command for the affected node should solve the problem.
hadbm restartnode --level=clear nodeno dbname |
Note that all devices for the node will be reinitialized. You may have to stop the node before reinitializing it.
When starting on a host running Solaris 8 with several NIC cards installed, if there is a mixture of cards with IPv6 and IPv4 enabled, the management agent may terminate with the exception "IPV6_MULTICAST_IF failed."
Set the environment variable JAVA_OPTIONS to -Djava.net.preferIPv4Stack=true; for example:
export JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true" |
Alternatively, use Solaris 9 or later, which do not exhibit this problem.
There is a bug in the 64-bit version of Red Hat Enterprise Linux 3.0 that makes the clu_trans_srv process end up in an uninterruptible mode when performing asynchronous I/O. This means that kill -9 does not work and the operating system must be rebooted.
Use a 32-bit version of Red Hat Enterprise Linux 3.0.
Capital letters in passwords are converted to lowercase when the password is stored in hadb.
Do not use passwords containing capital letters.
When downgrading to a previous HADB version, the management agent may fail with different error codes.
It is possible to downgrade the HADB database, however the management agent cannot be downgraded if there changes have been made in the repository objects. After a downgrade, you must keep use the management agent from the latest HADB version.
Regarding install/removal of HADB c package (Solaris: SUNWhadbc, Linux: sun-hadb-c) version <m.n.u-p>, the symlink /opt/SUNWhadb/<m> is never touched once it exists. Thus, it is possible that an orphaned symlink will exist.
Delete the symlink before install or after uninstall unless in use.
On Solaris 10, stopping a management agent by using the ma-initd script in a global zone stops the management agent in the local zone as well.
Do not install the management agent both in the global and local zone.
Sometimes, a resource contention problem on the server may cause a management client to become disconnected, When reconnecting, a misleading error message "hadbm:Error 22184: A password is required to connect to the management agent" may be returned.
Sometimes, a resource contention problem on the server may cause a management client to become disconnected, When reconnecting, a misleading error message "hadbm:Error 22184: A password is required to connect to the management agent" may be returned.
Check if there is a resource problem on the server, take proper action (e.g., add more resources), and retry the operation.
Installing with Java Enterprise System (as root) does not permit non-root users to manage HADB.
Always login as root to manage HADB.
Special use interfaces with IP addresses like 0.0.0.0 should not be registered as valid interfaces to be used for HADB nodes in the Management Agent. Registering such interfaces may cause problems if HADB nodes are set up on these interfaces by means of a user issuing a hadbm create command using host names instead of IP addresses. The nodes will then be unable to communicate, causing the create command to hang.
When using hadbm create on hosts with multiple interfaces, always specify the IP addresses explicitly using DDN notation.
On the Windows platform, with certain configurations and loads, there may be a large number of reassembly failures in the operating system. The problem has been seen with configurations of more than twenty nodes when running several table scans (select *) in parallel. The symptoms may be that transactions abort frequently, repair or recovery may take a long time to complete, and there may be frequent timeouts in various parts of the system.
To fix the problem, the Windows registry variable HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters can be set to a value higher than the default 100. It is recommended that you increase this value to 0x1000 (4096). For more information, see. article 811003 from the Microsoft support pages.
Cookies with a path equal to “/” interfere with the cookies of a highly available web application deployed at a context root other than “/” that uses in-memory replication as its persistence type, making it impossible for the highly available web application to maintain any HTTP session state. One common scenario where this may happen is when using the same browser to access both the Admin GUI (which is deployed at “/”) and the highly available web application.
Access the web application deployed at “/” from a different browser.
SASL32.DLL and ZLIB.DLL are required files for Load Balancer to work with Windows IIS 6. These files are currently not available under as-install/lib.
Copy the two DLL files manually to as-install/lib. These files can be downloaded from:
http://download.java.net/javaee5/external/OS/aslb/jars/ |
Where OS represents the desired platform, and can be one of the following values:
SunOS
SunOS_X86
Linux
WINNT
Two issues arise when installing or uninstalling Enterprise Server with High Availability packages in a Global Zone:
HA packages get installed in all zones, which may not be desirable.
When uninstalling, HA, MQ, JDK packages get removed from all zones, which may not be desirable.
This problem does not occur when installing or uninstalling from a root local zone.
Perform installation and uninstallations from a local root zone rather than a global zone.
Highly available web applications deployed at “/” are unable to maintain any HTTP sessions when using in-memory replication as their persistence type.
Deploy highly available web applications that use in-memory replication as their persistence type to a context root other than “/”. If you want to make such a web application available at “/”, you may designate it as the default-web-module of the virtual server to which the web application has been deployed.
During Enterprise Server Load Balancer installation for Apache on Solaris, the installer updates LD_LIBRARY_PATH in the apachectl script. However, the installer does not correctly write the /usr/lib/mps path. On Solaris, the Apache security instance will not start without this path in LD_LIBRARY_PATH.
This issue exists only on Solaris platforms. To work around the issue, add /opt/SUNWappserver/appserver/lib/lbplugin/lib to your LD_LIBRARY_PATH.
The Enable LoadBalance button is always enabled on the Clustered/Instance general page, regardless of what is saved in domain.xml.
For clustered instances, select Instances tab, and then click the Quiesce action from the table pull-down.
For standalone instances, make sure the instance is running, and then click the Quiesce button on instance General screen.
(Solaris only) After installing Enterprise Server 2.1 on SPARC Solaris 10 with HADB, you may receive the following error after starting Enterprise Server and then attempting to install JES 5 UR1 with Registry Server:
Dependency Error: Installation can not proceed because the version of HA Session Store 4.4.3 detected on this host is incomplete , and a compatible version is required by Servervice Registry Deployment Support. |
It is not possible to install Registry Server from JES 5 UR1 with Enterprise Server 2.1 IFR on Solaris machines. The Registry Server packages have to be installed manually using the pkgadd command from the following JES5 UR1 distribution directory:
path/OS/Products/registry-svr/Packages |
(Internet Explorer 6 and 7 only) When attempting to export the Load Balancer configuration file (loadbalancer.xml) from Internet Explorer 6 or 7, the browser displays an error message saying that the sun-loadbalancer_1_2.dtd DTD file cannot be located.
To save the file, use the following workaround:
Click Export on the Load Balancer page in Internet Explorer.
The “XML page cannot be displayed” message is displayed.
Click the error frame, and then choose File->Save As from the Internet Explorer.
Save the loadbalancer.xml file to the directory of your choice.