![]() |
|
Sun Java System Application Server Enterprise Edition 8.1 2005Q1 High Availability Administration Guide |
Chapter 3
Administering High Availability DatabaseThis chapter describes the high availability database (HADB) in the Sun Java System Application Server Enterprise Edition environment. It explains how to configure and administer the HADB. Before you can create and administer the HADB, you must first determine the topology of your systems and install the HADB software on the various machines.
This chapter discusses the following topics:
Using the HADB Management AgentThe management agent, ma, executes management commands on HADB hosts. The management agent also ensures availability of the HADB node supervisor processes by restarting them if they fail.
Management Agent Command Syntax
The syntax of the management agent ma command is:
ma [common-options] config-file
Where:
- common-options is one or more of the common options described in Table 3-1.
- config-file is the full path to the management agent configuration file. For more information, see Customizing Management Agent Configuration.
Table 3-1 Management Agent Common Options
Option
Description
Default
--define name=value
-DAssign value to property name, where property is one of the properties defined in Table 3-2. This option can be repeated multiple times.
None
--help
-?Display help information.
False
--javahome path
-jUse Java Runtime Environment (1.4 or later) located at path.
None
--systemroot path
-yPath to the operating system root as normally set in %SystemRoot%.
None
--version
-VDisplay version information.
False
Customizing Management Agent Configuration
HADB includes a configuration file that you can use to customize the management agent settings. When you start the management agent without specifying a configuration file, it uses default values. If you specify a configuration file, the management agent will use the settings in that file. You can re-use the configuration file on all hosts in a domain.
To customize the management agent configuration on each HADB host, follow this procedure:
Configuration File
The management agent configuration file is installed to the following location:
- Java Enterprise System. All the entries in the file are commented out. No changes are required to use the default configuration. To customize the management agent configuration, remove the comments from the file, change the values as desired, then start the management agent specifying the configuration file as an argument.
- Standalone installer
Table 3-2 describes the settings in the configuration file.
Starting the Management Agent
You can start the management agent two ways:
- As a service. In a production environment, start it as a service to ensure its availability. See Starting the Management Agent as a Service.
- As a regular process (in console mode), for testing or development. See Starting the Management Agent in Console Mode.
In each case, the procedures are different depending on whether you are using Java Enterprise System or the standalone Application Server.
Starting the Management Agent as a Service
For a production deployment, start the management agent as a service to ensure its availability in case the ma process fails or the operating system reboots.
Procedure for Java Enterprise System
This section describes how to start the management agent as a service when using Java Enterprise System.
This section assumes you have a basic understanding of operating system initialization and runlevels. For information on these topics, see your operating system documentation.
To check the default runlevel of your system, inspect the file /etc/inittab, and look for a line near the top similar to this:
id:5:initdefault:
This example shows a default runlevel of 5.
The following procedure ensures the management agent starts only when the system enters:
Entering other runlevels stops the management agent.
To start the management agent as a service:
To deactivate automatic start and stop of the agent, remove the links or change the letters K and S in the link names to lowercase.
Procedure for Standalone Application Server
This section describes how to start the management agent as a service when using the standalone Application Server.
To start the management agent as a service:
- In a shell, change your current directory to HADB_install_dir/lib.
- Edit the shell script ma-initd and replace the default values of HADB_ROOT and HADB_MA_CFG in the script to reflect your installation:
- HADB_ROOT is the HADB installation directory, HADB_install_dir.
- HADB_MA_CFG is the location of the management agent configuration file. For more information, see Customizing Management Agent Configuration.
- Copy ma-initd to the directory /etc/init.d
- Create the following soft links to the file /etc/init.d/ma-initd:
/etc/rc0.d/K20ma-initd
/etc/rc1.d/K20ma-initd
/etc/rc2.d/K20ma-initd
/etc/rc3.d/S99ma-initd
/etc/rc5.d/S99ma-initd
/etc/rcS.d/K20ma-initd
This procedure ensures the management agent starts only when the system enters runlevel 3 on Solaris or runlevel 5 on RedHat Linux. For a brief discussion of runlevels, see Procedure for Java Enterprise System.
To deactivate automatic start and stop of the agent, remove the links or change the letters K and S in the link names to lowercase.
Starting the Management Agent in Console Mode
You may wish to start the management agent manually in console mode for evaluation or testing. Do not start the management agent this way in a production environment, because the ma process will not restart after a system or process failure and will terminate when the command window is closed.
Java Enterprise System
To start the HADB management agent in console mode, use the command:
opt/SUNWhadb/bin/ma [config-file]
The default management agent configuration file is /etc/opt/SUNWhadb/mgt.cfg
To stop the management agent, kill the process or close the shell window.
Standalone Application Server
To start the HADB management agent in console mode, use the command:
HADB_install_dir/bin/ma [config-file]
The default management agent configuration file is HADB_install_dir/bin/ma.cfg
To stop the management agent, kill the process or close the shell window.
Using the hadbm Management CommandUse the hadbm command-line utility to manage an HADB domain, its database instances, and nodes. The hadbm utility (also called the management client) sends management requests to the specified management agent, acting as a management server, which has access to the database configuration from the repository.
This section describes the hadbm command-line utility, with the following topics:
Command Syntax
The hadbm utility is located in the HADB_install_dir/bin directory. The general syntax of the hadbm command is:
hadbm subcommand [-short-option [option-value]] [--long-option [option-value]] [operands]
The subcommand identifies the operation or task to perform. Subcommands are case-sensitive. Most commands have one operand (usually dbname), but some have none, and some have two.
Options modify how hadbm performs a subcommand. Options are case-sensitive. Each option has a long form and a short form. Precede the short form with a single dash (-); precede the long forms with two dashes (--). Most options require argument values, except for boolean options, which must be present to switch a feature on. Options are not required for successful execution of the command.
If a subcommand requires a database name, and you do not specify one, hadbm will use the default database, hadb.
For example, the following illustrates the status subcommand:
hadbm status --nodes
Security Options
For security reasons, all hadbm commands require an administrator password. Use the --adminpassword option to set the password when you create a database or domain. From then on, you must specify that password when you perform operations on the database or domain.
For enhanced security, use the --adminpasswordfile option to specify a file containing the password, instead of entering it on the command line. Define the password in the password file with the following line:
HADBM_ADMINPASSWORD=password
Replace password with the password. Any other content in the file is ignored.
If you specify both the --adminpassword and --adminpasswordfile options, the --adminpassword takes precedence. If a password is required, but is not specified in the command, hadbm prompts you for a password.
Note
You can change the administrator password, if required, using the command setadminpassword. For more information about the command, see the manpage for the command setadminpassword.
In addition to the administrator password, HADB also requires a database password to perform operations that modify the database schema. You must use both passwords when using the following commands: hadbm create, hadbm addnodes, and hadbm refragment.
Specify the database password on the command line with the --dbpassword option. Similar to the administrator password, you can also put the password in a file and use the --dbpasswordfile option, specifying the file location. Set the password in the password file with the following line:
HADBM_DBPASSWORD=password
For testing or evaluation, you can turn off password authentication with the --no-adminauthentication option when you create a database or domain. For more information, see Creating a Database and Creating a Management Domain.
Table 3-3 summarizes the hadbm security command line options.
General Options
General command options can be used with any hadbm subcommand. All are boolean options that are false by default. Table 3-4 describes the hadbm general command options.
Environment Variables
For convenience, you can set an environment variable instead of specifying a command option. Table 3-5 describes environment variables that correspond to hadbm command options.
Configuring HADBThis section describes the following basic HADB configuration tasks:
Creating a Management Domain
The command hadbm createdomain creates a management domain containing the specified HADB hosts. The command initializes internal communication channels between hosts and the persistence configuration store.
The syntax of the command is:
hadbm createdomain
[--adminpassword=password |--adminpasswordfile=file | --no-adminauthentication] [--agent=maurl]
hostlistThe hostlist operand is a comma-separated list of HADB hosts, each of which is a valid IPv4 network address. Include all the hosts that you want to be in the new domain in the hostlist.
See Table 3-4 for a description of the command options.
Before using this command, be sure an HADB management agent is running on every host in the hostlist. Additionally, the management agents must:
After hadbm creates the management domain, it enables all the hosts in the domain. Then the management agents are ready to manage databases. After creating HADB domains, the next step is to create the HADB database. For more information on creating HADB databases, see Creating a Database.
Example of Creating an HADB Management Domain
The following example creates a management domain on the four specified hosts:
hadbm createdomain --adminpassword=password host1,host2,host3,host4
After hadbm successfully executes the command, you will see the message:
Domain host1,host2,host3, host4 created.
After creating HADB domains, register the path and version of the HADB packages with the management agents.
Creating a Database
Use the hadbm create command to create a database manually. Before you use this command to create a database:
- Create the management domain. See Creating a Management Domain for more information.
- Register the HADB package. See Registering HADB Packages for more information.
If you have not performed the first two steps when you run hadbm create, it implicitly creates a domain and registers the HADB package. Although this might seem like less work, failures in any of the commands can make debugging difficult. Besides, hadbm create is not atomic, that is, if one of the implicit commands fails, the commands that executed successfully will not be rolled back. For example, if hadbm createdomain and hadbm registerpackage execute successfully but hadbm create database fails, the changes made by hadbm createdomain and hadbm registerpackage will persist.
Therefore, it is best to create the database only after creating the domain and registering the HADB package.
The command syntax is:
hadbm create
[--package=name]
[--packagepath=path]
[--historypath=path]
[--devicepath=path]
[--datadevices=number ]
[--portbase=number]
[--spares=number]
[--set=attr-val-list]
[--agent=maurl]
[--no-cleanup]
[--no-clear]
[--devicesize=size]
[--dbpassword=password | --dbpasswordfile=file ]
[--adminpassword=password |--adminpasswordfile=file | --no-adminauthentication]
--hosts=host list [dbname]The dbname operand specifies the database name, which must be unique. To make sure the database name is unique, use the hadbm list command to list existing database names. Use the default database name unless you need to create multiple databases. For example, to create multiple clusters with independent databases on the same set of HADB machines, use a separate database name for each cluster.
The hadbm create command writes error messages to the console, not log files.
Table 3-6 describes the special hadbm create command options. See Table 3-4 for a description of additional command options.
Table 3-6 hadbm create Options
Option
(Short Form)Description
Default
--datadevices
=number-a
Number of data devices on each node, between one and eight inclusive. Data devices are numbered starting at 0.
1
--devicepath=path
-d
Path to the devices. There are four devices:
This path must exist and be writable. To set this path differently for each node or each device, see Setting Heterogeneous Device Paths.
Solaris and Linux: /var/opt/SUNWhadb
Default is specified by ma.server.dbdevicepath in management agent configuration file. For more details, see Configuration File Settings.
--devicesize=size
-z
Device size for each node. For more information, see Specifying Device Size.
Increase the device size as described in Adding Storage Space to Existing Nodes.
1024MB
Maximum size is lesser of maximum operating system file size or 256 GB. Minimum size is:
(4 x LogbufferSize + 16MB) / n
Where n is the number of data devices given by the option --datadevices.
--historypath
=path-t
Path to the history files. This path must already exist and be writable.
For more information on history files, see Clearing and Archiving History Files.
Default is specified by ma.server.dbhistorypath in management agent configuration file. For more details, see Configuration File Settings.
Solaris and Linux:
/var/opt/SUNWhadb
--hosts=hostlist
-H
Comma-separated list of host names or IP addresses (IPv4 only) for the nodes in the database. Use IP addresses to avoid dependence on DNS lookups. Host names must be absolute. You cannot use localhost or 127.0.0.1 as a host name. Host names
See Specifying Hosts for more information.
None
--package=name
-kName of the HADB package (version). If the package is not found, a default package is registered.
This option is deprecated. Use the hadbm registerpackage command to register a package in the domain.
None
--packagepath=path
-L
Path to the HADB software package. Use only if the package is not registered in the domain.
This option is deprecated. Use the hadbm registerpackage command to register a package in the domain.
None
--portbase
=number-b
Port base number used for node 0. Successive nodes are automatically assigned port base numbers in steps of 20 from this number. Each node uses its port base number and the next five consecutively numbered ports.
To run several databases on the same machine, have a plan for allocating port numbers explicitly.
15200
--spares=number
-s
Number of spare nodes. This number must be even and must be less than the number of nodes specified in the --hosts option.
0
--set=attr-val-list
-S
Comma-separated list of database configuration attributes in name=value format. For explanations of database configuration attributes, see Clearing and Archiving History Files.
None
Specifying Hosts
Use the --hosts option to specify a comma-separated list of host names or IP addresses for the nodes in the database. The hadbm create command creates one node for each host name (or IP address) in the list. The number of nodes must be even. Use duplicate host names to create multiple nodes on the same machine with different port numbers. Make sure that nodes on the same machine are not mirror nodes.
Nodes are numbered starting at zero in the order listed in this option. The first mirrored pair are nodes zero (0) and one (1), the second two (2) and three (3), and so on. Odd numbered nodes are in one DRU, even numbered nodes in the other. With --spares option, spare nodes are those with the highest numbers.
For information about configuring double network interfaces, see Configuring Network Redundancy.
Specifying Device Size
Specify the device size using the --devicesize option. The recommended device size is:
(4x/nd + 4l/d) / 0.99
Where
If refragmentation might occur (for example, using hadbm addnodes), then the recommended device size is:
(8x/nd + 4l/d) / 0.99
Setting Heterogeneous Device Paths
To set a different device path for each node or service, use the --set option of hadbm create. There are four types of devices: the DataDevice, the NiLogDevice (node internal log device), the RelalgDevice (relational algebra query device), and the NoManDevice (node manager device). The syntax for each name=value pair is as follows, where -devno is required only if the device is DataDevice:
node-nodeno.device-devno.Devicepath
For example:
--set Node-0.DataDevice-0.DevicePath=/disk0,
Node-1.DataDevice-0.DevicePath=/disk 1You can also set a heterogeneous path to history files, as follows:
node-nodeno.historypath=path
For information on history files, see Clearing and Archiving History Files.
Any device path that is not set for a particular node or device defaults to the --devicepath value.
Example of Creating a Database
The following command is an example of creating a database:
hadbm create --spares 2 --devicesize 1024 --dbpassword secret123
--hosts n0,n1,n2,n3,n4,n5Troubleshooting
If you have difficulty creating a database, check the following:
- Ensure you have started the management agents on all the hosts and defined an HADB domain. For details, see Starting the Management Agent and Creating a Management Domain.
- File and directory permissions must be set to allow read, write, and execute access to the install, history, device, and config paths for the following users:
For details about setting user permissions, see Preparing for HADB Setup.
Viewing and Modifying Configuration Attributes
You can view and modify database configuration attributes with the hadbm get and hadbm set commands, respectively.
Getting the Values of Configuration Attributes
To get the values of configuration attributes (for a list, see Configuration Attributes), use the hadbm get command. The command syntax is:
hadbm get attribute-list | --all [dbname]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]The dbname operand specifies the database name. The default is hadb.
The attribute-list operand is a comma-separated or quote-enclosed space-separated list of attributes. The --all option displays values for all attributes. For a list of all attributes for hadbm get, see Table 3-7.
See Table 3-4 for a description of command options.
For example:
hadbm get JdbcUrl,NumberOfSessions
Setting the Values of Configuration Attributes
To set the values of configuration attributes (for a list, see Configuration Attributes), use the hadbm set command. The syntax is as follows:
hadbm set [dbname] attribute=value[,attribute=value...]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]The dbname operand specifies the database name. The default is hadb.
The attribute=value list is a comma-separated or quote-enclosed space-separated list of attributes.
See Table 3-4 for a description of command options.
If this command executes successfully, it restarts the database in the state it was in previously, or in a better state. For information about database states, see Getting the Status of HADB. If the command does not execute successfully, restart the HADB as described in Restarting the HADB.
You cannot set the following attributes with hadbm set. Instead, set them when you create a database (see Creating a Database):
- DatabaseName
- DevicePath
- HistoryPath
- NumberOfDatadevices
- Portbase
- JdbcUrl (its value is set during database creation based on the --hosts and --portbase options).
Configuration Attributes
Table 3-7 lists the configuration attributes that you can modify with hadbm set and and retrieve with hadbm get.
Table 3-7 Configuration Attributes
Attribute
Description
Default
Range
ConnectionTrace
If true, records a message in the HADB history files when a client connection (JDBC, ODBC) is initiated or terminated.
False
True or False
CoreFile
Do not change the default value.
False
True or False
DatabaseName
Name of the database.
hadb
DataBufferPoolSize
Size of the data buffer pool allocated in shared memory.
200MB
16 - 2047 MB
DataDeviceSize
Specifies the device size for the node. For information on the recommended DataDeviceSize, see Specifying Device Size.
The maximum value is the smaller of 256GB or the maximum operating system file size. The minimum value is:
(4 x LogbufferSize + 16MB) / n
where n is number of data devices.
1024MB
32 - 262144 MB
PackageName
Name of HADB software package used by the database.
V4.x.x.x
None
DevicePath
Location of the devices. Devices are:
Solaris and Linux: /var/opt/SUNWhadb
EagerSessionThreshold
Determines whether normal or eager idle session expiration is used.
In normal idle session expiration, sessions that are idle for more than SessionTimeout seconds are expired.
When the number of concurrent sessions exceeds the EagerSessionThreshold percentage of the maximum number of sessions, sessions that are idle for more than EagerSessionTimeout seconds are expired.
Half of NumberOfSessions attribute
0 - 100
EagerSessionTimeout
The time in seconds a database connection can be idle before it expires when eager session expiration is used.
120 seconds
0-2147483647 seconds
EventBufferSize
Size of the event buffer, where database events are logged. If set to 0, no event buffer logging is performed.
During failures, the event buffer is dumped. This gives valuable information on the cause of the failures and is useful during trial deployment.
Writing events to memory has a performance penalty.
0 MB
0-2097152 MB
HistoryPath
Location of the HADB history files, which contain information, warnings, and error messages.
This is a read-only attribute.
Solaris and Linux: /var/opt/SUNWhadb
InternalLogbufferSize
Size of the node internal log device, which keeps track of operations related to storing data.
12MB
4 - 128 MB
JdbcUrl
The JDBC connection URL for the database.
This is a read-only attribute.
none
LogbufferSize
Size of the log buffer, which keeps track of operations related to data.
48MB
4 - 2048 MB
MaxTables
Maximum number of tables allowed in an HADB database.
1100
100 - 1100
NumberOfDatadevices
Number of data devices used by an HADB node.
This is a read-only attribute.
1
1 - 8
NumberOfLocks
Number of locks allocated by an HADB node.
50000
20000-1073741824
NumberOfSessions
Maximum number of sessions (database connections) that can be opened for an HADB node.
100
1 - 10000
PortBase
Base port number used to create different port numbers for different HADB processes.
This is a read-only attribute.
15200
10000 - 63000
RelalgDeviceSize
Size of the device used in relational algebra queries.
128 MB
32 - 262144 MB
SessionTimeout
Amount of time a database connection can be idle before it expires when normal session expiration is used.
1800 seconds
0-2147483647 seconds
SQLTraceMode
Amount of information about executed SQL queries written to the history files.
If SHORT, login and logout of SQL sessions are recorded. If FULL, all SQL queries being prepared and executed, including parameter values, are recorded.
NONE
NONE/SHORT/ FULL
StartRepairDelay
Maximum time a spare node allows for a failed active node to perform a node recovery. If the failed node cannot recover within this time interval, the spare node starts copying data from the failed node's mirror and becomes active. Changing the default value is not recommended.
20 seconds
0 - 100000 seconds
StatInterval
Interval at which an HADB node writes throughput and response time statistics to its history file. To disable, set to 0.
Here is an example of a statistics line:
Req-reply time: # 123, min= 69 avg= 1160 max= 9311 %=100.0
The number after the has sign (#) is the number of requests serviced over the StatInterval. The next three numbers are the minimum, average, and maximum time in microseconds taken by transactions completed over the StatInterval. The number afer the percent sign (%) is the number of transactions completed successfully within 15 milliseconds over the StatInterval.
600 seconds
0 - 600 seconds
SyslogFacility
Facility used when reporting to syslog (see man syslog for details). The syslog daemon should be configured (see man syslogd.conf for details).
Use a facility that is not used by other applications running on the same machine.
Set to none to disable syslog logging.
local0
local0, local1, local2, local3, local4, local5, local6, local7, kern, user, mail, daemon, auth, syslog, lpr, news, uucp, cron, none
SysLogging
If true, an HADB node writes information to the operating system's syslog files.
True
True or False
SysLogLevel
Minimum level of HADB message saved to operating system's syslog files. All messages of that level or higher will be logged. For example, "info" logs all messages.
warning
none
alert
error
warning
infoSyslogPrefix
Text string inserted before all syslog messages written by the HADB.
hadb -dbname
TakeoverTime
Time between when a node fails and when its mirror takes over. Do not change the default value.
10000 (milliseconds)
500 - 16000 milliseconds
Configuring the JDBC Connection Pool
Application Server communicates with the HADB using the Java Database Connectivity (JDBC) API. The asadmin configure-ha-cluster command automatically creates a JDBC connection pool for use with HADB (for a cluster cluster-name):
The initial configuration of the connection pool is normally sufficient. When you add a node, change the steady pool size so that there are eight connections for each HADB active node. See Adding Nodes.
For general information about connection pools and JDBC resources, see Administration Guide.
Getting the JDBC URL
Before you can set up the JDBC connection pool, you need to determine the JDBC URL of the HADB using the hadbm get command as follows:
hadbm get JdbcUrl [dbname]
For example:
hadbm get JdbcUrl
This command displays the JDBC URL, which is of he following form:
jdbc:sun:hadb:host:port,host:port,...
Remove the jdbc:sun:hadb: prefix and use the host:port,host:port... part as the value of the serverList connection pool property, described in the next section.
Creating a Connection Pool
The following table summarizes connection pool settings required for the HADB. Change the Steady Pool Size when adding nodes, but do not change other settings.
The following table summarizes connection pool properties required for the HADB. Change serverList when adding nodes, but do not change other properties.
Table 3-9 HADB Connection Pool Properties
Property
Description
username
Name of the storeuser to use in the asadmin create-session-store command.
password
Password (storepassword ) to us in the asadmin create-session-store command.
serverList
JDBC URL of the HADB. To determine this value, see Getting the JDBC URL.
You must change this value if you add nodes to the database. See Adding Nodes.
cacheDatabaseMetaData
When false , as required, ensures that calls to Connection.getMetaData() make calls to the database, which ensures that the connection is valid.
eliminateRedundantEndTransaction
When true , as required, improves performance by eliminating redundant commit and rollback requests and ignoring these requests if no transaction is open.
maxStatement
Maximum number of statements per open connection that are cached in the driver statement pool. Set this property to 20.
Connection Pool Example
Here is an example asadmin create-jdbc-connection-pool command that creates an HADB JDBC connection pool.
asadmin create-jdbc-connection-pool --user adminname --password secret --datasourceclassname com.sun.hadb.jdbc.ds.HadbDataSource --steadypoolsize=32 --isolationlevel=repeatable-read --isconnectvalidatereq=true --validationmethod=meta-data --property username=storename:password=secret456:serverList=host\\:port,host\\:port,host\\:p ort,host\\:port,host\\:port,host\\:port:cacheDatabaseMetaData=false:eliminateRedund antEndTransaction=true hadbpool
On Solaris, escape colon characters (:) within property values with double backslashes (\\).
Creating a JDBC Resource
The following table summarizes JDBC resource settings required for the HADB.
Table 3-10 HADB JDBC Resource Settings
Setting
Description
JNDI Name
The following JNDI name is the default in the session persistence configuration: jdbc/hastore. You can use the default name or a different name.
You must also specify this JNDI name as the value of the store-pool-jndi-name Persistence Store property when you activate the availability service.
Pool Name
Select from the list the name (or ID) of the HADB connection pool used by this JDBC resource. For more information, see Configuring Network Redundancy.
Data Source Enabled
Checked/true
Managing the HADBYou generally need to perform management operations when you replace or upgrade your network, hardware, operating system, or HADB software. The following sections explain various management operations:
Managing Domains
You can perform the following operations on an HADB domain:
- Creating a Domain: For information on the creating a domain, see Creating a Management Domain.
See Table 3-3 and Table 3-4 for a description of command options. For more information on these commands, see the Sun Java Application Server Reference Manual or the corresponding man pages.
Extending a Domain
Use extenddomain to add hosts to an existing management domain. The command syntax is:
hadbm extenddomain
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
hostlistIP addresses of HADB hosts must be IPv4 addresses.
Deleting a Domain
Use deletedomain to remove a management domain. The command syntax is:
hadbm deletedomain
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]Removing Hosts from a Domain
Use reducedomain to remove hosts from the management domain. The command syntax is:
hadbm reducedomain
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
host_listListing Hosts in a Domain
Use listdomain to list all hosts defined in the management domain. The command syntax is:
hadbm listdomain
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]Managing Nodes
You can perform the following operations on individual nodes:
Starting a Node
You might need to manually start an HADB node that was stopped because its host was taken off-line for a hardware or software upgrade or replacement. Also, you might need to manually start a node if it fails to restart for some reason (other than a double failure). For more information on how to recover from double failures, see Clearing the HADB.
In most cases, you should first attempt to start the node using the normal start level. You must use the repair start level if the normal start level fails or times out.
To start a node in the database, use the hadbm startnode command. The syntax is:
hadbm startnode
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[--startlevel=level]
nodeno
[dbname]The dbname operand specifies the database name. The default is hadb.
The nodeno operand specifies the number of the node to start. Use hadbm status to display the numbers of all nodes in a database.
Start level option
This command has one special option, --startlevel (short form -l), that specifies the level at which to start the node.
Node start levels are:
- normal (default): starts the node with the data found locally on the node (in the memory and in the data device file on the disk) and synchronizes it with the mirror for recent updates it missed.
- repair: forces the node to discard local data and copy it from its mirror.
- clear: reinitializes the devices for the node and forces a repair of data from its mirror node. Use when the device files need to be initialized, necessary if they are damaged or the disk that contained the device files is replaced.
See Table 3-4 for a description of other command options.
For example:
hadbm startnode 1
Stopping a Node
You might need to stop a node to repair or upgrade the host machine's hardware or software. To stop a node, use the hadbm stopnode command. The command syntax is:
hadbm stopnode
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[--no-repair]
nodeno
[dbname]The nodeno operand specifies the number of the node to stop. The mirror node of this node number must be running. Use hadbm status to display the numbers of all nodes in a database.
The dbname operand specifies the database name. The default is hadb.
This command has one special option, --no-repair (short form -R) that indicates no spare no is to replace the stopped node. Without this option, a spare node starts up and takes over the functioning of the stopped node.
See Table 3-4 for a description of other command options.
For example:
hadbm stopnode 1
Restarting a Node
You might have to restart a node if you notice unusual behavior such as excessive CPU consumption.
To restart a node in the database, use the hadbm restartnode command. The command syntax is:
hadbm restartnode
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[--startlevel=level]
nodeno
[dbname]The dbname operand specifies the database name. The default is hadb.
The nodeno operand specifies the number of the node to restart. Use hadbm status to display the numbers of all nodes in a database.
This command has one special option, --startlevel (short form -l), that specifies the level at which to start the node. See Start level option for more information.
See Table 3-4 for a description of other command options.
For example:
hadbm restartnode 1
Managing Databases
You can perform the following operations on HADB databases:
Starting the HADB
To start a database, use the hadbm start command. This command starts all nodes that were running before the database was stopped. Individually stopped (offline) nodes are not started when the database is started after a stop.
The command syntax is:
hadbm start
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb.
See Table 3-4 for a description of command options.
For example:
hadbm start
Stopping the HADB
When you stop and start the HADB in separate operations, data is unavailable while the HADB is stopped. To keep data available, you can restart the HADB as described in Restarting the HADB.
Stop the HADB to:
Before stopping the HADB, either stop dependent Application Server instances that are using the HADB database, or configure them to use a Persistence Type other than ha.
When you stop the database, all the running nodes in the database are stopped and the status of the database becomes Stopped. For more information about database states, see Getting the Status of HADB.
To stop a database, use the hadbm stop command. The command syntax is:
hadbm stop
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb.
See Table 3-4 for a description of command options.
For example:
hadbm stop
Restarting the HADB
You might want to restart the HADB if you notice strange behavior (for example consistent timeout problems). In some cases, a restart may solve the problem.
When you restart the HADB, data and database services remain available. When you stop and start the HADB in separate operations, data and database services are unavailable while the HADB is stopped. This is because by default hadbm restart performs a rolling restart of nodes: it stops and starts the nodes one by one. In contrast, hadbm stop stops all nodes simultaneously.
To restart a database, use the hadbm restart command. The command syntax is:
hadbm restart
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[--no-rolling]
[dbname]The dbname operand specifies the database name. The default is hadb.
This command has one special option, --no-rolling (short form -g), that specifies to restart all nodes at once, which causes loss of service. Without this option, this command restarts each of the nodes in the database to the current state or a better state.
See Table 3-4 for a description of other command options.
For example:
hadbm restart
Listing Databases
To list all the databases in the HADB instance, use the hadbm list command. The command syntax is:
hadbm list
[--agent=maurl]
[--adminpassword=password | --adminpasswordfile=file]See Table 3-4 for a description of command options.
Clearing the HADB
Clear the HADB when:
- hadbm status command reveals that the database is non-operational or if multiple nodes do not respond and in the waiting state for a long time. See Getting the Status of HADB.
- Recovering from session data corruption. See Recovering from Session Data Corruption.
The hadbm clear command stops the database nodes, clears the database devices, then starts the nodes. This command erases the Application Server schema data store in HADB, including tables, user names, and passwords. After running hadbm clear, use asadmin configure-ha-cluster to recreate the data schema, reconfigure the JDBC connection pool, and reload the session persistence store.
The command syntax is:
hadbm clear
[--fast]
[--spares=number]
[--dbpassword=password | --dbpasswordfile=file]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb.
The following table describes the special hadbm clear command options. See Table 3-4 for a description of other options.
For example:
hadbm clear --fast --spares=2 --dbpassword secret123
Removing a Database
To remove an existing database from the HADB system, use the hadbm delete command. This command deletes the database's configuration files, device files, and history files, and frees shared memory resources. The database you want to remove must exist and must be stopped. See Stopping the HADB.
The command syntax is:
hadbm delete
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb.
See Table 3-4 for a description of command options.
For example, the command:
hadbm delete
deletes the default database, hadb.
Recovering from Session Data Corruption
The following are indications that session data may be corrupted:
- Error messages appear in the Sun Java System Application Server system log (server.log) every time you try to save the session state.
- Error messages are written to the server log indicating that the session could not be found or could not be loaded during session activation.
- Sessions that are activated after previously being passivated contain empty or incorrect session data.
- When an instance fails, failed-over sessions contain empty or incorrect session data.
- When an instance fails, instances that try to load a failed-over session cause an error in the server log indicating the session could not be found or could not be loaded.
To bring the session store back to a consistent state if you determine that the data has been corrupted, do the following:
- Clear the session store.
- If clearing the session store doesn't work or you continue to see errors in the server log, reinitialize the data space on all the nodes and clear the data in the database. See Clearing the HADB.
- If clearing the database doesn't work, delete and then recreate the database. See Removing a Database and Creating a Database.
Expanding the HADBThere are two reasons to expand your original HADB configuration:
This section describes how you can expand the HADB without shutting down your Application Server cluster or database, in particular:
Also see related information in Maintaining HADB Machines.
Adding Storage Space to Existing Nodes
Add HADB storage space:
- If user transactions repeatedly abort with one of the following error messages:
- If the hadbm deviceinfo command consistently reports insufficient free size. See Getting Device Information.
If there is unused space on the disks on which the HADB nodes reside, or if you added or upgraded the disks with larger capacity, increase the data device size. For information on the recommended data device size, see Specifying Device Size.
Changing the data device size for a database in a FaultTolerant or higher state upgrades the system without loss of data or availability. The database remains in operational during the reconfiguration. Changing device size on a system that is not FaultTolerant or better, causes loss of data. For more information about database states, see Database States.
The command syntax is:
hadbm set DataDeviceSize=size
where size is the data device size in MBytes.
See Table 3-4 for a description of command options.
For example:
hadbm set DataDeviceSize=1024
Adding Machines
You may want to add machines if the HADB requires more processing or storage capacity. To add a new machine on which to run the HADB, install the HADB packages with or without the Sun Java System Application Server as described in Chapter 2, "Installing and Setting Up High Availability Database." For an explanation of node topology alternatives, see the Sun Java System Application Sever Deployment Planning Guide.
To integrate new machines into the existing HADB instance:
- Start management agents on the new nodes.
- Extend the management domain to the new hosts. For details, see hadbm extenddomain command.
- Start the new nodes on these hosts. For more details, see Adding Nodes.
Adding Nodes
Increase processing and storage capacity of the HADB system by creating new nodes and adding them to the database.
After you add nodes, update the following properties of the HADB JDBC connection pool:
To add nodes, use the hadbm addnodes command. The command syntax is:
hadbm addnodes
[--no-refragment]
[--spares=sparecount]
[--historypath=path]
[--devicepath=path]
[--set=attr-name-value-list]
[--dbpassword=password | --dbpasswordfile=file ]
[--adminpassword=password | --adminpasswordfile=file]
--hosts=hostlist
[dbname]The dbname operand specifies the database name. The default is hadb. The database must be in the HA Fault Tolerant or Fault Tolerant state. For more information about database states, see Getting the Status of HADB.
If you do not specify the --devicepath and --historypath options, the new nodes will have the same device path and use the same history files as the existing database.
For example:
hadbm addnodes --dbpassword secret123 -adminpassword=password --hosts n6,n7,n8,n9
The following table describes the special hadbm addnodes command options. See Table 3-4 for a description of other options.
Table 3-12 hadbm addnodes Options
Option
(Short Form)Description
Default
--no-refragment
-r
Do not refragment the database during node creation; In this case, refragment the database later using the hadbm refragment command to use the new nodes. For details about refragmentation, see Refragmenting the Database.
If you do not have sufficient device space for refragmentation, recreate the database with more nodes. See Adding Nodes by Recreating the Database.
Not applicable
--spares=number
-s
Number of new spare nodes in addition to those that already exist. Must be even and not greater than the number of nodes added.
0
--devicepath=path
-d
Path to the devices. Devices are:
This path must already exist and be writable. To set this path differently for each node or each device, see Setting Heterogeneous Device Paths.
Solaris and Linux: HADB_install_dir
/device
--hosts=hostlist
-H
Comma-separated list of new host names for the new nodes in the database. One node is created for each comma-separated item in the list. The number of nodes must be even. IP addresses of HADB hosts must be IPv4 addresses.
Using duplicate host names creates multiple nodes on the same machine with different port numbers. Make sure that nodes on the same machine are not mirror nodes.
Odd numbered nodes are in one DRU, even numbered nodes in the other. If --spares is used, new spare nodes are those with the highest numbers.
If the database was created with double network interfaces, the new nodes must be configured in the same way. See Configuring Network Redundancy.
None
Refragmenting the Database
Refragment the database to store data in newly-created nodes. Refragmentation distributes data evenly across all active nodes. To refragment the database, use the hadbm refragment command. The command syntax is:
hadbm refragment
[--dbpassword=password | --dbpasswordfile=file]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb. The database must be in the HA Fault Tolerant or Fault Tolerant state. For more information about database states, see Getting the Status of HADB.
See Table 3-4 for a description of command options.
For example:
hadbm refragment --dbpassword secret123
Note
The best time to add nodes is when the system is lightly loaded.
Adding nodes performs a refragmentation and redistribution of the existing data to include the new nodes in the system. Online refragmenting requires that the disks for the HADB nodes have enough space to contain the old data and the new data simultaneously until refragmenting is finished, that is, the user data size must not exceed 50% of the space available for user data. For details, see Getting Device Information.
If this command fails after multiple attempts, see Adding Nodes by Recreating the Database.
Adding Nodes by Recreating the Database
If online refragment fails persistently when new nodes are added, either due to lack of data device space or other reasons, recreate the database with new nodes. This will lead to the loss of existing user data, as well as schema data.
To add nodes by recreating the database:
- For each Application Server instance:
- Disable the Application Server instance in the load balancer.
- Disable session persistence.
- Restart the server instance.
- Re-enable the Application Server instance in the load balancer.
If you do not need to maintain availability, you can disable and re-enable all the server instances at once in the load balancer. This saves time and prevents failover of outdated session data.
- Stop the database as described in Stopping the HADB.
- Delete the database as described in Removing a Database.
- Recreate the database with the additional nodes as described in Creating a Database.
- Reconfigure the JDBC connection pool as described in Configuring the JDBC Connection Pool.
- Reload the session persistence store.
- Perform the following tasks for each Application Server instance:
- Disable the Application Server instance in the load balancer.
- Enable session persistence.
- Restart the Application Server instance.
- Re-enable the Application Server instance in the load balancer.
If you do not need to maintain availability, you can disable and re-enable all the server instances at once in the load balancer. This saves time and prevents failover of outdated session data.
Monitoring HADBYou can monitor the activities of HADB by:
These sections briefly describe the hadbm status, hadbm deviceinfo, and hadbm resourceinfo commands. For details about interpreting HADB information, see the Sun Java System Application Server Performance Tuning Guide.
Getting the Status of HADB
Use the hadbm status command to display the status of the database or its nodes. The command syntax is:
hadbm status
[--nodes]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb.
The --nodes option (short form -n) displays information on each node in the database. See Node Status. See Table 3-4 for a description of other command options.
For example:
hadbm status --nodes
Database States
A database's state summarizes its current condition. Table 3-13 describes the possible database states.
Table 3-13 HADB States
Database State
Description
High-Availability Fault Tolerant (HAFaultTolerant)
Database is fault tolerant and has at least one spare node on each DRU.
Fault Tolerant
All the mirrored node pairs are up and running.
Operational
At least one node in each mirrored node pair is running.
Non Operational
One or more mirrored node pairs is missing both nodes.
If the database is non-operational, clear the database as described in Clearing the HADB.
Stopped
No nodes are running in the database.
Unknown
Cannot determine the state of the database.
Node Status
Use the --nodes option to make the hadbm status command display the following information for each node in the database:
- Node number
- Name of the machine where the node is running
- Port number of the node
- Role of the node. For a list of roles and their meanings, see Roles of a Node.
- State of the node. For a list of states and their meanings, see States of a Node.
- Number of the corresponding mirror node.
A node's role and state can change as described in these sections:
Roles of a Node
A node is assigned a role during its creation and can take any one of these roles:
- Active: Stores data and allows client access. Active nodes are in mirrored pairs.
- Spare: Allows client access, but does not store data. After initializing data devices, monitors other data nodes to initiate repair if another node becomes unavailable.
- Offline: Provide no services until their role changes. When placed back online, its role can change back to its former role.
- Shutdown: An intermediate step between active and offline, waiting for a spare node to take over its functioning. After the spare node has taken over, the node is taken offline.
States of a Node
A node can be in any one of the following states:
- Starting: The node is starting.
- Waiting: The node cannot decide its start level and is offline. If a single node is in this state for more than two minutes, stop the node and then start it at the repair level; see Stopping a Node and Starting a Node. If multiple nodes are in this state, clear the database as described in Clearing the HADB.
- Running: The node is providing all services that are appropriate for its role.
- Stopping: The node is in the process of stopping.
- Stopped: The node is inactive. Repair of a stopped node is prohibited.
- Recovering: The node is being recovered. When a node fails, the mirror node takes over the functions of the failed node. The failed node tries to recover by using the data and log records in main memory or on disk. The failed node uses the log records from the mirror node to catch up with the transactions performed when it was down. If recovery is successful, the node becomes active. If recovery fails, the node state changes to repairing.
- Repairing: The node is being repaired. This operation reinitializes the node and copies the data and log records from the mirror node. Repair is more time consuming than recovery.
Getting Device Information
Monitor free space in HADB data (disk storage) devices:
- Routinely, to check the trend in disk space use.
- As part of preventive maintenance: if the user load has increased and you want to resize or scale the database configuration.
- As part of scaling up the database: Before running hadbm addnodes to add new nodes to the system, check whether there is enough device space. Remember, you need around 40-50% free space on the existing nodes to add nodes.
- When you see messages in the history files and server.log file such as
Use the hadbm deviceinfo command to get information about free space in data devices. This command displays the following information for each node of the database:
The command syntax is:
hadbm deviceinfo
[--details]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb.
The --details option displays the following additional information:
See Table 3-4 for a description of other command options.
To determine the space available for user data, take the total device size, then subtract the space reserved for HADB: four times the LogBufferSize + 1% of the device size. If you do not know the size of the log buffer, use the command hadbm get logbufferSize. For example, if the total device size is 128 MB and the LogBufferSize is 24 MB, the space available for user data is 128 - (4 x 24) = 32 MB. Of the 32 MB, half is used for replicated data and around one percent is used for the indices, and only 25 percent is available for the real user data.
The space available for user data is the difference between the total size and reserved size. If the data is refragmented in the future, the free size must be approximately equal to 50% of the space available for user data.If refragmentation is not relevant, the data devices can be exploited to their maximum. Resource consumption warnings are written to the history files when the system is running short on device space.
For more information about tuning HADB, see the Sun Java System Application Server Performance Tuning Guide.
For example, the following command:
hadbm deviceinfo --details
Displays the following example results:
NodeNO Totalsize Freesize Usage NReads NWrites DeviceName
0 128 120 6% 10000 5000 C:\Sun\SUNWhadb\hadb.data.0
1 128 124 3% 10000 5000 C:\Sun\SUNWhadb\hadb.data.1
2 128 126 2% 9500 4500 C:\Sun\SUNWhadb\hadb.data.2
3 128 126 2% 9500 4500 C:\Sun\SUNWhadb\hadb.data.3Getting Runtime Resource Information
The hadbm resourceinfo command displays HADB runtime resource information. You can use this information to help identify resource contention, and reduce performance bottlenecks. For details, see the Sun Java System Application Sever Deployment Planning Guide and the Sun Java System Application Server Performance Tuning Guide.
The command syntax is:
hadbm resourceinfo
[--databuf]
[--locks]
[--logbuf]
[--nilogbuf]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]
[dbname]The dbname operand specifies the database name. The default is hadb.
Table 3-14 describes the hadbm resourceinfo special command options. See Table 3-4 for a description of other command options.
Table 3-14 hadbm resourceinfo Command Options
Option
(Short Form)Description
--databuf
-d
Display data buffer pool information.
See Data Buffer Pool Information below for more information.
--locks
-l
Display lock information.
See Lock Information below for more information.
--logbuf
-b
Display log buffer information.
See Log Buffer Information below for more information.
--nilogbuf
-n
Display node internal log buffer information.
See Node Internal Log Buffer Information below for more information.
Data Buffer Pool Information
Data buffer pool information contains the following:
- NodeNo: Node number.
- Avail: Total space available in the pool, in MBytes.
- Free: Free space available, in MBytes.
- Access: Cumulative number of accesses to the data buffer from database, from start until now.
- Misses: Cumulative number of page faults that have occurred from database start until now.
- Copy-on-Write: Cumulative number of pages copied internally in the data buffer due to checkpointing.
For example:
NodeNO Avail Free Access Misses Copy-on-Write
0 256 128 100000 50000 1000
1 256 128 110000 45000 950When a user transaction performs an operation on a record, the page containing the record must be in the data buffer pool. If it is not, a miss or a page fault occurs. The transaction then has to wait until the page is retrieved from the data device file on the disk.
If the miss rate is high, increase the data buffer pool. Since the misses are cumulative, run hadbm resourceinfo periodically and use the difference between two runs to see the trend of miss rate. Do not be concerned if free space is very small, since the checkpointing mechanism will make new blocks available.
Lock Information
Lock information is as follows:
For example:
NodeNO Avail Free Waits
0 50000 20000 10
1 50000 20000 0One single transaction cannot use more than 25% of the available locks on a node. Therefore, transactions performing operations in large scale should be aware of this limitation. It is best to perform such transactions in batches, where each batch must be treated as a separate transaction, that is, each batch commits. This is needed because read operations running with repeatable read isolation level, and delete, insert, and update operations use locks that are released only after the transaction terminates.
To change the NumberOfLocks, see Clearing and Archiving History Files.
Log Buffer Information
Log buffer information is:
For example:
NodeNO Avail Free
0 16 2
1 16 3Do not worry if free space is very small, since HADB starts compressing the log buffer. HADB starts compression from the head of the ring buffer and performs it on consecutive log records. Compression cannot proceed when HADB encounters a log record that has not been
Node Internal Log Buffer Information
Node internal log buffer information is:
For example:
NodeNO Avail Free
0 16 2
1 16 3
Maintaining HADB MachinesThe HADB achieves fault tolerance by replicating data on mirror nodes. Mirror nodes should be placed on separate DRUs in a production environment as described in Sun Java System Application Sever Deployment Planning Guide.
A failure is an unexpected event such as a hardware failure, power failure, or operating system reboot. The HADB tolerates single failures: of one node, one machine (that has no mirror node pairs), one or more machines belonging to the same DRU, or even one entire DRU. However, the HADB does not automatically recover from a double failure, which is the simultaneous failure of one or more mirror node pairs. If a double failure occurs, you must clear the HADB and recreate its session store, which erases all its data.
Maintaining HADB on a Single Machine
To perform planned or unplanned maintenance on a single machine without interrupting HADB service:
- Perform the maintenance procedure and get the machine up and running.
- Ensure that ma is running: If ma runs under init.d scripts (recommended for deployment), it should have been started by the operating system. If not start it manually.
See Starting the Management Agent.
- Start all nodes on the machine.
See Starting a Node.
- Check whether the nodes are active and running. See Getting the Status of HADB.
To perform planned maintenance, such as hardware or software upgrade, on all HADB machines without interrupting HADB service:
- For each spare machine in the first DRU, repeat the single machine procedure as described in Maintaining HADB on a Single Machine, one by one, for each machine.
- For each active machine in the first DRU, repeat the single machine procedure as described in Maintaining HADB on a Single Machine, one by one, for each machine.
To perform planned maintenance with HADB service interruption on all HADB machines, or when the entire HADB is on a single machine:
- Stop the HADB. See Stopping the HADB.
- Perform the maintenance procedure and get all the machines up and running.
- Ensure ma is running.
- Start the HADB. See Starting the HADB. The data stored in the database before the stop is available again.
To perform unplanned maintenance in the event of a failure, first check the database status. See Getting the Status of HADB.
- If the database state is Operational or better, this means the machines needing unplanned maintenance do not include mirror nodes. Follow the single machine procedure for each failed machine, one DRU at a time. HADB service is not interrupted.
- If the database state is Non-Operational, this means the machines needing unplanned maintenance include mirror nodes. One such case is when the entire HADB is on a single failed machine. Get all the machines up and running first. Then clear the HADB and recreate the session store. See Clearing the HADB. This interrupts HADB service.
Clearing and Archiving History Files
HADB history files record all database operations and error messages. HADB appends on to the end of existing history files, so the files grow over time. To save disk space and prevent files from getting too large, periodically clear and archive history files.
To clear a database's history files, use the hadbm clearhistory command. The command syntax is:
hadbm clearhistory
[--saveto=path]
[dbname]
[--adminpassword=password | --adminpasswordfile=file]
[--agent=maurl]The dbname operand specifies the database name. The default is hadb.
Use the --saveto (short form -o) to specify the directory in which to store the old history files. This directory must have appropriate write permissions. See Table 3-4 for a description of other command options
The --historypath option of the hadbm create command determines the location of the history files. The names of the history files are of the format dbname.out.nodeno. For information on hadbm create, see Creating a Database.
History File Format
Each message in the history file contains the following information:
Messages about resource shortages contain the string "HIGH LOAD."
You do not need a detailed knowledge of all entries in the history file. If for any reason you need to study a history file in greater detail, obtain help from Sun customer support.