| Sun ONE Application Server 7, Enterprise Edition Administrator's Guide |
Chapter 20
Administering the High-Availability DatabaseThis chapter describes the high-availability database (HADB) in the SunTM Open Net Environment (Sun ONE) Application Server 7, Enterprise Edition and explains how to configure and administer the HADB. The tasks described in this chapter fit into the overall steps to set up the HADB, which are summarized in the following table.
Note
If you have not configured rsh or ssh, the hadbm commands described in this chapter will not work. For details about rsh or ssh configuration, see the Sun ONE Application Server Installation Guide and "Setting the Management Protocol".
This chapter contains the following sections:
About the High-Availability DatabaseThis section introduces you to the high-availability database (HADB), which you can use to store persistent HTTPSession information. This section includes these topics:
HADB Architecture
High-availability means availability despite planned outages for upgrades or unplanned outages caused by hardware or software failures. The HADB is based on a simple data model and the Always-On technology. The HADB offers an ideal platform for delivering all types of session state persistence within a high performance enterprise application server environment.
The following figure shows the architecture of a database with four active nodes and two spare nodes. Nodes 0 and 1 are a mirror node pair, as are nodes 2 and 3.
Figure 20-1 HADB Architecture
The HADB achieves high data availability through fragmentation and replication of data. All tables in the database are partitioned to create subsets of approximately the same size called fragments. This process of fragmentation is based on a hash function. This hash function fragments and evenly distributes the data among the database’s nodes. Each fragment is stored twice in the database, in mirror nodes. This ensures fault tolerance and fast recovery of data. In addition, if a node fails or is shut down, a spare node can take over until the node is active again.
HADB nodes are organized into two Data Redundancy Units (DRUs), which mirror each other. Each DRU consists of half of the active and spare nodes, and contains one complete copy of the data. To ensure fault tolerance, the computers that support one DRU must be completely self-supported with respect to power (use of uninterruptible power supplies is recommended), processing units, and storage. If a power failure occurs in one DRU, the nodes in the other DRU can continue servicing requests until the power returns.
Without a session persistence mechanism, the HTTPSession state, including the passivated session state, is lost when a web container fails over to another. Use of the HADB for session persistence overcomes this situation. The HADB stores and retrieves state information in a separate but well-integrated persistent storage tier.
The HADB reclaims space when session data is deleted. The HADB places session data records in fixed size blocks. When all records of a block are deleted, the block is freed. Records of a block can be deleted randomly, creating “holes” in the block. When a new record is to be inserted into a block and contiguous space is needed, the holes are removed and thus the block is compacted.
This is a brief summary of the architecture. For details, see the Sun ONE Application Server System Deployment Guide.
HADB Nodes
A database node consists of a set of processes, a dedicated area of shared memory, and one or more secondary storage devices. It is used for storing and updating session data. Each node must have a mirror node, therefore nodes occur in pairs. In addition, to maximize availability, you should include two or more spare nodes, one in each DRU, so if a node fails a spare can take over while the node is repaired.
For an explanation of node topology alternatives, see the Sun ONE Application Server System Deployment Guide. For more general information about nodes and how to monitor them, see "Node Status".
The hadbm Command
Note
If you have not configured rsh or ssh, the hadbm command will not work. For details about rsh or ssh configuration, see the Sun ONE Application Server Installation Guide and "Setting the Management Protocol".
The hadbm command is located in the install_dir/SUNWhadb/4/bin directory. The commands you use to administer HADB are subcommands of the hadbm command. The general syntax is as follows:
hadbm subcommand [-short-form [argument]]* [--long-form [argument]]* [dbname]
For example, the following is one use of the hadbm status subcommand, which lets you check the status of HADB:
hadbm status --nodes
The subcommand identifies the operation or task you wish to perform. Subcommands are case-sensitive.
Options are also case-sensitive. Each option has a long form and a short form. Short forms have a single dash (-); while long forms have two dashes (--). Options modify how hadbm performs a subcommand. Most options require argument values, except for boolean options, which must be present to switch a feature on. Optional options are enclosed in square brackets [ ] in syntax lines.
For subcommands that take a database name, if a database is not identified, the default database is used. The default database is hadb, all lowercase.
You can set the password for a subcommand from a file instead of entering the password at the command line. The --dbpasswordfile option takes the file containing the passwords. The valid contents for the file are:
HADBM_DBPASSWORD=password
The rest of the contents of the file are ignored. If both the --dbpassword and --dbpasswordfile options are specified, the --dbpassword takes precedence. If a password is required, but is not specified in the command, you are prompted for a password.
The hadbm general command options, which you can use with any hadbm subcommand, are listed in the following table. All are booleans that are not present by default.
Configuring the HADBThis section describes the following basic HADB configuration tasks:
Setting the Management Protocol
Before you create the database, you need to determine whether the HADB uses rsh or ssh for remote execution. Because ssh is the default, you only need to set the management protocol if you need to use rsh instead. Use --set option of the hadbm create command as in the following example:
hadbm create --set managementProtocol=rsh --spares 2 --devicesize 1024 --dbpassword secret123 --hosts n1,n2,n3,n4,n5,n6
The hadbm command uses the rsh or ssh protocol to communicate with the database nodes. The management protocol is not used for database operations.
Creating a Database
The clsetup command creates an HADB database as part of cluster initialization and setup. This is the recommended way of creating a database. For details about clsetup, see the Sun ONE Application Server Installation Guide.
However, you can create a database outside of clsetup. To manually create a database to store the session data, and to start the database if inetd is not used, use the hadbm create command. The syntax is as follows.
hadbm create [--installpath=path] [--historypath=path] [--devicepath=path] [--configpath=path] [--datadevices=devices-per-node] [--portbase=base-no] [--spares=sparecount] [--set=attr-name-value-list] [--inetd] [--inetdsetupdir=path] --devicesize=size --dbpassword=password | --dbpasswordfile=file --hosts=node-list [dbname]
For example:
hadbm create --spares 2 --devicesize 1024 --dbpassword secret123 --hosts n1,n2,n3,n4,n5,n6
If you have difficulty creating a database, check the following:
Database creation errors are written to the Sun ONE Application Server’s log file and to the HADB history file. For more information about the history file, see "Clearing and Archiving History Files". If you still have difficulty creating a database, contact Sun customer support. See "Using Sun Customer Support for the HADB".
The hadbm create command options are listed in the following table.
Table 20-3 hadbm create Options
Long Form
Short Form
Default
Description
--installpath
-n
location of hadbm command
Specifies the HADB system installation path. This path must already exist. Use this option if the HADB server installation resides in a location different from the management client installation.
--historypath
-t
/var/tmp
Specifies the path to the history files. This path must already exist. For details about history files, see "Clearing and Archiving History Files".
--devicepath
-d
/var/opt/SUNWhadb
Specifies the path to the data and log devices. This path must already exist.
--configpath
-c
/etc/opt/SUNWhadb
Specifies the path to the configuration files used internally by the HADB. This path must already exist.
--datadevices
-a
1
Specifies the number of data devices on each node, between 1 and 8 inclusive.
--portbase
-b
15200
Specifies the port base number used for node 0. Successive nodes are automatically assigned port base numbers in steps of 20 from this number. Each node uses its port base number and the next four consecutively numbered ports.
If you want to run several databases on the same machine, you should have a plan for allocating port numbers and allocate them explicitly.
--spares
-s
0
Specifies the number of spare nodes. This number must be even and must be less than the number of nodes specified in the --hosts option. Spare nodes are optional, but having two or more ensures high availability.
--set
-S
none
Specifies a comma-separated list of database configuration attributes. For explanations of valid database configuration attributes, see "Viewing and Modifying Configuration Attributes".
For example, to specify the use of rsh instead of ssh (the default), use the following option:
--set managementProtocol=rsh
--inetd
-I
not specified
If specified, the database is configured to run with the inetd daemon, and is not automatically started after it is created. See "Additional Steps for inetd".
--inetdsetupdir
-u
current directory
Specifies the directory in which to store the inetd setup files. The directory must exist on the machine and must be writable.
--devicesize
-z
none
Specifies the size of the devices per node in MB. The device size should be as large as possible. The maximum size is the maximum operating system file size. The minimum size is derived as follows, based on the default LogbufferSize of 48MB:
4 x LogbufferSize + 16MB = 208MB
You can increase the device size later as described in "Adding Storage Space to Existing Nodes".
For more information on setting the LogbufferSize, see "Viewing and Modifying Configuration Attributes".
--dbpassword
-p
none
Creates a password for the HADB system user. Must be at least 8 characters. You can use --dbpasswordfile instead. For details, see "The hadbm Command".
--dbpasswordfile
-P
none
Specifies a file that stores the password to be created for the HADB system user. For details, see "The hadbm Command".
--hosts
-H
none
Specifies a comma-separated list of host names or IP addresses for the nodes in the database. Using IP addresses is recommended because there is no dependence on DNS lookups. Host names must be absolute. You cannot use localhost or 127.0.0.1 as a host name.
One node is created for each item in the list. The number of nodes must be even. Using duplicate host names creates nodes on the same host with different port numbers.
Nodes are numbered starting at 0 in the order listed in this option. The first mirrored pair are nodes 0 and 1, the second 2 and 3, and so on. Odd nodes are in one DRU, even nodes in the other. If --spares is used, spare nodes are those with the highest numbers.
dbname
none
hadb
Specifies the database name, which must be unique. To make sure the database name is unique, use the hadbm list command to list existing database names.
Use the default database name unless you need to create multiple databases. For example, to create multiple clusters with independent databases on the same set of HADB machines, use a separate database name for each cluster.
Additional Steps for inetd
If you did not specify --inetd when you created the database, the database is initialized and started from the --installpath location, which you can verify using the hadbm status command.
If you specified --inetd, the database exists, as shown by the hadbm list command, but is not running, as shown by hadbm status command.
Using inetd in production environments is recommended, but it is usually not needed in development or test environments. Creating a database using the --inetd option allows you to use inetd to automatically restart HADB nodes on a machine if the machine reboots. This allows for a more robust deployment, but it has its drawbacks in terms of administration.
In particular, if you want to stop a node, for example to perform some kind of maintenance, you must perform the following tasks:
Also, if you add nodes, you must take extra steps to update the inetd configuration files to take into account the new nodes.
Note
If a node and its mirror fail at the same time, you must clear the database as described in "Clearing the HADB". This is called a double-node failure and inetd cannot help in this situation.
To set up inetd for a new database or a database with new nodes, follow these steps:
- Stub files for inetd support are created in the directory specified in the --inetdsetupdir option and are named as follows:
dbname.hostname.inetd.conf
dbname.hostname.services
All stub files are placed locally on the machine where the hadbm create command is typed. Host names are further distinguished by numbers appended to the hostname.
Append the contents of these files to the /etc/inetd.conf and /etc/services files for each node’s machine, then reconfigure inetd as follows:
kill -HUP inetd-process-id
- After inetd has been reconfigured on each machine, you are ready to initialize your database using the hadbm clear command. See "Clearing the HADB".
- You can verify that inetd is working by stopping a node and checking to make sure it restarts. See "Stopping a Node" and "Getting the Status of the HADB".
Setting Up the JDBC Connection Pool
The Sun ONE Application Server communicates with the HADB in the same way that it communicates with relational databases used for data storage, therefore you need to set up a JDBC connection pool for the HADB as you would for any other database.
Using the clsetup command is recommended for configuring a JDBC connection pool and JDBC resource for the HADB in the Sun ONE Application Server. This command is described in the Sun ONE Application Server Installation Guide.
Manual configuration of a JDBC connection pool and JDBC resource for the HADB is briefly summarized in these sections:
For general information about connection pools and JDBC resources, see "About JDBC Resources".
Getting the JDBC URL
Before you can set up the JDBC connection pool, you need to determine the JDBC URL of the HADB using the hadbm get command as follows:
hadbm get jdbcURL [dbname]
For example:
hadbm get jdbcURL
The JDBC URL is displayed on the standard output device in the following form:
jdbc:sun:hadb:host:port,host:port,...
Remove the jdbc:sun:hadb: prefix and use the host:port,host:port... part as the value of the serverList connection pool property, described in the next section.
Creating a Connection Pool
The following table summarizes connection pool settings required for the HADB. Change Steady Pool Size when adding nodes, but do not change other settings.
The following table summarizes connection pool properties required for the HADB. Change serverList when adding nodes, but do not change other properties.
Table 20-5 HADB Connection Pool Properties
Property
Description
username
Specifies the name of the storeuser to be specified in the asadmin create-session-store command. See "Enabling Availability for an Application Server Instance".
password
Specifies the storepassword to be specified in the asadmin create-session-store command. See "Enabling Availability for an Application Server Instance".
serverList
Specifies the JDBC URL of the HADB. To determine this value, see "Getting the JDBC URL".
You must change this value if you add nodes to the database. See "Adding Nodes to the HADB".
cacheDatabaseMetaData
Setting this property to false as required ensures that calls to Connection.getMetaData() make calls to the database, which ensures that the connection is valid.
eliminateRedundantEndTransaction
Setting this property to true as required improves performance by eliminating redundant commit and rollback requests and ignoring these requests if no transaction is open.
Connection Pool Example
Here is an example asadmin create-jdbc-connection-pool command that creates an HADB JDBC connection pool. For more details about this command, see the Sun ONE Application Server Developer’s Guide to J2EE Services and APIs.
asadmin create-jdbc-connection-pool --user adminname --password secret --datasourceclassname com.sun.hadb.jdbc.ds.HadbDataSource --steadypoolsize=32 --isolationlevel=repeatable-read --isconnectvalidatereq=true --validationmethod=meta-data --property username=storename:password=secret456:serverList=host\\:port,host\\:port ,host\\:port,host\\:port,host\\:port,host\\:port:cacheDatabaseMetaData=false: eliminateRedundantEndTransaction=true hadbpool
Note that colon characters (:) within property values must be escaped with double backslashes (\\) on Solaris™ platforms, because otherwise they are interpreted as property delimiters. For details about using escape characters, see "Using Escape Characters".
Creating a JDBC Resource
The following table summarizes JDBC resource settings required for the HADB.
Table 20-6 HADB JDBC Resource Settings
Setting
Description
JNDI Name
The following JNDI name is the default in the session persistence configuration: jdbc/hastore. You can use the default name or a different name.
You must also specify this JNDI name as the value of the store-pool-jndi-name Persistence Store property when you activate the availability service. See "Default Values for Session Persistence".
Pool Name
Select from the list the name (or ID) of the HADB connection pool used by this JDBC resource. For more information, see "Creating a Connection Pool".
Data Source Enabled
Checked/true
Managing the HADBIn general, management operations are not necessary unless you are replacing or upgrading your network, hardware, operating system, or HADB software. For assistance with these operations, call Sun customer support. See "Using Sun Customer Support for the HADB". The following sections explain various management operations:
Starting a Node
Starting a node is not necessary if you are running with inetd, because inetd automatically restarts the node when the machine reboots.
You may want to start a node in the following circumstances:
In most cases, you should first attempt to start the node using the normal start level. You must use the repair start level in the following circumstances:
To start a node in the database, use the hadbm startnode command. The syntax is as follows:
hadbm startnode [--startlevel=level] node-number [dbname]
For example:
hadbm startnode 1
The hadbm startnode command options are listed in the following table.
Stopping a Node
You may want to stop a node if you want to replace hardware or software on the host, and you need to stop the host.
Note
Do not stop a node if its mirror node is not running, because this may force the database into a Non Operational state. For details about node status, see "Getting the Status of the HADB".
To stop a node in the database, use the hadbm stopnode command. The syntax is as follows:
hadbm stopnode [--no-repair] node-number [dbname]
For example:
hadbm stopnode 1
Note
If you have set up inetd, the node supervisor is automatically restarted when you stop a node. The result is that the node resumes the running state but is in the offline role.
Therefore, if you have set up inetd, you must perform these additional tasks when you stop a node:
For more information, see "Additional Steps for inetd".
The hadbm stopnode command options are listed in the following table.
Restarting a Node
You may want to restart a node if you notice strange behavior in a node (for example excessive CPU consumption) and want to check whether a restart cures the problem.
Note
Do not restart a node if its mirror node is not running, because this may force the database into a Non Operational state. For details about node status, see "Getting the Status of the HADB".
Repair of a node that does not contain data causes the repair to hang. This can happen if you attempt to start a new active node at the repair level before either refragmenting the database or clearing the database and recreating the session store.
To restart a node in the database, use the hadbm restartnode command. The syntax is as follows:
hadbm restartnode [--startlevel=level] node-number [dbname]
For example:
hadbm restartnode 1
The hadbm restartnode command options are listed in the following table.
Starting the HADB
To start a database, use the hadbm start command. The syntax is as follows:
hadbm start [dbname]
For example:
hadbm start
The default dbname is hadb, all lowercase.
This command starts all nodes that were running before the database was stopped. Individually stopped (offline) nodes are not started when the database is started after a stop.
Note
If you stop a database from the server machine, you must also start it from the server machine. Starting it from the client machine does not work in this case.
Stopping the HADB
When you stop and start the HADB in separate operations, data is unavailable while the HADB is stopped. To keep data available, you can restart the HADB as described in "Restarting the HADB".
You may want to stop the HADB in the following circumstances:
- If you want to remove the HADB database.
- If you want to perform system maintenance that affects all HADB nodes.
- Before executing the hadbm clear command to reinitialize the database. See "Clearing the HADB".
Before stopping the HADB, you should either stop dependent Sun ONE Application Server instances or configure them to use a different persistence method. For details, see Chapter 18, "Configuring Session Persistence".”
Note
If you stop a database from the server machine, you must also start it from the server machine. Starting it from the client machine does not work in this case.
To stop a database, use the hadbm stop command. The syntax is as follows:
hadbm stop [dbname]
For example:
hadbm stop
The default dbname is hadb, all lowercase. The database you want to stop must be in one of these states: HA Fault Tolerant, Fault Tolerant, Operational, or Non-Operational. For more information about database states, see "Getting the Status of the HADB".
When you stop the database, all the running nodes in the database are stopped and the status of the database is Stopped.
If you have set up inetd to automatically restart the HADB, you must perform these steps to stop the HADB:
Restarting the HADB
You may want to restart the HADB if you notice strange behavior in the HADB (for example consistent timeout problems) and want to check whether a restart cures the problem.
When you restart the HADB, data remains available. When you stop and start the HADB in separate operations, data is unavailable while the HADB is stopped. If a hadbm set command fails, restarting the HADB restores the previous configuration. For details about hadbm set, see "Viewing and Modifying Configuration Attributes".
To restart a database, use the hadbm restart command. The syntax is as follows:
hadbm restart [--no-rolling] [dbname]
For example:
hadbm restart
The default dbname is hadb, all lowercase. By default, this command restarts each of the nodes in the database to the current state or a better state. If you specify the --no-rolling or -g option, this command restarts all nodes at once, with loss of service.
Listing Databases
To list all the databases that have been created, use the hadbm list command. The syntax is as follows:
hadbm list
Clearing the HADB
You may want to clear the HADB in the following circumstances:
- If you are creating or adding new nodes to a database that uses inetd. See "Additional Steps for inetd".
- If the hadbm status command reveals that the database is Non Operational or that multiple nodes are in the Waiting state. See "Getting the Status of the HADB".
- If you are recovering from session data corruption. See "Recovering from Session Data Corruption".
You should stop the database before clearing it. See "Stopping the HADB".
The hadbm clear command stops the database nodes, clears the database devices, then starts the nodes. The syntax is as follows.
hadbm clear [--fast] [--spares=sparecount] --dbpassword=password | --dbpasswordfile=file [dbname]
For example:
hadbm clear --fast --spares=2 --dbpassword secret123
The hadbm clear command options are listed in the following table.
Table 20-10 hadbm clear Options
Long Form
Short Form
Default
Description
--fast
-F
not present
If present, skips device initialization while initializing the database. Do not use if the disk storage device is corrupted or if the database uses inetd.
--spares
-s
0
Specifies the number of spare nodes the reinitialized database will have. This number must be even and must be less than the number of nodes in the database. Spare nodes are optional, but having two or more ensures high availability.
--dbpassword
-p
none
Specifies the HADB system user password. You can use --dbpasswordfile instead. For details, see "The hadbm Command".
--dbpasswordfile
-P
none
Specifies a file that stores the password for the HADB system user. For details, see "The hadbm Command".
dbname
none
hadb
Specifies the database name.
Removing a Database
The database you want to remove must exist and must be in the Stopped state. See "Stopping the HADB". To remove an existing database from the HADB system, use the hadbm delete command. The syntax is as follows:
hadbm delete [dbname]
For example:
hadbm delete
The default database name is hadb, all lowercase. When you execute this command, the configuration files, device files, log files, and history files of the database are deleted, and shared memory resources are freed.
Expanding the HADBIf you determine that your system performance is limited because the HADB cannot persist data fast enough, you can expand the HADB to increase throughput without shutting down your Sun ONE Application Server cluster or the HADB. This section describes how you can expand the HADB in the following sections:
Adding Storage Space to Existing Nodes
You may want to add storage space to the HADB in the following circumstances:
- If there is unused space on the disks on which the HADB nodes reside, or if you have upgraded these disks.
- If one of the following messages appears:
4592: No free blocks on data devices
4593: No unreserved blocks on data devices
- If the hadbm deviceinfo command reports insufficient free size. See "Getting Device Information".
You can increase the device size per node in MB using the following hadbm set command:
hadbm set TotalDatadeviceSizePerNode=size
For example:
hadbm set TotalDatadeviceSizePerNode=2048
The maximum size allowed is the maximum operating system file size. The device size should be as large as possible.
Changing the TotalDatadeviceSizePerNode on a database in a FaultTolerant or higher state means that the system is upgraded without loss of data, and the database remains in an Operational state during the reconfiguration. If you change device size on a system that is not FaultTolerant or better, data is lost. For more information about database states, see "Database Status".
Adding Machines
You may want to add machines if the HADB requires more processing or storage capacity. For an explanation of node topology alternatives, see the Sun ONE Application Server System Deployment Guide.
To add a new machine on which to run the HADB, install the HADB packages with or without the Sun ONE Application Server as described in the Sun ONE Application Server Installation Guide.
Adding Nodes to the HADB
When you create new nodes and add them to the database, you increase processing and storage capacity. To add nodes, use the hadbm addnodes command. The syntax is as follows:
hadbm addnodes [--no-refragment] [--spares=sparecount] --dbpassword=password | --dbpasswordfile=file [--inetdsetupdir=path] --hosts=node-list [dbname]
For example:
hadbm addnodes --dbpassword secret123 --hosts n7,n8,n9,n10
After you have added nodes, you must perform these additional tasks:
For details, see "Setting Up the JDBC Connection Pool".
Note
If you created the database using --inetd, you must do the following:
- Use the --no-refragment option and refragment the database in a separate step using the hadbm refragment command.
- Take extra steps to update the inetd configuration files to take into account the new nodes you add. For more information, see "Additional Steps for inetd".
The hadbm addnodes command options are listed in the following table.
Table 20-11 hadbm addnodes Options
Long Form
Short Form
Default
Description
--no-refragment
-r
not specified
If specified, does not refragment the database during node creation; you can refragment the database later using the hadbm refragment command. For details about refragmentation, see "Refragmenting the HADB".
You must use this option if you created the database using --inetd. In this case, you must refragment the database in a separate step using hadbm refragment.
You can add nodes without refragmenting the database; see "Adding Nodes Without Refragmenting".
--spares
-s
0
Specifies the number of new spare nodes. This number must be even and must not be greater than the number of nodes added. Spare nodes are optional, but having two or more ensures high availability.
--dbpassword
-p
none
Specifies the HADB system user password. You can use --dbpasswordfile instead. For details, see "The hadbm Command".
--dbpasswordfile
-P
none
Specifies a file that stores the password for the HADB system user. For details, see "The hadbm Command".
--inetdsetupdir
-u
current directory
Specifies the directory in which to store the inetd setup files. The directory must exist on the machine and must be writable.
--hosts
-H
none
Specifies a comma-separated list of new host names for the nodes in the database. Duplicates are allowed; this creates nodes on the same host with different port numbers. One node is created for each item in the list. The number of nodes must be even.
dbname
none
hadb
Specifies the database name. The database must be in the HA Fault Tolerant or Fault Tolerant state. For more information about database states, see "Getting the Status of the HADB".
Refragmenting the HADB
You must refragment the database before new nodes can store data. Refragmentation is required to store data evenly across all active nodes. To refragment the database, use the hadbm refragment command. The syntax is as follows:
hadbm refragment --dbpassword=password | --dbpasswordfile=file [dbname]
For example:
hadbm refragment --dbpassword secret123
Refragmentation requires that the user data size not exceed 50% of the space available for user data. For details, see "Getting Device Information".
If this command fails even after multiple attempts, see "Adding Nodes Without Refragmenting".
The hadbm refragment command options are listed in the following table.
Table 20-12 hadbm refragment Options
Long Form
Short Form
Default
Description
--dbpassword
-p
none
Specifies the HADB system user password. You can use --dbpasswordfile instead. For details, see "The hadbm Command".
--dbpasswordfile
-P
none
Specifies a file that stores the password for the HADB system user. For details, see "The hadbm Command".
dbname
none
hadb
Specifies the database name. The database must be in the HA Fault Tolerant or Fault Tolerant state. For more information about database states, see "Getting the Status of the HADB".
Adding Nodes Without Refragmenting
If you don’t refragment the database when adding nodes, you must clear the database and recreate the session store instead, otherwise the session store can’t use the new nodes. Adding nodes without refragmenting the database is not generally recommended. However, it may be the best alternative if all of the following conditions are met:
- You don’t have enough disk space to expand each node as described in "Adding Storage Space to Existing Nodes".
- The user data size exceeds 50% of the space available for user data, which means you cannot refragment as described in "Refragmenting the HADB".
- You are not passivating the session state.
To add nodes without refragmenting, perform the following tasks:
- Perform the following tasks for each server instance:
- Disable the server instance in the load balancer, as described in "Load Balancer Configuration File".
- Disable session persistence as described in "Enabling Availability for an Application Server Instance".
- Restart the server instance.
- Re-enable the server instance in the load balancer.
If you do not need to maintain availability, you can disable and reenable all the server instances at once in the load balancer. This saves time and prevents failover of outdated session data.
- Clear the database as described in "Clearing the HADB".
- Reconfigure the JDBC connection pool as described in "Setting Up the JDBC Connection Pool". You can also use the cladmin command. See Chapter 19, "Using the cladmin Command."
- Reload the session persistence store as described in "Enabling Availability for an Application Server Instance". You can also use the cladmin command.
- Perform the following tasks for each server instance:
- Disable the server instance in the load balancer.
- Enable session persistence as described in "Enabling Availability for an Application Server Instance".
- Restart the server instance.
- Re-enable the server instance in the load balancer.
If you do not need to maintain availability, you can disable and reenable all the server instances at once in the load balancer. This saves time and prevents failover of outdated session data.
Monitoring the HADBYou can monitor the activities in the HADB by performing the following tasks:
These sections briefly describe the hadbm status, hadbm deviceinfo, and hadbm resourceinfo commands. For details about interpreting HADB information, see the Sun ONE Application Server Performance Tuning Guide.
Getting the Status of the HADB
To display the status of the database or its nodes, use the hadbm status command. The syntax is as follows:
hadbm status [--nodes] [dbname]
For example:
hadbm status --nodes
The hadbm status command options are listed in the following table.
Table 20-13 hadbm status Options
Long Form
Short Form
Default
Description
--nodes
-n
not present
If present, displays node status information. See "Node Status".
dbname
none
hadb
Specifies the database name.
Database Status
The possible states of a database are as follows:
- High-Availability Fault Tolerant (HAFT) - The database is fault tolerant and has at least one spare node on each DRU.
- Fault Tolerant (FT) - All the mirrored node pairs are up and running.
- Operational (O) - At least one node in each mirrored node pair is running.
- Non Operational (NO) - One or more mirrored node pairs is missing both nodes.
- Stopped (S) - No nodes are running in the database.
- Unknown (U) - The command cannot determine the state of the database.
If the database is Non Operational, clear the database using hadbm clear as described in "Clearing the HADB".
Node Status
If you specify the --nodes option, the following information is displayed for each node in the database:
- Node number
- Name of the machine where the node is running
- Port number of the node
- Role of the node. For a list of possible roles and their meanings, see "Roles of a Node".
- State of the node. For a list of possible states and their meanings, see "States of a Node".
- Number of the corresponding mirror node.
A node’s role and state can change as described in these sections:
Roles of a Node
A node is assigned a role during its creation and can take any one of these roles:
- Active: An active node allows data storage and client access. Active nodes are in mirrored pairs.
- Spare: After having their data devices initialized, spare nodes monitor other data nodes to initiate repair if another node becomes unavailable. A spare node allows client access, but not data storage.
- Offline: A node is taken offline prior to stopping it to prevent restart by inetd. Offline nodes provide no services until their role changes. An offline node’s role can change back to its former role.
- Shutdown: An intermediate step between active and offline, which a node occupies while waiting for a spare node to take over its functioning. After the spare node has taken over, the node is taken offline.
States of a Node
A node can be in any one of the following states:
- Starting: The node is starting.
- Waiting: The node cannot decide its start level and is offline. If a single node is in this state for more than two minutes, stop the node and then start it at the repair level; see "Stopping a Node" and "Starting a Node". If multiple nodes are in this state, clear the database as described in "Clearing the HADB".
- Running: The node is fully up and running, providing services appropriate for its role.
- Stopping: The node is in the process of stopping.
- Stopped: The node is inactive. Repair of a stopped node is prohibited.
- Recovering: The node is being recovered. When a node fails, the mirror node takes over the functions of the failed node. The failed node tries to recover by restarting from main memory or from disk. It uses the log files from the mirror node and recovers the lost data. If recovery is successful, the node becomes active. If recovery fails, the node state changes to Repairing.
- Repairing: The node is being repaired. This operation reinitializes the node and copies the data and log files from the mirror node. Repair is more time consuming than recovery.
Getting Device Information
Monitoring the HADB involves making sure that there is enough free space for the growth of the database. To get information about disk storage devices on each active node, use the hadbm deviceinfo command. The syntax is as follows:
hadbm deviceinfo [--details] [dbname]
For example:
hadbm deviceinfo --details
The default dbname is hadb.
The information displayed for each node of the database includes:
To determine the space available for user data, take the total device size, then subtract 4 times the LogBufferSize. If you do not know the size of the log buffer, use the command hadbm get logbufferSize. For example, if the total device size is 128 MB and the LogBufferSize is 24 MB, the space available for user data is 128 – (4 x 24) = 32 MB.
The difference between the total device size and the free size is the user data size. If the data may be refragmented in the future, the user data size should not exceed 50% of the space available for user data. If refragmentation is not relevant, close to 100% may be used. Resource consumption warnings are written to the history files when the system is running short on device space.
For more information about tuning the HADB, see the Sun ONE Application Server Performance Tuning Guide.
If the --details option is specified, additional information is displayed:
For example:
NodeNO Totalsize Freesize Usage NReads NWrites DeviceName
0 128 120 6% 10000 5000 /var/opt/hadb.data.0
1 128 124 3% 10000 5000 /var/opt/hadb.data.1
2 128 126 2% 9500 4500 /var/opt/hadb.data.2
3 128 126 2% 9500 4500 /var/opt/hadb.data.3
If you need additional information, you can use the hadbm resourceinfo command. This command displays HADB runtime resource information that helps to identify resource contention, which you can use to reduce performance bottlenecks. For details, see the Sun ONE Application Server System Deployment Guide and the Sun ONE Application Server Performance Tuning Guide. The syntax is as follows:
hadbm resourceinfo [--databuf] [--locks] [--logbuf] [--nilogbuf] [dbname]
The following database information is displayed based on the options you specify:
For example, data buffer pool information is as follows:
NodeNO Avail Free Access Misses Copy-on-Write
0 256 128 100000 50000 1000
1 256 128 110000 45000 950
Locks information is as follows:
For example:
NodeNO Avail Free Waits
0 50000 20000 10
1 50000 20000 0
No more than 50% of the available locks should be used. To change the NumberOfLocks, see "Viewing and Modifying Configuration Attributes".
Log buffer information is as follows:
For example:
NodeNO Avail Free
0 16 2
1 16 3
Node internal log buffer information is as follows:
For example:
NodeNO Avail Free
0 16 2
1 16 3
Viewing and Modifying Configuration AttributesYou can modify database configuration variables. This section describes the following tasks:
Getting the Values of Configuration Attributes
To get the values of configuration attributes (for a list, see "Configuration Attributes"), use the hadbm get command. The syntax is as follows:
hadbm get attribute-list | --all [dbname]
For example:
hadbm get jdbcURL,NumberOfSessions
The default dbname is hadb. The attribute-list is a comma-separated or quote-enclosed space-separated list of attributes. The --all option displays values for all attributes.
Setting the Values of Configuration Attributes
To set the values of configuration attributes (for a list, see "Configuration Attributes"), use the hadbm set command. The syntax is as follows:
hadbm set [dbname] attribute=value,attribute=value ...
The default dbname is hadb. The attribute-list is a comma-separated or quote-enclosed space-separated list of attributes.
If execution of this command is successful, the database is restarted in the state it was in previously, or in a better state. For information about database states, see "Getting the Status of the HADB".
If execution of this command is unsuccessful, restart the HADB as described in "Restarting the HADB".
Configuration Attributes
The following table lists the configuration attributes that you can get and set. Except where noted, sizes are in MB, and times are in seconds.
Clearing and Archiving History FilesHADB history files contain a record of database operations and error messages. The location of these files is determined by the --historypath option of the hadbm create command. The default location is /var/tmp. These files have names of the format dbname.out.nodenumber. For details about hadbm create, see "Creating a Database".
These history files grow over time. To save space and prevent files from getting too large, you should periodically clear and archive older history files. To clear the history files of a database, use the hadbm clearhistory command. The syntax is as follows:
hadbm clearhistory [--saveto=path] [dbname]
The default dbname is hadb.
Use the --saveto or -o option to specify a directory if you want to store the old history files. This directory must have write permissions set.
Each message in the history file contains the following information:
Messages about resource shortages contain HIGH LOAD.
You do not need a detailed knowledge of all the various types of entries in the history file. If for any reason you need to study a history file in greater detail, you should obtain help from Sun customer support. See "Using Sun Customer Support for the HADB".
Recovering from Session Data CorruptionThe following are indications that session data may be corrupted:
- Error messages appear in the Sun ONE Application Server system log (server log) every time you try to save the session state.
- Error messages are written to the server log indicating that the session could not be found or could not be loaded during session activation.
- Sessions that are activated after previously being passivated contain empty or incorrect session data.
- When an instance fails, failed-over sessions contain empty or incorrect session data.
- When an instance fails, instances that try to load a failed-over session cause an error in the server log indicating the session could not be found or could not be loaded.
To bring the session store back to a consistent state if you determine that the data has been corrupted, do the following:
- Clear the session store. For more information, see "Clearing the Session Store".
- If clearing the session store doesn’t work or you continue to see errors in the server log, reinitialize the data space on all the nodes and clear the data in the database. See "Clearing the HADB".
- If clearing the database doesn’t work, delete and then recreate the database. See "Removing a Database" and "Creating a Database".
Using Sun Customer Support for the HADBBefore calling Sun customer support about HADB issues, you should gather as much of the following information about your system as possible:
Environment VariablesThis table lists environment variables that correspond to hadbm command options.
