|
|
Troubleshooting Applications
Other chapters of this document discuss many diagnostic tools provided by your BEA WebLogic Enterprise or BEA Tuxedo system: commands and log files that help you monitor a running system, identify potential problems while there is still time to prevent them, and detect error conditions once they have occurred. This chapter provides additional information to help you identify and recover from various system errors.
This topic includes the following sections:
Distinguishing Between Types of Failures
The first step in troubleshooting is to determine the area in which the problem has occurred. In most applications, you must consider six possible sources of trouble:
To resolve the trouble in most of these areas, you must work with the appropriate administrator. If, for example, you determine that the trouble is being caused by a networking problem, you must work with the network administrator.
Determining the Cause of an Application Failure
To detect the source of an application failure, complete the following steps:
Determining the Cause of a BEA WebLogic Enterprise or BEA Tuxedo System Failure
To detect the source of a system failure, complete the following steps:
Broadcasting Unsolicited Messages (BEA Tuxedo System)
To send an unsolicited message, enter the following command:
broadcast (bcst) [-m machine] [-u usrname] [-c cltname] [text]
By default, the message is sent to all clients. You have the choice, however, of limiting distribution to one of the following recipients:
The text may not include more than 80 characters. The system sends the message in a buffer of type STRING. This means that the client's unsolicited message handling function (specified by tpsetunsol(0)) must be able to handle a message of this type. The tptypes() function may be useful in this case.
Performing System File Maintenance
This section provides instructions for the following tasks that you may need to perform in the course of maintaining your file system:
Complete the following steps to create a device list.
crdl [-z devicename] [-b blocks]
Note: Because 35 blocks are needed for the administrative overhead associated with a TLOG, be sure to assign a value higher than 35 when you create a TLOG.
Destroying a Device List
To destroy a device list with index devindx, enter the following command:
dsdl [-z devicename] [yes] [devindx]
To reinitialize a device on a device list, enter the following command:
initdl [-z devicename] [-yes] devindx
Printing the Universal Device List (UDL)
To print a UDL, enter the following command:
lidl
To specify the device from which you want to obtain the UDL, you have a choice of two methods:
-z device name [devindx]
To get information about all VTOC table entries, enter the following command:
livtoc
To specify the device from which you want to obtain the VTOC, you have a choice of two methods:
-z device name [devindx]
Repairing Partitioned Networks
A network partition exists if one or more machines cannot access the master machine. As the application administrator, you are responsible for detecting partitions and recovering from them. This section provides instructions for troubleshooting a partition, identifying its cause, and taking action to recover from it.
A network partition may be caused by the following:
The procedure you follow to recover from a partitioned network depends on the cause of the partition. Recovery procedures for these situations are provided in this section.
Detecting Partitioned Networks
There are several ways to detect a network partition:
When things go wrong with the network, BEA WebLogic Enterprise or BEA Tuxedo system administrative servers start sending messages to the ULOG. If the ULOG is set up over a remote file system, all messages are written to the same log. In such a case you can run the tail(1) command on one file and check the failure messages displayed on the screen.
If, however, the remote file system is using the same network, the remote file system may no longer be available.
Example
151804.gumby!DBBL.28446: ... : ERROR: BBL partitioned, machine=SITE2
Gathering Information about the Network, Server, and Service
Listing 22-1 provides an example of a tmadmin session in which information is being collected about a partitioned network, and a server and a service on that network. Three tmadmin commands are run:
Listing 22-1 Example of a tmadmin Session
$ tmadmin
> pnw SITE2
Could not retrieve status from SITE2
> psr -m SITE1
a.out Name Queue Name Grp Name ID Rq Done Load Done Current Service
BBL 30002.00000 SITE1 0 - - (- )
DBBL 123456 SITE1 0 121 6050 MASTERBB
simpserv 00001.00001 GROUP1 1 - - ( - )
BRIDGE 16900672 SITE1 0 - - ( DEAD )
>psc -m SITE1
Service Name Routine Name a.out Grp Name ID Machine # Done Status
------------ ------------ -------- -------- -- ------- ------------
ADJUNCTADMIN ADJUNCTADMIN BBL SITE1 0 SITE1 - PART
ADJUNCTBB ADJUNCTBB BBL SITE1 0 SITE1 - PART
TOUPPER TOUPPER simpserv GROUP1 1 SITE1 - PART
BRIDGESVCNM BRIDGESVCNM BRIDGE SITE1 1 SITE1 - PART
Restoring a Network Connection
This section provides instructions for recovering from transient and severe network failures.
Recovering from Transient Network Failures
Because the BRIDGE tries, automatically, to recover from any transient network failures and reconnects, transient network failures are usually not noticed. If, however, you do need to perform a manual recovery from a transient network failure, complete the following steps:
rco non-partioned_node1 partioned_node2
Recovering from Severe Network Failures
To recover from severe network failure, complete the following steps:
pcl partioned_machine
Restoring Failed Machines
The procedure you follow to restore a failed machine depends on whether that machine was the master machine.
Restoring a Failed Master Machine
To restore a failed master machine, complete the following procedure.
tmadmin
boot -B SITE1
The BBL will not boot if you have not executed pclean on SITE1.
MASTER
Restoring a Failed Nonmaster Machine
To restore a failed nonmaster machine, complete the following steps:
In Listing 22-2, SITE2, a nonmaster machine, is restored.
Listing 22-2 Example of Restoring a Failed Nonmaster Machine
$ tmadmin
tmadmin - Copyright © 1987-1990 AT&T; 1991-1993 USL. All rights reserved
> pclean SITE2
Cleaning the DBBL.
Pausing 10 seconds waiting for system to stabilize.
3 SITE2 servers removed from bulletin board
> boot -B SITE2
Booting admin processes ...
Exec BBL -A :
on SITE2 -> process id=22923 ... Started.
1 process started.
> q
Replacing System Components (BEA Tuxedo System)
To replace BEA Tuxedo system components, complete the following steps:
Replacing Application Components
To replace components of your application, complete the following steps:
Cleaning Up and Restarting Servers Manually
By default, the BEA WebLogic Enterprise or BEA Tuxedo system cleans up resources associated with dead processes (such as queues) and restarts restartable dead servers from the Bulletin Board (BB) at regular intervals during BBL scans. You may, however, request cleaning at other times.
Cleaning Up Resources
To request an immediate cleanup of resources associated with dead processes, complete the following procedure.
The bbclean command takes one optional argument: the name of the machine to be cleaned.
If You Specify . . . |
Then . . . |
---|---|
No machine |
The resources on the default machine are cleaned. |
A machine |
The resources on that machine are cleaned. |
DBBL |
The resources on the Distinguished Bulletin Board Listener (DBBL) and the Bulletin Boards at all sites are cleaned. |
To clean up other resources, complete the following steps:
Note: You must specify a value for machine; it is a required argument.
If the Specified Machine Is . . . |
Then . . . |
---|---|
Not partitioned |
pclean will invoke bbclean. |
Partitioned |
pclean will remove all entries for servers and services from all nonpartitioned Bulletin Boards. |
This command is useful for restoring order to a system after partitioning has occurred unexpectedly.
Checking the Order in Which Servers Are Booted (BEA WebLogic Enterprise Servers)
If a BEA WebLogic Enterprise application fails to boot, open the application's UBBCONFIG file with a text editor and check whether the servers are booted in the correct order in the SERVERS section. The following is the correct order in which to boot the servers on a BEA WebLogic Enterprise system. A BEA WebLogic Enterprise application will not boot if this order is not adhered to.
Boot the servers in the following order:
For a detailed example, see the section "Required Order in Which to Boot Servers (WLE Servers)" in Creating a Configuration File.
Checking Hostname Format and Capitalization (BEA WebLogic Enterprise Servers)
The network address that is specified by programmers in the Bootstrap object constructor or in TOBJADDR must exactly match the network address in the server application's UBBCONFIG file. The format of the address as well as the capitalization must match. If the addresses do not match, the call to the Bootstrap object constructor will fail with a seemingly unrelated error message:
ERROR: Unofficial connection from client at
<tcp/ip address>/<port-number>:
For example, if the network address is specified as //TRIXIE:3500 in the ISL command-line option string (in the server application's UBBCONFIG file), specifying either //192.12.4.6:3500 or //trixie:3500 in the Bootstrap object constructor or in TOBJADDR will cause the connection attempt to fail.
On UNIX systems, use the uname -n command on the host system to determine the capitalization used. On Windows NT systems, see the host system's Network control panel to determine the capitalization used.
Some Clients Fail to Boot (BEA WebLogic Enterprise Servers)
You may want to perform the following steps on a Windows NT server that is running a BEA WebLogic Enterprise application, if the following problem occurs: some Internet Inter-ORB Protocol (IIOP) clients boot, but some clients fail to create a Bootstrap object and return an InvalidDomain message, even though the //host:port address is correctly specified. (For related information, see the section "Checking Hostname Format and Capitalization (BEA WebLogic Enterprise Servers)" .)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Afd\Parameters
DynamicBacklogGrowthDelta: REG_DWORD : 0xa
EnableDynamicBacklog: REG_DWORD: 0x1
MaximumDynamicBacklog: REG_DWORD: 0x3e8
MinimumDynamicBacklog: REG_DWORD: 0x14
These values replace the static connection queue (that is, the backlog) of five pending connections with a dynamic connection backlog, that will have at least 20 entries (minimum 0x14), at most 1000 entries (maximum 0x3e8), and will increase from the minimum to the maximum by steps of 10 (growth delta 0xa).
These settings only apply to connections that have been received by the system, but are not accepted by an IIOP Listener. The minimum value of 20 and the delta of 10 are recommended by Microsoft. The maximum value depends on the machine. However, Microsoft recommends that the maximum value not exceed 5000 on a Windows NT server.
Aborting or Committing Transactions
This section provides instructions for aborting and committing transactions.
Aborting a Transaction
To abort a transaction, enter the following command:
aborttrans (abort) [-yes] [-g groupname] tranindex
This command is useful when the coordinating site is partitioned or when the client terminates before calling a commit or an abort. If the timeout is large, the transaction remains in the transaction table unless it is aborted.
Committing a Transaction
To commit a transaction, enter the following command:
committrans (commit) [-yes] [-g groupname] tranindex
Be careful about using this command. The only time you should need to run it is when both of the following conditions apply:
Also, a client may be blocked on tpcommit(), which will be timed out. If you are going to perform an administrative commit, be sure to inform this client.
Recovering from Failures When Transactions Are Used
When the application you are administering includes database transactions, you may need to apply an after-image journal (AIJ) to a restored database following a disk corruption failure. Or you may need to coordinate the timing of this recovery activity with your site's database administrator (DBA). Typically, the database management software automatically performs transaction rollback when an error occurs. When the disk containing database files has become permanently corrupt, however, you or the DBA may need to step in and perform the rollforward operation.
Assume that a disk containing portions of a database is corrupted at 3:00 P.M. on a Wednesday. For this example, assume that a shadow volume does not exist.
Refer to the documentation for the resource manager (database product) for specific instructions on the database rollforward process.
|
Copyright © 2000 BEA Systems, Inc. All rights reserved.
|