![]() |
![]() |
e-docs > Tuxedo > Administering a Tuxedo Application at Run Time > Troubleshooting a BEA Tuxedo Application |
Administering a Tuxedo Application at Run Time
|
Troubleshooting a BEA Tuxedo Application
This topic includes the following sections:
Determining Types of Failures
The first step in troubleshooting is determining problem areas. In most applications you must consider six possible sources of trouble:
Once you have determined the problem area, you must then work with the appropriate administrator to resolve the problem. If, for example, you determine that the trouble is caused by a networking problem, you must work with the network administrator.
How to Determine the Cause of an Application Failure
The following steps will help you detect the source of an application failure.
How to Determine the Cause of a BEA Tuxedo System Failure
The following steps will help you detect the source of a system failure.
How to Broadcast an Unsolicited Message
The EventBroker enhances troubleshooting by providing a system-wide summary of events and a mechanism whereby an event triggers notification. The EventBroker provides details about BEA Tuxedo system events, such as servers dying and networks failing, or application events, such as an ATM machine running out of money. A BEA Tuxedo client that receives unsolicited notification of an event, can name a service routine to be invoked, or name an application queue in which data should be stored for later processing. A BEA Tuxedo server that receives unsolicited notification can specify a service request or name an application queue to store data.
broadcast (bcst) [-m machine] [-u usrname] [-c cltname] [text]
The text may not include more than 80 characters. The system sends the message in a STRING type buffer, which means the client's unsolicited message handling function (specified by tpsetunsol(0)) must be able to handle this type of message. The tptypes() function may be useful in this case.
See Also
Maintaining Your System Files
Periodically, you may need to perform the following tasks to maintain your file system:
Note: This file format is used for TUXCONFIG, TLOG, and /Q.
How to Print the Universal Device List (UDL)
To print a UDL, complete the following procedure:
lidl
How to Print VTOC Information
To print VTOC information, complete the following procedure.
livtoc
How to Reinitialize a Device
To reinitialize a device that is included on a device list, complete the following procedure.
initdl [-z devicename] [-yes] devindx
How to Create a Device List
To create a device list, complete the following procedure.
crdl [-z devicename] [-b blocks]
How to Destroy a Device List
To destroy a device list with index devindx, complete the following procedure.
dsdl [-z devicename] [yes] [devindx]
Recovery Considerations
The BEA Tuxedo system requires a certain level of environmental stability to provide optimum functionality. Although the BEA Tuxedo administrative subsystem offers unparalleled capabilities of recovering from network, machine, and application process failures, it is not invulnerable. You should be aware of the following ways in which a BEA Tuxedo system works.
Application clients and servers that use the FASTPATH model of SYSTEM_ACCESS (the default) have direct memory access to the BEA Tuxedo shared data structures. Using the FASTPATH model helps ensure that the BEA Tuxedo system achieves its outstanding performance. The BEA Tuxedo system uses the IPC (InterProcess Communication and File System) facilities provided by the operating system.
If an application accidentally uses these facilities to write into the BEA Tuxedo shared memory or to a BEA Tuxedo file descriptor, or if it mistakenly uses any other BEA Tuxedo system resource, data may become corrupted, BEA Tuxedo functionality may be compromised, or an application may be brought down.
It is inappropriate for a user or administrator to directly terminate application clients, application servers, or BEA Tuxedo administrative processes because these processes may be executing within a critical section (that is, updating shared information in shared memory). Interrupting a critical section during a memory update could potentially cause inconsistent internal data structures. (This is characteristic not only of the BEA Tuxedo system, but of any system in which shared data is used.) Error messages in the BEA Tuxedo userlog that refer to locks or semaphores may indicate that such corruption has occurred.
For maximum application availability, you can take advantage of the BEA Tuxedo system's facilities for managing redundancy, such as its multiple server, machine, and domain facilities. Distributing an application's functionality allows continued operation if a failure occurs in one area.
Repairing Partitioned Networks
This topic provides instructions for troubleshooting a partition, identifying its cause, and taking action to recover from it. A network partition exists if one or more machines cannot access the MASTER machine. As the application administrator, you are responsible for detecting partitions and recovering from them.
A network partition may be caused by any the following failures:
The procedure you follow to recover from a partitioned network depends on the cause of the partition.
Detecting a Partitioned Network
You can detect a network partition in one of the following ways:
How to Check the ULOG
When problems occur with the network, BEA Tuxedo system administrative servers start sending messages to the ULOG. If the ULOG is set up over a remote file system, all messages are written to the same log. In this scenario, you can run the tail(1) command on one file and check the failure messages displayed on the screen.
If, however, the remote file system is using the network in which the problem has occurred, the remote file system may no longer be available.
Listing 9-1 Example of a ULOG Error Message
151804.gumby!DBBL.28446: ... : ERROR: BBL partitioned, machine=SITE2
How to Gather Information About the Network, Server, and Service
The following is an example of a tmadmin session in which information is being collected about a partitioned network, a server, and a service on that network. Three tmadmin commands are run:
Listing 9-2 Example tmadmin Session
$ tmadmin
> pnw SITE2
Could not retrieve status from SITE2
> psr -m SITE1
a.out Name Queue Name Grp Name ID Rq Done Load Done Current Service
BBL 30002.00000 SITE1 0 - - ( - )
DBBL 123456 SITE1 0 121 6050 MASTERBB
simpserv 00001.00001 GROUP1 1 - - ( - )
BRIDGE 16900672 SITE1 0 - - ( DEAD )
>psc -m SITE1
Service Name Routine Name a.out Grp Name ID Machine # Done Status
------------ ------------ -------- -------- -- ------- ------------
ADJUNCTADMIN ADJUNCTADMIN BBL SITE1 0 SITE1 - PART
ADJUNCTBB ADJUNCTBB BBL SITE1 0 SITE1 - PART
TOUPPER TOUPPER simpserv GROUP1 1 SITE1 - PART
BRIDGESVCNM BRIDGESVCNM BRIDGE SITE1 1 SITE1 - PART
Restoring a Network Connection
This topic provides instructions for recovering from transient and severe network failures.
How to Recover from Transient Network Failures
Because the BRIDGE tries, automatically, to recover from any transient network failures and reconnect, transient network failures are usually not noticed. If, however, you need to perform a manual recovery from a transient network failure, complete the following procedure.
rco non-partioned_node1 partioned_node2
How to Recover from Severe Network Failures
To recover from severe network failure, complete the following procedure.
pcl partioned_machine
Restoring Failed Machines
The procedure you follow to restore a failed machine depends on whether that machine was the MASTER machine.
How to Restore a Failed MASTER Machine
To restore a failed MASTER machine, complete the following procedure.
tmadmin
boot -B SITE1
MASTER
How to Restore a Failed Nonmaster Machine
To restore a failed nonmaster machine, complete the following procedure.
In the following list, SITE2, a nonmaster machine, is restored.
Listing 9-3 Example of Restoring a Failed Nonmaster Machine
$ tmadmin
tmadmin - Copyright © 1987-1990 AT&T; 1991-1993 USL. All rights reserved
> pclean SITE2
Cleaning the DBBL.
Pausing 10 seconds waiting for system to stabilize.
3 SITE2 servers removed from bulletin board
> boot -B SITE2
Booting admin processes ...
Exec BBL -A :
on SITE2 -> process id=22923 ... Started.
1 process started.
> q
How to Replace System Components
To replace BEA Tuxedo system components, complete the following procedure.
How to Replace Application Components
To replace components of your application, complete the following procedure.
Cleaning Up and Restarting Servers Manually
By default, the BEA Tuxedo system cleans up resources associated with dead processes (such as queues) and restarts restartable dead servers from the Bulletin Board (BB) at regular intervals during BBL scans. You may, however, request cleaning at other times.
How to Clean Up Resources Associated with Dead Processes
To request an immediate cleanup of resources associated with dead processes, complete the following procedure.
The bbclean command takes one optional argument: the name of the machine to be cleaned.
How to Clean Up Other Resources To clean up other resources, complete the following procedure.
Note: You must specify a value for machine; it is a required argument.
This command is useful for restoring order to a system after partitioning has occurred unexpectedly.
How to Check the Order in Which BEA Tuxedo CORBA Servers Are Booted
If a BEA Tuxedo CORBA application fails to boot, open the application's UBBCONFIG file with a text editor and check whether the servers are booted in the correct order in the SERVERS section. The following is the correct order in which to boot the servers in a BEA Tuxedo CORBA environment. A BEA Tuxedo CORBA application will not boot if this order is not adhered to.
Boot the servers in the following order:
For a detailed example, see the section "Required Order in Which to Boot CORBA C++ Servers" on page 3-73 in Setting Up a BEA Tuxedo Application.
How to Check the Hostname Format and Capitalization of BEA Tuxedo CORBA Servers
The network address that is specified by programmers in the Bootstrap object constructor or in TOBJADDR must exactly match the network address in the server application's UBBCONFIG file. The format of the address as well as the capitalization must match. If the addresses do not match, the call to the Bootstrap object constructor will fail with a seemingly unrelated error message:
ERROR: Unofficial connection from client at
<tcp/ip address>/<port-number>:
For example, if the network address is specified as //TRIXIE:3500 in the ISL command-line option string (in the server application's UBBCONFIG file), specifying either //192.12.4.6:3500 or //trixie:3500 in the Bootstrap object constructor or in TOBJADDR will cause the connection attempt to fail.
On UNIX systems, use the uname -n command on the host system to determine the capitalization used. On Windows 2000 systems, see the host system's Network control panel to determine the capitalization used.
Why Some BEA Tuxedo CORBA Clients Fail to Boot
You may want to perform the following steps on a Windows 2000 server that is running a BEA Tuxedo CORBA application, if the following problem occurs: some Internet Inter-ORB Protocol (IIOP) clients boot, but some clients fail to create a Bootstrap object and return an InvalidDomain message, even though the //host:port address is correctly specified. (For related information, see the section "How to Check the Hostname Format and Capitalization of BEA Tuxedo CORBA Servers" on page 9-17.)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Afd\Parameters
These values replace the static connection queue (that is, the backlog) of five pending connections with a dynamic connection backlog, that will have at least 20 entries (minimum 0x14), at most 1000 entries (maximum 0x3e8), and will increase from the minimum to the maximum by steps of 10 (growth delta 0xa).
These settings only apply to connections that have been received by the system, but are not accepted by an IIOP Listener. The minimum value of 20 and the delta of 10 are recommended by Microsoft. The maximum value depends on the machine. However, Microsoft recommends that the maximum value not exceed 5000 on a Windows 2000 server.
Aborting or Committing Transactions
This topic provides instructions for aborting and committing transactions.
How to Abort a Transaction
To abort a transaction, complete the following procedure.
aborttrans (abort) [-yes] [-g groupname] tranindex
This command is useful when the coordinating site is partitioned or when the client terminates before calling a commit or an abort. If the timeout is large, the transaction remains in the transaction table unless it is aborted.
How to Commit a Transaction
To commit a transaction, complete the following procedure.
The operation fails if the transaction is not precommitted or has been marked aborted. This message should be sent to all groups to fully commit the transaction.
Cautions About Using the committrans Command
Be careful about using the committrans command. The only time you need to run it is when both of the following conditions apply:
Also, a client may be blocked on tpcommit(), which will be timed out. If you are going to perform an administrative commit, be sure to inform this client.
How to Recover from Failures When Transactions Are Used
When the application you are administering includes database transactions, you may need to apply an after-image journal (AIJ) to a restored database following a disk corruption failure. Or you may need to coordinate the timing of this recovery activity with your site's database administrator (DBA). Typically, the database management software automatically performs transaction rollback when an error occurs. When the disk containing database files has become corrupted permanently, however, you or the DBA may need to step in and perform the rollforward operation.
Assume that a disk containing portions of a database is corrupted at 3:00 P.M. on a Wednesday. For this example, assume that a shadow volume (that is, you have disk mirroring) does not exist.
Refer to the documentation for the resource manager (database product) for specific instructions on the database rollforward process.
How to Use the IPC Tool When an Application Fails to Shut Down Properly
Inter-process communication (IPC) resources are operating system resources, such as message queues, shared memory, and semaphores. When a BEA Tuxedo application shuts down properly with the tmshutdown command, all IPC resources are removed from the system. In some cases, however, an application may fail to shut down properly and stray IPC resources may remain on the system. When this happens, it may not be possible to reboot the application.
One way to address this problem is to remove IPC resources with a script that invokes the system IPCS command and scan for all IPC resources owned by a particular user account. However, with this method, it is difficult to distinguish among different sets of IPC resources; some may belong to the BEA Tuxedo system; some to a particular BEA Tuxedo application; and others to applications unrelated to the BEA Tuxedo system. It is important to be able to distinguish among these sets of resources; unintentional removal of IPC resources can severely damage an application.
The BEA Tuxedo IPC tool (that is, the tmipcrm command) enables you to remove IPC resources allocated by the BEA Tuxedo system (that is, for core BEA Tuxedo and Workstation components only) in an active application.
The command to remove IPC resources, tmipcrm, resides in TUXDIR/bin. This command reads the binary configuration file (TUXCONFIG), and attaches to the bulletin board using the information in this file. tmipcrm works only on the local server machine; it does not clean up IPC resources on remote machines in a BEA Tuxedo configuration.
To run this command, enter it as follows on the command line:
tmipcrm [-y] [-n] [TUXCONFIG_file]
The IPC tool lists all IPC resources used by the BEA Tuxedo system and gives you the option of removing them.
Note: This command will not work unless you have set the TUXCONFIG environment variable correctly or specified the appropriate TUXCONFIG file on the command line.
Troubleshooting Multithreaded/
Multicontexted Applications
Debugging Multithreaded/Multicontexted Applications
Multithreaded applications can be much more difficult to debug than single-threaded applications. As the administrator, you may want to establish a policy governing whether such multithreaded applications should be created.
Limitations of Protected Mode in a Multithreaded Application
When running in protected mode, an application attaches to shared memory only when an ATMI call is being executed. Protected mode is used to guard against problems that arise when BEA Tuxedo shared memory is accidentally overwritten by stray application pointers.
If your multithreaded application is running in protected mode, some threads may be executing application code while others are attached to the BEA Tuxedo Bulletin Board's shared memory within a BEA Tuxedo function call. Therefore, as long as at least one thread is attached to the bulletin board in an ATMI call, the use of protected mode cannot guard against stray application pointers in threads executing application code, which may overwrite the BEA Tuxedo shared memory. As a result, the usefulness of protected mode is relatively limited in multithreaded applications.
There is no solution to this limitation. We simply want to warn you that when running a multithreaded application you cannot rely on protected mode as much as you do when running a single-threaded application.
![]() |
![]() |
![]() |
![]() |
||
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |