|
|
This topic includes the following sections:
The first step in troubleshooting is determining problem areas. In most applications you must consider six possible sources of trouble:
Once you have determined the problem area, you must then work with the appropriate administrator to resolve the problem. If, for example, you determine that the trouble is caused by a networking problem, you must work with the network administrator.
The following steps will help you detect the source of an application failure.
ULOG).ULOG.stdout and stderr, respectively).stdout and stderr files are located in the directory defined by the APPDIR variable.stdout and stderr files for your clients and servers may have been renamed. (You can rename the stdout and stderr files by specifying -e and -o in the appropriate client and server definitions in your configuration file. For details, see servopts(5) in the File Formats, Data Descriptions, MIBs, and System Processes Reference.)APPDIR.variable. Use a debugger such as dbx to get a stack trace. If you find core dumps, notify your application developer.sar(1) command) to determine why your system is not functioning properly. Consider the following reasons:The following steps will help you detect the source of a system failure.
ULOG):
The EventBroker enhances troubleshooting by providing a system-wide summary of events and a mechanism whereby an event triggers notification. The EventBroker provides details about BEA Tuxedo system events, such as servers dying and networks failing, or application events, such as an ATM machine running out of money. A BEA Tuxedo client that receives unsolicited notification of an event, can name a service routine to be invoked, or name an application queue in which data should be stored for later processing. A BEA Tuxedo server that receives unsolicited notification can specify a service request or name an application queue to store data.
The text may not include more than 80 characters. The system sends the message in a STRING type buffer, which means the client's unsolicited message handling function (specified by tpsetunsol(0)) must be able to handle this type of message. The tptypes() function may be useful in this case.
Periodically, you may need to perform the following tasks to maintain your file system:
| Note: | This file format is used for TUXCONFIG, TLOG, and /Q. |
To print a UDL, complete the following procedure:
To print VTOC information, complete the following procedure.
To reinitialize a device that is included on a device list, complete the following procedure.
tmadmin -c.initdl [-zdevicename] [-yes]devindx
| Note: | The value of devindx is the index to the file to be destroyed. |
-yes option on the command line, you are not prompted to confirm your intention to destroy the file before the file is actually destroyed.To create a device list, complete the following procedure.
tmadmin -c.crdl [-zdevicename] [-bblocks]
devicename [devindx] is the desired device name. (Another way to assign a name to a new device is by setting the FSCONFIG environment variable to the desired device name.)blocks is the number of blocks needed. The default is 1000 blocks.| Note: | Because 35 blocks are needed for the administrative overhead associated with a TLOG, be sure to assign a value higher than 35 when you create a TLOG. |
To destroy a device list with index devindx, complete the following procedure.
tmadmin -c.dsdl [-zdevicename] [yes] [devindx]
| Note: | The value of devindx is the index to the file to be destroyed. |
The BEA Tuxedo system requires a certain level of environmental stability to provide optimum functionality. Although the BEA Tuxedo administrative subsystem offers unparalleled capabilities of recovering from network, machine, and application process failures, it is not invulnerable. You should be aware of the following ways in which a BEA Tuxedo system works.
Application clients and servers that use the FASTPATH model of SYSTEM_ACCESS (the default) have direct memory access to the BEA Tuxedo shared data structures. Using the FASTPATH model helps ensure that the BEA Tuxedo system achieves its outstanding performance. The BEA Tuxedo system uses the IPC (InterProcess Communication and File System) facilities provided by the operating system.
If an application accidentally uses these facilities to write into the BEA Tuxedo shared memory or to a BEA Tuxedo file descriptor, or if it mistakenly uses any other BEA Tuxedo system resource, data may become corrupted, BEA Tuxedo functionality may be compromised, or an application may be brought down.
It is inappropriate for a user or administrator to directly terminate application clients, application servers, or BEA Tuxedo administrative processes because these processes may be executing within a critical section (that is, updating shared information in shared memory). Interrupting a critical section during a memory update could potentially cause inconsistent internal data structures. (This is characteristic not only of the BEA Tuxedo system, but of any system in which shared data is used.) Error messages in the BEA Tuxedo userlog that refer to locks or semaphores may indicate that such corruption has occurred.
For maximum application availability, you can take advantage of the BEA Tuxedo system's facilities for managing redundancy, such as its multiple server, machine, and domain facilities. Distributing an application's functionality allows continued operation if a failure occurs in one area.
This topic provides instructions for troubleshooting a partition, identifying its cause, and taking action to recover from it. A network partition exists if one or more machines cannot access the MASTER machine. As the application administrator, you are responsible for detecting partitions and recovering from them.
A network partition may be caused by any the following failures:
The procedure you follow to recover from a partitioned network depends on the cause of the partition.
You can detect a network partition in one of the following ways:
ULOG) for messages that may shed light on the origin of the problem.
When problems occur with the network, BEA Tuxedo system administrative servers start sending messages to the ULOG. If the ULOG is set up over a remote file system, all messages are written to the same log. In this scenario, you can run the tail(1) command on one file and check the failure messages displayed on the screen.
If, however, the remote file system is using the network in which the problem has occurred, the remote file system may no longer be available.
151804.gumby!DBBL.28446: ... : ERROR: BBL partitioned, machine=SITE2
The following is an example of a tmadmin session in which information is being collected about a partitioned network, a server, and a service on that network. Three tmadmin commands are run:
pnw (the printnetwork command)psr (the printserver command)psc (the printservice command)$ tmadmin
> pnw SITE2
Could not retrieve status from SITE2
> psr -m SITE1
a.out Name Queue Name Grp Name ID Rq Done Load Done Current Service
BBL 30002.00000 SITE1 0 - - ( - )
DBBL 123456 SITE1 0 121 6050 MASTERBB
simpserv 00001.00001 GROUP1 1 - - ( - )
BRIDGE 16900672 SITE1 0 - - ( DEAD )
>psc -m SITE1
Service Name Routine Name a.out Grp Name ID Machine # Done Status
------------ ------------ -------- -------- -- ------- ------------
ADJUNCTADMIN ADJUNCTADMIN BBL SITE1 0 SITE1 - PART
ADJUNCTBB ADJUNCTBB BBL SITE1 0 SITE1 - PART
TOUPPER TOUPPER simpserv GROUP1 1 SITE1 - PART
BRIDGESVCNM BRIDGESVCNM BRIDGE SITE1 1 SITE1 - PART
This topic provides instructions for recovering from transient and severe network failures.
Because the BRIDGE tries, automatically, to recover from any transient network failures and reconnect, transient network failures are usually not noticed. If, however, you need to perform a manual recovery from a transient network failure, complete the following procedure.
MASTER machine, start a tmadmin(1) session.reconnect command (rco), specifying the names of nonpartitioned and partitioned machines:rconon-partioned_node1 partioned_node2
To recover from severe network failure, complete the following procedure.
The procedure you follow to restore a failed machine depends on whether that machine was the MASTER machine.
To restore a failed MASTER machine, complete the following procedure.
tmadmin session on the ACTING MASTER (SITE2):tmadmin
MASTER (SITE1) by entering the following command:boot -B SITE1
(The BBL does not boot if you have not executed pclean on SITE1.)
tmadmin, start a DBBL running again on the MASTER site (SITE1) by entering the following:MASTER
To restore a failed nonmaster machine, complete the following procedure.
MASTER machine, start a tmadmin session.pclean, specifying the partitioned machine on the command line.MASTER machine.
In the following list, SITE2, a nonmaster machine, is restored.
$ tmadmin
tmadmin - Copyright © 1987-1990 AT&T; 1991-1993 USL. All rights reserved
> pclean SITE2
Cleaning the DBBL.
Pausing 10 seconds waiting for system to stabilize.
3 SITE2 servers removed from bulletin board
> boot -B SITE2
Booting admin processes ...
Exec BBL -A :
on SITE2 -> process id=22923 ... Started.
1 process started.
> q
To replace BEA Tuxedo system components, complete the following procedure.
To replace components of your application, complete the following procedure.
By default, the BEA Tuxedo system cleans up resources associated with dead processes (such as queues) and restarts restartable dead servers from the Bulletin Board (BB) at regular intervals during BBL scans. You may, however, request cleaning at other times.
To request an immediate cleanup of resources associated with dead processes, complete the following procedure.
The bbclean command takes one optional argument: the name of the machine to be cleaned.
To clean up other resources, complete the following procedure.
| Note: | You must specify a value for machine; it is a required argument. |
This command is useful for restoring order to a system after partitioning has occurred unexpectedly.
If a BEA Tuxedo CORBA application fails to boot, open the application's UBBCONFIG file with a text editor and check whether the servers are booted in the correct order in the SERVERS section. The following is the correct order in which to boot the servers in a BEA Tuxedo CORBA environment. A BEA Tuxedo CORBA application will not boot if this order is not adhered to.
Boot the servers in the following order:
TMSYSEVT.-N option and the -M option, which starts the NameManager service (as a MASTER). This service maintains a mapping of application-supplied names to object references.-N option only, to start a slave NameManager service.TMFFNAME server with the -F option, to start the FactoryFinder.For a detailed example, see the section Required Order in Which to Boot CORBA C++ Servers" in Setting Up a BEA Tuxedo Application.
The network address that is specified by programmers in the Bootstrap object constructor or in TOBJADDR must exactly match the network address in the server application's UBBCONFIG file. The format of the address as well as the capitalization must match. If the addresses do not match, the call to the Bootstrap object constructor will fail with a seemingly unrelated error message:
ERROR: Unofficial connection from client at
<tcp/ip address>/<port-number>:
For example, if the network address is specified as //TRIXIE:3500 in the ISL command-line option string (in the server application's UBBCONFIG file), specifying either //192.12.4.6:3500 or //trixie:3500 in the Bootstrap object constructor or in TOBJADDR will cause the connection attempt to fail.
On UNIX systems, use the uname -n command on the host system to determine the capitalization used. On Windows systems, see the host system's Network control panel to determine the capitalization used.
You may want to perform the following steps on a Windows server that is running a BEA Tuxedo CORBA application, if the following problem occurs: some Internet Inter-ORB Protocol (IIOP) clients boot, but some clients fail to create a Bootstrap object and return an InvalidDomain message, even though the //host:port address is correctly specified. (For related information, see the section
How to Check the Hostname Format and Capitalization of BEA Tuxedo CORBA Servers on page 9-14.)
regedt32, the Registry Editor.HKEY_LOCAL_MACHINE on Local Machine window.HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Afd\Parameters
DynamicBacklogGrowthDelta: REG_DWORD : 0xa
EnableDynamicBacklog: REG_DWORD: 0x1
MaximumDynamicBacklog: REG_DWORD: 0x3e8
MinimumDynamicBacklog: REG_DWORD: 0x14
These values replace the static connection queue (that is, the backlog) of five pending connections with a dynamic connection backlog, that will have at least 20 entries (minimum 0x14), at most 1000 entries (maximum 0x3e8), and will increase from the minimum to the maximum by steps of 10 (growth delta 0xa).
These settings only apply to connections that have been received by the system, but are not accepted by an IIOP Listener. The minimum value of 20 and the delta of 10 are recommended by Microsoft. The maximum value depends on the machine. However, Microsoft recommends that the maximum value not exceed 5000 on a Windows server.
This topic provides instructions for aborting and committing transactions.
To abort a transaction, complete the following procedure.
aborttrans (abort) [-yes] [-ggroupname]tranindex
tranindex, run the printtrans command (a tmadmin command).groupname is specified, a message is sent to the TMS of that group to mark as "aborted" the transaction for that group. If a group is not specified, a message is sent, instead, to the coordinating TMS, requesting an abort of the transaction. You must send abort messages to all groups in the transaction to control the abort.This command is useful when the coordinating site is partitioned or when the client terminates before calling a commit or an abort. If the timeout is large, the transaction remains in the transaction table unless it is aborted.
To commit a transaction, enter the following command:
committrans (commit) [-yes] [-ggroupname]tranindex
| Note: | Both groupname and tranindex are required arguments. |
The operation fails if the transaction is not precommitted or has been marked aborted. This message should be sent to all groups to fully commit the transaction.
Be careful about using the committrans command. The only time you need to run it is when both of the following conditions apply:
Also, a client may be blocked on tpcommit(), which will be timed out. If you are going to perform an administrative commit, be sure to inform this client.
When the application you are administering includes database transactions, you may need to apply an after-image journal (AIJ) to a restored database following a disk corruption failure. Or you may need to coordinate the timing of this recovery activity with your site's database administrator (DBA). Typically, the database management software automatically performs transaction rollback when an error occurs. When the disk containing database files has become corrupted permanently, however, you or the DBA may need to step in and perform the rollforward operation.
Assume that a disk containing portions of a database is corrupted at 3:00 P.M. on a Wednesday. For this example, assume that a shadow volume (that is, you have disk mirroring) does not exist.
Refer to the documentation for the resource manager (database product) for specific instructions on the database rollforward process.
Inter-process communication (IPC) resources are operating system resources, such as message queues, shared memory, and semaphores. When a BEA Tuxedo application shuts down properly with the tmshutdown command, all IPC resources are removed from the system. In some cases, however, an application may fail to shut down properly and stray IPC resources may remain on the system. When this happens, it may not be possible to reboot the application.
One way to address this problem is to remove IPC resources with a script that invokes the system IPCS command and scan for all IPC resources owned by a particular user account. However, with this method, it is difficult to distinguish among different sets of IPC resources; some may belong to the BEA Tuxedo system; some to a particular BEA Tuxedo application; and others to applications unrelated to the BEA Tuxedo system. It is important to be able to distinguish among these sets of resources; unintentional removal of IPC resources can severely damage an application.
The BEA Tuxedo IPC tool (that is, the tmipcrm command) enables you to remove IPC resources allocated by the BEA Tuxedo system (that is, for core BEA Tuxedo and Workstation components only) in an active application.
The command to remove IPC resources, tmipcrm, resides in TUXDIR/bin. This command reads the binary configuration file (TUXCONFIG), and attaches to the bulletin board using the information in this file. tmipcrm works only on the local server machine; it does not clean up IPC resources on remote machines in a BEA Tuxedo configuration.
To run this command, enter it as follows on the command line:
tmipcrm [-y] [-n] [TUXCONFIG_file]The IPC tool lists all IPC resources used by the BEA Tuxedo system and gives you the option of removing them.
| Note: | This command will not work unless you have set the TUXCONFIG environment variable correctly or specified the appropriate TUXCONFIG file on the command line. |
Multithreaded applications can be much more difficult to debug than single-threaded applications. As the administrator, you may want to establish a policy governing whether such multithreaded applications should be created.
When running in protected mode, an application attaches to shared memory only when an ATMI call is being executed. Protected mode is used to guard against problems that arise when BEA Tuxedo shared memory is accidentally overwritten by stray application pointers.
If your multithreaded application is running in protected mode, some threads may be executing application code while others are attached to the BEA Tuxedo Bulletin Board's shared memory within a BEA Tuxedo function call. Therefore, as long as at least one thread is attached to the bulletin board in an ATMI call, the use of protected mode cannot guard against stray application pointers in threads executing application code, which may overwrite the BEA Tuxedo shared memory. As a result, the usefulness of protected mode is relatively limited in multithreaded applications.
There is no solution to this limitation. We simply want to warn you that when running a multithreaded application you cannot rely on protected mode as much as you do when running a single-threaded application.
|
