This topic includes the following sections:
The first step in troubleshooting is determining problem areas. In most applications you must consider six possible sources of trouble:
Once you have determined the problem area, you must then work with the appropriate administrator to resolve the problem. If, for example, you determine that the trouble is caused by a networking problem, you must work with the network administrator.
The following steps will help you detect the source of an application failure.
stderrfiles are located in the directory defined by the
stderrfiles for your clients and servers may have been renamed. (You can rename the
stderrfiles by specifying
-oin the appropriate client and server definitions in your configuration file. For details, see servopts(5) in the File Formats, Data Descriptions, MIBs, and System Processes Reference.)
APPDIR.variable. Use a debugger such as
dbxto get a stack trace. If you find core dumps, notify your application developer.
sar(1) command) to determine why your system is not functioning properly. Consider the following reasons:
The following steps will help you detect the source of a system failure.
The EventBroker enhances troubleshooting by providing a system-wide summary of events and a mechanism whereby an event triggers notification. The EventBroker provides details about Oracle Tuxedo system events, such as servers dying and networks failing, or application events, such as an ATM machine running out of money. An Oracle Tuxedo client that receives unsolicited notification of an event, can name a service routine to be invoked, or name an application queue in which data should be stored for later processing. An Oracle Tuxedo server that receives unsolicited notification can specify a service request or name an application queue to store data.
The text may not include more than 80 characters. The system sends the message in a
STRING type buffer, which means the client’s unsolicited message handling function (specified by
tpsetunsol(0)) must be able to handle this type of message. The
tptypes() function may be useful in this case.
Periodically, you may need to perform the following tasks to maintain your file system:
|Note:||This file format is used for
To print a UDL, complete the following procedure:
To print VTOC information, complete the following procedure.
To reinitialize a device that is included on a device list, complete the following procedure.
|Note:||The value of
-yesoption on the command line, you are not prompted to confirm your intention to destroy the file before the file is actually destroyed.
To create a device list, complete the following procedure.
devindx] is the desired device name. (Another way to assign a name to a new device is by setting the
FSCONFIGenvironment variable to the desired device name.)
blocksis the number of blocks needed. The default is 1000 blocks.
|Note:||Because 35 blocks are needed for the administrative overhead associated with a TLOG, be sure to assign a value higher than 35 when you create a TLOG.|
To destroy a device list with index
devindx, complete the following procedure.
devicename] [yes] [
|Note:||The value of
The Oracle Tuxedo system requires a certain level of environmental stability to provide optimum functionality. Although the Oracle Tuxedo administrative subsystem offers unparalleled capabilities of recovering from network, machine, and application process failures, it is not invulnerable. You should be aware of the following ways in which an Oracle Tuxedo system works.
Application clients and servers that use the
FASTPATH model of
SYSTEM_ACCESS (the default) have direct memory access to the Oracle Tuxedo shared data structures. Using the
FASTPATH model helps ensure that the Oracle Tuxedo system achieves its outstanding performance. The Oracle Tuxedo system uses the IPC (InterProcess Communication and File System) facilities provided by the operating system.
If an application accidentally uses these facilities to write into the Oracle Tuxedo shared memory or to an Oracle Tuxedo file descriptor, or if it mistakenly uses any other Oracle Tuxedo system resource, data may become corrupted, Oracle Tuxedo functionality may be compromised, or an application may be brought down.
It is inappropriate for a user or administrator to directly terminate application clients, application servers, or Oracle Tuxedo administrative processes because these processes may be executing within a critical section (that is, updating shared information in shared memory). Interrupting a critical section during a memory update could potentially cause inconsistent internal data structures. (This is characteristic not only of the Oracle Tuxedo system, but of any system in which shared data is used.) Error messages in the Oracle Tuxedo userlog that refer to locks or semaphores may indicate that such corruption has occurred.
For maximum application availability, you can take advantage of the Oracle Tuxedo system’s facilities for managing redundancy, such as its multiple server, machine, and domain facilities. Distributing an application’s functionality allows continued operation if a failure occurs in one area.
This topic provides instructions for troubleshooting a partition, identifying its cause, and taking action to recover from it. A network partition exists if one or more machines cannot access the
MASTER machine. As the application administrator, you are responsible for detecting partitions and recovering from them.
A network partition may be caused by any the following failures:
The procedure you follow to recover from a partitioned network depends on the cause of the partition.
You can detect a network partition in one of the following ways:
ULOG) for messages that may shed light on the origin of the problem.
When problems occur with the network, Oracle Tuxedo system administrative servers start sending messages to the
ULOG. If the
ULOG is set up over a remote file system, all messages are written to the same log. In this scenario, you can run the
tail(1) command on one file and check the failure messages displayed on the screen.
If, however, the remote file system is using the network in which the problem has occurred, the remote file system may no longer be available.
151804.gumby!DBBL.28446: ... : ERROR: BBL partitioned, machine=SITE2
The following is an example of a
tmadmin session in which information is being collected about a partitioned network, a server, and a service on that network. Three
tmadmin commands are run:
> pnw SITE2
Could not retrieve status from SITE2
> psr -m SITE1
a.out Name Queue Name Grp Name ID Rq Done Load Done Current Service
BBL 30002.00000 SITE1 0 - - ( - )
DBBL 123456 SITE1 0 121 6050 MASTERBB
simpserv 00001.00001 GROUP1 1 - - ( - )
BRIDGE 16900672 SITE1 0 - - ( DEAD )
>psc -m SITE1
Service Name Routine Name a.out Grp Name ID Machine # Done Status
------------ ------------ -------- -------- -- ------- ------------
ADJUNCTADMIN ADJUNCTADMIN BBL SITE1 0 SITE1 - PART
ADJUNCTBB ADJUNCTBB BBL SITE1 0 SITE1 - PART
TOUPPER TOUPPER simpserv GROUP1 1 SITE1 - PART
BRIDGESVCNM BRIDGESVCNM BRIDGE SITE1 1 SITE1 - PART
This topic provides instructions for recovering from transient and severe network failures.
BRIDGE tries, automatically, to recover from any transient network failures and reconnect, transient network failures are usually not noticed. If, however, you need to perform a manual recovery from a transient network failure, complete the following procedure.
MASTERmachine, start a tmadmin(1) session.
rco), specifying the names of nonpartitioned and partitioned machines:
To recover from severe network failure, complete the following procedure.
The procedure you follow to restore a failed machine depends on whether that machine was the
To restore a failed
MASTER machine, complete the following procedure.
tmadminsession on the
SITE1) by entering the following command:
boot -B SITE1
tmadmin, start a DBBL running again on the
SITE1) by entering the following:
To restore a failed nonmaster machine, complete the following procedure.
MASTERmachine, start a
pclean, specifying the partitioned machine on the command line.
In the following list,
SITE2, a nonmaster machine, is restored.
tmadmin - Copyright © 1987-1990 AT&T; 1991-1993 USL. All rights reserved
> pclean SITE2
Cleaning the DBBL.
Pausing 10 seconds waiting for system to stabilize.
3 SITE2 servers removed from bulletin board
> boot -B SITE2
Booting admin processes ...
Exec BBL -A :
on SITE2 -> process id=22923 ... Started.
1 process started.
To replace Oracle Tuxedo system components, complete the following procedure.
To replace components of your application, complete the following procedure.
By default, the Oracle Tuxedo system cleans up resources associated with dead processes (such as queues) and restarts restartable dead servers from the Bulletin Board (BB) at regular intervals during BBL scans. You may, however, request cleaning at other times.
To request an immediate cleanup of resources associated with dead processes, complete the following procedure.
bbclean command takes one optional argument: the name of the machine to be cleaned.
To clean up other resources, complete the following procedure.
|Note:||You must specify a value for
This command is useful for restoring order to a system after partitioning has occurred unexpectedly.
If a Oracle Tuxedo CORBA application fails to boot, open the application’s
UBBCONFIG file with a text editor and check whether the servers are booted in the correct order in the
SERVERS section. The following is the correct order in which to boot the servers in a Oracle Tuxedo CORBA environment. A Oracle Tuxedo CORBA application will not boot if this order is not adhered to.
Boot the servers in the following order:
-Noption and the
-Moption, which starts the NameManager service (as a
MASTER). This service maintains a mapping of application-supplied names to object references.
-Noption only, to start a slave NameManager service.
TMFFNAMEserver with the
-Foption, to start the FactoryFinder.
For a detailed example, see the sectionto Boot CORBA C++ Servers” in Setting Up an Oracle Tuxedo Application.
The network address that is specified by programmers in the Bootstrap object constructor or in
TOBJADDR must exactly match the network address in the server application’s
UBBCONFIG file. The format of the address as well as the capitalization must match. If the addresses do not match, the call to the Bootstrap object constructor will fail with a seemingly unrelated error message:
ERROR: Unofficial connection from client at
For example, if the network address is specified as //TRIXIE:3500 in the ISL command-line option string (in the server application’s
UBBCONFIG file), specifying either //220.127.116.11:3500 or //trixie:3500 in the Bootstrap object constructor or in TOBJADDR will cause the connection attempt to fail.
On UNIX systems, use the uname -n command on the host system to determine the capitalization used. On Windows systems, see the host system’s Network control panel to determine the capitalization used.
You may want to perform the following steps on a Windows server that is running a Oracle Tuxedo CORBA application, if the following problem occurs: some Internet Inter-ORB Protocol (IIOP) clients boot, but some clients fail to create a Bootstrap object and return an
InvalidDomain message, even though the
//host:port address is correctly specified. (For related information, see the section .)
regedt32, the Registry Editor.
HKEY_LOCAL_MACHINE on Local Machinewindow.
These values replace the static connection queue (that is, the backlog) of five pending connections with a dynamic connection backlog, that will have at least 20 entries (minimum 0x14), at most 1000 entries (maximum 0x3e8), and will increase from the minimum to the maximum by steps of 10 (growth delta 0xa).
These settings only apply to connections that have been received by the system, but are not accepted by an IIOP Listener. The minimum value of 20 and the delta of 10 are recommended by Microsoft. The maximum value depends on the machine. However, Microsoft recommends that the maximum value not exceed 5000 on a Windows server.
This topic provides instructions for aborting and committing transactions.
To abort a transaction, complete the following procedure.
aborttrans (abort) [-yes] [-g
tranindex, run the
groupnameis specified, a message is sent to the TMS of that group to mark as “aborted” the transaction for that group. If a group is not specified, a message is sent, instead, to the coordinating TMS, requesting an abort of the transaction. You must send abort messages to all groups in the transaction to control the abort.
This command is useful when the coordinating site is partitioned or when the client terminates before calling a commit or an abort. If the timeout is large, the transaction remains in the transaction table unless it is aborted.
To commit a transaction, enter the following command:
committrans (commit) [-yes] [-g
The operation fails if the transaction is not precommitted or has been marked aborted. This message should be sent to all groups to fully commit the transaction.
Be careful about using the
committrans command. The only time you need to run it is when both of the following conditions apply:
Also, a client may be blocked on
tpcommit(), which will be timed out. If you are going to perform an administrative commit, be sure to inform this client.
When the application you are administering includes database transactions, you may need to apply an after-image journal (AIJ) to a restored database following a disk corruption failure. Or you may need to coordinate the timing of this recovery activity with your site’s database administrator (DBA). Typically, the database management software automatically performs transaction rollback when an error occurs. When the disk containing database files has become corrupted permanently, however, you or the DBA may need to step in and perform the rollforward operation.
Assume that a disk containing portions of a database is corrupted at 3:00 P.M. on a Wednesday. For this example, assume that a shadow volume (that is, you have disk mirroring) does not exist.
Refer to the documentation for the resource manager (database product) for specific instructions on the database rollforward process.
Inter-process communication (IPC) resources are operating system resources, such as message queues, shared memory, and semaphores. When a Oracle Tuxedo application shuts down properly with the
tmshutdown command, all IPC resources are removed from the system. In some cases, however, an application may fail to shut down properly and stray IPC resources may remain on the system. When this happens, it may not be possible to reboot the application.
One way to address this problem is to remove IPC resources with a script that invokes the system
IPCS command and scan for all IPC resources owned by a particular user account. However, with this method, it is difficult to distinguish among different sets of IPC resources; some may belong to the Oracle Tuxedo system; some to a particular Oracle Tuxedo application; and others to applications unrelated to the Oracle Tuxedo system. It is important to be able to distinguish among these sets of resources; unintentional removal of IPC resources can severely damage an application.
The Oracle Tuxedo IPC tool (that is, the
tmipcrm command) enables you to remove IPC resources allocated by the Oracle Tuxedo system (that is, for core Oracle Tuxedo and Workstation components only) in an active application.
The command to remove IPC resources,
tmipcrm, resides in
TUXDIR/bin. This command reads the binary configuration file (
TUXCONFIG), and attaches to the bulletin board using the information in this file.
tmipcrm works only on the local server machine; it does not clean up IPC resources on remote machines in a Oracle Tuxedo configuration.
To run this command, enter it as follows on the command line:
tmipcrm [-y] [-n] [
The IPC tool lists all IPC resources used by the Oracle Tuxedo system and gives you the option of removing them.
|Note:|| This command will not work unless you have set the
Multithreaded applications can be much more difficult to debug than single-threaded applications. As the administrator, you may want to establish a policy governing whether such multithreaded applications should be created.
When running in protected mode, an application attaches to shared memory only when an ATMI call is being executed. Protected mode is used to guard against problems that arise when Oracle Tuxedo shared memory is accidentally overwritten by stray application pointers.
If your multithreaded application is running in protected mode, some threads may be executing application code while others are attached to the Oracle Tuxedo Bulletin Board’s shared memory within an Oracle Tuxedo function call. Therefore, as long as at least one thread is attached to the bulletin board in an ATMI call, the use of protected mode cannot guard against stray application pointers in threads executing application code, which may overwrite the Oracle Tuxedo shared memory. As a result, the usefulness of protected mode is relatively limited in multithreaded applications.
There is no solution to this limitation. We simply want to warn you that when running a multithreaded application you cannot rely on protected mode as much as you do when running a single-threaded application.