[Top] [Previous Page] [Next Page] [Bottom]



Chapter 12. Monitoring a Running System

As an administrator, you must ensure that once your application is up and running, it meets (and continues to meet) the performance, availability, and security requirements your company has set for it. To perform this task, you need to monitor the resources (such as shared memory), activities (such as transactions), and potential problems (such as security breaches) in your configuration, and take any corrective actions that are necessary.

To help you meet this responsibility, the BEA TUXEDO system provides tools that enable you to oversee both system events and application events. This chapter explains how to use these tools to keep your application performing fast, well, and securely.

Specifically, this chapter discusses the following topics:

Overview of System and Application Data

This section describes the types of data available for monitoring a running system and explains how to use that data.

Components and Activities for Which Data Is Available

Your BEA TUXEDO system maintains parameter settings and generates statistics for the following system components:

Where the Data Resides

To ensure that you have the information necessary for monitoring your system, the BEA TUXEDO system provides the following three data repositories:

This chapter describes the data stored in the UBBCONFIG file and in the Bulletin Board, and provides instructions for monitoring that data. For a description of the log files, see Chapter 13, "Monitoring Log Files."

How You Can Use the Data

The administrative data provided by your BEA TUXEDO system allows you to monitor a multitude of potential trouble areas on your system. For example, this data allows you to:

Moreover, you can set up your system so that it is able to use the statistics in the Bulletin Board to make decisions and to modify system components dynamically, without your help. With proper configuration, your system may be able to do tasks such as the following (when indicated by Bulletin Board statistics):

Thus, by monitoring the administrative data for your system, you can prevent and resolve problems that threaten the performance, availability, and security of your application.

Types of Data

Two types of administrative data are available on every running BEA TUXEDO system: static and dynamic.

Static Data

Static data about your configuration consists of configuration settings that you assign when you first configure your system and application. These settings are never changed without intervention (either in real-time or through a program you have provided). Examples include system-wide parameters (such as the number of machines being used) and the amount of IPC resources (such as shared memory) allocated to your system on your local machine. Static data is kept in the UBBCONFIG file and in the Bulletin Board.

At times you will need to check the static data about your configuration. For example:

Dynamic Data

Dynamic data about your configuration consists of information that changes in real time, that is, while an application is running. For example, the load (the number of requests sent to a server) and the state of various configuration components (such as servers) change frequently. Dynamic data is kept in the Bulletin Board.

You will need to check the dynamic data about your configuration frequently. For example:

Monitoring Methods

To monitor a running application, you need to keep track of the dynamic aspects of your configuration and sometimes check the static data. Thus, you need to be able to watch the Bulletin Board on an ongoing basis and consult the UBBCONFIG file when necessary. The BEA TUXEDO system provides the following methods of doing both tasks, as shown in this table.

You Can Use the . . . By . . . For Instructions, See . . .
tmadmin command Entering commands after a prompt This chapter
AdminAPI Using the MIB (and the commands described in this chapter) to write programs that monitor your run-time application Chapter 18, "Event Broker/Monitor."
BEA TUXEDO Web-based GUI Using a graphical interface The Help accessed directly from the GUI

Which method is best for you? The answer depends on your answers to several questions.

If you decide to monitor your system at run time through the tmadmin command interpreter, continue reading; this chapter describes tmadmin and explains how to use it.

Using the tmadmin Command Interpreter

This section provides the following information:

What Is tmadmin?

The tmadmin command is an interpreter for 50 commands that let you view and modify a Bulletin Board and its associated entities.

Note: tmadmin is supported on UNIX and Windows NT platforms.

How might you want to use tmadmin to modify your system while it is running? Consider the following sample scenario. Suppose you want to check the current values for all the parameters listed in the Bulletin Board, such as maximum number of servers and services. You can do this by running the tmadmin command, bbparms.

How a tmadmin Session Works

  1. A tmadmin session starts when you (the administrator) enter the tmadmin command at a shell prompt. The shell prompt($) is replaced by the tmadmin prompt (>) which is used until you quit tmadmin.
    $ tmadmin [operating_mode_option]
    >

    You can request one of three operating modes on the command line: the default mode (which allows you to view and change the Bulletin Board and associated entities), read-only mode (-r), or configuration mode (-c).

  2. tmadmin verifies that the configuration is running. If the configuration is not running the following message is displayed:
    No bulletin board exists. Entering boot mode
    >
  3. tmadmin checks the TUXCONFIG and TUXOFFSET environment variables to get the location and offset at which the configuration file has been loaded. (Be sure you have defined these variables before beginning a tmadmin session.)
  4. tmadmin enters the Bulletin Board in one of the following three states, depending on which operating mode you have requested.
  5. The > prompt is displayed on your screen and you enter a tmadmin command.

    Not all tmadmin commands are available on every machine at all times. Which commands are available depends on several factors:

    For details, see the tmadmin(1) reference page in the BEA TUXEDO Reference Manual.

Summary of tmadmin Options

Whenever you start a tmadmin session, you have a choice of operating modes for that session: read-only mode, configuration mode, or the default operating mode. In addition, you can generate a report of the BEA TUXEDO version and license numbers.

Read-only Mode

In this mode, you can view the data in the Bulletin Board, but you cannot make any changes. The advantage of working in read-only mode is that your administrator process is not tied up by tmadmin; the tmadmin process attaches to the Bulletin Board as a client, leaving your administrator slot available for other work.

To start a tmadmin session in read-only mode, specify the -r option on the command line:

$ tmadmin -r
Configuration Mode

In this mode, you can view the data in the Bulletin Board and, if you are the BEA TUXEDO application administrator, you can make changes. You can start a tmadmin session in configuration mode on any machine, including an inactive machine. On most inactive machines, configuration mode is required. (The only inactive machine on which you can start a tmadmin session without requesting configuration mode is the MASTER machine.)

To start a tmadmin session in configuration mode, specify the -c option on the command line:

$ tmadmin -c
Default Operating Mode

If you want to view and change Bulletin Board data during a tmadmin session, you must:

  1. Have administrator privileges (that is, your effective UID and GID must be those of the administrator).
  2. Invoke the command interpreter without any options:
    $ tmadmin
Version number and license number report

To find out which version of the BEA TUXEDO system you are running and to get the license number for it, specify the -v option on the command line:

$ tmadmin -v

After displaying the version and license numbers, tmadmin exits, even if you have specified -c or -r in addition to -v. When -v is requested, all other options are ignored.

tmadmin Meta-commands

The tmadmin command interpreter is equipped with a set of meta-commands, commands that help you use tmadmin. Table 12-1 lists the tmadmin meta-commands.

Note: The tables and examples in this chapter include the abbreviated forms of the tmadmin command names.

Table 12-1 tmadmin Meta-commands

Use This Command Or its Abbreviation To
default d Set defaults for arguments of other commands
dump du Download the current Bulletin Board into a file
echo e Display input command lines
help h Display command list or command syntax
paginate page Pipe output of commands to a pager
quit q Terminate the session
verbose v Show output in verbose mode (a toggle key)
!shlcmd (n/a) Escape to the shell and run the specified shell command
!! (n/a) Repeat the previous shell command
<RETURN> (n/a) Repeat the last tmadmin command
Default

The default meta-command (d) lets you set and unset defaults for the following frequently used parameters for most tmadmin commands: group name, server ID, machine, user name, client name, queue address, service name, device blocks, device offset, and UDL configuration device path. (For details, see the tmadmin(1) reference page in the BEA TUXEDO Reference Manual.)

Note: You cannot assign defaults to any parameters for the boot and shutdown commands.

Once defaults are set, they remain in effect until the session ends or until the parameters are reset to different values. The remainder of this section provides a list of instructions for checking, setting, and unsetting defaults:

Optional versus Required Arguments

Most tmadmin commands require explicit information about the resource on which the command is to act. Required arguments can always be specified on the command line, and can often be set via the default command, as well. tmadmin reports an error if the required information is not available from either source.

Some tmadmin statistical commands interpret unspecified default parameters as all.

Running tmadmin Commands

This section provides the basic procedure for running tmadmin commands. Commands for doing specific monitoring tasks through tmadmin are provided in the section "Monitoring a Running System with tmadmin" in this chapter.

Note: For complete details about tmadmin, see the tmadmin(1) reference page in the BEA TUXEDO Reference Manual.

Perform the following steps to run the tmadmin commands.

  1. Make sure the TUXCONFIG and TUXOFFSET environment variables have been set.
  2. Enter tmadmin in the appropriate operating mode.
  3. When the tmadmin session prompt (>) is displayed, enter your first tmadmin command. Specify, on the command line, how much information from the Bulletin Board you want to have displayed.
  4. After viewing the output of your first tmadmin command, continue entering tmadmin commands until you are ready to end the session.
  5. End the tmadmin session by entering:
    quit

Monitoring a Running System with tmadmin

Table 12-2 provides a list of potential problems that you might want to check while monitoring your run-time system, along with a list of the tmadmin commands that enable you to perform such a check. The table also suggests follow-up actions you might take if the tmadmin command you run generates a particular type of output.

Note: For a comprehensive list of the tmadmin commands, see the tmadmin(1) reference page in the BEA TUXEDO Reference Manual.

.

Table 12-2 Commands for Monitoring Tasks

To Determine
Whether . . .
Run this Command . . . If . . . Then . . .
Any servers are stalled in a service $ tmadmin -r
> printserver
The Current Service and Request fields do not change The server is spending excessive time on the current service.

In a development environ- ment, the server might be stalled in an infinite loop; you may want to stop it.

The load distribution is appropriate $ tmadmin -r
> printserver
The values in the Load Done field are not reasonably similar Check the layout of the MSSQs and the data-dependent routing.

If the current servers have too heavy a load, you may want to boot more servers.

A particular service is doing any work $ tmadmin -r
> printservice
The value in the Requests Completed field is 0 Data-dependent routing may be preventing requests from being sent to that server for that service. You can:
Any clients are inactive $ tmadmin -r
> printclient
Tell the client-via a broadcast message-to exit
The work is distributed in such a way that it is flowing smoothly through the system $ tmadmin -r
> printqueue
Some queues are always heavy and others are not Check the arrangement of services within servers, data-dependent routing, and/or queue organization.
A client is tying up a connection and preventing a server from doing any work for another client $ tmadmin -r
> printconn
A client is maintaining control of a connection and is not issuing any requests
  1. Suspend the client by using the client MIB. (We recommend using the BEA TUXEDO Web-based GUI for this task.)
  2. Terminate the client.
The network is stable $ tmadmin -r
> printnet
A machine is no longer connected You may want to:
  1. Partition the machine (that is, take it off the network).
  2. Resolve the problem.
  3. Reconnect the machine.
You must manually commit or abort a transaction $ tmadmin -r
> printtrans
For example, the status is TMGDECIDED The first phase of the two-phase commit has completed successfully. This means you must find out why the second phase cannot be completed.

For example, you may find that the coordinating TMS cannot complete the transaction because a participating site has gone down.
Your operating system resources (such as shared memory and semaphores) on a local machine are sufficient $tmadmin -r
> bbsread
You do not have sufficient resources in the operating system Increase the IPC resources (semaphores, shared memory segments, and so on) in the operating system.
You want to keep the current values for system-wide parameters (in the RESOURCES section of your UBBCONFIG file) $ tmadmin -r
> bbparms
You do not have sufficient resources for your application
  1. Stop the application.
  2. Configure additional IPC resources (assuming you have enough available) by increasing the values of relevant parameters (such as MAXSERVERS and MAXCLIENTS) in the RESOURCES section of the configuration file.
  3. Re-boot the application.

Example: Output from tmadmin Commands

This section provides examples of output from the following tmadmin monitoring commands:

printqueue Output

The following output from the printqueue command lets you check the distribution of work in the bankapp application.

printqueue [qaddress]

tmadmin - Copyright © 1987-1990 AT&T; 1991-1993 USL.  All rights reserved.

>pq
 
a.out Name Queue Nam # Svrs Wk Q'd # Queued Ave. Len Machine
TLR

TLR

28706

28706

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

TMS_SQL

TMS_SQL

BANKB1_T

BANKB1_T

2

2

0

0

0

0

0.0

0.0

SITE1

SITE1

TLR

TLR

24946

24946

1

1

0

0

0

0

0.1

0.1

SITE1

SITE1

BAL

BAL

8533

8533

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

BAL

BAL

24915

24915

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

BTADD

BTADD

28897

28897

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

XFER

XFER

4380

4380

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

XFER

XFER

28840

28840

1

1

100

100

0

0

1.0

1.0

SITE1

SITE1

TLR

TLR

12519

12519

1

1

100

100

2

2

0.0

0.0

SITE1

SITE1

BBL

BBL

24846

24846

1

1

0

0

2

2

0.0

0.0

SITE1

SITE1

ACCT

ACCT

71

71

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

TMS_SQL

TMS_SQL

BANKB3_T

BANKB3_T

2

2

0

0

0

0

0.0

0.0

SITE1

SITE1

BAL

BAL

28958

28958

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

ACCT

ACCT

254

254

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

BTADD

BTADD

12310

12310

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

XFER

XFER

16494

16494

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

TMS_SQL

TMS_SQL

BANKB2_T

BANKB2_T

2

2

0

0

0

0

0.0

0.0

SITE1

SITE1

BTADD

BTADD

8430

8430

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

ACCT

ACCT

24641

24641

1

1

0

0

0

0

0.0

0.0

SITE1

SITE1

Note: By default, information is supplied for all queues. If you want your output to be limited to information about only one queue, specify the address for the desired queue.

The output of this command includes the following information.

printconn Data

The following (verbose) output from the printconn command shows that the client process has:

printnet Command Output

This section shows the output from the following procedure.

  1. The printnet command was run. (The output shows the number of messages sent and received by both sites.)
  2. The BRIDGE process at SITE2 was stopped.
  3. The printnet command was re-entered. (The output shows that SITE2 is no longer connected to the master machine, SITE1.)
    printnet [-m machine_list]
    
    tmadmin - Copyright © 1987-1990 AT&T; 1991-1993 USL. All rights reserved.
    
    > echo 
    Echo now on.
    
    > pnw
    SITE1     Connected To:   msgs sent   msgs received
    SITE2	     100103
    
    SITE2     Connected To:   msgs sent   msgs received
    SITE1	     104             101
    
    > pnw
    SITE1     Connected To:   msgs sent   msgs received
    
               Could not retrieve status from SITE2
    
    >

printtrans Command Output

The printtrans command reports statistics only for transactions that are currently in progress, specifically, statistics on the number of rollbacks, commits, and aborts that have been executed on your machine, group, or server.

This section shows the output produced by running the printtrans command in terse and verbose modes:

Case Study: Monitoring Run-time bankapp

This section presents a sample configuration for a multiprocessor (MP) version of the bankapp application. This section also shows the output that was returned when the local IPC resources and system-wide parameters were checked by running the appropriate tmadmin commands.

Configuration File for bankapp

For this case study, we will use the configuration defined in the UBBCONFIG file shown in Listing 12-3.

Listing 12-3 UBBCONFIG File for bankapp (MP Version)


#Copyright (c) 1997, 1998 BEA Systems, Inc.#All rights reserved
RESOURCESIPCKEY            80952UID               4196GID               601PERM              0660MAXACCESSERS      40MAXSERVERS        35MAXSERVICES       75MAXCONV           10MAXGTT            20MASTER            SITE1,SITE2
SCANUNIT          10SANITYSCAN        12BBLQUERY          180BLOCKTIME         30DBBLWAIT          6OPTIONS           LAN,MIGRATEMODEL             MPLDBAL             Y#MACHINESmchn1             LMID=SITE1                  TUXDIR="/home/tuxroot"                  APPDIR="/home/apps/bank"                  ENVFILE="/home/apps/bank/ENVFILE"                  TLOGDEVICE="/home/apps/bank/TLOG"                  TLOGNAME=TLOG                  TUXCONFIG="/home/apps/bank/tuxconfig"                  TYPE="3B2"                  ULOGPFX="/home/apps/bank/ULOG"wgs386            LMID=SITE2                  TUXDIR="/home2/tuxroot"                  APPDIR="/home2/apps/bank"                  ENVFILE="/home2/apps/bank/ENVFILE"                  TLOGDEVICE="/home2/apps/bank/TLOG"                  TLOGNAME=TLOG                  TUXCONFIG="/home2/apps/bank/tuxconfig"                  TYPE="386"                  ULOGPFX="/home2/apps/bank/ULOG"#GROUPSDEFAULT:  TMSNAME=TMS_SQL   TMSCOUNT=2# For NT/Netware,  :bankdb: becomes ;bankdb;BANKB1              LMID=SITE1         GRPNO=1          OPENINFO="TUXEDO/SQL:/home/apps/bank/bankdl1:bankdb:readwrite"BANKB2              LMID=SITE2         GRPNO=2          OPENINFO="TUXEDO/SQL:/home2/apps/bank/bankdl2:bankdb:readwrite"
NETWORKSITE1     NADDR="//mach1.beasys.com:1900"          BRIDGE="/dev/tcp"          NLSADDR="//mach1.beasys.com:1900"SITE2     NADDR="//mach386.beasys.com:1900"          BRIDGE="/dev/tcp"          NLSADDR="//mach386.beasys.com:1900"SERVERS#DEFAULT:  RESTART=Y MAXGEN=5 REPLYQ=Y CLOPT="-A"
TLR       SRVGRP=BANKB1    SRVID=1    RQADDR=tlr1    CLOPT="-A -- -T 100"TLR       SRVGRP=BANKB1    SRVID=2    RQADDR=tlr1    CLOPT="-A -- -T 200"TLR       SRVGRP=BANKB2    SRVID=3    RQADDR=tlr2    CLOPT="-A -- -T 600"TLR       SRVGRP=BANKB2    SRVID=4    RQADDR=tlr2    CLOPT="-A -- -T 700"XFER      SRVGRP=BANKB1    SRVID=5XFER      SRVGRP=BANKB2    SRVID=6ACCT      SRVGRP=BANKB1    SRVID=7ACCT      SRVGRP=BANKB2    SRVID=8BAL       SRVGRP=BANKB1    SRVID=9     BAL       SRVGRP=BANKB2    SRVID=10BTADD     SRVGRP=BANKB1BTADD     SRVGRP=BANKB2    SRVID=12AUDITC    SRVGRP=BANKB1    SRVID=13 CONV=Y MIN=1 MAX=10BALC      SRVGRP=BANKB1    SRVID=24BALC      SRVGRP=BANKB2    SRVID=25#
SERVICESDEFAULT:  LOAD=50          AUTOTRAN=NWITHDRAWAL         PRIO=50          ROUTING=ACCOUNT_IDDEPOSIT            PRIO=50          ROUTING=ACCOUNT_IDTRANSFER           PRIO=50          ROUTING=ACCOUNT_IDINQUIRY            PRIO=50          ROUTING=ACCOUNT_IDCLOSE_ACCT         PRIO=40          ROUTING=ACCOUNT_IDOPEN_ACCT          PRIO=40          ROUTING=BRANCH_IDBR_ADD             PRIO=20          ROUTING=BRANCH_IDTLR_ADD            PRIO=20          ROUTING=BRANCH_IDABAL               PRIO=30          ROUTING=b_idTBAL               PRIO=30          ROUTING=b_idABAL_BID           PRIO=30          ROUTING=b_idTBAL_BID           PRIO=30          ROUTING=b_idABALC_BID          PRIO=30          ROUTING=b_idTBALC_BID          PRIO=30          ROUTING=b_id
ROUTINGACCOUNT_ID         FIELD=ACCOUNT_ID                   BUFTYPE="FML"                   RANGES="10000-59999:BANKB1,                            60000-109999:BANKB2,                             *:*"BRANCH_ID FIELD=BRANCH_ID                   BUFTYPE="FML"                   RANGES="1-5:BANKB1,                            6-10:BANKB2,                            *:*"b_id               FIELD=b_id                   BUFTYPE="VIEW:aud"                   RANGES="1-5:BANKB1,                            6-10:BANKB2,                            *:*"

Output from Checking the Local IPC Resources

To check the local IPC resources for this configuration, a tmadmin session was started, and the bbsread command was run. The output of bbsread is shown in Listing 12-4.

Listing 12-4 bbsread Output


SITE1> bbsread
IPC resources for the bulletin board on machine SITE1:SHARED MEMORY:          Key: 0x1013c38SEGMENT 0:                         ID: 15730                       Size: 36924         Attached processes: 12      Last attach/detach by: 4181
This semaphore is the system semaphoreSEMAPHORE:              Key: 0x1013c38                         Id: 15666       |  semaphore |  current |   last   | # waiting |       |   number   |  status  | accesser | processes |       |----------------------------------------------|       |     0      |   free   |   4181   |     0     |       |------------|----------|----------|-----------|This semaphore set is part of the user-level semaphoreSEMAPHORE:              Key: IPC_PRIVATE                         Id: 11572
       |  semaphore |  current |   last   | # waiting |       |   number   |  status  | accesser | processes |       |----------------------------------------------|       |      0     |  locked  |   4181   |     0     |       |      1     |  locked  |   4181   |     0     |       |      2     |  locked  |   4181   |     0     |       |      3     |  locked  |   4181   |     0     |       |      4     |  locked  |   4181   |     0     |       |      5     |  locked  |   4181   |     0     |       |      6     |  locked  |   4181   |     0     |       |      7     |  locked  |   4181   |     0     |       |      8     |  locked  |   4181   |     0     |       |      9     |  locked  |   4181   |     0     |       |     10     |  locked  |   4181   |     0     |       |     11     |  locked  |   4181   |     0     |       |     12     |  locked  |   4181   |     0     |       |     13     |  locked  |   4181   |     0     |       |------------|----------|----------|-----------|

Note: The display is the same with verbose mode on or off.

Output from Checking System-wide Parameter Settings

To check the current values of the system-wide parameters for this configuration, we started a tmadmin session and ran the bbparms command. The output of bbparms is shown in Listing 12-5.

Listing 12-5 Sample bbparms Output


> bbparmsBulletin Board Parameters:        MAXSERVERS: 35       MAXSERVICES: 75      MAXACCESSERS: 40            MAXGTT: 20           MAXCONV: 10        MAXBUFTYPE: 16       MAXBUFSTYPE: 32            IPCKEY: 35384            MASTER: SITE1,SITE2             MODEL: MP             LDBAL: Y           OPTIONS: LAN,MIGRATE          SCANUNIT: 10        SANITYSCAN: 12          DBBLWAIT: 6          BBLQUERY: 180           BLOCKTIME: 30

Note: The display is the same with verbose mode on or off.



[Top] [Previous Page] [Next Page] [Bottom]