8 Monitoring Data Manager Activity

Learn how to check the status of Oracle Communications Billing and Revenue Management (BRM) Data Managers (DMs) at regular intervals to monitor resource usage. You can also make inferences about the operation of the DM by checking the status at intervals and comparing the results with what you expect.

Topics in this document:

• Manually Checking the Status of the DM

• Monitoring DM Shared Memory Usage

• Monitoring DM Transaction Queues

• Monitoring DM Back Ends

• Monitoring DM Front Ends

• Increasing the Level of Reporting for a DM

• Replacing Failed DM Child Processes

Manually Checking the Status of the DM

You can check and view the status of the DM in flist format and in a report format.

• Checking the DM Status in flist Format

• Checking the DM Status in Report Format

Note:

You can also use Oracle Enterprise Manager to check the status of the DM.

Checking the DM Status in flist Format

To check the status of the DM in flist format:

1. Go to the BRM_home/sys/test directory.

2. Enter the following commands:

   testnap
   robj - database_number /status_dm 1
   

   where database_number is the database number of the DM for which you want the status.

   Note:

   You can check the status of only one DM at a time.

BRM displays the status of the DM in flist format.

Table 8-1 describes the fields in /status_dm.

Table 8-1 /status_dm Object Fields

/status_dm Object Field	Description
PIN_FLD_DM_BIGSIZE	Specifies the size, in bytes, of the big part of the DM shared memory.
PIN_FLD_SM_PASSTHRU_NAME	Specifies the current value of the dm_sm_pass_thru_obj entry in the DM pin.conf file.
PIN_FLD_SM_SHMSIZE	Specifies the maximum shared memory size, in bytes, for a custom DM. Note: Ignore this field if your system uses Oracle DM.
PIN_FLD_TRANS_OP_QUEUED	Specifies the number of transactions currently queued. This is an instantaneous counter.
PIN_FLD_DM_BACKEND	Array that defines the DM back end.
PIN_FLD_FLAGS	Specifies the internal state of the DM back end. These states are used for the internal working of the Oracle DM. DMSHM_ALIVE 0x00001000 DMSHM_DYING 0x00002000 DMSHM_DEAD 0x00004000 DMSHM_INITING 0x00008000 DMSHM_RESTARTING 0x00010000
PIN_FLD_TATTLE_TALE	This flag is reset each time you retrieve the DM status report. This allows you to see what happened since the last DM report. DM_TATTLE_SELECT 0x0001 DM_TATTLE_CMD 0x0002 DM_TATTLE_IO_IN 0x0004 DM_TATTLE_IO_OUT 0x0008 DM_TATTLE_ALL (DM_TATTLE_SELECT | DM_TATTLE_CMD | DM_TATTLE_IO_IN | DM_TATTLE_IO_OUT)
PIN_FLD_DM_FRONTEND	Array that defines the DM back end.
PIN_FLD_FLAGS	Specifies the internal state of the DM front end. These states are used for the internal working of the Oracle DM. DMSHM_ALIVE 0x00001000 DMSHM_DYING 0x00002000 DMSHM_DEAD 0x00004000 DMSHM_INITING 0x00008000 DMSHM_RESTARTING 0x00010000
PIN_FLD_TATTLE_TALE	This flag is reset each time you retrieve the DM status report. This allows you to see what happened since the last DM report. DM_TATTLE_SELECT 0x0001 DM_TATTLE_CMD 0x0002 DM_TATTLE_IO_IN 0x0004 DM_TATTLE_IO_OUT 0x0008 DM_TATTLE_ALL (DM_TATTLE_SELECT | DM_TATTLE_CMD | DM_TATTLE_IO_IN | DM_TATTLE_IO_OUT)
PIN_FLD_CONNECTS	Specifies the number of concurrent connections the front end has received. This is an instantaneous counter.
PIN_FLD_HIWAT	Specifies the maximum number of concurrent connections the front end received during the life of the DM. This is the maximum value reached by PIN_FLD_CONNECTS for this front end
PIN_FLD_DM_FE_CONNECT	Array that defines the front-end connection.
PIN_FLD_FLAGS	Specifies the internal state for a DM context: DM_FLAGS_OPEN 0x00000001 DM_FLAGS_LOST_MSG_SENT 0x00000004 DM_FLAGS_DOING_TRANS 0x01000000 DM_FLAGS_DIED 0x00000008 DM_FLAGS_HEAP_GOT 0x00000010 DM_FLAGS_SOFT_SHUTDOWN 0x00001000 DM_FLAGS_SHUTDOWN 0x00002000
PIN_FLD_DM_FE_STATE	Specifies the current front-end state in the DM context. 0: Waiting to receive an operation from the CM. 1: Receiving an operation from the CM. 2: Sent an operation to be processed and waiting for the back end. 3: The operation is complete. 4: Sending a response to the CM.
PIN_FLD_DM_BE_STATE	Specifies the current back-end state in the DM context: 1: Busy; currently doing an operation. 2: Locked to a transaction.
PIN_FLD_DM_BE_IDX	Specifies the back-end index that is performing this connection (transaction).
PIN_FLD_DM_BACKEND	Array that defines the DM back end.
PIN_FLD_OPCODE	Specifies the number of the opcode that is being run. Note: To find an opcode's number, see the opcode header files in the BRM_home/include/ops directory.
PIN_FLD_DM_USED	Specifies the memory, in bytes, dedicated to a DM context.
PIN_FLD_DM_LOW	Specifies the smallest available free memory, in bytes, in the connection's heap.
PIN_FLD_DM_HIGH	Specifies the largest available free memory, in bytes, in the connection's heap.
PIN_FLD_DM_BIG	Specifies how much big memory this connection has allocated.

Checking the DM Status in Report Format

To check the status of the DM in report format:

1. Find the process ID (PID) of the DM master process by looking in the pid file for the DM in BRM_home/sys/dm_oracle.

2. Enter the following command:

   kill -USR1 PID_of_DM
   

   where PID_of_DM is the process ID of the DM master process.

BRM displays the status of the DM in the dm_oracle.log file. The log file shows information about the DM, such as the PID, memory usage, transaction queue, and information about the back ends and the front ends.

Monitoring DM Shared Memory Usage

You can check shared memory usage by looking in the master overview section of the DM report. The number of used and free heap blocks (# used and # free) shows memory usage, expressed in 8-KB blocks. To prevent failures associated with insufficient memory, verify that # free is a relatively large number. If # free is a small portion of # used, you should increase the size of the shared memory area. Otherwise, operations might fail, returning PIN_ERR_NO_MEM.

Monitoring DM Transaction Queues

To check the status of transactions, look in the master overview section of the DM report. The trans_op_cnt entry shows the number of transactions currently being processed, and the trans_op_queued entry shows the number waiting to be processed. For applications that require rapid response times, you can adjust the load on the system to keep to a minimum the number of transactions waiting to be processed. See "Improving Data Manager and Queue Manager Performance".

Monitoring DM Back Ends

You can use the back-end report to identify each back-end process ID (PID), the back-end status, and the number of operations processed. A value of 0x1000 (4096) for FLAGS shows that the back end is active. The report also gives information on resource usage.

The second flag value is reset each time the DM status report is received. Therefore, you can tell what has happened (at least once) since the last DM report by a flag bit being clear.

Table 8-2 shows the flag bit values:

Table 8-2 DM Flag Bit Values

Value	Flag	Description
0x8	IO output	Never cleared for back ends.
0x4	IO input	Cleared when the back end starts an operation.
0x2	CMD	Cleared when the back end is given a command or transaction.
0x1	SELECT	Cleared when the back end wakes up using select(2).

On a quiet back end, the second flag value stays at f. The counters of most interest are those that keep track of the total number of operations and total transactions.

As shown in Table 8-3, the back-end state values are a bit mask flag:

Table 8-3 Back-End State Bit Mask Values

Value	Description
0x1	Busy; currently doing an operation.
0x2	Locked to a transaction.

The back-end index and operation may be left over from the previous operation and may be no longer valid. The used field indicates memory usage. When idle, one 8-KB chunk is normally used. During an operation or transaction, this amount varies.

Monitoring DM Front Ends

You can use the front-end report to identify each front-end process ID (PID), the front-end status, and the number of operations processed. A value of 0x1000 (4096) for FLAGS shows that the front end is active.

For each connection, the report also gives a snapshot of the connection status. When idle, the state values should each be 0 (zero).

Table 8-4 describes the front-end state values:

Table 8-4 Front-End State Values

Value	Description
0	Waiting to receive an operation from the CM.
1	Receiving from the CM.
2	Sent an operation to be processed, waiting for back end.
3	The operation is done.
4	Sending a response to the CM.

The front-end flags are the same as the back-end flags, except that the front ends clear the IO output value when they send a reply back to the CM. The information in the connection report is a snapshot of the connection status.

Increasing the Level of Reporting for a DM

By default, DMs report errors and warnings. You can have a DM report debugging messages as well.

You can specify which debugging messages you want written to the log. The following debug settings control which debugging information is logged:

• DM_DEBUG controls which opcode-processing debug messages are logged.

• DM_DEBUG2 controls which data dictionary processing debug messages are logged.

• DM_DEBUG3 controls which SQL statement debug messages are logged based on which parts of the DM produced the SQL statements.

• DM_DEBUG5 controls which queuing debug messages are logged for the Oracle DM (dm_oracle).

The BRM_home/include/dm_debug.h file contains definitions of the flags you can enable for each debug setting. You enable multiple flags for one debug setting by summing the values of the flags and including the sum in the debug setting's corresponding DM configuration (pin.conf) file entry (see "Editing the Configuration File to Set Debug Logging Options") or environment variable (see "Using Environment Variables to Set Debug Logging Options").

For example, to log information about transaction tracing, set DM_DEBUG to 0x70, which is the sum of the following flags:

DM_DEBUG_TRANS_IN_PR     0x10
DM_DEBUG_TRANS_OUT_PR    0x20
DM_DEBUG_TRANS_TRACE     0x40

Depending on what information you want to log, you can include values for any combination of the debug settings (DM_DEBUG, DM_DEBUG2, DM_DEBUG3, and DM_DEBUG5).

The way you increase the level of reporting depends on the DM:

• For all DMs other than dm_oracle and dm_tax, you can include debug settings in the DM's configuration (pin.conf) file. Specify each debug setting separately as in the following example:

  - dm dm_debug    0xFFF003FF
  - dm dm_debug2   0x10
  - dm dm_debug3   0x10

  See "Editing the Configuration File to Set Debug Logging Options".

• For dm_oracle and dm_tax, you must specify the debug settings as environment variables. Set a separate environment variable for each debug setting.

  See "Using Environment Variables to Set Debug Logging Options".

Editing the Configuration File to Set Debug Logging Options

To set debug logging options for all DMs except dm_oracle and dm_tax:

1. Stop the DM. See "Starting and Stopping the BRM System".

2. Open the configuration file (pin.conf) for this DM.

3. Edit the debug setting entries to set the level of debugging reporting.

4. Save and close the file.

5. Start the DM. See "Starting and Stopping the BRM System".

6. Run the DM operations for which you want debugging information.

7. Stop the DM.

8. Open the log file for the DM (for example, dm_fusa.pinlog) and review the messages.

9. Return DM logging to its normal level by commenting out the debug setting entries in the configuration file. Otherwise, subsequent DM activity will generate large log files.

Using Environment Variables to Set Debug Logging Options

To set debug logging options for dm_oracle and dm_tax:

1. Stop the DM. See "Starting and Stopping the BRM System".

2. In the environment from which the DM starts, set an environment variable for each debug setting. For example:

   C shell:

   setenv DM_DEBUG3 0xFFFF003F

   Korn shell:

   DM_DEBUG3=0xFFFF003F
   export DM_DEBUG3

3. Start the DM. See "Starting and Stopping the BRM System".

4. Run the DM operations for which you want to display debugging information.

5. Stop the DM.

6. Open the log file for the DM (for example, dm_oracle.pinlog) and review the messages.

7. Return DM logging to its normal level. Otherwise, subsequent DM activity will generate large log files.

Logging the DM Process Time Information for Performance Diagnostics

To diagnose performance problems with the DM process, you can configure the DM to log the time it takes to process each opcode. You can use this information to determine the time the DM spends on its internal operations and the time it spends on the database operations.

Before the DM starts processing an opcode, it logs the current time. Then for each SQL statement that the DM sends to the database for the opcode, it logs the following information:

• Session ID

• Statement ID

• Time taken by the database to process the SQL statement

To log the timing information for the SQL statement, set the DM_DEBUG3 flag to 0x00010000, which corresponds to the DM_DEBUG3_TIME_INFO variable defined in the BRM_home/include/dm_debug.h file.

Replacing Failed DM Child Processes

All DMs, such as Paymentech DM and Email DM are set to automatically replace child processes that have stopped. This feature prevents the system from losing DM processes as a result of transient failures over time. For initial testing, or if you have recurring errors that would cause a “fork and die" endless loop (in an Oracle database, for example), you can tell the DM to not replace failed child processes:

1. Open the configuration file (pin.conf) for this DM.

2. Change the value of the dm_restart_children entry to 0.

3. Save and close the file.

4. Stop and restart the DM.

When a child process stops and is replaced, BRM notes the event in the error log file for the DM.

Note:

BRM does not automatically replace child processes that are hung. See "Problem: Hung and or Looping Processes".