Installing and Configuring the Sun MTP HA Data Service

This document provides instructions for setting up and administering the Sun MTP high availability (HA) data service on your Sun Cluster nodes. It contains the following topics:


Sun MTP HA Data Service Overview

The Sun MTP HA data service works with the Sun Cluster hardware and software to make one or more Sun MTP regions highly available.

A Sun Cluster system can host multiple Sun MTP HA data service instances distributed across the entire cluster. Even when the same cluster node hosts several instances, each instance uses only the resources available on that node.

The Sun MTP data service starts, stops, restarts, and fails over the Sun MTP application among the nodes of the cluster in response to cluster events, such as administrative actions, application failures, or cluster node failure.

The Sun MTP application restart is managed by the Sun Cluster resource group manager (RGM).

If the Sun MTP application deaths exceed the failure count within the failure time window, the resource group containing the Sun MTP application is automatically failed over to another cluster node.

The fault monitor is provided using the probe method.

The Sun MTP data service provides:

For general information about data services, resource groups, resources, and other related topics, refer to the Sun Cluster 3.1 Concepts document.


Task Map: Installing and Configuring the Sun MTP HA Data Service

TABLE 1 lists the tasks for installing and configuring Sun MTP HA data service. Perform these tasks in the order that they are listed.

TABLE 1 Task Map: Installing and Configuring Sun MTP HA Data Service

Task

For Instructions, Go To

Plan the installation

Chapter 1 of Sun Cluster 3.1 Data Service Planning and Administration Guide

 

Planning the Sun MTP HA Data Service Installation and Configuration

Prepare to install the Sun MTP HA data service

Preparing to Install the Sun MTP HA Data Service

 

Install Sun MTP

Installing Sun MTP

 

Install Sun MTP HA data service package

Installing the Sun MTP HA Data Service

 

Register and configure Sun MTP HA data service as a failover data service

Registering and Configuring the Sun MTP HA Data Service

 

Verify Sun MTP HA data service Installation and configuration

Verifying the Installation and Configuration

 

Understand Sun MTP HA data service fault monitor

Understanding the Sun MTP HA Data Service Fault Monitor


To upgrade a previous version of the Sun MTP HA data service, see Upgrading the Sun MTP HA Data Service.


Planning the Sun MTP HA Data Service Installation and Configuration

This section contains the information you need to plan your Sun MTP HA data service installation and configuration.



Note - Check the SunSolveSM web site at http://sunsolve.sun.com on a regular basis for any patches that are available, and apply the recommended patches. Refer to the patch README files for additional information about latest releases and any software interdependencies.



Configuration Requirements

To use this version of the Sun MTP HA data service, your environment must have the following:

If you are also using Sun MBM and the Sun MBM HA data service, you must use Sun MBM 10.0.0, minimum patch level of 7, and the Sun MBM HA data service, release 1.0, minimum patch level of 1. Make sure to configure the Sun MBM HA data service as described in the Sun Mainframe Batch Manager Software High Availability Data Service for Sun Cluster document (part number 817-3505-10).

Configuration Considerations

This section describes the configuration components that have been qualified with this release. It also lists components and packages that will be qualified in a future release, and components or functionality that are not supported.

Currently Qualified

The Sun MTP HA data service has been qualified with the following components and functionality:

Not Yet Qualified

The following components and functionality have not yet been qualified:

The following Sun mainframe rehosting software components will be qualified when there are highly available versions:

Not Supported

The following functionality is not supported:

Usage Considerations

Your data service configuration might not be supported if you do not observe these guidelines.



Note - For restrictions that apply to all data services, refer to the Sun Cluster Release Notes.




Preparing to Install the Sun MTP HA Data Service

This section describes the tasks you must perform before you install and configure the Sun MTP HA data service.

Setting Up the Sun MTP Administrator

Contact your system administrator to create the Sun MTP administrator (using the useradd command), if one does not already exist. This user is specified in the EMP_ADMIN property when the data service is registered. See Registering and Configuring the Sun MTP HA Data Service.



Note - The Sun MTP administrator's login shell must be Korn shell (/bin/ksh).



Updating NIS/NIS+ Configuration File

Because the Sun MTP HA data service uses the su EMP_ADMIN command when starting and stopping the Sun MTP system, the su(1M) command must not refer to the network information name service (NIS/NIS+). This is because NIS/NIS+ might not be available because of a failure of the public network on the cluster node, and it is, therefore, a potential single point of failure for a Sun Cluster system.

The following entries in the /etc/nsswitch.conf configuration file removes the dependency for contacting an NIS/NIS+ master server over public network adapters. However, the functionality these fields provide will not be available from the NIS server. Adding the entries ensures that the su command does not refer to the NIS/NIS+ name services if the network information name service is unavailable.

passwd: files [NOTFOUND=return] nis
group: files [NOTFOUND=return] nis
publickey: files [NOTFOUND=return] nis
project: files [NOTFOUND=return] nis

Updating the Sun MTP Setup File

This section describes how to update the region setup file to include the appropriate startup options. It also provides information about where to maintain the region setup file in the Sun Cluster environment.

Setting the Startup Options

To ensure that the Sun MTP region is restarted with the appropriate unikixmain (kixstart) options, add the HA_MTP_START_OPTS environment variable to your region's setup file. This setup file is specified in the EMP_SETUP property when the data service is registered. See Registering and Configuring the Sun MTP HA Data Service.

The last line in the following example setup file shows the HA_MTP_START_OPTS environment variable. The value of this variable is a list of unikixmain options you normally use to start the region. The value must be enclosed in double quotes.

UNIKIX=/pkgs/mtp/MTP8.0.0;export UNIKIX
COBDIR=/opt/cobol/2.10;export COBDIR
COBCPY=$COBDIR/cpylib:$UNIKIX/copy;export COBCPY
LD_LIBRARY_PATH=$COBDIR/lib:$UNIKIX/lib;export LD_LIBRARY_PATH
PATH=$UNIKIX/local/bin:$UNIKIX/bin:$COBDIR/bin:$PATH;export PATH
KIXSYS=/mtp_data/appl2/sys;export KIXSYS
KIXPROGS=/mtp_data/appl2/cobol/online;export KIXPROGS
KIXMAPS=/mtp_data/appl2/maps;export KIXMAPS
KIXDATA=/mtp_data/appl2/data/vsam;export KIXDATA
KIXLICDIR=/licenses/8.0;export KIXLICDIR
HA_MTP_START_OPTS="-S 4m -j 300 -N 20 -p 8001";export HA_MTP_START_OPTS



caution icon

Caution - Make sure to include $PATH in the PATH statement so that the inherited Sun Cluster environment is not overridden.



Selecting a Location for the Setup File

The setup file is dynamically loaded at execution time by the Sun MTP probe process. Changes to the setup file while the Sun MTP resource is enabled can result in unpredictable behavior. Therefore, if the setup file is located on the HAStoragePlus file service, you must disable the Sun MTP resource before changing any information in the setup file.

1. Log in to the system as superuser.

2. Disable the resource by typing the following command:

# scswitch -n -j MTP-resource-name

3. Edit the setup file and save it.

4. Enable the resource by typing the following command:

# scswitch -e -j MTP-resource-name



Note - If you maintain the region setup file on the HAStoragePlus file service, it is a good idea to locate it in the $KIXSYS directory.



A less preferable alternative is to have a copy of the setup file on each local node. This alternative requires the contents of the setup files to be consistent. The advantage of maintaining separate setup files is that you can modify the setup file on the secondary node without disabling the resource on the primary node. If you want to keep separate setup files, use the following procedure to maintain them:

1. On the secondary node, modify the setup file.

2. As superuser, move the Sun MTP resource to the secondary node.

# scswitch -z -g resource-group -h cluster-node-host-name

3. Check that Sun MTP is up and running with the new setup file.

4. Move the Sun MTP resource back to the primary node.


Installing Sun MTP

The installation procedure for Sun MTP in a Sun Cluster environment is the same as the normal installation procedure. Refer to the Sun Mainframe Transaction Processing Software Installation Guide for detailed information.

Also, refer to Usage Considerations, which contains information about where to locate the Sun MTP binaries ($UNIKIX), configuration directory ($KIXSYS), and the region setup file. The $UNIKIX directory is specified in the EMP_BASEDIR property when the data service is registered. See Registering and Configuring the Sun MTP HA Data Service.

Make sure to verify your installation on each node as described in the Sun Mainframe Transaction Processing Software Installation Guide.

Upgrading Sun MTP

When upgrading Sun MTP in the Sun Cluster environment, you must disable the Sun MTP resource before performing the normal upgrade process. Refer to the Sun Mainframe Transaction Processing Software Installation Guide for information about upgrading the software.



Note - Make sure to upgrade Sun MTP on each node of the cluster.




Installing the Sun MTP HA Data Service

You must be superuser to perform this procedure.


procedure icon  To Install the Sun MTP HA Data Service

1. Load the distribution CD into the CD-ROM, or download the SUNWscmtp package from SunSolve.

2. Run the pkgadd command on each cluster node on which the data service will be configured to run.

Refer to the pkgadd(1M) man page for information on how to install a package using this administrative command.

The installation creates the /opt/SUNWscmtp directory, which contains the following:


Registering and Configuring the Sun MTP HA Data Service

The Sun MTP HA data service must be registered and configured as a failover data service. To ensure optimal performance and data integrity, you must associate the data service with the HAStoragePlus file service (SUNW.HAStoragePlus).

You must be superuser to perform the procedures in this section.


procedure icon  To Register and Configure the Sun MTP HA Data Service

1. Register the SUNW.scmtp resource type:

# scrgadm -a -t SUNW.scmtp

where:

-a

Adds the data service resource type.

-t SUNW.scmtp

Specifies the predefined resource type name for your data service.


2. Create the SUNW.HAStoragePlus resource type:

# scrgadm -a -t SUNW.HAStoragePlus

3. Create a resource group:

# scrgadm -a -g resource-group-name [-h nodelist]

where:

-g resource-group-name

Specifies the name of the resource group. This name must be unique for resource groups within the cluster.

-h nodelist

Specifies an optional comma-separated list of physical node names. The order of the name determines the order in which nodes are considered primary during failover.


4. Define the logical host name.

a. Make sure that the logical host name is defined in the cluster's /etc/hosts file.

b. Set up the logical host name on the client system.

5. Add the logical host name to the failover resource group:

# scrgadm -a -L -j logical-host-resource-name -g resource-group-name -l logical-hostname

where:

-L

Specifies logical host name resources.

-j logical-host-resource-name

An optional name for the logical host name resource.

-l logical-hostname

Specifies a host name that can apply to any node in the cluster, and which is assigned dynamically.


6. Create a disk device group resource:

# scrgadm -a -j disk-device-group-name -g resource-group-name \
-t SUNW.HAStoragePlus -x FileSystemMountPoints=ha-filesystem \
-x AffinityOn=True

The -x options used in this command are extension properties for SUNW.HAStoragePlus.



Note - The -x FileSystemMountPoints option identifies the cluster failover file system.



7. Turn the resource group online:

# scswitch -Z -g resource-group-name

8. Create the Sun MTP application resource in the failover resource group:

# scrgadm -a -j MTP-resource-name -g resource-group-name \
-t SUNW.scmtp \
-y Resource_Dependencies=disk-device-group-name \
-x EMP_BASEDIR=MTP-directory \
-x EMP_ADMIN=MTP-administrator-name \
-x EMP_SETUP=fully-qualified-setup-pathname \
-x Monitor=monitor-level

where the -x values are the following extension properties:

MTP-directory

Identifies the fully-qualified directory path name where the Sun MTP product is installed (usually referred as $UNIKIX).

MTP-administrator-name

Identifies the user ID that will be used to start and stop the Sun MTP system.

fully-qualified-setup-pathname

Points to a setup file in Korn shell format, located on the cluster failover file system, that contains the environment variables of the Sun MTP region. Examples of variables contained in this file are: KIXSYS, KIXPROGS, KIXLICDIR, and so on.

monitor-level

Specifies the monitor level to be performed. Options are:

APPL_DEFS: Monitors for the availability of the application definitions, which is the information related to the Sun MTP resources that are contained in the Sun MTP shared memory.

SYS_PROCS: Monitors for the existence of the Sun MTP main process (unikixmain).

NONE: Does not perform any monitoring on the Sun MTP data resource.

You must use the colon-separated combination of APPL_DEFS and SYS_PROCS to have full monitoring capability.


9. Turn the Sun MTP resource online:

# scswitch -e -j MTP-resource-name

See Configuring the Sun MTP HA Data Service Properties for a complete list of the extension and standard properties and their valid values.


Configuration Example

The following example shows the commands needed to register and configure the mtp-rg-1 resource group and associate it with a SUNW.HAStoragePlus resource type.

# scrgadm -a -t SUNW.scmtp
# scrgadm -a -t SUNW.HAStoragePlus
# scrgadm -a -g mtp-rg-1
# scrgadm -a -L -j mtp-lh-1 -g mtp-rg-1 -l mtp-logical
# scrgadm -a -j mtp-dg-1 -g mtp-rg-1                     \
-t SUNW.HAStoragePlus -x FileSystemMountPoints=/mtp_data \
-x AffinityOn=True
# scswitch -Z -g mtp-rg-1
# scrgadm -a -j mtp-rs-1 -g mtp-rg-1 -t SUNW.scmtp       \
-y ResourceDependencies=mtp-dg-1                         \
-x EMP_BASEDIR=/pkgs/mtp/MTP8.0.0                        \
-x EMP_ADMIN=mtpadmin                                    \
-x EMP_SETUP=/mtp_data/appl2/sys/Setup.ksh               \
-x Monitor=SYS_PROCS:APPL_DEFS
# scswitch -e -j mtp-rs-1


Verifying the Installation and Configuration


procedure icon  To Verify the Installation and Configuration

1. Log in as the Sun MTP administrator.

2. Type the following command to bring online the resource group where the Sun MTP application resource is defined:

# scswitch -Z -g resource-group

where the -Z option enables all resources of the resource group and their monitors, and moves the resource group into the managed state.

3. Shut down the Sun MTP application by killing one of the Sun MTP system processes, such as the unikixstrt process.

4. Verify that Sun Cluster restarts the region.

See To Verify the Failover.


procedure icon  To Switch to Another Node

single-step bulletExecute the following command to move the Sun MTP application to another node:

# scswitch -z -g resource-group -h cluster-node-host-name

where:

-z

Specifies a change in master of the resource group.

-h cluster-node-host-name

Identifies the Sun Cluster node you are switching to.



procedure icon  To Verify the Failover

1. Execute the following command to determine if the Sun MTP application is online on the selected cluster node:

# scstat -g

2. Confirm that the resource group is now online on the new primary node.


Understanding the Sun MTP HA Data Service Fault Monitor

The Sun MTP HA data service contains a built-in fault monitor (or fault probe). The fault monitor is a process that probes the health of the Sun MTP HA data service.

Monitor Start Method

The fault monitor is invoked by the RGM when you bring the resource group and its associated resources online. This invocation causes the RGM to internally call the Monitor_Start method for the Sun MTP HA data service.

Monitor Stop Method

The fault monitor is stopped by RGM when you bring the resource group and its associated resources offline. This invocation causes the RGM to internally call the Monitor_Stop method for the Sun MTP HA data service.

Standard Properties

The fault monitor uses the following standard properties:

You might be able to tune these properties. Refer to the Sun Cluster documentation to review or set resource properties. Also see Configuring the Sun MTP HA Data Service Properties to review or set resource properties.

Extension Properties

The fault monitor uses the following Sun MTP extension properties:

You might be able to tune these properties. Refer to Configuring the Sun MTP HA Data Service Properties to review or set extension properties.

Probing Algorithm and Functionality

The monitor functionality of the Sun MTP HA data service consists of two components: the Process Monitor Facility (PMF) and the fault monitor.



Note - These components are not activated if the Monitor extended property is set to None.



PMF

PMF monitors the abnormal exit of all Sun MTP system processes. On any abnormal exit, the PMF tries to restart the data service up to PMF_Retry_Count times in a period of Retry_interval. If Sun MTP crashes more often than the retry interval period, the PMF stops trying to restart it.

This feature relies on the Sun MTP main server process (unikixmain) monitoring capabilities. If any Sun MTP system process dies, unikixmain evaluates whether the failure is critical, and decides whether to restart the dead process or shut down Sun MTP.

Fault Monitor

The fault monitor performs health checks on the data service. The logic followed by the fault monitor is as follows:

1. Sleep Probe_start_interval number of seconds before going into an infinite loop.

2. Sleep Thorough_probe_interval number of seconds.

3. Perform health check under the time out property Probe_timeout using the Sun MTP entry point mtp_probe.

The mtp_probe command checks the health of the Sun MTP system by performing the following checks according to the options set in the Monitor extended property:


Configuring the Sun MTP HA Data Service Properties

Typically, you configure properties using the options on the scrgadm command at the time you create the Sun MTP resource.

The following table describes the standard tunable properties that you can configure for the Sun MTP HA data service. Standard properties are default resource properties. To set or modify this property, use the -y parameter=value option.

TABLE 2 Tunable Standard Properties for Sun MTP

Property Name

Description

Retry_interval

The number of seconds over which to count attempts to restart a failed Sun MTP resource. The monitor uses this properties in conjunction with Retry_count.

Default: 300

Tunable: Any time

Category: Optional

Retry_count

The number of times the monitor attempts to restart the Sun MTP resource if it fails on the primary node.

Default: 2

Tunable: Any time

Category: Optional

Thorough_probe_interval

The number of seconds between invocations of an high-overhead fault probe of the Sun MTP resource.

Default: 60

Tunable: Any time

Category: Optional

Start_timeout

Timeout value for the start method in seconds.

Default: 300

Tunable: Any time

Category: Optional

Stop_timeout

Timeout value for the stop method in seconds.

Default: 300

Tunable: Any time

Category: Optional

Failover_mode

Controls whether the RGM relocates a resource group or aborts a node in response to a failure of the start or stop method. None indicates that the RGM should just set the resource state on method failure and wait for operator intervention. Soft indicates failure of the start method should cause the RGM to relocate the resource's group to a different node, while failure of the stop method should cause the RGM to set the resource state and wait for operator intervention. Hard indicates that failure of the start method should cause the relocation of the group and failure of the stop method should cause the forcible stop of the resource by aborting the cluster node.

Default: Soft

Tunable: Any time

Category: Optional


The following table describes the tunable extension properties that you can configure for the Sun MTP HA data service. Extension properties are additional properties specific to the Sun MTP resource. To set or modify this property, use
the -x parameter=value option.

TABLE 3 Tunable Extension Properties for Sun MTP

Property Name

Description

Probe_timeout

Timeout value for the probe in seconds.

Default: 30

Tunable: Any time

Category: Optional

Child_mon_level

Child process monitoring level for the Process Management Facility (the -C option of pmfadm). Default=-1 means monitor all Sun MTP processes, a value >0 indicates the level of child process monitoring that is desired.

Do not change this parameter unless you are directed to do so by your authorized service provider.

Default: -1

Tunable: When disabled

Category: Optional

Probe_start_interval

Probe startup waiting time before starting monitoring Sun MTP in seconds.

Default: 60

Tunable: When disabled

Category: Optional

Debug

Debug option to enable/disable the internal debugging messages. Valid values are ON and OFF.

Default: OFF

Tunable: Any time

Category: Optional

PMF_Retry_Count

Number of retries used by PMF to restart Sun MTP if it fails.

Default: 2

Tunable: When disabled

Category: Optional

EMP_BASEDIR

The directory where the Sun MTP product resides. Usually $UNIKIX.

Default: None

Tunable: When disabled

Category: Required

EMP_ADMIN

The Sun MTP administrator ID.

Default: None

Tunable: When disabled

Category: Required

EMP_SETUP

The Sun MTP setup file.

Default: None

Tunable: When disabled

Category: Required

Monitor

Monitor level; can be NONE or a colon-separated combination of SYS_PROCS and APPL_DEFS, which is the recommended setting.

Default: None

Tunable: When disabled

Category: Required


The terms used in the tables are defined as follows:

Property: Name of the resource property.

Description: A brief description of the property.

Default: The default value for the property.

Tunable: Indicates when the cluster administrator can set or modify the value of this property for the Sun MTP resource. Values that allow administrator tuning are: Anytime (at any time), At creation (only when the resource is created), or When disabled (when the resource is offline).

Optional: These properties are set to the default values at creation time. They can be changed after the resource is created.

Required: The property must be specified when creating the resource.


Using Sun MTP as a Sun Cluster Resource

When managing Sun MTP as a Sun Cluster resource, Sun Cluster management facilities must be used to start and stop the region. Do not use the standard Sun MTP commands, such as kixstart, unikixmain, kixclean, and kixstop. Using these commands will conflict with the Sun Cluster management facilities, and might fail to provide a highly available operational environment.


Upgrading the Sun MTP HA Data Service


procedure icon  To Install an Update of the Sun MTP Data Service

1. Log in to the system as the superuser.

2. Disable and remove all Sun MTP resources on the entire cluster.

Type the following commands for each resource, using the names you gave to your resources when you originally configured them:

# scswitch -n -j MTP-resource-name
# scrgadm -r -j MTP-resource-name

3. Remove the Sun MTP HA data service.

# scrgadm -r -t SUNW.scmtp

4. Remove the SUNWscmtp package from each node of the cluster.

# pkgrm SUNWscmtp

5. Verify that the package removal was successful for each node of the cluster by executing the following command:

$ cd /opt/SUNWscmtp

An error message indicating that the directory does not exist should be displayed.

6. Install the new SUNWscmtp package.

See Installing the Sun MTP HA Data Service.


Configuring the COBOL Animator

This procedure enables you to run the Micro Focus COBOL Animator on a node in a Sun Cluster environment. However, due to the Sun Cluster restriction on animating processes while they are being monitored by PMF, using Animator to debug COBOL programs on a production Sun Cluster region is strongly discouraged.



caution icon

Caution - Failure to observe this guideline could result in data corruption.




procedure icon  To Set Up the Data Service to Allow Animation

1. Log in to the system as superuser.

2. Change directory to $UNIKIX/scbin.

3. Change the owner of the kxchg_dbg module to root:

# chown root kxchg_dbg

4. Change permissions on the kxchg_dbg module:

# chmod 4755 kxchg_dbg

5. Log out as superuser.



Note - This procedure must be performed on each node where you are using Sun MTP and Animator.




Troubleshooting

If you encounter problems that cannot be resolved by means of the Sun Cluster or Sun MTP troubleshooting facilities, run the Sun Explorer facility (SUNWexplo). You can download the SUNWexplo package, including information on installing and using it, from the SunSolve web site:

http://sunsolve.sun.com

Also run the kixsnap utility on the node where Sun MTP is running.

Both utilities generate zip files.

Collect all the output and send the files, along with the information noted in Help, to your authorized service provider.