Sun Cluster Data Services Developer's Guide for Solaris OS

Chapter 1 Overview of Resource Management

This book provides guidelines for creating a resource type for a software application, such as Oracle®, Sun Java^TM System Web Server (formerly Sun ONE Web Server), or DNS. As such, this book is intended for developers of resource types.

To understand the contents of this book, you must be thoroughly familiar with the concepts that are presented in the Sun Cluster Concepts Guide for Solaris OS.

This chapter provides an overview of the concepts that you need to understand to develop a data service. This chapter covers the following topics:

Note –

This book uses the terms resource type and data service interchangeably. The term agent, though rarely used in this book, is equivalent to resource type and data service.

Sun Cluster Application Environment

The Sun Cluster system enables applications to be run and administered as highly available and scalable resources. The Resource Group Manager (RGM) provides the mechanism for high availability and scalability.

The following elements form the programming interface to this facility:

A set of callback methods that you write that enable the RGM to control an application in the cluster.
The Resource Management API (RMAPI), a set of low-level API commands and functions that you can use to write the callback methods. These APIs are implemented in the libscha.so library.
Process Monitor Facility (PMF) for monitoring and restarting processes in the cluster.
The Data Service Development Library (DSDL), a set of library functions that encapsulates the low-level API and process-management functionality at a higher level. The DSDL adds some additional functionality to ease the writing of callback methods. These functions are implemented in the libdsdev.so library.

The following figure shows the interrelationship of these elements.

Figure 1–1 Programming Architecture of the Sun Cluster Application Environment

Diagram showing interrelationship between callback methods,
RMAPI, Process Monitor Facility (PMF), and DSDL

Sun Cluster Agent Builder, which is described in Chapter 9, Sun Cluster Agent Builder, is a tool in the Sun Cluster package that automates the process of creating a data service. Agent Builder generates data service code in either C (by using DSDL functions to write the callback methods) or in the Korn (ksh) shell command language (by using low-level API commands to write the callback methods).

The RGM runs as a daemon on each cluster node and automatically starts and stops resources on selected nodes or zones according to preconfigured policies. The RGM makes a resource highly available in the event of a node or zone failure or reboot. The RGM does so by stopping the resource on the affected node or zone and starting it on another node or zone. The RGM also automatically starts and stops resource-specific monitors. These monitors detect resource failures and relocate failing resources onto other nodes or zones or monitor other aspects of resource performance.

The RGM supports both failover resources and scalable resources. A failover resource can be online on only one node or zone at a time. A scalable resource can be online on multiple nodes or zones simultaneously. However, a scalable resource that uses a shared address to balance the service load between nodes can be online in only one zone per physical node.

Resource Group Manager Model

This section introduces some fundamental terminology and explains in more detail the RGM and its associated interfaces.

The RGM handles three major kinds of interrelated objects: resource types, resources, and resource groups. One way to introduce these objects is by means of an example, as follows.

You implement a resource type, ha-oracle, that makes an existing Oracle DBMS application highly available. An end user defines separate databases for marketing, engineering, and finance, each of which is a resource of type ha-oracle. The cluster administrator places these resources in separate resource groups so that they can run on different nodes or zones and fail over independently. You create a second resource type, ha-calendar, to implement a highly available calendar server that requires an Oracle database. The cluster administrator places the resource for the finance calendar into the same resource group as the finance database resource. The cluster administrator does so to ensure that both resources run on the same node or in the same zone and fail over together.

Resource Types

A resource type consists of the following elements:

A software application to be run in the cluster
Control programs that are used as callback methods by the RGM to manage the application as a cluster resource
A set of properties that form part of the static configuration of a cluster

The RGM uses resource type properties to manage resources of a particular type.

Note –

In addition to a software application, a resource type can represent other system resources, such as network addresses.

You specify the properties for the resource type and set property values in a resource type registration (RTR) file. The RTR file follows the format that is described in Setting Resource and Resource Type Properties and in the rt_reg(4) man page. See also Defining the Resource Type Registration File for a description of a sample RTR file.

Resource Type Properties provides a list of the resource type properties.

The cluster administrator installs and registers the resource type implementation and underlying application on a cluster. The registration procedure enters the information from the RTR file into the cluster configuration. The Sun Cluster Data Services Planning and Administration Guide for Solaris OS describes the procedure for registering a data service.

Resources

A resource inherits the properties and values of its resource type. In addition, you can declare resource properties in the RTR file. Resource Properties contains a list of resource properties.

The cluster administrator can change the values of particular properties depending on how the properties are specified in the RTR file. For example, property definitions can specify a range of allowable values. Property definitions can also specify when the property is tunable: never, any time, at creation (when the resource is added to the cluster), or when the resource is disabled. Within these specifications, the cluster administrator can make changes to properties by using administration commands.

The cluster administrator can create many resources of the same type, with each resource having its own name and set of property values, so that more than one instance of the underlying application can run in the cluster. Each instantiation requires a unique name within the cluster.

Resource Groups

Each resource must be configured in a resource group. The RGM brings all resources in a group online and offline together on the same node or in the same zone. When the RGM brings a resource group online or offline, it runs callback methods on the individual resources in the group.

The nodes or zones where a resource group is currently online are called its primaries or primary nodes or primary zones. A resource group is mastered by each of its primaries. Each resource group has an associated Nodelist property that identifies all potential primaries or masters of the resource group. The cluster administrator sets the Nodelist property.

A resource group also has a set of properties. These properties include configuration properties that can be set by the cluster administrator and dynamic properties, set by the RGM, that reflect the active state of the resource group.

The RGM defines two types of resource groups: failover and scalable. A failover resource group can be online on only one node or zone at any time. A scalable resource group can be online on multiple nodes or zones simultaneously. The RGM provides a set of properties to support the creation of each type of resource group. See Transferring a Data Service to a Cluster and Implementing Callback Methods for details about these properties.

Resource Group Properties contains a list of resource group properties.

Resource Group Manager

The Resource Group Manager (RGM) is implemented as a daemon, rgmd, that runs in the global zone on each member node of a cluster. All of the rgmd processes communicate with each other and act as a single cluster-wide facility.

The RGM supports the following functions:

Whenever a node or zone boots or crashes, the RGM attempts to maintain the availability of all managed resource groups. The RGM does so by automatically bringing them online on correct masters.
If a particular resource fails, its monitor program can request that the resource group be restarted on the same master or switched to a new master.
The cluster administrator can issue an administrative command to request one of the following actions:
- Change mastery of a resource group.
- Enable or disable a particular resource within a resource group.
- Create, delete, or modify a resource type, a resource, or a resource group.

Whenever the RGM activates configuration changes, it coordinates its actions across all member nodes of the cluster. This kind of activity is known as a reconfiguration. To effect a state change on an individual resource, the RGM runs a resource type-specific callback method on that resource.

Callback Methods

The Sun Cluster framework uses a callback mechanism to provide communication between a data service and the RGM. The framework defines a set of callback methods, including their arguments and return values, and the circumstances under which the RGM calls each method.

You create a data service by coding a set of individual callback methods and implementing each method as a control program that the RGM can call. That is, the data service does not consist of a single executable, but a number of executable scripts (ksh) or binaries (C), each of which the RGM can call directly.

Callback methods are registered with the RGM through the RTR file. In the RTR file you identify the program for each method that you have implemented for the data service. When a cluster administrator registers the data service on a cluster, the RGM reads the RTR file, which provides the identity of the callback programs and other information.

The only required callback methods for a resource type are a start method (Start or Prenet_start) and a stop method (Stop or Postnet_stop).

The callback methods can be grouped into the following categories:

Control and initialization methods
- The Start and Stop methods start and stop resources in a group that is being brought online or offline.
- The Init, Fini, and Boot methods execute initialization and termination code on resources.
Administrative support methods
- The Validate method verifies properties that are set by administrative action.
- The Update method updates the property settings of an online resource.
Net-relative methods

Prenet_start and Postnet_stop perform special startup or shutdown operations before network addresses in the same resource group are configured to go up or after they are configured to go down.
Monitor control methods
- Monitor_start and Monitor_stop start or stop the monitor for a resource.
- Monitor_check assesses the reliability of a node or zone before a resource group is moved to the node or zone.

See Chapter 3, Resource Management API Reference and the rt_callbacks(1HA) man page for more information about the callback methods. Also see Chapter 5, Sample Data Service and Chapter 8, Sample DSDL Resource Type Implementation for callback methods in sample data services.

Programming Interfaces

For writing data service code, the resource management architecture provides a low-level or base API, a higher-level library that is built on top of the base API, and Sun Cluster Agent Builder, a tool that automatically generates a data service from basic input that you provide.

Resource Management API

The Resource Management API (RMAPI) provides a set of low-level functions that enable a data service to access information about the resource types, resources, and resource groups in the system, to request a local restart or failover, and to set the resource status. You access these functions through the libscha.so library. The RMAPI provides these callback methods both in the form of shell commands and in the form of C functions. See the scha_calls(3HA) man page and Chapter 3, Resource Management API Reference for more information about the RMAPI functions. Also see Chapter 5, Sample Data Service for examples of how to use these functions in sample data service callback methods.

Data Service Development Library

Built on top of the RMAPI is the Data Service Development Library (DSDL), which provides a higher-level integrated framework while retaining the underlying method-callback model of the RGM. The libdsdev.so library contains the DSDL functions.

The DSDL brings together various facilities for data service development, including the following:

libscha.so. The low-level resource management APIs.
PMF. The Process Monitor Facility (PMF), which provides a means of monitoring processes and their descendants, and restarting them if they die. See the pmfadm(1M) and rpc.pmfd(1M) man pages.
hatimerun. A facility for running programs under a timeout. See the hatimerun(1M) man page.

For the majority of applications, the DSDL provides most or all of the functionality you need to build a data service. Note, however, that the DSDL does not replace the low-level API but encapsulates and extends it. In fact, many DSDL functions call the libscha.so functions. Likewise, you can directly call libscha.so functions while using the DSDL to code the bulk of your data service.

See Chapter 6, Data Service Development Library and the scha_calls(3HA) man page for more information about the DSDL.

Sun Cluster Agent Builder

Agent Builder is a tool that automates the creation of a data service. You input basic information about the target application and the data service to be created. Agent Builder generates a data service, which includes source and executable code (C or Korn shell), a customized RTR file, and a Solaris package.

For most applications, you can use Agent Builder to generate a complete data service with only minor manual changes on your part. Applications with more sophisticated requirements, such as adding validation checks for additional properties, might require work that Agent Builder cannot do. However, even in these cases, you might be able to use Agent Builder to generate the bulk of the code and manually code the rest. At a minimum, you can use Agent Builder to generate the Solaris package for you.

Resource Group Manager Administrative Interface

Sun Cluster provides both a graphical user interface (GUI) and a set of commands for administering a cluster.

Sun Cluster Manager

Sun Cluster Manager, formerly SunPlex^TM Manager, is a web-based tool that enables you to perform the following tasks:

Install a cluster.
Administer a cluster.
Create and configure resources and resource groups.
Configure data services with the Sun Cluster software.

See the Sun Cluster Software Installation Guide for Solaris OS for instructions on how to install Sun Cluster Manager and how to use Sun Cluster Manager to install cluster software. Sun Cluster Manager provides online help for most unique administrative tasks.

`clsetup` Utility

You can perform most Sun Cluster administration tasks interactively with the clsetup(1CL) utility.

You can administer the following Sun Cluster elements with the clsetup utility:

Quorum
Resource groups
Data services
Cluster interconnect
Device groups and volumes
Private host names
New nodes
Other cluster tasks

You can also perform the following operations with the clsetup utility:

Create a resource group
Add a network resource to a resource group
Add a data service resource to a resource group
Register a resource type
Bring a resource group online or offline
Switchover a resource group
Suspend or resume the automatic recovery actions of a resource group
Enable or disable a resource
Change resource group properties
Change resource properties
Remove a resource from a resource group
Remove a resource group
Clear the Stop_failed error flag from a resource

Administrative Commands

The Sun Cluster commands for administering RGM objects are clresourcetype, clresourcegroup, clresource, clnode, and cluster.

The clresourcetype, clresourcegroup, and clresource commands enable you to view, create, configure, and delete a resource type, a resource group, and the resource objects that are used by the RGM. These commands are part of the administrative interface for the cluster, but are not to be used in the same programming context as the application interface that is described in the rest of this chapter. However, the clresourcetype, clresourcegroup, and clresource commands are the tools for constructing the cluster configuration in which the API operates. Understanding the administrative interface sets the context for understanding the application interface. See the clresourcetype(1CL), clresourcegroup(1CL), and clresource(1CL) man pages for details about the administrative tasks that you can perform with these commands.