test

ChorusOS 5.0 Application Developer's Guide

Chapter 15 Recovering From Application Failure: Hot Restart

This chapter deals with the process restart and persistent memory services available on ChorusOS systems and demonstrates the use of the ChorusOS APIs for increasing the availability of application processes.

After implementing the examples provided in this chapter, you will understand how these APIs might be used in an application.

Note -

For additional information on ChorusOS hot restart and its features, refer to the ChorusOS 5.0 Features and Architecture Overview.

Getting Started with Hot Restart

This section describes how to set up a ChorusOS system to use the hot restart feature. It covers the following:

Note -

This chapter assumes that you have already correctly installed the Sun Embedded Workshop software on a host machine, and that you have a target machine which can be booted from a network boot server. You should also be familiar with configuring your ChorusOS system and building a system image. For more information on these topics, see Chapter 4, Building Makefiles and Configuring the System Image.

System configuration

Before beginning to program and run processes that will use the hot restart feature, you must update and configure your system for hot restart. System configuration for hot restart involves the following steps:

These steps are described in the following sections.

Features

To incorporate hot restart into your ChorusOS system, use the ews graphical tool or the configurator(1CC) command line utility to include the following optional features in your system profile:

Memory Requirements and Design Constraints

The HOT_RESTART feature implements persistent memory as a portion of the random access memory (RAM) on the target device. Although the persistent memory bank does not use virtual memory or swapping, HOT_RESTART is compatible with all three main memory models: flat, protected, and virtual.

The size of the persistent memory bank is defined in bytes by the system tunable parameter, pmm.rambankSize. The value of this parameter is static and cannot be modified while the system is running. In addition, because the RAM persistent memory bank does not use virtual memory or swapping, objects in persistent memory are locked in memory until they are freed. For these two reasons, it is important to ensure that pmm.rambankSize is set to a value realistic for the amount of data likely to be stored in persistent memory at any given time.

A portion of space reserved for an object in the persistent memory bank is known as a persistent memory block. A block is a contiguous set of memory pages, which means that the size of a block is always a multiple of the page size. For more information on the page size relevant to your platform, see the vmPageSize(2K) man page.

For each restartable process that is running, the system stores the following data in persistent memory:

The persistent memory blocks used to store the process image and executing image are freed only when the originating group terminates cleanly. Note that this may be some time after the process itself has terminated. The process can also allocate its own blocks of persistent memory to store run-time data while executing.

Although it can be difficult to predict the likely required value of pmm.rambankSize in the early stages of the development cycle, the following rule of thumb, derived from the previously mentioned statements, may be of use to developers at the system design stage:

Note -

Sharing persistent memory blocks between user processes, or between user and supervisor processes is not supported. Persistent memory blocks can only be shared between supervisor processes.

The default value of the pmm.rambankSize tunable parameter is one megabyte.

Tunable Parameters

The HOT_RESTART feature uses a number of system tunable parameters. Each parameter has a default value which can serve as a guideline, and is generally suitable for getting started with hot restart programming. All tunable parameters are static -- they cannot be modified while the system is running.

Two parameters define the limits for persistent memory occupation of the system's persistent memory bank:

Two parameters control the maximum number of restartable processes and restart groups permitted in a system:

Two parameters define the system's restart policy (see "Site Restart"). These parameters are fairly sensitive -- different values can produce very different behavior in the system. The system manages a restart counter for each restart group. Each time a group is restarted, the system increases its restart counter by one.

Building the System Image

After updating your system's features for hot restart and setting the tunable parameters to suit your requirements, you are ready to build the system image.

To run the examples and hot restart demonstration, include the examples directory and X11 library in your system build paths (if they are not already included). For information on building a system image for your particular target platform, see the corresponding document in the ChorusOS 5.0 Target Platform Collection.

After the system image has been correctly built, copy the image to your boot server and reboot the target machine. You are now ready to begin programming and running applications that can use the hot restart feature.

Running the Hot Restart Demonstration Program

The Sun Embedded Workshop software includes a graphical demonstration of the hot restart feature. The demonstration is based on a well-known program, Xmaze, which has been slightly modified to make it hot restartable. Some of the program's data is stored in persistent memory, which means that when the program is restarted, it starts at a point close to the point it had reached prior to the restart. The resulting application is a ChorusOS process called xdemo.

To run the hot restart demonstration program, do the following:

  1. Ensure that your system features are correctly set for hot restart (see "Features".

  2. Adjust the following system parameters to suit the memory requirements of the Xmaze demonstration program using Ews or the configurator(1CC) command line utility, as shown in the following table:

    Tunable parameter 

    Description 

    Required Value 

    pmm.rambankSize 

    Size of persistent memory bank (in bytes) 

    0x400000 

    kern.exec.dflSysStackSize 

    Default system stack size (in bytes) 

    0x8000 

  3. Configure your system image build to include the X11 library and ChorusOS examples directory (if this is not already the case):


    % make reconfigure NEWCONF '_s <src_dir>/opt/X11'

  4. If you have made changes to the system image since the previous build, rebuild the system and copy the system image to the appropriate location (for example, the boot directory if you are using tftp-based boot). Reboot the target machine.

  5. Ensure that a copy of the xdemo process is present in a directory mounted on the target machine. If you use the make root command, a copy of the process is already stored in build_dir/root/bin/examples. If this directory is not mounted, or to use a different mounted directory, do the following:

    $ cp build_dir/BUILD_EXAMPLES/restartDemo/xdemo example_directory

  6. Set the target machine's DISPLAY environment variable to the host machine:

    $ rsh target setenv DISPLAY host_IP_address:0.0

  7. Run the restartable process:

    $ rsh target arun -g 0 example_directory/xdemo

    The process will be run as a member of the restart group with a group ID 0.

The Xmaze demonstration appears on the screen. As the demonstration runs, it periodically stores its state as data in persistent memory. Allow the demonstration to run for a short time, then restart the process by typing the following command on the host console:

$ rsh target akill pid

The process identifier (PID) is printed on the host console when the process starts.

The process is restarted, and the Xmaze demonstration continues from a point close to where it left off before the restart.

The akill command provoked the restart because it was not called with the restart-specific option -g. To kill the Xmaze demonstration process without restarting it, type:

$ rsh target akill -g 0

Because the xdemo process runs from the command line it is a direct process, and will be started automatically by the system when the site is restarted. To confirm this, rerun the process, and provoke a site restart by typing the following:

$ rsh target restart

After the system has been re-initialized, the demonstration will be restarted.

This is a basic illustration of the use of the hot restart feature. The site restart is provoked manually from the command line. As an alternative, try restarting the process using akill -g sufficiently frequently to trigger an automatic site restart. To do this, set the system's restart policy to be more sensitive to process failure. The following example configuration will invoke a site restart if the process is restarted twice within four seconds:

Tunable parameter 

Value 

hrCtrl.interval 

4 

hrCtrl.maxBadness  

2 

Using Persistent Memory

This section provides a description of the API exported by the Persistent Memory Manager. In particular, it covers the following topics:

In this section, an example "hello world" program is used to illustrate different aspects of the Persistent Memory Manager interface. The code for this example is provided in "A Basic Application".

To run the example, compile the code and copy it to a directory which is mounted on the target machine. See "Compiling and Running the Examples" for information about compiling and running the hot restart examples.

Introduction to Persistent Memory Programming

Within a running ChorusOS system, access to persistent memory is provided by a ChorusOS process known as the Persistent Memory Manager. The Persistent Memory Manager exports a specific API for allocating and freeing blocks of memory in the persistent memory bank. This API differs from the API used for allocating and deallocating traditional ChorusOS memory regions (rgnAllocate(2K), rgnFree(2K), svPagesAllocate(2K) and svPagesFree(2K)) for the following reasons:

The Persistent Memory Manager API is available to all ChorusOS processes (not just restartable processes). This section describes the use of this API.

A Basic Application

Before proceeding with a description of the different functions in the Persistent Memory Manager API, consider the following basic restartable application, an implementation of the "hello world" example. When the process is run for the first time, it displays the following message on the host console:


Hello world!

When the process is restarted, it will display the following message on the target console:


Hello again! I have been restarted.

The basic flow of execution is as follows:

Example 15-1 The "Hello world" Restartable process

#include <stdio.h>
#include <pmm/chPmm.h>
#include <hr/hr.h>
#define HR_GROUP "HELLO_GROUP"
    int 
main()
{
  int res;
 int any = 1;
 int* counter_p;  /* It will be stored in persistent memory */
 long *p;
 PmmName name; 
 KnRgnDesc rgn;
             /* 
              * Initialize the name and medium fields 
              * to identify the persistent memory block in the system.
              */
 bzero(&name, sizeof(name));
 strcpy(name.medium,"RAM");
 strcpy(name.name,"PM1");
             /* 
              * Initialize the block fields
              */ 
 bzero(&rgn, sizeof(rgn));
 rgn.options = K_ANYWHERE | K_RESERVED;
 rgn.size    = vmPageSize();
 res = rgnAllocate(K_MYprocess, &rgn);
 if (res != K_OK) {
            printf("rgnAllocate() failed res=%d\n", res);
            HR_EXIT_HDL();
            exit(-1);
 }
 p = (long*) rgn.startAddr;
            /*
             * From now on p is a bad pointer, since 
             * VIRTUAL_ADDRESS_SPACE is true.
             */
            /* 
             * Allocate the persistent memory block that stores
             * counter_p.
             */
 res=pmmAllocate((VmAddr *)&counter_p,
                        &name,sizeof(int),
                        HR_GROUP,
                        sizeof(HR_GROUP));
  if (res != K_OK) {
      printf("Cannot allocate or map the persistent memory block called %s."
             " Error = %d\n", name.name, res);
        HR_EXIT_HDL();
        exit(-1);
 }
             /* 
              * From the value of *counter_p the process detects  
              * whether it has been hot restarted or not.
              */
 if ( *counter_p==0 ) {
             /*
              * This is the first time the process is run.
              */
            printf("Hello world!\n");
               /* 
                * Increment the counter
                */
            (*counter_p)++;
               /*
                * Normally the next instruction causes a core dump and
                * a hot restart of the process
                */
            *p = 0xDeadBeef;
 } else {
               /* 
                * The process has been restarted
                * NOTE: this message will appear on the console!
                */
            printf("The process has been restarted.\n");
               /*
                * Free the persistent memory block before exiting
                */
            res = pmmFree(&name);
            if (res != K_OK) {
                printf(" pmmFree failed, res=%d. Exit\n", res);
                HR_EXIT_HDL();
                exit(-1);
            } 
               /* 
                * Terminate cleanly. 
                */ 
            printf("Example finished. Exit.\n");
            HR_EXIT_HDL();
            exit(0);
 }
            /* Never reached */
}

The aspects of this program that are of interest to users of the Persistent Memory Manager API are discussed in the rest of this section.

Allocating and Retrieving a Persistent Memory Block

The "hello world" application uses a block of persistent memory to store a counter indicating whether it has been restarted. The value of the counter controls the program's flow of execution. This is a common use of persistent memory. A counter or flag such as this is usually necessary because it is the only way a process can know whether it has been restarted.

A block of persistent memory is described in the system by a structure of the following type:

#include <pmm/chPmm.h>
typedef	struct {
			 	PmmMedium   medium;
         PmmMemName  name;
} PmmName;
PmmName MyPmmName = { "RAM", "myname" };

Within the structure, medium is a character string which identifies the memory bank to be used. In the current implementation, it must always be set to RAM. The name parameter is a user-defined, NULL-terminated character string that uniquely identifies the block of memory in the memory bank. The lifetime of a block name is identical to the lifetime of the block itself in persistent memory. The system parameter, pmm.maxBlocks, defines the number of distinct persistent memory blocks (and therefore names) that can be allocated at any one time. The default value is 30.

Caution - Caution -

Sharing persistent memory blocks between user processes, or between user and supervisor processes is not supported. Persistent memory blocks can only be shared between supervisor processes.

To allocate or retrieve a block of persistent memory, use the pmmAllocate() function call, defined as follows:

#include <pmm/chPmm.h>
KnError pmmAllocate(  VmAddr     *addr,
                      PmmName    *name,
                      size_t      size,
                      PmmDelKey   delKey,
                      size_t      delKeySize);

If no memory block corresponding to the specified PmmName structure is found in persistent memory, pmmAllocate() allocates a block of size size in persistent memory, fills it with nulls, and returns the pointer *addr to the address of the block. The address is determined by the system and cannot be manually specified or changed.

If a block identified with the specified PmmName already exists in persistent memory, pmmAllocate() returns a pointer to the existing memory block as an address (*addr), and the size parameter is ignored. Persistent memory blocks are always mapped to the same address. In other words, the address returned by the first and subsequent calls to pmmAllocate() is always the same for a given block.

As a result of this dual functionality of the pmmAllocate() call, the difference between initially allocating and subsequently retrieving a persistent memory block is transparent at the programming level. The first time the code of the "hello world" example is run, the call to pmmAllocate() will allocate an integer-sized block of persistent memory that contains the initialized value of counter (0). 

res=pmmAllocate((VmAddr *)&counter_p,
                          &name,sizeof(int),
                          HR_GROUP,
                          sizeof(HR_GROUP));

The second time the code is run, pmmAllocate() returns a pointer to the value of counter in persistent memory.

The delKey and delKeySize parameters passed to pmmAllocate() are used to define the deletion key associated with a memory block. A deletion key is a user-defined binary array, used to mark a set of persistent memory blocks which can be freed simultaneously using the pmmFreeAll(2RESTART) function, described in "Freeing a Persistent Memory Block Explicitly".

Freeing a Persistent Memory Block

This section describes the API calls used to free persistent memory blocks.

Responsibility

A persistent memory block can remain in memory beyond the lifetime of a run-time instance of the process that allocates the block. This immediately raises the question of responsibility for freeing blocks of persistent memory. When a traditional ChorusOS user process terminates, any memory regions it previously allocated (using rgnAllocate(2K) are freed automatically. Clearly, this basic rule makes little sense in the case of persistent memory blocks (which can survive beyond such a termination).

The hot restart feature provides two solutions to this problem:

In both cases, freeing a persistent memory block has the same effect -- the block is freed immediately and permanently and cannot be retrieved. The name of the block becomes available for reuse and can be used to identify a different memory block.

Freeing a Persistent Memory Block Explicitly

Use the pmmFree() or pmmFreeAll() function to free a persistent memory block explicitly. The explicit freeing of a given memory block can be performed by any process (not necessarily the process that originally allocated the block). It is the programmer's responsibility to ensure that any persistent memory block that has been freed is no longer in use.

Use pmmFree() to free a single memory block identified by a PmmName:

#include <pmm/chPmm.h>
int pmmFree( PmmName *name )

Use pmmFreeAll() to free a group of persistent memory blocks that were allocated with the same deletion key. The deletion key for a persistent memory block is specified when the block is allocated with pmmAllocate().

#include <pmm/chPmm.h>
int pmmFreeAll( PmmDelKey    delkey,
                size_t       delKeySize );

A typical use of a deletion key is to mark all persistent memory blocks used by a process or a group of processes with the same key, and then have a separate, independent process that frees all the blocks when a particular job is completed (or a specific event occurs). The "hello world" example uses pmmFree() to free the single memory block it allocated before terminating. If the "hello world" process did not free its own persistent memory block, the following call to pmmFreeAll() from another process would free the block, and also with any other blocks marked with the deletion key HR_GROUP:

pmmFreeAll( HR_GROUP, sizeof(HR_GROUP) );

Using Restartable Processes

This section discusses programming and running restartable processes on the ChorusOS operating system. The following topics are presented:

Introduction

A restartable process can be reconstructed rapidly from a process image (text and data) without accessing stable storage. The management of restartable processes is handled by a ChorusOS supervisor process known as the Hot Restart Controller. The Hot Restart Controller is responsible for:

The following section looks at the API provided by the Hot Restart Controller and the corresponding restart-related commands provided by the C_INIT process. Before proceeding to a description of the API, however, it is important to understand how a restartable process is managed within the system.

Types of Restartable Processes

Processes do not explicitly declare themselves restartable, that is, there is no function call to declare a process restartable at the start of its main() program. Instead, a process can be run as a restartable process. Specifically, a process can be run as either a direct or indirect restartable process:

The distinction between direct and indirect processes is important in understanding the automatic restart mechanism provided by the Hot Restart Controller. When an error occurs, the Hot Restart Controller first stops all processes in the group. After the processes are stopped, only the direct restartable processes will be restarted. These processes (re-executed from their initial entry point) are responsible for restarting any indirect processes they may have spawned.

Restartable Process Credentials

Restartable processes, just like traditional ChorusOS processes, are identified in the system by a unique capability and PID. Restartable processes also run in a user group (with a user ID) like traditional ChorusOS processes. The life of each of these credentials is the same as the life of a specific run-time instance of the process -- when a restartable process is restarted, it is given a new capability, PID and user ID.

Hot restartable processes also have two additional credentials which persist across a process restart, and characterize the processes in the Hot Restart Controller:

Restartable Processes and Persistent Memory

As previously discussed in "Memory Requirements and Design Constraints", the system uses persistent memory to store the following data for each executing restartable process:

This data is stored in three persistent memory blocks. One memory block is used for the process image, one is used for the executed text, and the last is used for the process data. These blocks are allocated and freed upon requests from the Hot Restart Controller to the Persistent Memory Manager. Other processes cannot access or free these persistent memory blocks. However, restartable processes can allocate additional blocks under the control of the Hot Restart Controller. This is described in "Freeing Persistent Memory".

The Restartable Process Lifecycle

One approach to understanding how the Hot Restart Controller API is used, is to consider it in the context of the run-time life-cycle of a restartable process. A restartable process's code is not executed just once (from the start of the main() program to its final return). The code may be re-executed several times if there are multiple restarts. Data that is initialized, and processes that are initially loaded during the first execution will only need to be retrieved or restarted on subsequent executions. Therefore, it is important to view the restart API in the context of this first execution, and then of subsequent executions.

This section looks at the way the Hot Restart Controller API is used in the context of the life-cycle of a typical restartable process.

Initial Load

Use the C_INIT command arun with the -g option, or the function call hrfexec() to load a restartable process from stable storage into persistent memory. Both arun and hrfexec() provide support for specifying the persistent credentials of a restartable process when the process is initially loaded.

Processes that were created directly using acreate(2K) or actorCreate(2K) are not hot restartable and are unable to use the Hot Restart Controller API.

When a process is run as a restartable process, the Hot Restart Controller checks whether a process identified with the specified name is already registered. If this is not the case (as with the initial load), the Hot Restart Controller first solicits the Persistent Memory Manager to allocate the persistent memory blocks which will store the process's process image and executing image. If successful, the Hot Restart Controller registers the name of the new process as a restartable process, running in the specified group.

The subsequent load and start of the persistent process is the same as for a process run using a member of the afexec(2K) function family (see the man page for a description of this process). The only difference is that the process is loaded from its process image (in persistent memory) and not from stable storage.

Note -

A restartable process's name remains registered in the Hot Restart Controller for the life of its process group. The lifespan of the group may extend beyond the lifespan of the process. It is the programmer's responsibility to ensure that no two restartable processes will attempt to register with the same name in the Hot Restart Controller.

After a restartable process has been registered and loaded, it runs under the control of the Hot Restart Controller. If the process fails, the failure will invoke the restart of all direct members of its restart group. These direct processes will be responsible for restarting any indirect processes registered in the group. To query a process's restart group, use hrGetprocessGroup(2RESTART):

#include <hr/hr.h>
hrGetprocessGroup(int pid)

Group Restart

In the context of hot restart, a process is considered to have terminated abnormally (and will therefore invoke the restart of its group) if any of the following occur:

There is no single API call that can explicitly force a group of processes to restart. For cases in which it may be desirable to provoke a restart (for example, for testing purposes). The easiest way to do so is to deliberately provoke one of the previous cases. In the "hello world" example introduced in the previous chapter, this was done by causing a segmentation fault.

When a process fails, all processes in the failed restart group stop running and the Hot Restart Controller restarts all direct processes in the group from their initial entry point. The direct processes are responsible for restarting any indirect processes, using hrfexec(). When hrfexec() is called with a name that is already registered in the Hot Restart Controller, the Controller recognizes the process name and restarts the process from the process image, instead of loading it from stable storage.

A restartable process is always restarted at the same address. Its capability, process ID and user ID are not guaranteed to be will not necessarily be the same after restart. All system resources obtained before the restart are lost: in particular, open files, including those that were inherited at the time of initial creation are lost. This may include standard I/O connected to an rsh connection.

A restarted process uses the same arguments and environment parameters that were specified when the process was initially started. For direct restartable processes, a new set of pre-open stdin/stdout/stderr has been provided, which is connected to /dev/console. For indirect members, a new set of pre-open stdin/stdout/stderr is provided by the invoker of hrfexec(), just as for afexec(2K).

Freeing Persistent Memory

Just like any process, a restartable process can free persistent memory blocks using pmmFree() or pmmFreeAll(). This is described in "Freeing a Persistent Memory Block".

Restartable processes which allocate memory with pmmAllocate() can also use a basic automatic deallocation mechanism provided by the Hot Restart Controller. This saves the process from having to free its persistent memory explicitly. Instead, the persistent memory will remain allocated for the lifespan of the process's group, and then be freed automatically by the Hot Restart Controller when the last member of the process's restart group terminates cleanly. The disadvantage of this system is that the lifespan of the restart group may extend well beyond the point at which the memory block is no longer required. In this situation, the memory block will take up space in persistent memory unnecessarily.

To mark a persistent memory block for automatic de-allocation by the Hot Restart Controller, pass the macros HR_GROUP_KEY and HR_GROUP_KEYSIZE as the delKey and delKeySize arguments respectively in the call to pmmAllocate(). These macros tie the lifespan of the persistent memory block to the lifespan of the calling process's restart group.

A block marked for automatic de-allocation by the Hot Restart Controller can still be freed explicitly by calling pmmFree() with the block's PmmName. However, attempting to call pmmFreeAll() by passing the HR_GROUP_KEY and HR_GROUP_KEYSIZE macros will result in an error because this is not permitted.

Clean Termination

Any process that exits before the expected completion of its task is considered to have aborted abnormally and will cause a restart of its process group. This can be useful for cases where the process exits prematurely as a result of an error. This mechanism can also be useful for invoking a process restart where this is required, for example, if an execution problem is detected.

To enable a restartable process to terminate cleanly without causing a restart, use the HR_EXIT_HDL() macro prior to the call to exit(3STDC):

#include <hr/hr.h>
HR_EXIT_HDL();

The purpose of the preceding macro is to add an additional hot restart exit handler to the process's atexit(3STDC) function. The hot restart exit handler effectively removes the process in question from the Hot Restart Controller's responsibility. After a process has called HR_EXIT_HDL(), the Hot Restart Controller will no longer monitor the process for abnormal termination. As a result, when the process exits, it will terminate cleanly and not trigger a restart.

The HR_EXIT_HDL() macro should be called shortly before the process exits. Calling this macro earlier in the process code will mean that any unexpected exit between the macro call and the final exit will not be detected by the Hot Restart Controller. As a result, the process will not be restarted if it exits abnormally.

Cleanly terminating a process does not unregister the process in the Hot Restart Controller or remove the process's process image and executing image from persistent memory. This is because a cleanly terminated process will still be restarted if its group is restarted (because a group is always restarted in its initial state). In other words, when a group is restarted, all direct restartable processes will recommence execution at their initial entry point, regardless of whether or not they had already exited before the restart occurred. This is demonstrated by the following diagram. Both direct process one (DP1) and indirect process two (IP2) terminate cleanly, but are automatically restarted when direct process two (DP2) crashes.

Figure 15-1 Restart of Cleanly Terminated Processes

Graphic

Because of this behavior, it is useful to record the clean termination of restartable processes that will never require being reexecuted completely during a group's life by setting a flag in persistent memory. A restarted process can check the state of this flag at the start of its execution, and therefore detect whether it should re-execute or not.

Group Termination

For each group of restartable processes present in a ChorusOS system, the Hot Restart Controller stores a list of the processes for each group in a persistent memory block. A process is added to the list when it is first started. When a process cleanly terminates, the Hot Restart Controller notes this in the list. When all processes in the list have terminated cleanly, the Hot Restart Controller performs the following:

A group of processes can only terminate if all of its member processes terminate cleanly. This is important to remember in situations where not all indirect processes are restarted after a group restart. This is a matter of execution flow: if certain conditions in a direct process change the process's flow from one execution to another, the direct process may not restart an indirect process that was running prior to the restart. As a result, the indirect process will never terminate cleanly and so the group will not be able to terminate.

For example, consider the situation in the following diagram. The direct process spawns the indirect process only after certain conditions are met. These conditions are met the first time the direct process runs. After the direct process restarts, the conditios are no longer satisfied, so the indirect process is no longer spawned.

Figure 15-2 Conditional Spawning of a Restartable Process

Graphic

In the preceding diagram, the process group will not be able to terminate until the indirect process has been rerun using hrfexec(), and has terminated cleanly.

When a restart group cannot terminate because of one or more direct processes, the Hot Restart Controller detects this situation and displays the following message on the target console:


HR_CTRL: group gid blocked, some members have 
not terminated: list_of_processes

gid is the ID of the group in question, and list_of_processes provides the name of each process which prevents the group from terminating. When this message is displayed, a common solution is to kill the process group using the akill command with the -g option. However, this solution is useful only if none of the indirect processes need to be run to complete the group's task.

A better solution is to use careful application design. If the preceding situation is likely to occur, flags can be stored in persistent memory to identify indirect processes that have not terminated cleanly. A process can then be made responsible for cleaning up the group, that is, restarting each indirect process that is flagged. This clean-up process can be run using the arun -g command when the Hot Restart Controller notification is displayed on the target console. Alternatively, the group could be designed so that the clean-up process is always run just before the group is expected to terminate. In this case the problem is solved without accessing the C_INIT console.

Killing Restartable Processes

At times it may be necessary to circumvent the automatic restart mechanism provided by the Hot Restart Controller and explicitly terminate (kill) a restartable process. Processes which are killed will not be restarted. Killing a process automatically kills all processes within the process's restart group. This is because a restart group must remain consistent. The restart group may not be able to function correctly if a process is no longer available.

Restartable processes can be explicitly killed using either of the following:

Either method produces the same result: all processes in the associated restart group are killed. The Hot Restart Controller terminates the group as though all processes had exited cleanly (see "Group Termination" Group Termination).

Site Restart

A site restart is a hot restart of the entire system. All data of boot processes are reset to their original values from the previously loaded system image and the system enters its start-up phase again. As C_INIT restarts, sysadm.ini is reexecuted. Any calls to start restartable processes in the sysadm.ini file are ignored for a site restart because all direct restartable processes are automatically restarted by the system after the sysadm.ini file has been read.

Note -

When the system is restarted, previously mounted disks are not automatically remounted. To resolve this issue, ensure that the disks are mounted in the sysadm.ini file, or create a hot restartable process that will automatically mount the disks.

A site restart can be invoked automatically by the Hot Restart Controller, according to the tunable parameters that define the system's restart policy. For more information, see "Tunable Parameters".

To invoke a site restart programmatically, use the sysShutdown(2K) function call with the -i 1 arguments:

int sysShutdown (int argc, char** argv)

To provoke a site restart from the C_INIT command-line console, use the command shutdown -i 1 or restart(1M) command.

Putting It All Together: the restartSpawn Example Program

The following code example, restartSpawn, illustrates many of the function calls covered in this and previous chapters. The example is provided as an overview of the restart mechanism and the use of persistent memory. Specific parts of the example could be used as the basis of a more complex user application that incorporates hot restart.

The restartSpawn example uses two restartable processes, a parent process, HR_parent.r and a child, HR_child.r which is spawned by the parent. Both processes should be compiled as supervisor processes. The source code for the two processes is provided in "Example Application Code". The example can be summarized as follows:

The message is displayed independently of the number of times the parent process crashes, or the site is restarted.

Hot Restart Programming Environment

This section describes the environment used for programming and compiling applications that use the API exported by the hot restart feature. For additional general information about compiling and linking ChorusOS processes, see Chapter 6, Building Applications for ChorusOS Systems.

Hot Restart Header Files and Directories

The hot restart programming interface is declared in the following files:

For the Persistent Memory Manager API:

install_dir/chorus-family/kernel/include/chorus/pmm/chPmm.h

For the Hot Restart Controller API:

install_dir/chorus-family/os/include/chorus/hr/hr.h

install_dir/chorus-family/os/include/chorus/hr/hrCtrl.h

Detailed descriptions of each function call are available in the ChorusOS man pages.

Make Environment

A restartable process can be compiled using any of the following standard Imakefile macros:

Processes that use dynamic or shared libraries (compiled with Imake macros of the type Dynamic...Target or Shared...Target) are not hot restartable.

Use the following table to link processes that use the API exported by the hot restart feature. Note that all ChorusOS processes are automatically linked with the libc.a library.

API Function 

Library 

hrfexec()

HR_EXIT_HDL()

hrKillGroup()

hrGetprocessGroup()

libc.a

pmmAllocate()

pmmFree()

pmmFreeAll()

pmmlib.a

The following is an example Imakefile for a restartable process that uses the Persistent Memory Manager API:

SRCS = HR_process.c
UserprocessTarget(HR_process_s, HR_process.o, 
                  $(NUCLEUS_DIR)/lib/pmm/pmmlib.a)
Depend($(SRCS))

Example Application Code

This section provides the following:

The source code for the example applications is also provided in install_dir/chorus-family/src/opt/examples after the examples package has been installed on your system.

Compiling and Running the Examples

Two examples have been designed to illustrate the use of the hot restart API and are provided with the Sun Embedded WorkShop software. The examples are as follows:

To compile the examples, ensure that the examples directory is included in your system image build configuration. Binaries for all examples are provided in build_dir/build-EXAMPLES after the building of the examples directory.

To run the examples, first copy them to a directory which is mounted on the target, or use the make root command to build a root directory to mount.

Use the C_INIT command arun with the -g option to run a restartable process from the command line. For example, to run the 'hello world' restart example:


$ rsh target arun -g 0 example_directory/HR_hello_u

Where target is the target name, and example_directory is the directory mounted on the target machine where the restartable hello world process binary is stored. The -g 0 option runs the hello world restartable process as a member of a restart group with ID 0.

The "hello world" Restartable process

The restartable "hello world" process is a basic illustration of the use of persistent memory.

See "Using Persistent Memory" for further information on this process.

helloRestart.c 

#include <stdio.h>
#include <pmm/chPmm.h>
#include <hr/hr.h>

#define HR_GROUP "HELLO_GROUP"

    int 
main()
{
	
	int res;
	int any = 1;
	int* counter_p;  /* It will be stored in persistent memory */
	long *p;
	PmmName name; 
	KnRgnDesc rgn;
	
            /* 
             * Initialize the name and medium fields 
             * to identify the persistent memory block in the system.
             */
	bzero(&name, sizeof(name));
	strcpy(name.medium,"RAM");
	strcpy(name.name,"PM1");
	
            /* 
             * Initialize the block fields
             */	
	bzero(&rgn, sizeof(rgn));
	rgn.options = K_ANYWHERE | K_RESERVED;
	rgn.size    = vmPageSize();
	res = rgnAllocate(K_MYprocess, &rgn);
	if (res != K_OK) {
            printf("rgnAllocate() failed res=%d\n", res);
            HR_EXIT_HDL();
            exit(-1);
	}

	p = (long*) rgn.startAddr;

            /*
             * From now on p is a bad pointer, since 
             * VIRTUAL_ADDRESS_SPACE is true.
             */

            /* 
             * Allocate the persistent memory block that stores
             * counter_p.
             */
	res=pmmAllocate((VmAddr *)&counter_p,
                        &name,sizeof(int),
                        HR_GROUP,
                        sizeof(HR_GROUP));
	
	if (res != K_OK) {
            printf("Cannot allocate or map the persistent 
	            memory block called %s."
                   " Error = %d\n", name.name, res);
            HR_EXIT_HDL();
            exit(-1);
	}
	
            /* 
             * From the value of *counter_p the process detects  
             * whether it has been hot restarted or not.
             */
	if ( *counter_p==0 ) {
                /*
                 * This is the first time the process is run.
                 */
            printf("Hello world!\n");
		
                /* 
                 * Increment the counter
                 */
            (*counter_p)++;
		
                /*
                 * Normally the next instruction causes a core dump and
                 * a hot restart of the process
                 */
            *p = 0xDeadBeef;

	} else {
                /* 
                 * The process has been restarted
                 * NOTE: this message will appear on the console!
                 */
            printf("The process has been restarted.\n");
		
                /*
                 * Free the persistent memory block before exiting
                 */
            res = pmmFree(&name);
            if (res != K_OK) {
                printf(" pmmFree failed, res=%d. Exit\n", res);
                HR_EXIT_HDL();
                exit(-1);
            }	
                /* 
                 * Terminate cleanly. 
                 */ 
            printf("Example finished. Exit.\n");
            HR_EXIT_HDL();
            exit(0);
	}
            /* Never reached */
}

Imakefile for helloRestart.c 

SRCS = helloRestart.c

SupprocessTarget(helloRestart.r, helloRestart.o, 
                 $(NUCLEUS_DIR)/lib/pmm/pmmlib.a)
UserprocessTarget(helloRestart_u, helloRestart.o, 
                  $(NUCLEUS_DIR)/lib/pmm/pmmlib.a)

Depend($(SRCS))

The restartSpawn Example

The restartSpawn example comprises two processes: HR_parent.r and HR_child.r. Both processes must be compiled as supervisor processes (see "Putting It All Together: the restartSpawn Example Program" for an overview of the restartSpawn example).

HR_parent.c 

#include <stdio.h>
#include <stdlib.h>
#include <strings.h>
#include <am/afexec.h>
#include <pmm/chPmm.h>
#include <exec/chModules.h>
#include <hr/hr.h>
#include <err.h>
#include <errno.h>

#define PM_MEDIUM "RAM"
#define PM_NAME "PARENT_PM"
#define MAX_LOOPS 8

/*
 * Some static variables
 */
char baseName[PATH_MAX];
char last_global_data;

/*
 * Declaration of objects that will be stored in persistent memory.
 * restarted: number of times the process has been restarted.
 * counter: number of times the process's main loop is run.
 */
typedef struct _HR_Status {
    int restarted;
    int counter;
} HR_Status;

    /*
     * Wait "sec" seconds.
     */
    void
waitSec(int sec)
{
    KnTimeVal delay;
    delay.tmSec = sec;
    delay.tmNSec = 0;
	
    (void) threadDelay(&delay);
}

    /*
     * Create a child hot restartable process.
     * Start the child process only if the parent has
     * not been hot restarted.
     */
    void
childCreate()
{
    KnCap childCap;
    KnprocessPrivilege curActPriv;
    PmmName childName;
    int res;
    int childPid = -1;
    char path[PATH_MAX];
    char* argv[3];
	
    res = processPrivilege(K_MYprocess, &curActPriv, NULL);
    if (res != K_OK) {
        printf("processPrivilege failed, res=%d\n", res);
	HR_EXIT_HDL();
	exit(-1);
    }
    
    if (curActPriv != K_SUPprocess) {
    	argv[0] = "HR_child";
    } else {
        argv[0] = "HR_child_u";
    }
	
    argv[1] = NULL;
    argv[2] = NULL;
    
    strcpy(childName.medium, "RAM");
    strcpy(childName.name, "CHILD");
	
    strcpy(path, baseName);
    if (curActPriv == K_SUPprocess) {
        strcat(path, "HR_child");
    } else {
        strcat(path, "HR_child_u");
    }
	
    childPid = hrfexecv(&childName, path, &childCap, NULL, argv);
	
    if (childPid == -1) {
        printf("Cannot hrfexecv(%s), error=%d\n", path, errno);
        HR_EXIT_HDL();
        exit(-1);
    } 
}

    /*
     * Cause a hot restart by exiting without
     * first calling HR_EXIT_HDL().
     */
    void
crash_exit()
{
    printf("\nPARENT hot-restarts (exits with no HR_EXIT_HDL)!\n");
    exit(1);
}

    /*
     * Cause a segmentation fault.
     */
    void
crash_seg()
{
    KnRgnDesc      rgn;
    unsigned long* badSupPtr;
    int            res;
	
    rgn.options = K_ANYWHERE | K_RESERVED;
    rgn.size    = vmPageSize();
    rgn.opaque1 = NULL;
    rgn.opaque2 = 0;
    res = rgnAllocate(K_MYprocess, &rgn);
    if (res != K_OK) {
        printf("unable to allocate a page res=%d\n", res);
        return;
    }
    
    badSupPtr = (unsigned long*) rgn.startAddr;

    printf("\nPARENT crashes (segmentation fault)!\n");

        /* 
         * Generate an unrecoverable page fault, since 
         * VIRTUAL_ADDRESS_SPACE is true
         */
    *badSupPtr = (unsigned long) 0xffffffff;
	
        /*
         * it should never return with 
         */
    printf("Can't generate a crash\n");
    return;
}

    /*
     * Cause a failure due to division by 0.
     * Note: This does not crash on some platforms.
     */
    int
crash_div()
{
    int i;
    int z;
    int x = 1;
	
    printf("\nPARENT tries to crash with division by 0!\n");
    for (i = 10; i > -1; i--) {
        z = x/i;
    }
    return z;
}

    /*
     * Perform a site restart.
     */
    void
site_restart()
{
    char*       argv[3];
    int         res;
	
    argv[0] = "shutdown";
    argv[1] = "-i";
    argv[2] = "1";
	
    res = sysShutdown (3, argv);
	
    if (res) {
        printf("parent error=%d\n", res);

    } else {
		
        waitSec(5);
        printf("Timeout ! \n");
    }
}

    /*
     * Kill the group processes and free persistent memory
     * blocks allocated by the parent process.
     */
    void
clean_up(PmmName *np)
{
    int res;
    int group = 1;
    int actId;
	
    actId = agetId();
	
    res=pmmFree(np);
    if (res != K_OK) {
        printf("\nCannot free the persistent memory block called %s."
               " Error = %d\n", np->name, res);
        HR_EXIT_HDL();
        exit(-1);
    }
    printf("\nPersistent memory has been freed.\n");

    group=hrGetprocessGroup(actId);
    if (group < 0) {
        printf("Cannot get process group. Error = %s\n", errno);
        HR_EXIT_HDL();
        exit(-1);
    }

    printf("Example finished. Exit.\n");

    res=hrKillGroup(group);
    if (res != K_OK) {
        printf("Cannot kill process group %d. Error = %d\n", group, res);
        HR_EXIT_HDL();
        exit(-1);
    }
}

    /*
     * main
     */
    int main(int argc, char** argv, char**envp)
{
	
    int res;
    int counter;
    int ref;
    int* mem_version;
    static PmmName name; 
    HR_Status* st;
    char* endPath;
    KnprocessPrivilege curActPriv;
	
        /* 
         * Check that argc != 0. Otherwise exit.
         */
    if(argc==0) {
        printf("Cannot start this test. argc == %d. Exit.\n", argc);
        HR_EXIT_HDL();
        exit(-1);
    }

    res = processPrivilege(K_MYprocess, &curActPriv, NULL);
    if (res != K_OK) {
        printf("processPrivilege failed, res=%d\n", res);
        HR_EXIT_HDL();
        exit(-1);
    }

    if (curActPriv != K_SUPprocess) {
        printf("This example can only be run in supervisor mode. Exit.\n");
        HR_EXIT_HDL();
        exit(-1);
    }

        /*
         * If the example runs in flat memory mode, it will not work.
         * Some of the failures will not always cause a hot-restart.
         * Print an error message and exit.
         */
    res = sysGetConf(K_MODULE_MEM_NAME, K_GETCONF_VERSION, mem_version);
    if (res != K_OK) {
        printf("Cannot get memory configuration."
               " res=%d\n", res);
        HR_EXIT_HDL();
        exit(-1);
    }
	
    if (*mem_version==K_MEM_VERSION_FLM) {
        printf("Sorry. The example cannot be run in flat memory"
               " configuration. Exit.\n");
        HR_EXIT_HDL();
        exit(-1);
    }
	
        /*
         * Get the directory of the current process.
         */
    strcpy(baseName, argv[0]);
    endPath = strrchr(baseName, '/');
    *(endPath+1) = '\0';
	
        /* 
         * Initialize the name and medium fields to identify 
         * the HR_Status structure.
         */
    bzero(&name, sizeof(name));
    strcpy(name.medium,PM_MEDIUM);
    strcpy(name.name,PM_NAME);
	
        /*
         * Allocate or map the data in st in persistent memory.
         */
    res=pmmAllocate((VmAddr *)&st,
                    &name,
                    sizeof(HR_Status),
                    HR_GROUP_KEY,
                    HR_GROUP_KEYSIZE);
	
    if (res != K_OK) {
        printf("Cannot allocate or map the persistent memory block 
	        called %s."
               " Error = %d, errno=%d\n", name.name, res, errno);
        HR_EXIT_HDL();
        exit(-1);
    }
	
        /*
         * If the process has been restarted, print out a message.
         */
    if (st->restarted>0) {
        printf("PARENT RESTARTS (%d-th time)\n", st->restarted);
    }
	
        /*
         * Increase the "restarted" counter.
         */
    st->restarted++;
	
	
        /*
         * Create a child hot-restartable process.
         */
    childCreate();
	
        /* 
         * main loop
         * provokes different faults in the parent process.
         * This causes the parent AND the child to hot restart.
         */
    while ( st->counter<MAX_LOOPS ) {
        waitSec(2 + rand() % 2);
        st->counter++;
        ref = (st->counter%5);
        switch ( ref ) {
            case 1:
                crash_seg();
                break;
            case 2:
                res = crash_div();
                    /* 
                     * If you get here, it means that division by 0 does not
                     * crash your system! 
                     */
                printf("The parent process does not crash"
                       " with division by 0. Continue.\n");
                break;
            case 3:
                crash_exit();
                break;
            case 4:
                site_restart();
                break;
            default:
                break;
        }
    }
	
        /*
         * Example complete. Free persistent memory blocks and exit.
         */
    clean_up(&name);

}

HR_child.c 

#include <stdio.h>
#include <strings.h>
#include <pmm/chPmm.h>
#include <hr/hr.h>
#include <exec/chExec.h>
#include <pd/chPd.h>
#include <errno.h>

#define PM_MEDIUM "RAM"
#define PM_NAME "CHILD_PM"
#define MESSAGE_NAME "CHILD_MESSAGE"
#define MESSAGE_SIZE 100

typedef struct _HR_Status {
    int restarted;
    int checkpoint;
} HR_Status;

/* 
 * Static variables
 */
static HR_Status *st;

    /*
     * Wait "sec" seconds.
     */
    void 
waitSec(int sec)
{
    KnTimeVal delay;
    delay.tmSec = sec;
    delay.tmNSec = 0;
	
    (void) threadDelay(&delay);
}

    /*
     * General operations in all steps.
     */
    void 
gen_step (char** message, char* m_out) 
{
    strcat(*message, m_out);
    printf("%s", m_out);
    fflush(NULL);
	
        /* 
         * st is stored in persistent memory. 
         * If the process does not reach the end of the next instruction
         * before a hot restart, the current step will be repeated.
         */
    st->checkpoint=++(st->checkpoint) % 4;
}

    /*
     * step1
     */
    void 
step1 (char** message) 
{
    gen_step(message, " STEP 1 ");  
}

    /*
     * step2
     */
    void 
step2 (char** message) 
{
    gen_step(message, " STEP 2 ");
}

    /*
     * step3
     */
    void 
step3 (char** message) 
{
    gen_step(message, " STEP 3 ");
}

    /*
     * step4
     */
    void 
step4 (char** message) 
{	
    gen_step(message, " STEP 4 ");
	
        /*
         * Print out the entire message at the end of the cycle.
         * The entire message is printed even if the child process
	 * is restarted during a cycle.
				* 
         * =========== Message ===========
         *  STEP 1  STEP 2  STEP 3  STEP 4 
         * ======== End of message========
         *
         * Note that output from the parent process may garble 
				* this output.
         */
    printf("\n\n=========== Message ===========\n");
    printf("%s", *message);
    printf("\n======== End of message========\n\n");
	
        /*
         * Reset the message.
         */
    bzero(*message, MESSAGE_SIZE);
}

    /*
     * Function to be executed before the process exits for any reason.
     */
    void 
before_exit() 
{
    printf("CHILD EXITS!\n");
}

    /*
     * main
     */
    int 
main(int argc, char** argv, char**envp)
{
	
    int	res;
    int	counter;
    static PmmName name;
    static PmmName m_name; 
    size_t size;
    PdKey key;  
    char message[MESSAGE_SIZE];
    KnprocessPrivilege curActPriv;
	
    res = processPrivilege(K_MYprocess, &curActPriv, NULL);
    if (res != K_OK) {
        printf("processPrivilege failed, res=%d\n", res);
        HR_EXIT_HDL();
        exit(-1);
    }
    if (curActPriv == K_SUPprocess) {
            /*
             * Create a private process data key with a
					 * destructor associated with it.
             */
        res = padKeyCreate(&key, (KnPdHdl)before_exit);
        if(res != 0) {
            printf("Couldn't create PD key. Exit with errno %d\n", errno);
            HR_EXIT_HDL();

            exit(-1);
        }
		
        res = padSet(K_MYprocess, key, "M");
        if (res != K_OK) {
            printf("Cannot set the PD key, error %d\n", res);
            HR_EXIT_HDL();
            exit(-1);
        }
    } else {
        res=atexit(&before_exit);

            /*
             * atexit() accepts up to 32 functions so this cannot fail.
             */
    }
	
        /* 
         * Initialize the name and medium fields for the 
				* HR_Status structure.
         */
    bzero(&name, sizeof(name));
    strcpy(name.medium,PM_MEDIUM);
    strcpy(name.name,PM_NAME);
	
        /*
         * Allocate or map the data in st in persistent memory.
         */
    res=pmmAllocate((VmAddr *)&st,
                    &name,
                    sizeof(HR_Status),
                    HR_GROUP_KEY,
                    HR_GROUP_KEYSIZE);
	
	
    if (res != K_OK) {
        printf("Cannot allocate or map the persistent memory block 
	        called %s." " Error = %d\n", name.name, res);
        HR_EXIT_HDL();
        exit(-1);
    }
	
        /* 
         * Initialize the name and medium fields for the
				* message char buffer.
         */
    bzero(&m_name, sizeof(m_name));
    strcpy(m_name.medium,PM_MEDIUM);
    strcpy(m_name.name,MESSAGE_NAME);
	
        /*
         * Allocate or map the message data in persistent memory.
         */
    res=pmmAllocate((VmAddr *)&message, 
                    &m_name, 
                    MESSAGE_SIZE, 
                    HR_GROUP_KEY,
                    HR_GROUP_KEYSIZE);
	
    if (res != K_OK) {
        printf("Cannot allocate or map the persistent memory block 
	        called %s." " Error = %d\n", name.name, res);
        HR_EXIT_HDL();
        exit(-1);
    }
	
        /*
         * If the process has been restarted, print out a message.
         */
    if (st->restarted>0) {
        printf("CHILD RESTARTS (%d-th time)\n", st->restarted);
    }
	
        /*
         * Increase the "restarted" counter.
         */
    st->restarted++;
	
        /*
         * Loop forever.
         * Each time the parent process crashes, the child process will be
         * stopped with it since they belong 
         * to the same group.
         */
    while ( 1 ) {
        waitSec(1);
        switch ( st->checkpoint ) {
            case 0:
                step1(&message);
                break;
            case 1:
                step2(&message);
                break;
            case 2:
                step3(&message);
                break;
            case 3:
                step4(&message);
                break;
            default:
                break;
        }
    }
    return 0;
	
}

Imakefile for HR_parent.c and HR_child.c 

SRCS = HR_child.c HR_parent.c

SupprocessTarget(HR_child.r,HR_child.o,$(NUCLEUS_DIR)/lib/pmm/pmmlib.a)
SupprocessTarget(HR_parent.r,HR_parent.o,$(NUCLEUS_DIR)/lib/pmm/pmmlib.a)

Depend($(SRCS))