Hot Restart Overview

The ChorusOS hot restart feature comprises an API and run-time architecture that offer the following services:

• Persistent memory allocation

  The hot restart API allows actors to allocate and free portions of persistent memory while they are executing. This service is available to all ChorusOS actors after hot restart is configured.

• Actor restart

  With hot restart, the system is capable of detecting the abnormal termination of one or more actors and restarting them automatically from persistent memory. In addition, actors are organized into restart groups, enabling the simultaneous restart of all actors in a predefined group when a single actor in the group fails.

• Site restart

  With hot restart, in addition to restarting one or more actors, the system is capable of restarting all restartable actors, plus the microkernel and boot actors, for a given ChorusOS site.

The combination of these services provides a powerful framework for highly-available systems and applications, dramatically reducing the time it takes for a failed system or component to return to service.

Hot Restart API

The hot restart API is summarized in the following table:

Restartable Actors

A restartable actor is any actor that can be restarted rapidly without accessing stable storage, when it terminates abnormally. A restartable actor is restarted from an actor image that comprises the actor's text and initialized data regions. The actor image is stored in persistent memory (unless the actor is executed in place, in which case the actor image is the actor's executable file, stored in non-persistent, physical memory). Restartable actors can use additional blocks of persistent memory to store their own data.

Figure 3-2 shows the state of a typical restartable actor at its initialization, during execution, and once it has been hot restarted as the result of an error. The actor uses persistent memory to store state data. After hot restart, the actor is reconstructed from its actor image, also in persistent memory. It is then re-executed from its initial entry point, and can retrieve the persistent state data that has been stored.

Figure 3-2 A typical restartable actor

In the hot restart architecture, restartable actors are managed by a ChorusOS supervisor actor called the Hot Restart Controller (HR_CTRL). The HRC monitors restartable actors to detect abnormal termination and automatically take the appropriate restart action. In the context of hot restart, abnormal termination includes unrecoverable errors, such as division by zero, a segmentation fault, unresolved page fault, or invalid operation code.

Restartable actors, like traditional ChorusOS actors, can be run in either user or supervisor mode. They can be executed from the sysadm.ini file, from the C_INIT console, or spawned dynamically during system execution. The restartable nature of restartable actors remains transparent to system actors because restartable actors do not declare themselves restartable, but are run as restartable actors. More specifically, the way in which a restartable actor is initially run determines how it will be restarted when a restart occurs:

• Restartable actors, run from the sysadm.ini file or directly from the C_INIT console, are restarted directly by the system when a restart occurs. These actors are known as direct restartable actors.

• Restartable actors, spawned dynamically during system execution, are restarted by the actor that initially spawned them. These actors are known as indirect restartable actors.

The distinction between direct and indirect restartable actors provides a useful framework for the construction of restartable groups of actors, described in "Restart Groups".

C_INIT and the Hot Restart Controller provide an interface specifically for running and spawning restartable actors.

Restart Groups

Many applications are made up of not one but several actors, that cooperate to provide a service. As these actors cooperate closely, a failure in one of them can have repercussions in the others. For example, assume that actors A and B cooperate closely, using the ChorusOS operating system over IPC, and that A fails. Simply terminating, reloading, or hot-restarting A will probably not be sufficient, and will almost certainly cause B to fail, or to go through some special recovery action. This recovery action may in turn affect other actors that cooperate with actor B. Building cooperating applications that can cope with the large number of potential fault scenarios is a complex task that grows exponentially with the number of actors.

In response to this problem, the hot restart feature uses restart groups. A restart group is essentially a group of cooperating restartable actors that can be restarted in the event of the failure or abnormal termination of one or more actors within the group. In other words, when one actor in the group fails, all actors in the group are stopped and then restarted (either directly, by the system, or indirectly, through spawning). In this way, closely cooperating actors are guaranteed a consistent, combined operating state.

Every restartable actor in a ChorusOS operating system is a member of a restart group. Restart groups of actors are mutually exclusive and, as such, a running actor can only be a member of one actor group (declared when the actor is run), and group containment is not permitted. A restart group is created dynamically when a direct actor is declared to be a member of the group. Thus, each group contains at least one direct actor. An indirect actor is always a member of the same group as the actor that spawned it. A restart group is therefore populated through spawning from one or more direct, restartable actors.

The following figure illustrates the possible organization of restartable actors into groups within a system.

Figure 3-3 Restart Groups in a ChorusOS Operating System

When a group is restarted, it is restarted from the point at which it initially started. Figure Figure 3-4 shows the state of a group of restartable actors when the group is initially created, during execution, and when it is restarted following the failure of one of its member actors. The group contains two direct actors and one indirect (spawned) actor. The failure of the indirect actor causes a group restart. The two direct actors automatically execute their code again from their initial entry point. Time runs vertically down the page.

Figure 3-4 Group restart

Simply restarting a group of actors may still not bring the system to the error-free state desired. Such a situation is possible when the failure that provokes an actor group restart is, in fact, the consequence of an error or failure elsewhere in the system. For this reason, the hot restart feature supports the concept of site restart, described in the next section.

Site Restart

A site restart is the reinitialization of an entire ChorusOS site (system) following the repeated failure of a group of restartable actors. It is the most severe action that can be invoked automatically by the Hot Restart Controller. A site restart involves the following:

• The microkernel and boot actors are reinitialized from the system image. This step is sometimes called a 'hot reboot' of the system, as opposed to a cold reboot, which involves a board reset and initial system loading steps. See the ChorusOS 5.0 Board Support Package Developer's Guide for details on a cold reboot.

• All restartable actor groups are restarted.

The precise number of group restarts to invoke a site restart is determined by the system's restart policy. The policy implemented by the hot restart feature is based on a set of system tunable parameters. You can extend the basic restart policy within your own applications, by choosing to invoke a group or site restart when particular application-specific exceptions are raised, or when particular events occur.

Hot Restart Components

The ChorusOS hot restart feature uses the following two restart-specific actors to implement hot restart services:

• A supervisor actor called the persistent memory manager (PMM), which offers services for allocating and freeing persistent memory blocks.

• A supervisor actor called the hot restart controller, (HR_CTRL). It offers the system calls that create and kill restartable actors, monitors restartable actors for abnormal termination, and takes the appropriate restart action when a failure occurs.

The Persistent Memory Manager and Hot Restart Controller principally use the services of the following:

• The C_INIT actor, for the interpretation of hot restart-specific commands entered on the target or host console.

• The system actor C_OS, solicited by the hot restart controller for loading and running restartable actors.

• The ChorusOS microkernel, for the low-level allocation of persistent memory, and for support for site restart.

The resulting architecture is summarized in Figure 3-5. Hot restart-specific components appear in gray, together with the API calls they provide. Other components appear in white. Arrows from A to B depict A calling functions which are implemented in B.

Figure 3-5 Hot Restart Architecture

For details of how to implement hot restart, see the ChorusOS 5.0 Application Developer's Guide .