Advance Reservation

C H A P T E R 5

Advance Reservation

Advance Reservation (AR) is a feature of some queuing software systems, but this feature is not present in Sun Grid Engine release 6.1. (If you are using a later release of Sun Grid Engine, check whether that version includes an Advance Reservation feature.) AR schedules compute and visualization resources when the computer resources and the people to use the resources are both available. Reservations must not be scheduled to conflict with each other (by oversubscribing available resources), nor with other Sun Grid Engine uses of the same resources.

This chapter details information the system administrator needs to know about AR. The Sun Shared Visualization 1.1 Software Client Administration Guide, 820-3257, contains information for the end user.

Topics in this section include:

Advance Reservation Overview

Architecture of Advance Reservation Facility

Advance Reservation File Structure

Initial Configuration of Advance Reservation

Using Advance Reservation

Reservation States

Advance Reservation Troubleshooting

Advance Reservation Overview

A user can reserve specified resources at a given time, for a given duration. Once confirmed, the resources are available to that user’s Sun Grid Engine jobs during that given reservation period. Jobs intended to run during the reservation period can be submitted to Sun Grid Engine (as with Sun Grid Engine’s qsub command) right after the reservation is confirmed, or anytime before the end of the reserved period.

Implementing Advance Reservations outside of Sun Grid Engine requires creating a dynamic Sun Grid Engine queue to represent each confirmed reservation. Resources are allocated to the reservation’s queue by temporarily removing the resources from the execution host’s generic queue (such as all.q or interactive). Resources are removed in advance of the reservation, so that non-AR jobs that use the resources are finished prior to when the reservation is scheduled to start.

Architecture of Advance Reservation Facility

The Advance Reservation package has two kinds of programs:

An AR server

The AR server assures that reservations consume only enabled resources. This server assures that confirmed reservations’ resources are available to the users. For each confirmed reservation, the server dynamically creates a Sun Grid Engine queue that becomes active (that is, the queue’s jobs can run) at the reservation time. The server runs as the Sun Grid Engine administrator (sgeadmin) on a Sun Grid Engine administration host.

An AR client

The AR client is used by any Sun Grid Engine user to create, list, and delete reservations. The client communicates with the AR server. The AR client exists in two forms. The Reserve client is a simple command-line program. The Reserve GUI is a graphical user interface that eases use.

Additional components involved in AR, shown in the architectural diagram (FIGURE 5-1), are:

The reservation database

This database is currently a Berkeley database, which is used only in primitive ways to make reservations persistent.

Sun Grid Engine

This application is the software that actually allocates resources to jobs, including jobs submitted against the reservation’s queue.

FIGURE 5-1 shows the implementation architecture.

FIGURE 5-1 Advance Reservation Architecture

Advance Reservation File Structure

The Advance Reservation package installs an ar subdirectory under the Sun Grid Engine SGE_ROOT. The default SGE_ROOT is /gridware/sge. If your SGE_ROOT differs from the default, you must set the SGE_ROOT environment variable, typically by sourcing the Sun Grid Engine /gridware/sge/default/common/settings.csh or /gridware/sge/default/common/settings.sh file.

Under $SGE_ROOT/ar you find these components:

TABLE 5-1 Directory Tree Under `$SGE_ROOT/ar`
Directory or Filename		Description or Comment
`bin/`
	`SERVER`	Script that starts the Advance Reservation server.
	`runar`	Script to set up the necessary environment and then start Java. This script might need editing for purposes such as: Indicating the correct location of a Java that is at least 1.5 Reflecting the `$SGE_ROOT/ar` location. A copy of this script is needed by users to start the AR client easily.
`config/`		The Advance Reservation server is configured using these files.
	`queue.template`	This file is the basis for creation of a Sun Grid Engine dynamic queue to represent a reservation. This file is similar to the output of the Sun Grid Engine `qconf -sa all.q@‘hostname‘` command, but with spots ready to be replaced. You can edit this file to be more similar to your output of that command.
	`users.template`	This file is the basis for creation of a Sun Grid Engine user list, and enables only the user creating a reservation to use the reservation. Adding users to this template (prior to `#Users#`) enables all such users to submit jobs to any reservation.
	`defaults.prop`	A Java properties file that provides the execution host name and domain, the generic queue name (default `all.q`), and some configuring durations. For example: `#Advance Reservation Configuration PropertiesAdvanceReservation.ServerHost=my1AdvanceReservation.ServerPort=6789AdvanceReservation.ExecutionHostList=my1 my2 anotherAdvanceReservation.ExecutionHostDomain=my.company.comAdvanceReservation.GenericQueueName=all.qAdvanceReservation.MaximumNonreservedJobDuration=2\:0\:0AdvanceReservation.FinishToDeleteQueueDuration=12\:0\:0`
	`complex`	Similar to `qconf -sc` output, this file contains the vocabulary of resources that can be reserved. Only integer consumable resources are currently supported. A resource could be required (as in Sun Grid Engine), meaning a reservation request would need to specify a value for that resource.
	`limits`	Similar to `qconf -sq all.q` output, this file gives a maximum value of a resource that can be allocated by the AR server. For example, a host with 10 graphics resources might enable at most six to be used for AR, keeping four for temporary use (through Sun Grid Engine or outside of Sun Grid Engine entirely).
`lib/`		Directory of files needed by the Advance Reservation server and clients.
	`je.jar`	Berkeley Database Java Edition (needed by server).
	*.jar	Additional Java archives needed by the Advance Reservation server and clients.
	`*.perl`	Scripts used by server (in preparing files for Sun Grid Engine).

Planning Configuration of Advance Reservation

Specifying a Nondefault `SGE_ROOT`

If during installation or configuration you choose a different location for SGE_ROOT than the default of /gridware/sge, and if the Solaris software package SUNWsgeec is not installed on the Advance Reservation server host, the Advance Reservation facility needs your SGE_ROOT value. You must edit several files to include the nondefault location so that the Sun Grid Engine and Advance Reservation feature function properly.

To Edit the Files to Match a Nondefault SGE_ROOT

1. Install the Sun Grid Engine software and optionally configure for Advance Reservation.

2. As superuser, use an editor to edit the following files:

$SGE_ROOT/ar/bin/runar

$SGE_ROOT/ar/bin/SERVER

/lib/svc/method/n1gear (on a Solaris 10 or later system)

/etc/init.d/sgear (on all other systems)

3. Within the files, locate each occurrence of the string:

/gridware/sge

4. Replace each occurrence with:

/your-sge-root-path

where your-sge-root-path is your specific Sun Grid Engine root path.

Determining a Maximum Nonreserved Job Duration

If a reservation relies on certain resources being available at its start time T, no job can start (on the same host as the reservation) shortly before T using those resources and still be using them at time T. However, a job could start shortly before T using additional resources not required by the reservation.

defaults.prop contains a Java property called AdvanceReservation.MaximumNonreservedJobDuration. The value of this property is a duration. The default value is 2\:0\:0, which means 2 hours, 0 minutes, and 0 seconds.

This property determines the amount of time preceding any existing reservation during which the AR server will not allow another job to start running if that job would call for the reserved resources. That duration prior to the job start, the reserved resources will be set aside for the reservation by removing them from the generic queue given by the Java property AdvanceReservation.GenericQueueName (the default value is all.q). These reserved resources will be released by the reservation at the end of the reservation. The released resources are returned to the generic queue, unless they are already known to be needed soon afterwards by a subsequent reservation.

To guarantee the reservation, you must preclude jobs started prior to the resource reservation (that is more than AdvanceReservation.MaximumNonreservedJobDuration before the reservation start time T is still running at time T). To achieve this, the grid administrator should assure that jobs specify a maximum runtime (wall clock time) limit no greater than the duration of the AdvanceReservation.MaximumNonreservedJobDuration property.

The maximum runtime limit is specified using the h_rt resource (whose values are also in the hours:minutes:seconds format). You specify the maximum h_rt resource limit in the cluster-wide sge_request file. (You also can place a .sge_request file in the current working directory or in $HOME.) For the default cell, this file is $SGE_ROOT/default/common/sge_request.

The specification of a maximum h_rt equal to 2 hours would look like:

-l h_rt=2:0:0

Note that an effective specification does not start with the # that appears in the comments already in the file.

Initial Configuration of Advance Reservation

The first time the Advance Reservation server runs, the server creates initial versions of the configuration files described in TABLE 5-1. The initial configuration enables Advance Reservation on all execution hosts in the grid of the server host. Initially, the limits file knows only of graphics resources that were assigned using Sun Grid Engine to a specific queue (for example, all.q@mygraphicsserver). The file does not know of those resources allocated to an execution host. To correct this situation, perform the procedure for your respective operating system.

To Perform Initial Configuration for Solaris 10 and Later Operating Systems

1. As superuser, configure the Advance Reservation server with the Sun Grid Engine administrative user.

For example:

# svccfg -s n1ge_ar setprop config/admin_user= astring: sgeadmin

Replace sgeadmin with your Sun Grid Engine administrator username, if different.

2. Start the Advance Reservation service by typing:

# svcadm -v enable n1ge_ar

The service is started and configured to start whenever the host reboots.

The files described in TABLE 5-1 are created.

3. Edit the $SGE_ROOT/ar/config/limits file to reflect any resources that are assigned to an execution host, rather than to a queue.

4. Restart the Advance Reservation service by typing:

# svcadm -v refresh n1ge_ar

The output of the AR server is saved in a log file named:
/var/svc/log/network-n1ge_ar:default.log

To Perform Initial Configuration for Solaris 9 and Earlier and Linux Operating Systems

1. As superuser, open the /etc/init.d/sgear script in an editor.

2. Set the AR_USER variable to your Sun Grid Engine administrator username.

For example:

AR_USER=sgeadmin

Replace sgeadmin with your Sun Grid Engine administrator username, if different.

3. Save and close the file.

4. Start the Advance Reservation service by typing:

# /etc/init.d/sgear start

The files described in TABLE 5-1 are created.

5. Edit the $SGE_ROOT/ar/config/limits file to reflect any resources that are assigned to an execution host, rather than to a queue.

Restart the Advance Reservation service by typing:
The output of the AR server is saved in a log file named:
/var/tmp/ARS.$$ ($$ is the process ID for the script).

# /etc/init.d/sgear restart

Using Advance Reservation

Starting an AR Server or Client

To start the AR server or client, the bin/runar script is used. Normally, the server startup is performed by the RC script /etc/init.d/sgear.

The AR server must run on a Sun Grid Engine administrative host. The server runs as the Sun Grid Engine administrator (sgeadmin), and must have access to the Sun Grid Engine executables.

To Manually Start the Advance Reservation Script

Type:

myserver% /gridware/sge/ar/bin/runar [arguments]

If you have configured your $SGE_ROOT variable to something other than the default, you can also type:

myserver% $SGE_ROOT/ar/bin/runar [arguments]

For more simplicity, you can alias the complete path to the runar script to a single command. For example:

myserver% alias advance $SGE_ROOT/ar/bin/runar

Using an AR Client

Use of the Advance Reservation client is described in the Sun Shared Visualization 1.1 Software Client Administration Guide, 820-3257.

Reservation States

The following table lists the states a reservation normally passes through, in sequence:

TABLE 5-2 Reservation States
Reservation State	Description
`Specified`	Minimal user data has been specified for this reservation request.
`Confirmed`	Reservation request granted (the reservation is compatible with confirmed reservations).
`QueueMade`	A Sun Grid Engine queue has been created (with resources) for the reservation. A reservation should move to this state immediately after the server confirms the reservation. Sun Grid Engine’s `qmon` or `qstat` shows the reservation’s queue in state `C` for "suspended by calendar."
`Reserved`	Sun Grid Engine’s execution host’s generic queue’s resources have been reduced to those needed for this reservation. This situation ensures that no other Sun Grid Engine job is using these resources when the reservation requires the resources. A reservation should move to this state before the reservation’s start time. The amount of time before is equal to the configuration property `AdvanceReservation.MaximumNonreservedJobDuration` (in `defaults.prop`, 1 hour is a suggestion). A nonreservation job should start during that window only if the job has other resources available to it (that is, only if the job is compatible with all reservations).
`Started`	The reservation’s Sun Grid Engine queue is active (or the queue’s Sun Grid Engine calendar should have made the queue active). This situation should occur at the reservation’s start time. Sun Grid Engine’s `qmon` or `qstat` shows the reservation’s queue in state " " (the space means active).
`Finished`	The reservation’s Sun Grid Engine queue is suspended. (That is, the queue should be inactive or Sun Grid Engine’s calendar should have suspended the queue, which includes suspending any jobs still running on the queue). This situation should occur at the reservation’s finish time (its duration period after the reservation’s start time). Sun Grid Engine’s `qmon` or `qstat` shows the reservation’s queue in state `C` again (suspended by calendar). You won’t see a reservation in this state, because the reservation immediately transitions to `Returned`. A Sun Grid Engine administrator can resume any suspended jobs.
`Returned`	The reservation’s resources have been returned to the execution host’s generic queue, because the reservation is done. This situation occurs at the reservation’s finish time.
`QueueGone`	The reservation’s Sun Grid Engine queue has been deleted (along with any jobs pending for this queue and the job’s subordinate calendar and user set). A reservation should move to this state considerably after the reservation’s finish time, to enable the status of the queue to be evaluated or pending jobs to be moved to another queue, if needed. The amount of time after is equal to the configuration property `AdvanceReservation.FinishToDeleteQueueDuration` (in `defaults.prop`, 24 hours is a suggestion).

Advance Reservation Troubleshooting

General Troubleshooting

Both the client and server contain both assertions and Assume code. The Assume code prints stack traces if the code encounters an error, but then continues to execute (rather than aborting).

Client Troubleshooting

The client displays the following message to indicate the client doesn’t find the server running (on the port specified):

Cannot connect to server java.net.ConnectException:
Connection refused on port 6789

Server Troubleshooting

The server prints every message received from a client and sent to a client, as well as other messages. These messages are helpful in identifying problem cases.

When the server starts, the server deletes any reservations from the database that are sufficiently past the reservation stop time (the reservation’s start time plus the reservation’s duration). Additionally, the reservations are no longer visible in the Reserve GUI.

The server is multithreaded, but only one thread accepts client connections, so reservations are currently made one at a time.

Graphics allocated to a host do not appear in -sq output. Configure consumable attributes by queue (all.q@myhost) rather than by host, or edit the limits file to reflect the correct attributes, as displayed for any $hostname using:

% qconf -se $hostname | more +/complex_values -1