This chapter describes how to use files and scripts to add or modify grid engine system objects such as queues, hosts, and environments.
This chapter includes the following sections:
The grid engine system provides two kinds of reporting and accounting facilities:
Accounting and Reporting Console (ARCo)
qacct command and the accounting file
You can use the optional Accounting and Reporting Console (ARCo) to generate live accounting and reporting data from the grid engine system and store the data in the reporting database, which is a standard SQL database. ARCo supports the following SQL database systems:
PostgreSQL
Oracle
MySQL
ARCo also provides a web-based tool for generating information queries on the reporting database and for retrieving the results in tabular or graphical form. ARCo enables you to store queries for later use, to run predefined queries, and to run queries in batch mode. For more information about how to use ARCo, see Chapter 5, Accounting and Reporting, in Sun N1 Grid Engine 6.1 User’s Guide. For information about how to install ARCo, see Chapter 8, Installing the Accounting and Reporting Console, in Sun N1 Grid Engine 6.1 Installation Guide.
Raw reporting data is generated by sge_qmaster. This raw data is stored in a reporting file. The dbwriter program reads the raw data in the reporting file and writes it to the SQL reporting database, where it can be accessed by ARCo.
The dbwriter program performs the following tasks:
Reads raw data from the reporting file and writes this raw data to the reporting database.
Calculates derived values. You can configure which values to calculate, as well as the rules that govern the calculations.
Deletes outdated data. You can configure how long to keep data.
When dbwriter starts up, it calculates derived values. dbwriter also deletes outdated records at startup. If dbwriter runs in continuous mode, dbwriter continues to calculate derived values and to delete outdated records at hourly intervals, or at whatever interval you specify.
You can specify in an XML file the values that you want to calculate and the records that you want to delete. Use the -calculation option of the dbwriter command to specify the path to this XML file.
For detailed information about calculating derived values, see Calculating Derived Values With dbwriter.
For detailed information about deleting outdated records, see Deleting Outdated Records With dbwriter.
The reporting file contains the following types of data:
Host load values and consumable resources
Queue consumable resources
Job logging
Job accounting
Share-tree usage
When the grid engine system is first installed, the reporting file is disabled. To use ARCo, you must enable the reporting file for the cluster. Once enabled, the reporting file will be generated by sge_qmaster. By default, the reporting file is located in sge-root/cell/common. You can change the default with the -reporting option of the dbwriter command.
For information about configuring the generation of the reporting file, see the reporting_params parameter of the sge_conf(5) man page, and the report_variables parameter of the host_conf(5) man page.
To enable the reporting file with QMON, on the Main Control window click the Cluster Configuration button, select the global host, and then click Modify.
On the Cluster Settings dialog box, click the Advanced Settings tab.
In the Reporting Parameters field, set the following parameters:
Set accounting to true. true is the default value.
Set reporting to true.
Set flush_time to 00:00:15. 00:00:15 is the default value.
Set joblog to true.
Set sharelog to 00:00:00. 00:00:00 is the default value.
To enable the reporting file from the command line, use the qconf –mconf command to set the reporting_params attributes, as described in the preceding paragraph.
Once the reporting file is enabled, the dbwriter can read raw data from the reporting file and write it to the reporting database.
For more information about configuring the reporting file, see the reporting(5) man page. For complete details about installing and setting up ARCo, see Chapter 8, Installing the Accounting and Reporting Console, in Sun N1 Grid Engine 6.1 Installation Guide.
The rules for calculating derived values are specified in a derived tag, which is a sub tag of the DbWriterConfig tag. The following table lists the attributes of the derived tag:
Description
The object for which data is aggregated. The object is one of the following:
host
queue
project
department
user
group
The time range specifying how often to calculate the derived values. The time range is one of the following:
hour
day
month
year
The name of the variable to hold the calculated data.
The following table lists the subelements of the derived tag:
The SQL statement that calculates the derived values. The statement must produce the following columns:
time_start – Together with time_end, specifies the time period for the calculated value
time_end
value – The calculated derived value
The SQL statement can contain the following placeholders. dbwriter replaces the placeholders for each query, based on a rule:
__time_start__ – Start time for the query. dbwriter searches for the last previously calculated derived value from this rule, and uses this timestamp as the start time for the next query.
__time_end__ – End time for the query. This timestamp specifies the end of the last passed time range. For example, if the time range is day, and if derived values are calculated at 00:30, 00:00 is taken as time_end.
__key_0__, __key_1__, . . . , __key_n__ – Components of the primary key for the specified object type. For example, the sge_hosts table has the primary h_hostname. If a rule is processed for the host object type, one query is executed per entry in the sge_hosts table, the __key_0__ placeholder in the SQL statement is replaced by the hostname.
The sge_queue table has a composed primary key that is made up of q_qname and q_hostname.
dbwriter generates the SQL statement for the calculation of derived values.
The autogenerated SQL statement looks like the following template:
SELECT time_start, time_end, <function>(<value_field>) as value FROM ( SELECT TRUNC( <timestart_field>, <interval> ) as time_start TRUNC( <timestart_field>, <interval> ) + INTERVAL '1' <interval> as time_end, <value_field> FROM <object value table> WHERE <primary key field 0> = __key_0__ AND <primary key field 1> = __key_1__ AND . . . AND <parent key field> = (SELECT <parent key field> FROM <parent table> WHERE <parent filter> ) AND <timestart_field> <= {ts __time_start__ } AND <timeend_field> > {ts __time_end__ } GROUP BY time_start, time_end |
The SQL template parameters are as follows:
Description
Aggregate function for calculating the derived value. Comes from the function attribute of the auto tag in the XML file.
Depends on the object of the derived value.
Depends on the object of the derived value.
Depends on the object of the derived value.
Comes from the interval attribute of the derived tag
Name of the database table where the values are stored. Depends on the object (host => host_values, user => user_values, . . .)
Primary key that is necessary to join the value table to the parent table. Depends on the object.
Name of the field that holds the ID of the parent. Depends on the object.
Name of the parent database table. Depends on the object (host => host, user => user)
Filter for the parent table. A derived value for each entry of the parent table is calculated, for example, u_user = 'user1').
Here is an example of an autogenerated SQL statement:
<derive object="host" interval="day" variable="d_load"> <auto function="AVG" variable="h_load" /> </derive> SELECT time_start, time_end, AGE(hv_dvalue) FROM ( SELECT TRUNC( hv_time_start, 'day' ) as time_start, TRUNC( hv_time_start, 'day' ) + INTERVAL '1' day as time_end, hv_dvalue FROM sge_host_values WHERE hv_variable = 'h_load' AND hv_parent = (SELECT h_id FROM sge_host WHERE h_hostname = 'foo.bar') AND hv_time_start <= {ts '2004-05-21 00:00:00.0'} AND hv_time_end > {ts '2004-05-17 00:00:00.0'} ) GROUP BY time_start, time_end |
To delete outdated records in the reporting database, you must specify a deletion rule in the delete tag. The following table lists the attributes of the delete tag:
Description
The type of data to delete. Valid entries are the following:
job
job_log
share_log
host_values
queue_values
project_values
department_values
user_values
group_values
The unit of time_amount:
Number of units (time_range) during which a record is to be kept.
The following table lists a subelement of the delete tag:
For certain scopes, a subscope can be configured. The subscope specifies an additional condition for deletion. A subscope can be configured for all *_values scopes and for the share_log scope.
If a subscope is configured for a *_values rule, it contains a list of variables to delete, separated by spaces.
If a subscope is specified for the share_log, it contains a list of share-tree nodes to delete, separated by spaces.
If subscope are used, you should always have a fallback rule without subscope, which will delete all objects that are not explicitly named by the subscope.
Here is an example of a delete tag:
<?xml version="1.0" encoding="UTF-8"?> <DbWriterConfig> <!-- keep host values for 2 years --> <delete scope="host_values" time_range="year" time_amount="2"/> <!-- keep queue values one month --> <delete scope="queue_values" time_range="month" time_amount="1"> <sub_scope>slots</sub_scope> <sub_scope>state</sub_scope> </delete> </DbWriterConfig> |
You can use the qacct command to generate alphanumeric accounting statistics. If you specify no options, qacct displays the aggregate usage on all machines of the cluster, as generated by all jobs that have finished and that are contained in the cluster accounting file sge-root/cell/common/accounting. In this case, qacct reports three times, in seconds:
Several options are available for reporting accounting information about queues, users, and the like. In particular, you can use the qacct -l command to request information about all jobs that have finished and that match a resource requirement specification.
Use the qacct -j [job-id | job-name] command to get direct access to the complete resource usage information stored by the grid engine system. This information includes the information that is provided by the getrusage system call.
The -j option reports the resource usage entry for the jobs with job-id or with job-name. If no argument is given, all jobs contained in the referenced accounting file are displayed. If a job ID is specified, and if more than one entry is displayed, one of the following is true:
Job ID numbers have wrapped around. The range for job IDs is 1 through 999999.
A checkpointing job that migrated is displayed.
See the qacct(1) man page for more information.
You can back up your grid engine system configuration files automatically. The automatic backup process uses a configuration file called backup_template.conf. The backup configuration file is located by default in sge-root/util/install_modules/backup_template.conf.
The backup configuration file must define the following elements:
The grid engine system root directory.
The grid engine system cell directory.
The grid engine system backup directory.
Type of backup. Your backup can be just the grid engine system configuration files, or the backup can be a compressed tar file that contains the configuration files.
The file name of the backup file.
The backup template file looks like the following example:
################################################## # Autobackup Configuration File Template ################################################## # Please, enter your SGE_ROOT here (mandatory) SGE_ROOT="" # Please, enter your SGE_CELL here (mandatory) SGE_CELL="" # Please, enter your Backup Directory here # After backup you will find your backup files here (mandatory) # The autobackup will add a time /date combination to this dirname # to prevent an overwriting! BACKUP_DIR="" # Please, enter true to get a tar/gz package # and false to copy the files only (mandatory) TAR="true" # Please, enter the backup file name here. (mandatory) BACKUP_FILE="backup.tar" |
To start the automatic backup process, type the following command on the sge_qmaster host:
inst_sge -bup -auto backup-conf |
backup-conf is the full path to the backup configuration file.
You do not need to shut down any of the grid engine system daemons before you back up your configuration files.
Your backup is created in the directory specified by BACKUP_FILE. A backup log file called install.pid is also created in this directory. pid is the process ID number.
Type the following command to start a manual backup:
inst_sge -bup |
Enter the <sge-root> directory or use the default.
SGE Configuration Backup ------------------------ This feature does a backup of all configuration you made within your cluster. Please enter your SGE_ROOT directory. Default: [/home/user/ts/u10] |
Enter the <sge-cell> name or use the default.
Please enter your SGE_CELL name. Default: [default] |
Enter the backup destination directory or use the default.
Where do you want to save the backupfiles? Default: [/home/user/ts/u10/backup] |
Choose whether to create a compressed tar backup file.
If you are using different tar versions (gnu tar/ solaris tar), this option can make some trouble. In some cases the tar packages may be corrupt. Using the same tar binary for packing and unpacking works without problems.
Shall the backup function create a compressed tarpackage with your files? (y/n) [y] >> |
Enter file name of backup file, default is given.
... starting with backup Please enter a filename for your backupfile. Default: [backup.tar] >> |
Backup will be perfomed, info output is printed.
2007-01-11_22_43_22.dump bootstrap qtask settings.sh act_qmaster sgemaster settings.csh sgeexecd jobseqnum ... backup completed All information is saved in [/home/user/ts/u10/backup/backup.tar.gz[Z]] |
Shutdown the qmaster daemon before you start the restore process.The spooling database will be changed during restore which might lead to dataloss, if restore and qmaster are trying to access the same data.
Type the following command to start the restore process:
inst_sge -rst |
Read the messages on the screen and respond.
SGE Configuration Restore ------------------------- This feature restores the configuration from a backup you made previously. Hit, <ENTER> to continue! |
Enter the <sge-root> directory or use the default.
Please enter your SGE_ROOT directory. Default: [/home/user/ts/u10] |
Enter the <sge-cell> name or use the default.
Please enter your SGE_CELL name. Default: [default] |
Confirm the backup file format.
The backup file can be in a format other than a compressed tar file.
Is your backupfile in tar.gz[Z] format? (y/n) [y] |
Enter the full path to the backup file.
Please enter the full path and name of your backup file. Default: [/home/user/ts/u10/backup/backup.tar.gz] |
Verify the spooling database information.
The restore feature unpacks the backup file, and reads system information. To prevent data loss, you need to conform that the correct spooling database is detected.
Copying backupfile to /tmp/bup_tmp_22_51_40 /home/user/ts/u10/backup/backup.tar.gz 2007-01-11_22_43_22.dump bootstrap qtask settings.sh act_qmaster sgemaster settings.csh sgeexecd jobseqnum Spooling Method: berkeleydb detected! The path to your spooling db is [/tmp/dom/spooldb] If this is correct hit <ENTER> to continue, else enter the path. >> |
Restart qmaster.
This section describes how to use files and scripts to add or modify grid engine system objects such as queues, hosts, and environments.
You can use the QMON graphical user interface to perform all administrative tasks in the grid engine system. You can also administer a grid engine system through commands you type at a shell prompt and call from within shell scripts. Many experienced administrators find that using files and scripts is a more flexible, quicker, and more powerful way to change settings.
Use the qconf command with the following options to add objects according to specifications you create in a file:
qconf -Ae qconf -Aq qconf -Au qconf -Ackpt qconf -Ap |
Use the qconf command with the following options to modify objects according to specifications you create in a file:
qconf -Me qconf -Mq qconf -Mu qconf -Mckpt qconf -Mp |
The –Ae and –Me options add or modify execution hosts.
The –Aq and –Mq options add or modify queues.
The –Au and –Mu options add or modify usersets.
The –Ackpt and –Mckpt options add or modify checkpointing environments.
The –Ap and –Mp options add or modify parallel environments.
Use these options in combination with the qconf –s command to take an existing object and modify it. You can then update the existing object or create a new object.
#!/bin/sh # ckptmod.sh: modify the migration command # of a checkpointing environment # Usage: ckptmod.sh <checkpoint-env-name> <full-path-to-command> TMPFILE=tmp/ckptmod.$$ CKPT=$1 MIGMETHOD=$2 qconf -sckpt $CKPT | grep -v '^migr_command' > $TMPFILE echo "migr_command $MIGMETHOD" >> $TMPFILE qconf -Mckpt $TMPFILE rm $TMPFILE
You can modify individual queues, hosts, parallel environments, and checkpointing environments from the command line. Use the qconf command in combination with other commands.
If you have already prepared a file, type the qconf command with appropriate options:
qconf -Me qconf -Mq qconf -Mckpt qconf -Mp |
If you have not prepared a file, type the qconf command with appropriate options:
qconf -me qconf -mq qconf -mckpt qconf -mp |
The –Me and –me options modify execution hosts.
The –Mq and –mq options modify queues.
The –Mckpt and –mckpt options modify checkpointing environments.
The –Mp and –mp options modify parallel environments.
The difference between the uppercase –M options and the lowercase –m options controls the qconf command's result. Both –M and –m mean modify, but the uppercase –M denotes modification from an existing file, whereas the lowercase –m does not. Instead, the lowercase –m opens a temporary file in an editor. When you save any changes you make to this file and exit the editor, the system immediately reflects those changes.
However, when you want to change many objects at once, or you want to change object configuration noninteractively, use the qconf command with the options that modify object attributes (such as –Aattr, –Mattr, and so forth).
The following commands make modifications according to specifications in a file:
qconf -Aattr {queue | exechost | pe | ckpt} filename qconf -Mattr {queue | exechost | pe | ckpt} filename qconf -Rattr {queue | exechost | pe | ckpt} filename qconf -Dattr {queue | exechost | pe | ckpt} filename |
The following commands make modifications according to specifications on the command line:
qconf -aattr {queue | exechost | pe | ckpt} attribute value {queue-list | host-list} qconf -mattr {queue | exechost | pe | ckpt} attribute value {queue-list | host-list} qconf -rattr {queue | exechost | pe | ckpt} attribute value {queue-list | host-list} qconf -dattr {queue | exechost | pe | ckpt} attribute value {queue-list | host-list} |
The –Aattr and –aattr options add attributes.
The –Mattr and –mattr options modify attributes.
The –Rattr and –rattr options replace attributes.
The –Dattr and –dattr options delete attributes.
filename is the name of a file that contains attribute-value pairs.
attribute is the queue or host attribute that you want to change.
value is the value of the attribute you want to change.
The –aattr, –mattr, and –dattr options enable you to operate on individual values in a list of values. The –rattr option replaces the entire list of values with the new one that you specify, either on the command line or in the file.
The following command changes the queue type of tcf27–e019.q to batch only:
% qconf -rattr queue qtype batch tcf27-e019.q |
The following command uses the file new.cfg to modify the queue type and the shell start behavior of tcf27–e019.q:
% cat new.cfg qtype batch interactive checkpointing shell_start_mode unix_behavior % qconf -Rattr queue new.cfg tcf27-e019.q |
The following command adds the resource attribute scratch1 with a value of 1000M and the resource attribute long with a value of 2:
% qconf -rattr exechost complex_values scratch1=1000M,long=2 tcf27-e019 |
The following command attaches the resource attribute short to the host with a value of 4:
% qconf -aattr exechost complex_values short=4 tcf27-e019 |
The following command changes the value of scratch1 to 500M, leaving other values unchanged:
% qconf -mattr exechost complex_values scratch-=500M tcf27-e019 |
The following command deletes the resource attribute long:
% qconf -dattr exechost complex_values long tcf27-e019 |
The following command adds tcf27–b011.q to the list of queues for the checkpointing environment sph:
% qconf -aattr ckpt queue_list tcf27-b011.q sph |
The following command changes the number of slots in the parallel environment make to 50:
% qconf -mattr pe slots 50 make |
The qselect command outputs a list of queue instances. If you specify options, qselect lists only the queue instances that match the criteria you specify. You can use qselect in combination with the qconf command to target specific queue instances that you want to modify.
The following command lists all queue instances on Linux machines:
% qselect -l arch=glinux |
The following command lists all queue instances on machines with two CPUs:
% qselect -l num_proc=2 |
The following command lists all queue instances on all four-CPU 64–bit Solaris machines:
% qselect -l arch=solaris64,num_proc=4 |
The following command lists queue instances that provide an application license. The queue instances were previously configured.
% qselect -l app_lic=TRUE |
You can combine qselect with qconf to do wide-reaching changes with a single command line. To do this, put the entire qselect command inside backward quotation marks (` `) and use it in place of the queue-list variable on the qconf command line.
The following command sets the prolog script to sol_prolog.sh on all queue instances on Solaris machines:
% qconf -mattr queue prolog /usr/local/scripts/sol_prolog.sh `qselect -l arch=solaris` |
The following command sets the attribute fluent_license to two on all queue instances on two-processor systems:
% qconf -mattr queue complex_values fluent_license=2 `qselect -l num_proc=2` |
The most flexible way to automate the configuration of queue instances is to use the qconf command with the qselect command. With the combination of these commands, you can build up your own custom administration scripts.
To change a global configuration, use the qconf –mconf command. To change the scheduler, use the qconf –msconf command.
Both of these commands open a temporary file in an editor. When you exit the editor, any changes that you save to this temporary file are processed by the system and take effect immediately. The editor used to open the temporary file is the editor specified by the EDITOR environment variable. If this variable is undefined, the vi editor is used by default.
You can use the EDITOR environment variable to automate the behavior of the qconf command. Change the value of this variable to point to an editor program that modifies a file whose name is given by the first argument. After the editor modifies the temporary file and exits, the system reads in the modifications, which take effect immediately.
If the modification time of the file does not change after the edit operation, the system sometimes incorrectly assumes that the file was not modified. Therefore you should insert a sleep 1 instruction before writing the file, to ensure a different modification time.
You can use this technique with any qconf –m... command. However, the technique is especially useful for administration of the scheduler and the global configuration, as you cannot automate the procedure in any other way.
The following example modifies the schedule interval of the scheduler:
#!/bin/ksh # sched_int.sh: modify the schedule interval # usage: sched_int.sh <n>, where <n> is # the new interval, in seconds. n < 60 TMPFILE=/tmp/sched_int.$$ if [ $MOD_SGE_SCHED_INT ]; then grep -v schedule_interval $1 > $TMPFILE echo "schedule_interval 0:0:$MOD_SGE_SCHED_INT" >> $TMPFILE # sleep to ensure modification time changes sleep 1 mv $TMPFILE $1 else export EDITOR=$0 export MOD_SGE_SCHED_INT=$1 qconf -msconf fi |
This script modifies the EDITOR environment to point to itself. The script then calls the qconf –msconf command. This second nested invocation of the script modifies the temporary file specified by the first argument and then exits. The grid engine system automatically reads in the changes, and the first invocation of the script terminates.