NAME
dmake - Distributed Make
SYNOPSIS
dmake [-c dmake-rcfile] [-g dmake-group] [-j dmake-max-jobs]
[-m {serial | parallel | distributed| grid}] [-o dmake-odir]
DESCRIPTION
Note: This man page supplements the Solaris make utility man
page. The dmake utility will run under the Linux operating
system, but the dmake options will be independent of any
other make utility.
Distributed Make (dmake) parses your makefiles and deter-
mines which target can be built concurrently, and distri-
butes the build of those targets over a number of hosts set
by you.
If you use the standard make utility, the transition to
dmake requires little if any alteration to your makefiles.
dmake is a superset of the make utility. With nested makes,
if a top-level makefile calls "make", you need to use
$(MAKE).
You execute dmake on a "dmake host" and distribute "jobs" to
"build servers." You can also distribute jobs to the dmake
host, in which case the dmake host also becomes a build
server. dmake distributes jobs based on makefile targets
that dmake determines (based on your makefiles) can be built
concurrently.
You can use any machine as a build server as long as it
meets the following requirements:
- From the dmake host (the machine you are using) you must
be able to use rsh, without being prompted for a password,
to remotely execute commands on the build server. The fol-
lowing example shows the result of using the which command
on dmake if the machine is running the Solaris operating
system:
demo% rsh <machine_name> which dmake
/bin/dmake
The rsh must be "clean", with no additional output.
- The bin directory in which the dmake software is installed
must be accessible from the build server. By default,
dmake assumes that the logical path to the dmake execut-
ables on the build server is the same as the dmake host.
This assumption can be overridden by specifying a path
name a an attribute of the host entry in the runtime
configuration file.
From the dmake host you can control which build servers are
used and how many dmake jobs are allotted to each build
server. The number of dmake jobs that can run on a given
build server can also be limited on that server.
OPTIONS
-c dmake-rcfile
Specifies an alternate runtime configuration file. The
default runtime configuration file is $(HOME)/.dmakerc.
-g dmake-group
Specifies the name of the build server group to which
jobs are distributed. You define server groups in the
runtime configuration file. The default server group
is the first group in the runtime configuration file.
-j dmake-max-jobs
Specifies the maximum total number of jobs that are
distributed to the specified group of build servers in
the runtime configuration file. The default maximum
number of jobs is the sum of all the specified jobs in
a build server group. The jobs are subtracted from, or
added to, hosts by 1 in the order they appear in the
runtime configuration file. For example, if all jobs
specified in the runtime configuration file total 8:
host earth { jobs = 3 }
host mars { jobs = 5 }
and dmake-max-jobs is specified as 11, dmake adds three
more jobs to the current total maximum number of jobs
(which is eight) as follows:
host earth { jobs = 5 }
host mars { jobs = 6 }
Also, if dmake-max-jobs is specified as 4, dmake sub-
tracts 4 jobs (from the original eight) as follows:
host earth { jobs = 1 }
host mars { jobs = 3 }
-m {serial | parallel | distributed | grid}
Specify one of the following key words:
serial Causes dmake to behave like the standard
serial version of make.
parallel Causes dmake to distribute jobs to only the
dmake host.
distributed Causes dmake to behave in fully distributed
mode. This is the dmake default.
grid Causes dmake to use SGE (Solaris Grid
Engine) to distribute build jobs.
-o dmake-odir
Specifies a common physical directory that dmake can
write temporary output files to and read temporary out-
put files from. The directory used is $(HOME)/.dmake
and this or whichever directory is specified, must be
visible to all build servers. Use this option only if
the $(HOME) directory on your local host and the
$(HOME) directory on all of your remote hosts are NOT
the same physical $(HOME) directory. For example, a
root user would use this option.
Note: These options and the environment variables and
makefile macros described later in this man page
modify the same behavior. Their order of precedence is
defined as:
1. Command-line options
2. Makefile macros
3. Environment variables
4. dmake default
USAGE
Special-purpose Targets
dmake allows targets to be built concurrently on a number of
build servers. Concurrent processing can greatly reduce the
time required to build a large system or project. dmake sup-
plies the special makefile targets .PARALLEL, .NO_PARALLEL,
.LOCAL, and .WAIT for controlling concurrency and timing.
.NO_PARALLEL: Use this target to indicate which targets
are to be processed serially.
.PARALLEL: Use this target to indicate which targets
are to be processed in parallel.
.LOCAL: Use this target to indicate which targets
are to be processed serially on the local
host.
.WAIT When you specify this target in a depen-
dency list, dmake waits until the
dependencies that precede it are finished
before processing those that follow, even
when processing is parallel.
Makefiles that you write using these targets remain compati-
ble with the standard version of make distributed with
Solaris 1.x and Solaris 2.x. Standard make accepts these
targets without error (and without action).
Controlling dmake Jobs
The distribution of dmake jobs is controlled in two ways:
1. A dmake user on a dmake host can specify the
machines they want to use as build servers and the
number of jobs they want to distribute to each build
server.
2. The owner (a user that can alter the
/etc/opt/SPROdmake/dmake.conf file) on a build server
can control the maximum total number of dmake jobs that
can be distributed to that build server.
The dmake Host
When dmake begins execution it searches for a runtime confi-
guration file to know where to distribute jobs. Generally,
this file is located in your home directory on the dmake
host and is named .dmakerc. dmake searches for the runtime
configuration file in the following locations and in the
following order:
1. The path name you specify on the command line using the
-c option
2. The path name you specify using the DMAKE_RCFILE
makefile macro
3. The path name you specify using the DMAKE_RCFILE
environment variable
4. $(HOME)/.dmakerc
If a runtime configuration file is not found, dmake switches
to parallel mode and distributes two jobs (the default) to
the dmake host. You can change this using the -j option, or
DMAKE_MAX_JOBS.
The runtime configuration file may contain a list of build
servers and the number of jobs you want distributed to each
build server. The following is a sample of a simple runtime
configuration file:
# My machine. This entry causes dmake to distribute to it
falcon { jobs = 1 }
hawk
eagle { jobs = 3 }
# Manager's machine. She's usually at meetings
heron { jobs = 4 }
avocet
The entries: falcon, hawk, eagle, heron, and avocet are
listed as build servers.
You can specify the number of jobs you want distributed to
each build server. The default number of jobs is two.
Any line that begins with the "#" character is interpreted
as a comment.
Note: This list of build servers includes falcon which is
also the dmake host. The dmake host can also be speci-
fied as a build server. If you do not include it in
the runtime configuration file, no dmake jobs are dis-
tributed to it.
You can also construct groups of build servers in the run-
time configuration file. This provides you with the flexi-
bility of easily switching between different groups of build
servers as circumstances warrant. For instance you may
define a different group of build servers for builds under
different operating systems, or on groups of build servers
that have special software installed on them. The build
servers must be all the same architecture and have the same
SunOS installed.
The following runtime configuration file contains groups:
earth { jobs = 2 }
mars { jobs = 3 }
group lab1 {
host falcon { jobs = 3 }
host hawk
host eagle { jobs = 3 }
}
group lab2 {
host heron
host avocet { jobs = 3 }
host stilt { jobs = 2 }
group labs {
group lab1
group lab2
}
group sunos5.x {
group labs
host jupiter
host venus { jobs = 2 }
host pluto { jobs = 3 }
}
Formal groups are specified by the "group" directive and
lists of their constituents are delimited by braces ({}).
Build servers that are constituents of groups are specified
by the optional "host" directive.
Groups can be constituents of other groups.
Individual build servers can be listed in runtime configura-
tion files that also contain groups of build servers. In
this case dmake treats these build servers as constituents
of the unnamed group.
dmake distributes jobs to a single group of hosts specified
by the following list and in precedence from 1 to 4.
1. The group specified on the command-line as an argument to
the -g option
2. The group specified by the DMAKE_GROUP makefile macro
3. The group specified by the DMAKE_GROUP environment vari-
able
4. The first formal group listed in the runtime configura-
tion file
The names of groups and hosts specified in the runtime con-
figuration file may be enclosed in double quotes. This is to
allow more flexibility with respect to the character
sequences that may appear as part of the group and host
names. For example, if the name of the group starts with a
digit it should be double-quoted:
group "123_sparc"
As mentioned above, the bin directory in which the dmake
software is installed must be accessible from the build
server. By default, dmake assumes that the logical path to
the dmake executables on the build server is the same as the
dmake host. This assumption can be overridden by specifying
a path name as an attribute of the host entry in the runtime
configuration file. For example:
group sparc-cluster {
host wren { jobs = 10 , path = "/export/sunstudio/bin" }
host stimpy { path = "/bin" }
}
The Build Server
The /etc/opt/SPROdmake/dmake.conf file is located in the
file system of build servers. Use this file to specify the
following:
Required The maximum total number of dmake jobs (from all
users) that can run concurrently on that build
server.
Optional The /usr/bin/ priority under which all dmake jobs
are to be run.
The following is a sample of a dmake.conf file:
max_jobs: 8
nice_prio: 5
This file sets the maximum number of dmake jobs permitted to
run on that build server (from all dmake users) to be eight.
You can change the priority of the jobs to be run by using
the nice_prio command. See nice(1).
Note: If the /etc/opt/SPROdmake/dmake.conf file does not
exist on a build server, no dmake jobs will be allowed
to run on that server.
ENVIRONMENT/MACROS
The following can be defined as either environment variables
or makefile macros:
DMAKE_RCFILE
Defines an alternate runtime configuration file. The
default runtime configuration file is $(HOME)/.dmakerc
DMAKE_GROUP
Defines the name of the build server group to which
jobs are distributed. Server groups are defined in the
runtime configuration file. The default server group
is the first group in the runtime configuration file.
DMAKE_MAX_JOBS
Defines the maximum total number of jobs that are dis-
tributed to the specified group of build servers in the
runtime configuration file. The default maximum number
of jobs is the sum of all the specified jobs in a build
server group. The jobs are subtracted from, or added
to, hosts by 1 in the order they appear in the runtime
configuration file. See the -j option in this man page
for an example.
DMAKE_ADJUST_MAX_JOBS
May contain one of the following key words:
YES Allows dmake to adjust the limit of parallel jobs
according to the current loading of the system. If
the system is not overloaded, dmake will use the
limit defined by the user. If the system is over-
loaded, dmake will set the "current" limit to less
than the limit defined by the user.
If this variable is not set, dmake will adjust the
limit of parallel jobs according to the current
loading of the system. This is the dmake default.
NO Causes dmake to switch off the autoadjustment
mechanism.
DMAKE_MODE
May contain one of the following key words:
serial Causes dmake to behave like the standard
serial version of make.
parallel Causes dmake to distribute jobs to only the
dmake host.
distributed Causes dmake to behave in fully distributed
mode. This is the dmake default.
grid Causes dmake to use SGE (Solaris Grid
Engine) to distribute build jobs.
DMAKE_ODIR
Defines a common physical directory that dmake can
write temporary output files to and read temporary out-
put files from. Use this environment variable, or
macro, only if the $(HOME) directory on your local host
and the $(HOME) directory on all of your remote hosts
are NOT the same physical $(HOME) directory. For exam-
ple, a root user would use this option:
DMAKE_OUTPUT_MODE
Defines the format of the log file. May contain one of
the following keywords:
TEXT1 On starting each build job, dmake prints the
name of the system and command to the log file.
Also, if the command itself prints any output,
then as the job finishes, dmake prints the name
of the system and command to the log file again
along with the command output.
Example:
host1 --> 1 job
echo "Done host1"
host2 --> 1 job
echo "Done host2"
host1 --> Job output
echo "Done host1"
Done host1
host2 --> Job output
echo "Done host2"
Done host2
This is the dmake default.
TEXT2 Allows dmake to serialize the output of paral-
lel jobs, which makes the log file more read-
able. In this mode, dmake just prints the com-
mand once, immediately followed by the command
output, as each job finishes.
Example:
echo "Done host1"
Done host1
echo "Done host2"
Done host2
FILES
$(HOME)/.dmakerc The default runtime configuration file.
Contains the names of build servers and
groups of build servers.
/etc/opt/SPROdmake/dmake.conf
Located on build servers, this file is
used to specify the maximum total number
of jobs that can be distributed to it by
all dmake users. It is also used to
specify the /usr/bin/nice priority all
dmake jobs are to be run under.
SEE ALSO
Access the complete Solaris Studio documentation set by
pointing an HTML browser at:
http://developers.sun.com/sunstudio/documentation/
Consult the following man pages for additional information:
make(1), rsh(1), hosts(4), hosts.equiv(4), attributes(5),
largefile(5), qrsh(1)