dmake [-c dmake-rcfile] [-g dmake-group] [-j dmake-max-jobs]
[-m {serial | parallel | distributed | grid}]
[-o dmake-odir]
Note: This man page supplements the Solaris make utility man page. The dmake utility will run under the Linux operating system, but the dmake options will be independent of any other make utility.
Distributed Make (dmake) parses your makefiles and determines which target can be built concurrently, and distributes the build of those targets over a number of hosts set by you.
If you use the standard make utility, the transition to dmake requires little if any alteration to your makefiles. dmake is a superset of the make utility. With nested makes, if a top-level makefile calls "make", you need to use $(MAKE).
You execute dmake on a "dmake host" and distribute "jobs" to "build servers." You can also distribute jobs to the dmake host, in which case the dmake host also becomes a build server. dmake distributes jobs based on makefile targets that dmake determines (based on your makefiles) can be built concurrently.
You can use any machine as a build server as long as it meets the following requirements:
From the dmake host (the machine you are using) you must be able to use rsh (or ssh) without being prompted for a password to remotely execute commands on the build server. (See below on how to specify which remote shell to use.)
The following example shows the result of using rsh or ssh with the which command as a test on a machine running the Solaris operating system:
demo% rsh machine-name which dmake /bin/dmake demo% ssh machine-name which dmake /bin/dmake
The rsh or ssh call must be clean, returning no additional output.
The bin directory in which the dmake software is installed must be accessible from the build server. By default, dmake assumes that the logical path to the dmake executables on the build server is the same as the dmake host. This assumption can be overridden by specifying a path name a an attribute of the host entry in the runtime configuration file.
The /etc/opt/SPROdmake/dmake.conf file exists on the host, is readable, and contains the correct information. If this file does not exist, dmake will distribute only one job on this system
From the dmake host you can control which build servers are used and how many dmake jobs are allotted to each build server. The number of dmake jobs that can run on a given build server can also be limited on that server.
Specifies an alternate runtime configuration file. The default runtime configuration file is $(HOME)/.dmakerc.
Specifies the name of the build server group to which jobs are distributed. You define server groups in the runtime configuration file. The default server group is the first group in the runtime configuration file.
Specifies the maximum total number of jobs that are distributed to the specified group of build servers in the runtime configuration file. The default maximum number of jobs is the sum of all the specified jobs in a build server group. The jobs are subtracted from, or added to, hosts by 1 in the order they appear in the runtime configuration file. For example, if all jobs specified in the runtime configuration file total 8:
host earth { jobs = 3 }
host mars { jobs = 5 }
and dmake-max-jobs is specified as 11, dmake adds three more jobs to the current total maximum number of jobs (which is eight) as follows:
host earth { jobs = 5 }
host mars { jobs = 6 }
Also, if dmake-max-jobs is specified as 4, dmake subtracts 4 jobs (from the original eight) as follows:
host earth { jobs = 1 }
host mars { jobs = 3 }
Specify one of the following key words:
Causes dmake to behave like the standard serial version of make.
Causes dmake to distribute jobs to only the dmake host.
Causes dmake to behave in fully distributed mode. This is the dmake default.
Causes dmake to use SGE (Solaris Grid Engine) to distribute build jobs.
Specifies a common physical directory that dmake can write temporary output files to and read temporary output files from. The directory used is $(HOME)/.dmake and this or whichever directory is specified, must be visible to all build servers. Use this option only if the $(HOME) directory on your local host and the $(HOME) directory on all of your remote hosts are NOT the same physical $(HOME) directory. For example, a root user would use this option.
These options and the environment variables and makefile macros described later in this man page modify the same behavior. Their order of precedence is defined as:
Command-line options
Makefile macros
Environment variables
dmake default
dmake allows targets to be built concurrently on a number of build servers. Concurrent processing can greatly reduce the time required to build a large system or project. dmake supplies the special makefile targets .PARALLEL, .NO_PARALLEL, .LOCAL, and .WAIT for controlling concurrency and timing.
Use this target to indicate which targets are to be processed serially.
Use this target to indicate which targets are to be processed in parallel.
Use this target to indicate which targets are to be processed serially on the local host.
When you specify this target in a dependency list, dmake waits until the dependencies that precede it are finished before processing those that follow, even when processing is parallel.
Makefiles that you write using these targets remain compatible with the standard version of make distributed with Solaris 1.x and Solaris 2.x. Standard make accepts these targets without error (and without action).
The distribution of dmake jobs is controlled in two ways:
A dmake user on a dmake host can specify the machines they want to use as build servers and the number of jobs they want to distribute to each build server.
The owner (a user that can alter the /etc/opt/SPROdmake/dmake.conf file) on a build server can control the maximum total number of dmake jobs that can be distributed to that build server.
When dmake begins execution it searches for a runtime configuration file to know where to distribute jobs. Generally, this file is located in your home directory on the dmake host and is named .dmakerc. dmake searches for the runtime configuration file in the following locations and in the following order:
The path name you specify on the command line using the -c option
The path name you specify using the DMAKE_RCFILE makefile macro
The path name you specify using the DMAKE_RCFILE environment variable
$(HOME)/.dmakerc
If a runtime configuration file is not found, dmake switches to parallel mode and distributes two jobs (the default) to the dmake host. You can change this using the -j option, or DMAKE_MAX_JOBS.
The runtime configuration file may contain a list of build servers and the number of jobs you want distributed to each build server. The following is a sample of a simple runtime configuration file:
# My machine. This entry causes dmake to distribute to it
falcon { jobs = 1 }
hawk
eagle { jobs = 3 }
# Manager's machine. She's usually at meetings
heron { jobs = 4 }
avocet
The entries: falcon, hawk, eagle, heron, and avocet are listed as build servers.
You can specify the number of jobs you want distributed to each build server. The default number of jobs is two.
Any line that begins with the "#" character is interpreted as a comment.
This list of build servers includes falcon which is also the dmake host. The dmake host can also be specified as a build server. If you do not include it in the runtime configuration file, no dmake jobs are distributed to it.
You can also construct groups of build servers in the runtime configuration file. This provides you with the flexibility of easily switching between different groups of build servers as circumstances warrant. For instance you may define a different group of build servers for builds under different operating systems, or on groups of build servers that have special software installed on them. The build servers must be all the same architecture and have the same version of SunOS installed.
The following runtime configuration file contains groups:
earth { jobs = 2 }
mars { jobs = 3 }
group lab1 {
host falcon { jobs = 3 }
host hawk
host eagle { jobs = 3 }
}
group lab2 {
host heron
host avocet { jobs = 3 }
host stilt { jobs = 2 }
group labs {
group lab1
group lab2
}
group sunos5.x {
group labs
host jupiter
host venus { jobs = 2 }
host pluto { jobs = 3 }
}
Formal groups are specified by the "group" directive and lists of their constituents are delimited by braces ({}).
Build servers that are constituents of groups are specified by the optional "host" directive.
Groups can be constituents of other groups.
Individual build servers can be listed in runtime configuration files that also contain groups of build servers. In this case dmake treats these build servers as constituents of the unnamed group.
dmake distributes jobs to a single group of hosts specified by the following list and in precedence from 1 to 4.
The group specified on the command-line as an argument to the -g option
The group specified by the DMAKE_GROUP makefile macro
The group specified by the DMAKE_GROUP environment variable
The first formal group listed in the runtime configuration file
The names of groups and hosts specified in the runtime configuration file may be enclosed in double quotes. This is to allow more flexibility with respect to the character sequences that may appear as part of the group and host names. For example, if the name of the group starts with a digit it should be double-quoted:
group "123_sparc"
As mentioned above, the bin directory in which the dmake software is installed must be accessible from the build server. By default, dmake assumes that the logical path to the dmake executables on the build server is the same as the dmake host. This assumption can be overridden by specifying a path name as an attribute of the host entry in the runtime configuration file. For example:
group sparc-cluster {
host wren { jobs = 10 , path = "/opt/solarisstudio12.4/bin" }
host stimpy { path = "/bin" }
}
The /etc/opt/SPROdmake/dmake.conf file is located in the file system of build servers. Use this file to specify the following:
The maximum total number of dmake jobs (from all users) that can run concurrently on that build server.
The /usr/bin/ priority under which all dmake jobs are to be run.
The following is a sample of a dmake.conf file:
max_jobs: 8 nice_prio: 5
This file sets the maximum number of dmake jobs permitted to run on that build server (from all dmake users) to be eight. You can change the priority of the jobs to be run by using the nice_prio command. See nice (1) .
If the /etc/opt/SPROdmake/dmake.conf file does not exist on a build server, no dmake jobs will be allowed to run on that server.
The path to the remote shell can be specified in the dmakerc file. Example:
host earth { jobs = 3 }
host mars { jobs = 5 , rsh = "/bin/ssh" }
If rsh= is not specified, dmake will use /bin/rsh by default.
As with rsh, it is the user's responsibility to ensure that ssh can login in to the remote host without requiring a password, and does not issue any warnings or errors.
The following can be defined as either environment variables or makefile macros:
Defines an alternate runtime configuration file. The default runtime configuration file is $(HOME)/.dmakerc
Defines the name of the build server group to which jobs are distributed. Server groups are defined in the runtime configuration file. The default server group is the first group in the runtime configuration file.
Defines the maximum total number of jobs that are distributed to the specified group of build servers in the runtime configuration file. The default maximum number of jobs is the sum of all the specified jobs in a build server group. The jobs are subtracted from, or added to, hosts by 1 in the order they appear in the runtime configuration file. See the -j option in this man page for an example.
May contain one of the following key words:
Allows dmake to adjust the limit of parallel jobs according to the current loading of the system. If the system is not overloaded, dmake will use the limit defined by the user. If the system is overloaded, dmake will set the "current" limit to less than the limit defined by the user.
If this variable is not set, dmake will adjust the limit of parallel jobs according to the current loading of the system. This is the dmake default.
Causes dmake to switch off the autoadjustment mechanism.
May contain one of the following key words:
Causes dmake to behave like the standard serial version of make.
Causes dmake to distribute jobs to only the dmake host.
Causes dmake to behave in fully distributed mode. This is the dmake default.
Causes dmake to use SGE (Solaris Grid Engine) to distribute build jobs.
Defines a common physical directory that dmake can write temporary output files to and read temporary output files from. Use this environment variable, or macro, only if the $(HOME) directory on your local host and the $(HOME) directory on all of your remote hosts are NOT the same physical $(HOME) directory. For example, a root user would use this option:
Defines the format of the log file. May contain one of the following keywords:
On starting each build job, dmake prints the name of the system and command to the log file. Also, if the command itself prints any output, then as the job finishes, dmake prints the name of the system and command to the log file again along with the command output.
Example:
host1 --> 1 job echo "Done host1" host2 --> 1 job echo "Done host2" host1 --> Job output echo "Done host1" Done host1 host2 --> Job output echo "Done host2" Done host2
This is the dmake default.
Allows dmake to serialize the output of parallel jobs, which makes the log file more readable. In this mode, dmake just prints the command once, immediately followed by the command output, as each job finishes.
Example:
echo "Done host1" Done host1 echo "Done host2" Done host2
The default runtime configuration file. Contains the names of build servers and groups of build servers.
Located on build servers, this file is used to specify the maximum total number of jobs that can be distributed to it by all dmake users. It is also used to specify the /usr/bin/nice priority all dmake jobs are to be run under.
If there are any problems with using dmake in distributed mode, verify the following:
The $HOME environment variable is set to an accessible directory.
% ls -la $HOME
The file $HOME/.dmakerc exists, is readable, and contains correct information.
% cat $HOME/.dmakerc
All hosts mentioned in the $HOME/.dmakerc file are alive. Use the /usr/sbin/ping command to check each host.
% /usr/sbin/ping $HOST
where $HOST is the name of the system that is listed as the host in $HOME/.dmakerc file.
Verify that the path to the dmake binaries is correct by using the which command to find the dmake, rxm, and rxs commands:
% which dmake % which rxm % which rxs
The remote login (rsh or ssh) on each host works without a password, and each remote login takes an acceptable time (less than 2 seconds).
% time rsh $HOST uname -a
The file /etc/opt/SPROdmake/dmake.conf exists on each host and contains the correct information. If this file does not exist, dmake will distribute only one job on this system:
% rsh $HOST cat /etc/opt/SPROdmake/dmake.conf
The path to the dmake binaries is correct for each host:
% rsh $HOST `which dmake` % rsh $HOST `which rxm` % rsh $HOST `which rxs`
The build area is available from each host (rwx):
% cd $BUILD % rm $HOST.check.tmp % echo "Build area is available from host $HOST" > $HOST.check.tmp % rsh $HOST cat $BUILD/$HOST.check.tmp
where $BUILD is the full path to the build area.
That $HOME is available from each host:
% cd $HOME % rm $HOST.check.tmp % echo "HOME is available from host $HOST" > $HOST.check.tmp % rsh $HOST cat $HOME/$HOST.check.tmp
make (1S) , rsh (1) , ssh (1) , hosts (4) , hosts.equiv (4) , attributes (5) , largefile (5) , qrsh(1)
Access the complete Oracle Solaris Studio documentation set at:
http://www.oracle.com/technetwork/server-storage/solarisstudio/documentation