Sun Studio 12: Distributed Make (dmake)

Usage

Special-purpose Targets

dmake allows targets to be built concurrently on a number of build servers. Concurrent processing can greatly reduce the time required to build a large system or project. dmake supplies the special makefile targets for controlling concurrency and timing.

.NO_PARALLEL

Use this target to indicate which targets are to be processed serially.

.PARALLEL

Use this target to indicate which targets are to be processed in parallel.

.LOCAL

Use this target to indicate which targets are to be processed serially on the local host.

.WAIT

When you specify this target in a dependency list, dmake waits until the dependencies that precede it are finished before processing those that follow, even when processing is parallel.

Makefiles that you write using these targets remain compatible with the standard version of make distributed with Solaris 1.x and Solaris 2.x. Standard make accepts these targets without error (and without action).

Controlling dmake Jobs

The distribution of dmake jobs is controlled in two ways:

  1. A dmake user on a dmake host can specify the machines they want to use as build servers and the number of jobs they want to distribute to each build server.

  2. The owner (a user that can alter the /etc/opt/SPROdmake/dmake.conf file) on a build server can control the maximum total number of dmake jobs that can be distributed to that build server.

The dmake Host

When dmake begins execution it searches for a runtime configuration file to know where to distribute jobs.

Generally, this file is located in your home directory on the dmake host and is named dmakerc.

dmake searches for the runtime configuration file in the following locations and in the following order:

  1. The path name you specify on the command line using the -c option

  2. The path name you specify using the DMAKE_RCFILE makefile macro

  3. The path name you specify using the DMAKE_RCFILE environment variable

  4. In $(HOME)/.dmakerc

If a runtime configuration file is not found, dmake switches to parallel mode and distributes two jobs (the default) to the dmake host. You can change this using the -j option, or DMAKE_MAX_JOBS.

The runtime configuration file may contain a list of build servers and the number of jobs you want distributed to each build server.

The following is a sample of a simple runtime configuration file:


# My machine. This entry causes dmake to distribute to it
falcon { jobs = 1 }
hawk
eagle { jobs = 3 }
# Manager’s machine. She’s usually at meetings
heron { jobs = 4 }
avocet

The entries: falcon, hawk, eagle, heron, and avocet are listed as build servers.

You can specify the number of jobs you want distributed to each build server. The default number of jobs is two.

Any line that begins with the "#" character is interpreted as a comment.


Note –

This list of build servers includes falcon which is also the dmake host. The dmake host can also be specified as a build server. If you do not include it in the runtime configuration file, no dmake jobs are distributed to it.


You can also construct groups of build servers in the runtime configuration file.

This provides you with the flexibility of easily switching between different groups of build servers as circumstances warrant. For instance you may define a different group of build servers for builds under different operating systems, or on groups of build servers that have special software installed on them. The build servers must be all the same architecture and have the same SunOS installed.

The following runtime configuration file contains groups:


earth                   { jobs = 2 }
mars                    { jobs = 3 }

group lab1 {
            host falcon    { jobs = 3 }
            host hawk
            host eagle    { jobs = 3 }
}
            
group lab2 {
            host heron
            host avocet    { jobs = 3 }
            host stilt    { jobs = 2 }

            
group labs {
            group lab1
            group lab2
}

group sunos5.x {
            group labs
            host jupiter
            host venus    {     jobs = 2 }
            host pluto     {     jobs = 3 }
}

User Commands

Formal groups are specified by the "group" directive and lists of their constituents are delimited by braces ({}).

Build servers that are constituents of groups are specified by the optional "host" directive.

Groups can be constituents of other groups.

Individual build servers can be listed in runtime configuration files that also contain groups of build servers. In this case dmake treats these build servers as constituents of the unnamed group.

dmake distributes jobs to a single group of hosts specified by the following list and in precedence from 1 to 4.

  1. The group specified on the command-line as an argument to the -g option.

  2. The group specified by the DMAKE_GROUP makefile macro.

  3. The group specified by the DMAKE_GROUP environment variable.

  4. The first formal group listed in the runtime configuration file.

The names of groups and hosts specified in the runtime configuration file may be enclosed in double quotes. This is to allow more flexibility with respect to the character sequences that may appear as part of the group and host names. For example, if the name of the group starts with a digit it should be double-quoted:


group "123_sparc"

As mentioned above, the bin directory in which the dmake software is installed must be accessible from the build server. By default, dmake assumes that the logical path to the dmake executables on the build server is the same as the dmake host. This assumption can be overridden by specifying a path name as an attribute of the host entry in the runtime configuration file. For example:


group sparc-cluster {
   host wren   { jobs = 10 , path = "/export/SUNWspro/bin" }
   host stimpy { path = "/opt/SUNWspro/bin"                }
}

The Build Server

The /etc/opt/SPROdmake/dmake.conf file is located in the file system of build servers.

Use this file to specify the following:

The following is a sample of a dmake.conf file:


max_jobs: 8
nice_prio: 5

This file sets the maximum number of dmake jobs permitted to run on that build server (from all dmake users) to be eight. You can change the priority of the jobs to be run by using the nice_prio command. See nice(1).


Note –

If the /etc/opt/SPROdmake/dmake.conf file does not exist on a build server, no dmake jobs will be allowed to run on that server.