Running Tests with make

Shell scripts are often helpful for running tests and performing other routine tasks that are either interactive or don't require make's dependency checking. Test suites, in particular, often entail providing a program with specific, repeatable input that a program might expect to receive from a terminal.

In the case of a library, a set of programs that exercise its various functions can be written in C, and then executed in a specific order, with specific inputs from a script. In the case of a utility program, there can be a set of benchmark programs that exercise and time its functions. In each of these cases, the commands to run each test can be incorporated into a shell script for repeatability and easy maintenance.

After you have developed a test script that suits your needs, including a target to run it is easy. Although make's dependency checking might not be needed within the script itself, you can use it to make sure that the program or library is updated before running those tests.

In the following target entry for running tests, test depends on lib.a. If the library is out of date, make rebuilds it and proceeds with the test. This insures that you always test with an up-to-date version:

#This is the library we're testing
LIBRARY= lib.a

test: $(LIBRARY) testscript 
        	set -x ; testscript > /tmp/test.\$\$ 

testscript: testscript.sh test_1 test_2 test_3 

#rules for building the library
$(LIBRARY):
        	@ echo Building $(LIBRARY)
        (library-building rules here)

#test_1 ... test_3 exercise various library functions
test_1 test_2 test_3: $$@.c $(LIBRARY) 
        	$(LINK.c) -o $@ $< 

test also depends on testscript, which in turn depends on the three test programs.

This ensures that they too are up-to-date before make initiates the test procedure. lib.a is built according to its target entry in the makefile; testscript is built using the .sh implicit rule; and the test programs are built using the rule in the last target entry, assuming that there is just one source file for each test program. (The .c implicit rule doesn't apply to these programs because they must link with the proper libraries in addition to their .c files).

Escaped References to a Shell Variable

The string \$\$ in the rule for test illustrates how to escape the dollar-sign from interpretation by make. make passes each $ to the shell, which expands the $$ to its process ID. This technique allows each test to write to a unique temporary filename. The set -x command forces the shell to display the commands it runs on the terminal, which allows you to see the actual file name containing the results of the specific test.

Shell Command Substitutions

You can supply shell command substitutions within a rule as in the following example:

do: 
	        @echo `cat Listfile`

You can even place the backquoted expression in a macro:

DO= `cat Listfile` 
do: 
        	@echo $(DO)

However, you can only use this form of command substitution within a rule.

Command-Replacement Macro References

If you supply a shell command as the definition of a macro:

COMMAND= cat Listfile

you can use a command-replacement macro reference to instruct make to replace the reference with the output of the command in the macro's value. This form of command substitution can occur anywhere within a makefile:

COMMAND= cat Listfile 
$(COMMAND:sh): $$(@:=.c)

This example imports a list of targets from another file and indicates that each target depends on a corresponding .c file.

As with shell command substitution, a command replacement reference evaluates to the standard output of the command. NEWLINE characters are converted to SPACE characters. The command is performed whenever the reference is encountered. The command's standard error is ignored. However, if the command returns a non-zero exit status, make halts with an error.

A workaround for this is to append the true command to the command line:

COMMAND = cat Listfile ; true

Command-Replacement Macro Assignment

A macro assignment of the form

cmd_macro:sh = command

assigns the standard output of the indicated command to cmd_macro; for instance:

COMMAND:sh = cat Listfile 
$(COMMAND): $$(@:=.c)

is equivalent to the previous example. However, with the assignment form, the command is only performed once per make run. Again, only the standard output is used, NEWLINE characters are converted to SPACE characters, and a non-zero exit status halts make with an error.

Alternate forms of command-replacement macro assignments are:

macro:sh += command

Append command output to the value of macro.

target := macro:sh = command

Conditionally define macro to be the output of command when processing target and its dependencies.

target := macro:sh += command

Conditionally append the output of command to the value of macro when processing target and its dependencies.