Redo & GCC: Automatic Dependencies

Build Dependency Differences

Most build systems handle only one kind of dependency relation: Target files depend on source files in the sense that a target file should be rebuilt whenever a relevant source file changes. Examples are binaries depending on source code, bitmaps depending on vector graphics, documents depending on inlined pictures and, all kinds of files depending on their build rules (something often disregarded by authors of build systems).

Especially when using C or C++, often target files depend on nonexistent files as well, meaning that a target file should be rebuilt when a previosly nonexistent file is created: If the preprocessor includes /usr/include/stdio.h because it could not find /usr/local/include/stdio.h, the creation of the latter file should trigger a rebuild.

My implementation of redo handles both kinds of dependency relations: A dependency on /usr/include/stdio.h would be declared with redo-ifchange /usr/include/stdio.h in the target's dofile, while a dependency of a target on the non-existence of /usr/local/include/stdio.h would be declared with redo-ifcreate /usr/local/include/stdio.h.

GCC Dependency Generation

Many build systems require the user to list dependencies before starting the build. Users of Make, for example, must provide a text file with dependencies, the Makefile. As listing dependencies manually can be tedius and error-prone, determining them has often been automated. However, knowing all dependencies before a build is rarely possible.

GCC can provide a list of dependencies as a side-effect: Invoking gcc with the -MD and -MF command-line options outputs all dependencies used during compilation to an external file. In a dofile for the game Liberation Circuit I used this approach to determine dependencies:

#!/bin/sh
gcc -o $3 -c ${1%.o}.c -MD -MF $2.deps
read DEPS <$2.deps
redo-ifchange ${DEPS#*:}
Liberation Circuit dofile src/default.o.do, modified (if this seems unclear, read the explanation of redo parameters $1, $2, $3)

GCC Non-Existence Dependency Generation

Though the preprocessor must have access to the information, GCC does not provide a ready-made list of non-existence dependencies. However, one can deduce what files the build process is looking for by intercepting and recording system calls with strace. Filtering strace output for stat(2) (get file status) system calls that fail with ENOENT (No such file or directory) yields a list of non-existent files the build process tried to use. The following dofile (foo.do) uses that approach to compile a C program (foo.c) and record dependencies on non-existing header files:

#!/bin/sh
redo-ifchange $2.c
strace -e stat,stat64,fstat,fstat64,lstat,lstat64 -f 2>&1 >/dev/null\
 gcc $2.c -o $3 -MD -MF $2.deps\
 |grep '1 ENOENT'\
 |grep '\.h'\
 |cut -d'"' -f2 2>/dev/null\
 >$2.deps_ne

read d <$2.deps
redo-ifchange ${d#*:}

while read -r d_ne; do
 redo-ifcreate $d_ne
done <$2.deps_ne

chmod a+x $3
dofile foo.do

Dependency Graph Visualization

For demonstration purposes, I wrote a simple “Hello world!” program:

#include<stdio.h>
main() {
 printf("hello, world\n");
 return 0;
}
C program foo.c

Using the dofile foo.do, I built a binary from foo.c with redo foo. I then generated a dependency graph with redo-dot |sed s_$(pwd)/__g >foo-deps.dot && dot -Tpng <foo-deps.dot >foo-deps.png. Even with such a simple program, the chosen approach yields a lot of dependency relations:

Dependencies of binary foo on existing (solid edges) and non-existing (dotted edges) files. Note that foo depends on its source code (foo.c) and its build rules (foo.do). (GraphViz file)