Chapter 17. Generating Test Suites with Autotest

Note: This section describes an experimental feature which will
be part of Autoconf in a forthcoming release.  Although we believe
Autotest is stabilizing, this documentation describes an interface which
might change in the future: do not depend upon Autotest without
subscribing to the Autoconf mailing lists.

It is paradoxical that portable projects depend on nonportable tools to run their test suite. Autoconf by itself is the paragon of this problem: although it aims at perfectly portability, up to 2.13, its test suite was using DejaGNU, a rich and complex testing framework, but which is far from being standard on Unix systems. Worse yet, it was likely to be missing on the most fragile platforms, the very platforms that are most likely to torture Autoconf and exhibit deficiencies.

To circumvent this problem many package maintainers have developed their own testing framework, based on simple shell scripts whose sole output are their exit status: the test succeeded, or failed. In addition, most of these tests share some common patterns, what results in lots of duplicated code, tedious maintenance etc.

Following exactly the same reasoning that yielded to the inception of Autoconf, Autotest provides a test suite generation frame work, based on M4 macros, building a portable shell script. The suite itself is equipped with automatic logging and tracing facilities which greatly diminish the interaction with bug reporters, and simple timing reports.

Autoconf itself has been using Autotest for years, and we do attest that it has considerably improved the strength of the test suite, and the quality of bug reports. Other projects are known to use some generation of Autotest, such as Bison, Free Recode, Free Wdiff, GNU Tar, each of them having different needs, what slowly polishes Autotest as a general testing framework.

Nonetheless, compared to DejaGNU, Autotest is inadequate for interactive tool testing, which is probably its main limitation.

Using an Autotest Test Suite

testsuiteScripts

Generating testing or validation suites using Autotest is rather easy. The whole validation suite is held in a file to be processed through autom4te, itself using GNU m4 under the scene, to produce a stand-alone Bourne shell script which then gets distributed. Neither autom4te nor GNU m4 are not needed anymore at the installer end.

Each test of the validation suite should be part of some test group. A test group is a sequence of interwoven tests that ought to be executed together, usually because one test in the group creates data files than a later test in the same group needs to read. Complex test groups make later debugging more tedious. It is much better keeping keep only a few tests per test group, and if you can put only one test per test group, this is just ideal.

For all but the simplest packages, some file such as testsuite.at does not fully hold all test sources, as these are often easier to maintain in separate files. Each of these separate files holds a single test group, or a sequence of test groups all addressing some common functionality in the package. In such cases, file testsuite.at only initializes the whole validation suite, and sometimes do elementary health checking, before listing include statements for all other test files. The special file package.m4, containing the identification of the package, is automatically included if found.

The validation scripts that Autotest produces are by convention called testsuite. When run, testsuite executes each test group in turn, producing only one summary line per test to say if that particular test succeeded or failed. At end of all tests, summarizing counters get printed. If any test failed, one debugging script gets automatically generated for each test group which failed. These debugging scripts are named testsuite.nn, where nn is the sequence number of the test group. In the ideal situation, none of the tests fail, and consequently, no debugging script is generated out of validation.

The automatic generation of debugging scripts for failed test has the purpose of easing the chase for bugs.

It often happens in practice that individual tests in the validation suite need to get information coming out of the configuration process. Some of this information, common for all validation suites, is provided through the file atconfig, automatically created by AC_CONFIG_TESTDIR. For configuration informations which your testing environment specifically needs, you might prepare an optional file named atlocal.in, instantiated by AC_CONFIG_FILES. The configuration process produces atconfig and atlocal out of these two input files, and these two produced files are automatically read by the testsuite script.

Here is a diagram showing the relationship between files.

Files used in preparing a software package for distribution:

subfile-1.at -.
    ...         \
subfile-i.at ------ testsuite.at --.
    ...         /                     \
subfile-n.at -'                       -- autom4te* --testsuite
                                      /
                      [package.m4] -'

Files used in configuring a software package:

                                     .-- atconfig
                                    /
[atlocal.in] --  config.status* --
                                    \
                                     `-- [atlocal]

Files created during the test suite execution:

atconfig --.                    .-- testsuite.log
             \                  /
              -- testsuite* --
             /                  \
[atlocal] -'                    `-- [testsuite.nn*]

Autotest Logs

When run, the test suite creates a log file named after itself, e.g., a test suite named testsuite creates testsuite.log. It contains a lot of information, usually more than maintainers actually need, but therefore most of the time it contains all that is needed:

command line arguments

A very bad Unix habit which is unfortunately wide spread consists of setting environment variables before the command, such as in CC=my-home-grown-cc ./testsuite. This results in the test suite not knowing this change, hence (i) it can't report it to you, and (ii) it cannot preserve the value of CC for subsequent runs[22]. Autoconf faced exactly the same problem, and solved it by asking users to pass the variable definitions as command line arguments. Autotest requires this rule too, but has no means to enforce it; the log then contains a trace of the variables the user changed.

ChangeLog excerpts

The topmost lines of all the ChangeLogs found in the source hierarchy. This is especially useful when bugs are reported against development versions of the package, since the version string does not provide sufficient information to know the exact state of the sources the user compiled. Of course this relies on the use of a ChangeLog.

build machine

Running a test suite in a cross-compile environment is not an easy task, since it would mean having the test suite run on a machine build, while running programs on a machine host. It is much simpler to run both the test suite and the programs on host, but then, from the point of view of the test suite, there remains a single environment, host = build. The log contains relevant information on the state of the build machine, including some important environment variables.

tested programs

The absolute path and answers to -version of the tested programs (see the section called “Writing testsuite.at”, AT_TESTED).

configuration log

The contents of config.log, as created by configure, are appended. It contains the configuration flags and a detailed report on the configuration itself.



[22] When a failure occurs, the test suite is rerun, verbosely, and the user is asked to "play" with this failure to provide better information. It is important to keep the same environment between the first run, and bug-tracking runs.