The Automake test suite User interface ============== Running all tests ----------------- make check Interpretation -------------- Successes: PASS - success XFAIL - expected failure Failures: FAIL - failure XPASS - unexpected success Other: SKIP - skipped tests (third party tools not available) Getting details from failures ----------------------------- Each test is a script. In a non-VPATH build you can run them directly, they will be verbose. Otherwise, you can invoke make as follows, just replace the list of tests by those you want to check. env VERBOSE=x TESTS='first.test second.test ...' make -e check Reporting failures ------------------ Send verbose output of failing tests to , along with the usual version numbers (which Automake, which Autoconf, which operating system, which make version, which shell, etc.) Writing test cases ================== Do -- If you plan to fix a bug, write the test case first. This way you'll make sure the test catches the bug, and that it succeeds once you have fixed the bug. Add a copyright/license paragraph. Explain what the test does. Cite the PR number (if any), and the original reporter (if any), so we can find or ask for information if needed. Use `required=...' for required tools. Include ./defs (see other tests). Use `set -e' to catch failures you might not have thought of. ./defs sets a skeleton configure.in. If possible, append to this file. In some cases you'll have to overwrite it, but this should be the exception. Note that configure.in registers Makefile.in but do not output anything by default. If you need ./configure to create Makefile, append AC_OUTPUT to configure.in. Use $ACLOCAL, $AUTOMAKE, $AUTOCONF, $AUTOUPDATE, $AUTOHEADER, $PERL, $MAKE, $EGREP, and $FGREP, instead of the corresponding commands. Use $sleep when you have to make sure that some file is newer than another. Use `cat' or `grep' to display (part of) files that may be interesting for debugging, so that when a user send a verbose output we don't have to ask him for more details. It's more important to make sure that a feature works, than make sure that Automake's output looks correct. It might look correct and still fails to work. In other words, prefer running `make' over grepping `Makefile.in' (or do both). If you run $AUTOMAKE or $AUTOCONF several times in the same test and change `configure.in' by the meantime, do rm -rf autom4te.cache before the following runs. On fast machines the new `configure.in' could otherwise have the same timestamp as the old `autom4te.cache'. Use filenames with two consecutive spaces when testing that some code preserves filenames with spaces. This will catch errors like `echo $filename | ...`. Before commit: make sure the test is executable, add the tests to TESTS in Makefile.am, add it to XFAIL_TESTS in addition if needed, write a ChangeLog entry, send the diff to . Do not ------ Do not test an Automake error with `$AUTOMAKE && exit 1', or in three years we'll discover that this test failed for some other bogus reason. This happened many times. Better use something like AUTOMAKE_fails grep 'expected diagnostic' stderr (Note this doesn't prevent the test from failing for another reason, but at least it makes sure the original error is still here.) Do not override Makefile variables using make arguments, as in $MAKE ANSI2KNR=./ansi2knr U=_ all this is not portable for recursive targets (targets that call a sub-make may not pass `ANSI2KNR=./ansi2knr U=_' along). Use the following instead. ANSI2KNR=./ansi2knr U=_ $MAKE -e all Do not send a test case without signing a copyright disclaimer. See http://sources.redhat.com/automake/contribute.html or ask for details.